Re: WIP: About CMake v2

Hi Yury,

On Fri, Aug 19, 2016 at 9:46 AM, Yury Zhuravlev
<u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
> Christian Convey wrote:
>>
>> I'm interested in helping with your CMake effort. I don't have any
>> experience contributing to PG, but I do have some free time at the
>> moment. Please let me know if I can help.
>
> I glad to hear it. I suppose you can just try build postgres and send all
> problems to github tracker.
> https://github.com/stalkerg/postgres_cmake/issues

Thanks, I'll be happy to do that. There's been a lot of discussion on
this thread regarding the minimum required CMake version. If you'd
like me to test with a particular version (or versions) of CMake,
please let me know.

- Christian

Hi Yury,

>> I glad to hear it. I suppose you can just try build postgres and send all
>> problems to github tracker.
>> https://github.com/stalkerg/postgres_cmake/issues

FYI, I had success using your "postgres_cmake" repo. I tested it up
through "make check" and "make install".

Here are the details:

* postgres_cmake commit e7e75160d4533cd8caa9f3f0dd7b485dbd4e7bdf
* compiler = cc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
* cmake version = 3.5.3

Kind regards,
Christian

Yury,

On Mon, Aug 22, 2016 at 7:48 AM, Christian Convey
<christian(dot)convey(at)gmail(dot)com> wrote:
>>> I glad to hear it. I suppose you can just try build postgres and send all
>>> problems to github tracker.
>>> https://github.com/stalkerg/postgres_cmake/issues
>
> FYI, I had success using your "postgres_cmake" repo. I tested it up
> through "make check" and "make install".
>
> Here are the details:
>
> * postgres_cmake commit e7e75160d4533cd8caa9f3f0dd7b485dbd4e7bdf
> * compiler = cc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
> * cmake version = 3.5.3

Could it be possible to get a refreshed patch on this thread for at
least the sake of the archives? I'd really like to see somehting
happening here and do some progress for this CF.
--
Michael

Michael Paquier wrote:
> Could it be possible to get a refreshed patch on this thread for at
> least the sake of the archives? I'd really like to see somehting
> happening here and do some progress for this CF.

Sure, I will do it on Friday.
Today I finished mingw+msys support. (mingw without msys has not passed
some tests)
--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Fri, Sep 16, 2016 at 4:38 AM, Yury Zhuravlev
<u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
> Michael Paquier wrote:
>>
>> Could it be possible to get a refreshed patch on this thread for at
>> least the sake of the archives? I'd really like to see somehting
>> happening here and do some progress for this CF.
>
>
> Sure, I will do it on Friday.
> Today I finished mingw+msys support. (mingw without msys has not passed some
> tests)

Okay. That sounds good to me. I don't recall what your patch is
exactly doing but could you still keep the vanilla Makefiles around?
This will reduce the diff of the patch, and we'd need anyway to keep
the former Makefile methods around until the buildfarm scripts are
updated, the animals do the switch and then get green. So a period
where both live in parallel is unavoidable.

I heard as well that MySQL is using cmake... It may be interesting to
see how they are integrating with it as a large project and avoid
their past mistakes.
--
Michael

On Fri, Sep 16, 2016 at 9:58 AM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> Okay. That sounds good to me. I donas well a look around, and cmake really looks like a robust alternative to ./configure. Now I am aware of the fact that your patch recommends to build 't recall what your patch is
> exactly doing but could you still keep the vanilla Makefiles around?
> This will reduce the diff of the patch, and we'd need anyway to keep
> the former Makefile methods around until the buildfarm scripts are
> updated, the animals do the switch and then get green. So a period
> where both live in parallel is unavoidable.
>
> I heard as well that MySQL is using cmake... It may be interesting to
> see how they are integrating with it as a large project and avoid
> their past mistakes.

I could not resist so I just had a look at your patch. I had as well a
look around, and cmake really looks like a robust alternative to
./configure. In short, people doing now that:
./configure
make
make install
Would just do that:
cmake .
make
make install

I am no cmake guru yet, but VPATH builds are supported out of the box,
which is cool.

Your patch recommends to build with cmake after creating build/, now I
would expect most users to run cmake from the root folder. However
this causes all the Makefiles to get overwritten. As supporting all
platforms at once with cmake is going to be uncommitable, we are going
to need both methods able to live together for a while. Well, they can
coexist with this patch as long as cmake is not run from the root of
the code tree, which is acceptable for me as long as the switch is not
completed. However others may think differently on the matter.

Instead of getting support for all existing platforms, I would
recommend as well focusing only on one platform for the time being,
Linux, and get the work done correctly for that first. Once there is
something committed, we will be able to patch the buildfarm, and get
machines to switch to cmake one by one. After those are switched, we
could extend that. Another point of contention is support for
extensions. How long should we keep support for the existing PGXS? How
external extensions would compile with the new thing infrastructure?

Which brings me to another point, your patch is focused on features,
meaning that per-OS checks are all grouped by feature, but it may be a
good idea to split checks by OS if necessary, with for example
per-platform files and scripts in cmake/os/. And we could have just
something for Linux now.

A couple of issues I have noticed with your patch after a first set of tests:
- root's .gitignore needs to add entries for CMakeFiles and
cmake_install.cmake. You need as well to ignore CMakeCache
- A couple of headers are generated, like cubeparse.h (?)
- Currently a lot of users do things like that:
cd src/test/regress/ && make check
But this patch breaks that, and that's not cool. Recovery tests in
src/test/regress won't run either.

That's all I have for now. Looking forward to seeing some progress here.
--
Michael

Michael Paquier wrote:
> Your patch recommends to build with cmake after creating build/, now I
> would expect most users to run cmake from the root folder. However
> this causes all the Makefiles to get overwritten. As supporting all
> platforms at once with cmake is going to be uncommitable, we are going
> to need both methods able to live together for a while. Well, they can
> coexist with this patch as long as cmake is not run from the root of
> the code tree, which is acceptable for me as long as the switch is not
> completed. However others may think differently on the matter.

It's really a good point. Forbidden run cmake from root it is better
decision for me (of course for start).

>Instead of getting support for all existing platforms, I would
>recommend as well focusing only on one platform for the time being,
>Linux, and get the work done correctly for that first. Once there is
>something committed, we will be able to patch the buildfarm, and get
>machines to switch to cmake one by one. After those are switched, we
>could extend that.

You mean in first version of patch I can focus on Linux systems?

>Another point of contention is support for
>extensions. How long should we keep support for the existing PGXS? How
>external extensions would compile with the new thing infrastructure?

As long as possible. I hope I can make PGXS Makefiles generator.

>Which brings me to another point, your patch is focused on features,
>meaning that per-OS checks are all grouped by feature, but it may be a
>good idea to split checks by OS if necessary, with for example
>per-platform files and scripts in cmake/os/. And we could have just
>something for Linux now.

Currently I do not have a lot OS specific tests. All checks are doing in
same manner.

>- root's .gitignore needs to add entries for CMakeFiles and
>cmake_install.cmake. You need as well to ignore CMakeCache

Thanks I done this in last commit.

>- A couple of headers are generated, like cubeparse.h (?)

Because BISON generate header by default. I suppose author of cube launched
bison by hand but I made it automatic.

>- Currently a lot of users do things like that:
>cd src/test/regress/ && make check
>But this patch breaks that, and that's not cool. Recovery tests in
>src/test/regress won't run either.

It seems restriction by design because in CMake you have only one enter
point.

>That's all I have for now. Looking forward to seeing some progress here.

I merged master to my branch and I spent time to porting all changes. I
hope send patch in the weekend without terrible flaws.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Sat, Sep 17, 2016 at 1:40 AM, Yury Zhuravlev
<u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
> Michael Paquier wrote:
> I merged master to my branch and I spent time to porting all changes. I hope
> send patch in the weekend without terrible flaws.

By the way, I noticed that you did not register this patch in the current CF..
--
Michael

Hi,

On 2016-09-16 22:33:50 +0900, Michael Paquier wrote:
> As supporting all platforms at once with cmake is going to be
> uncommitable, we are going to need both methods able to live together
> for a while.

I very strongly disagree with this. It's way too likely that we end up
releasing with multiple buildsystems that way. I think we'll have to
require support for the most common OSs, and then fix up the fallout
afterwards.

Andres

On Sat, Sep 17, 2016 at 8:11 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi,
>
> On 2016-09-16 22:33:50 +0900, Michael Paquier wrote:
>> As supporting all platforms at once with cmake is going to be
>> uncommitable, we are going to need both methods able to live together
>> for a while.
>
> I very strongly disagree with this. It's way too likely that we end up
> releasing with multiple buildsystems that way. I think we'll have to
> require support for the most common OSs, and then fix up the fallout
> afterwards.

Ok, then if we go through the hard method the switch patch had better
remove as well all the Makefiles in the tree and recommend cmake run
from base root as the default build method. By the way, there will be
a strong need for docs in the patch sooner or later...
--
Michael

Michael Paquier wrote:
> On Sat, Sep 17, 2016 at 1:40 AM, Yury Zhuravlev
> <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>> Michael Paquier wrote:
>> I merged master to my branch and I spent time to porting all
>> changes. I hope
>> send patch in the weekend without terrible flaws.
>
> By the way, I noticed that you did not register this patch in
> the current CF..

Now, I published the first version of the patch.
This patch not ready for commit yet and all current task you can read here:
https://github.com/stalkerg/postgres_cmake/issues

I hope we realy close to end.

In this patch I forbade in-source build for avoid overwriting current
Makefiles.
We will can remove all Makefiles only after shall see in CMake. You don't
need support two system. During commitfests CMake build system will be
supported by me.

I need help with buildfarm because my knowledge of Perl is very bad
(thought about rewrite buildfarm to Python).

I hope for your support.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Andres Freund wrote:
> It's way too likely that we end up
> releasing with multiple buildsystems that way.

If we fail in the CMake integration we can easily remove all CMake files
before code freeze.
I hope step by step CMake integration better way for current situation.
But I do not insist, just I trying to find the easiest way.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 9/16/16 9:33 AM, Michael Paquier wrote:
> I am no cmake guru yet, but VPATH builds are supported out of the box,
> which is cool.
>
> Your patch recommends to build with cmake after creating build/, now I
> would expect most users to run cmake from the root folder.

My understanding is that cmake strongly recommends building in a
separate directory, which is usually a subdirectory named something like
"build". So the above is likely going to be the standard workflow.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

> On 17 Sep 2016, at 20:21, Yury Zhuravlev <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>
> Michael Paquier wrote:
>> On Sat, Sep 17, 2016 at 1:40 AM, Yury Zhuravlev
>> <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>>> Michael Paquier wrote:
>>> I merged master to my branch and I spent time to porting all changes. I hope
>>> send patch in the weekend without terrible flaws.
>>
>> By the way, I noticed that you did not register this patch in the current CF..
>
> Now, I published the first version of the patch. This patch not ready for commit yet and all current task you can read here:
> https://github.com/stalkerg/postgres_cmake/issues
>
> I hope we realy close to end.
> In this patch I forbade in-source build for avoid overwriting current Makefiles. We will can remove all Makefiles only after shall see in CMake. You don't need support two system. During commitfests CMake build system will be supported by me.
> I need help with buildfarm because my knowledge of Perl is very bad (thought about rewrite buildfarm to Python).
>
> I hope for your support.

Tried to generate Xcode project out of cmake, build fails on genbki.pl: can't locate Catalog.pm (which itself lives in src/backend/catalog/Catalog.pm)

--
Stas Kelvich
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

Stas Kelvich wrote:
>> On 17 Sep 2016, at 20:21, Yury Zhuravlev
>> <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>>
>> Michael Paquier wrote: ...
>
>
> Tried to generate Xcode project out of cmake, build fails on
> genbki.pl: can't locate Catalog.pm (which itself lives in
> src/backend/catalog/Catalog.pm)
>

Can you try again? On my Xcode 7.3 / ElCapitan everything is working
correctly.
--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Stas Kelvich wrote:
> Tried to generate Xcode project out of cmake, build fails on
> genbki.pl: can't locate Catalog.pm (which itself lives in
> src/backend/catalog/Catalog.pm)

Thanks again! I have corrected your issue.
--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 9/17/16 1:21 PM, Yury Zhuravlev wrote:
> Now, I published the first version of the patch.

I tried this out. Because of some file moves in initdb and
pg_basebackup, the build fails:

[ 74%] Linking C executable initdb
Undefined symbols for architecture x86_64:
"_fsync_pgdata", referenced from:
_main in initdb.c.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [src/bin/initdb/CMakeFiles/initdb.dir/build.make:177:
src/bin/initdb/initdb] Error 1
make[1]: *** [CMakeFiles/Makefile2:2893:
src/bin/initdb/CMakeFiles/initdb.dir/all] Error 2
make: *** [Makefile:128: all] Error 2

Please submit an updated patch.

I suggest you use git format-patch to produce patches. This is easier
to apply, especially when there are a lot of new files involved. Also
use the git facilities to check for whitespace errors.

Please supply some documentation, such as

- what are the basic commands
- how to set include/library paths, choose configure options
- how to set CFLAGS
- how to see raw build commands
- what are the targets for all/world/check/docs etc.
- explain directory structure

I suggest for now you could put this into a README.cmake file in your
patch. We don't need to commit it that way, but it would help in the
meantime.

When I run cmake without options, it seems to do opportunistic feature
checking. For example, it turns on OpenSSL or Python support if it can
find it, otherwise it turns it off. We need this to be deterministic.
Without options, choose the basic feature set, require all other
features to be turned on explicitly, fail if they can't be found.
Whatever the Autoconf-based build does now has been fairly deliberately
tuned, so there should be very little reason to deviate from that.

The Python check appears to be picking up pieces from two different
Python installations:

-- Found PythonInterp: /usr/local/bin/python (found version "2.7.12")
-- Found PythonLibs: /usr/lib/libpython2.7.dylib (found version "2.7.10")

The check results otherwise look OK, but I'm a bit confused about the
order. It checks for some functions before all the header files are
checked for. Is that intentional?

There are a number of changes in .[ch] and .pl files that are unclear
and not explained. Please explain them. You can also submit separate
preliminary patches if you need to do some refactoring. Ultimately, I
would expect this patch not to require C code changes.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hello.

Peter Eisentraut wrote:
> I tried this out. Because of some file moves in initdb and
> pg_basebackup, the build fails:
Yes, I need rebase patch to latest master. I do it as soon as possible.
Also I have some new achievements.

>I suggest you use git format-patch to produce patches. This is easier
>to apply, especially when there are a lot of new files involved. Also
>use the git facilities to check for whitespace errors.
I agree.

>I suggest for now you could put this into a README.cmake file in your
>patch. We don't need to commit it that way, but it would help in the
>meantime.
Good idea, currently I write all documentation on github. I will try to
move it to patch.

>When I run cmake without options, it seems to do opportunistic feature
>checking. For example, it turns on OpenSSL or Python support if it can
>find it, otherwise it turns it off. We need this to be deterministic.
>Without options, choose the basic feature set, require all other
>features to be turned on explicitly, fail if they can't be found.
>Whatever the Autoconf-based build does now has been fairly deliberately
>tuned, so there should be very little reason to deviate from that.
This approach I see only in Postgres project and not fully understood.
Can you explain me more what reasons led to this approach?
Not big deal to make behavior like postgres Autoconf but I think it's
important clear view here.

>The Python check appears to be picking up pieces from two different
>Python installations:
Ooops you right. For Python detecting I use standard CMake module and
his behavior depending on CMake version. We should really careful here.

>The check results otherwise look OK, but I'm a bit confused about the
>order. It checks for some functions before all the header files are
>checked for. Is that intentional?
I have plans to change order checks.

>There are a number of changes in .[ch] and .pl files that are unclear
>and not explained. Please explain them. You can also submit separate
>preliminary patches if you need to do some refactoring. Ultimately, I
>would expect this patch not to require C code changes.
I suppose separate patches with comments will be best way. I will do it.
I think we can talks about details after that.

Big thanks for your remarks it's very important for me and this "small"
project.
I will try to do all tasks quickly.

Best regards.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 9 Nov. 2016 06:37, "Yury Zhuravlev" <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>
>> When I run cmake without options, it seems to do opportunistic feature
>> checking. For example, it turns on OpenSSL or Python support if it can
>> find it, otherwise it turns it off. We need this to be deterministic.
>> Without options, choose the basic feature set, require all other
>> features to be turned on explicitly, fail if they can't be found.
>> Whatever the Autoconf-based build does now has been fairly deliberately
>> tuned, so there should be very little reason to deviate from that.
>
> This approach I see only in Postgres project and not fully understood.
> Can you explain me more what reasons led to this approach?

It's predictable. The default has the same result for everyone. I quite
like it myself.

On Wed, Nov 9, 2016 at 7:54 AM, Craig Ringer
<craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
> On 9 Nov. 2016 06:37, "Yury Zhuravlev" <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>> This approach I see only in Postgres project and not fully understood.
>> Can you explain me more what reasons led to this approach?
>
> It's predictable. The default has the same result for everyone. I quite like
> it myself.

+1. Let's tell to the system what we want him to do and not let him
guess what we'd like to be done or it will get harder to test and
develop code for all kind of code paths with #ifdef's. That's one step
away from Skynet.
--
Michael

On 9 November 2016 at 10:12, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
> On Wed, Nov 9, 2016 at 7:54 AM, Craig Ringer
> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>> On 9 Nov. 2016 06:37, "Yury Zhuravlev" <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>>> This approach I see only in Postgres project and not fully understood.
>>> Can you explain me more what reasons led to this approach?
>>
>> It's predictable. The default has the same result for everyone. I quite like
>> it myself.
>
> +1. Let's tell to the system what we want him to do and not let him
> guess what we'd like to be done or it will get harder to test and
> develop code for all kind of code paths with #ifdef's. That's one step
> away from Skynet.

Er... ok then. (Backs away slowly).

More seriously, I like it for development where a stable and
predictable default is great.

For users it slightly sucks, as most users will want us to find
whatever is on the system without being manually told to enable each
feature. "Of course I want SSL, I have openssl installed don't I?"
It's not like we require users to specify --enable-largefile
--enable-atomics --enable-getopt --enable-ipv6 .... we do detect a
lot automatically.

So personally I think it'd be fine if a cmake build defaulted to
finding and using what it could, but offered a --minimal mode or
whatever that gets us just core postgres + whatever we enable
explicitly. But our current behaviour is OK too.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Craig Ringer <craig(dot)ringer(at)2ndquadrant(dot)com> writes:
> On 9 Nov. 2016 06:37, "Yury Zhuravlev" <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>> This approach I see only in Postgres project and not fully understood.
>> Can you explain me more what reasons led to this approach?

> It's predictable. The default has the same result for everyone. I quite
> like it myself.

It's advantageous from a packaging standpoint, precisely because it's
predictable: either you get the set of features you expect, or you get
a build failure. Years ago we were more on the "opportunistic" side
of things, and we had problems with packages sometimes silently losing
features. A pretty-recent example is that OpenSSL changed their APIs
in 1.1.0 so that our configure probe failed. With an opportunistic
approach, that would have meant builds silently going from "ssl yes"
to "ssl no". That's not good.

So this is really not open for negotiation. As Peter said upthread,
what we are looking for in a CMake reimplementation is that it behaves
exactly like the Autoconf version does. To the extent that you are unable
or unwilling to duplicate that behavior, you increase the odds that
we'll reject this work.

regards, tom lane

Tom Lane wrote:
> So this is really not open for negotiation. As Peter said upthread,
> what we are looking for in a CMake reimplementation is that it behaves
> exactly like the Autoconf version does. To the extent that you are unable
> or unwilling to duplicate that behavior, you increase the odds that
> we'll reject this work.
Who asking about negotiation? I just wanted an explanation for the clear
understanding and nothing more.
Now I know about reasons. Thanks.

regards
--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Craig Ringer wrote:
> So personally I think it'd be fine if a cmake build defaulted to
> finding and using what it could, but offered a --minimal mode or
> whatever that gets us just core postgres + whatever we enable
> explicitly. But our current behaviour is OK too.

To me it's best way. But I'm not sure what Tom Lane will accept this.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Craig Ringer wrote:
> So personally I think it'd be fine if a cmake build defaulted to
> finding and using what it could, but offered a --minimal mode or
> whatever that gets us just core postgres + whatever we enable
> explicitly. But our current behaviour is OK too.

To me it's best way. But I'm not sure what Tom Lane will accept this.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 11/11/16 08:15, Yury Zhuravlev wrote:

> Craig Ringer wrote:
>> So personally I think it'd be fine if a cmake build defaulted to
>> finding and using what it could, but offered a --minimal mode or
>> whatever that gets us just core postgres + whatever we enable
>> explicitly. But our current behaviour is OK too.
>
> To me it's best way. But I'm not sure what Tom Lane will accept this.
>
>

I just had a play with this patch - nice! (ok so it needs a fix so that
the compile completes as mentioned prev).

I would recommend making it behave as Tom suggested. *Then* add an
--autodetect or similar option that makes
behave in the 'finding and using what it could' manner as a 2nd patch.

regards

Mark

Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> writes:
> I would recommend making it behave as Tom suggested. *Then* add an
> --autodetect or similar option that makes
> behave in the 'finding and using what it could' manner as a 2nd patch.

An "--autodetect" switch would be fine with me. I just don't want it
to behave that way by default, because then you get into the unexpected
results problem.

regards, tom lane

I had a bit a play around to see if I could get this building properly
again with v10 (i.e master). I've attached a minimal *incremental* patch
that needs to be applied after v1. This gets everything under the src
tree building (added the new file_utils.c and reordered some link libs
and removed some duplicates).

I haven't made any changes with respect to it trying to detect and build
everything. One added nit I see is that unless I'm missing something
there appears to be no way to stop it trying to build all the
contribs...so an option to enable/disable their build is needed.

To make it display what options there are I use:

$ mkdir build; cd build ; cmake .. -LH

And to do a build that works:

$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/pgsql/10 -DWITH_PERL=OFF
-DWITH_OPENSSL=OFF -DWITH_PYTHON=OFF

With reference to Tom's comments, I'm thinking that these should all
default to 'OFF' plus an additional variable WITH_CONTRIB (or similar)
should default to OFF too.

regards

Mark

On 09/11/16 08:37, Peter Eisentraut wrote:
> On 9/17/16 1:21 PM, Yury Zhuravlev wrote:
>> Now, I published the first version of the patch.
> I tried this out. Because of some file moves in initdb and
> pg_basebackup, the build fails:
>
> [ 74%] Linking C executable initdb
> Undefined symbols for architecture x86_64:
> "_fsync_pgdata", referenced from:
> _main in initdb.c.o
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status
> make[2]: *** [src/bin/initdb/CMakeFiles/initdb.dir/build.make:177:
> src/bin/initdb/initdb] Error 1
> make[1]: *** [CMakeFiles/Makefile2:2893:
> src/bin/initdb/CMakeFiles/initdb.dir/all] Error 2
> make: *** [Makefile:128: all] Error 2
>
> Please submit an updated patch.
>
> I suggest you use git format-patch to produce patches. This is easier
> to apply, especially when there are a lot of new files involved. Also
> use the git facilities to check for whitespace errors.
>
> Please supply some documentation, such as
>
> - what are the basic commands
> - how to set include/library paths, choose configure options
> - how to set CFLAGS
> - how to see raw build commands
> - what are the targets for all/world/check/docs etc.
> - explain directory structure
>
> I suggest for now you could put this into a README.cmake file in your
> patch. We don't need to commit it that way, but it would help in the
> meantime.
>
> When I run cmake without options, it seems to do opportunistic feature
> checking. For example, it turns on OpenSSL or Python support if it can
> find it, otherwise it turns it off. We need this to be deterministic.
> Without options, choose the basic feature set, require all other
> features to be turned on explicitly, fail if they can't be found.
> Whatever the Autoconf-based build does now has been fairly deliberately
> tuned, so there should be very little reason to deviate from that.
>
> The Python check appears to be picking up pieces from two different
> Python installations:
>
> -- Found PythonInterp: /usr/local/bin/python (found version "2.7.12")
> -- Found PythonLibs: /usr/lib/libpython2.7.dylib (found version "2.7.10")
>
> The check results otherwise look OK, but I'm a bit confused about the
> order. It checks for some functions before all the header files are
> checked for. Is that intentional?
>
> There are a number of changes in .[ch] and .pl files that are unclear
> and not explained. Please explain them. You can also submit separate
> preliminary patches if you need to do some refactoring. Ultimately, I
> would expect this patch not to require C code changes.
>

Yeah, there seems to be a lot of these. Looking through them almost all
concern the addition of piece of code to wrap putenv. e.g:

--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -1350,7 +1350,7 @@ exec_command(const char *cmd,
char *newval;

newval = psprintf("%s=%s", envvar, envval);
- putenv(newval);
+ pg_putenv_proxy(newval);
success = true;

/*

Where pg_putenv_proxy either calls putenv or pgwin32_putenv (the latter
on windows I'd guess). I wonder if this could have been avoided, since
the original code handles this sort of thing. There are also some minor
- and not immediately obvious - changes to a number of macros in various
includes...If I'm feeling keen I'll experiment to see how far I can get
without any source changes at all.

regards

Mark

On 09/11/16 08:37, Peter Eisentraut wrote:
>
> There are a number of changes in .[ch] and .pl files that are unclear
> and not explained. Please explain them. You can also submit separate
> preliminary patches if you need to do some refactoring. Ultimately, I
> would expect this patch not to require C code changes.
>

Actually, it was not that tricky to separate out the cmake only changes,
and test this on unmodified sources. It appears to work fine for me -
passes 'make check' (needs the v1_1 incremental patch applied of
course). The Patch is attached. I wonder if the original had some
changes for building under latest Windows...(I'm using Ubuntu 16.10,
with cmake 3.5).

regards

Mark

On 16/11/16 16:00, Mark Kirkwood wrote:
> Yeah, there seems to be a lot of these. Looking through them almost
> all concern the addition of piece of code to wrap putenv. e.g:
>
> --- a/src/bin/psql/command.c
> +++ b/src/bin/psql/command.c
> @@ -1350,7 +1350,7 @@ exec_command(const char *cmd,
> char *newval;
>
> newval = psprintf("%s=%s", envvar, envval);
> - putenv(newval);
> + pg_putenv_proxy(newval);
> success = true;
>
> /*
>
> Where pg_putenv_proxy either calls putenv or pgwin32_putenv (the
> latter on windows I'd guess). I wonder if this could have been
> avoided, since the original code handles this sort of thing. There are
> also some minor - and not immediately obvious - changes to a number of
> macros in various includes...If I'm feeling keen I'll experiment to
> see how far I can get without any source changes at all.
>
>
> regards
>
>
> Mark
>
>
> On 09/11/16 08:37, Peter Eisentraut wrote:
>>
>> There are a number of changes in .[ch] and .pl files that are unclear
>> and not explained. Please explain them. You can also submit separate
>> preliminary patches if you need to do some refactoring. Ultimately, I
>> would expect this patch not to require C code changes.
>>
>

Mark Kirkwood wrote:
> Actually, it was not that tricky to separate out the cmake only
> changes, and test this on unmodified sources. It appears to work
> fine for me - passes 'make check' (needs the v1_1 incremental
> patch applied of course). The Patch is attached. I wonder if the
> original had some changes for building under latest
> Windows...(I'm using Ubuntu 16.10, with cmake 3.5).
>
Thanks for all your works! Can you make push request here:
https://github.com/stalkerg/postgres_cmake

I have rebased (merge) to master and make other important fix.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Mark Kirkwood wrote:
> Yeah, there seems to be a lot of these. Looking through them
> almost all concern the addition of piece of code to wrap putenv.
> e.g:
I made this small wrapper special for MSVC 2015 without Service Packs
because postgres macross were in conflict with MS internal functions.
After some time and some updates for MSVC Michael Paquier could not
reproduce my problem but I keep this patch to avoid problems in the
future.
I can check old behavior again and revert all changes if needed and
ofcourse I have plans to make separate patch for this changes.

Thanks!
--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On Tue, Nov 8, 2016 at 9:12 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> On Wed, Nov 9, 2016 at 7:54 AM, Craig Ringer
> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>> On 9 Nov. 2016 06:37, "Yury Zhuravlev" <u(dot)zhuravlev(at)postgrespro(dot)ru> wrote:
>>> This approach I see only in Postgres project and not fully understood.
>>> Can you explain me more what reasons led to this approach?
>>
>> It's predictable. The default has the same result for everyone. I quite like
>> it myself.
>
> +1. Let's tell to the system what we want him to do and not let him
> guess what we'd like to be done or it will get harder to test and
> develop code for all kind of code paths with #ifdef's. That's one step
> away from Skynet.

Exaggerate much?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

I see there are some patches for the putenv issue with Visual studio
2013 in progress on this list - it is probably best to work with the
author to see if 2015 has any new issues and keep all changes for that
*out* of the cmake patches.

regards

Mark

On 16/11/16 21:22, Yury Zhuravlev wrote:
> I made this small wrapper special for MSVC 2015 without Service Packs
> because postgres macross were in conflict with MS internal functions.
> After some time and some updates for MSVC Michael Paquier could not
> reproduce my problem but I keep this patch to avoid problems in the
> future. I can check old behavior again and revert all changes if
> needed and ofcourse I have plans to make separate patch for this changes.
> Thanks!

On Wed, Nov 16, 2016 at 2:14 PM, Mark Kirkwood
<mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> wrote:
> I see there are some patches for the putenv issue with Visual studio 2013 in
> progress on this list - it is probably best to work with the author to see
> if 2015 has any new issues and keep all changes for that *out* of the cmake
> patches.

I don't recall all the details here, but no wrappers should be needed,
particularly on Windows where we already do that:
src/include/port/win32.h:#define putenv(x) pgwin32_putenv(x)
--
Michael

Michael Paquier wrote:
> src/include/port/win32.h:#define putenv(x) pgwin32_putenv(x)
and my MSVC2015 drop down here because pgwin32_putenv has wrong signature.
I hope it is not true now.

--
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Hello hackers.

----

I think you notice what I changed email, it is happened because I quit from
PgPro.

Good news is that I have time to work on CMake. Bad news I have to find a
new job. ;)

----

What is new in that patchset:

1.

I separated changes to several patches. Main patch include only new
CMake files without postgres files changes.
2.

I cleaned all the unnecessary postgres changes.
3.

I update cmake with latest master, add STRONG_RANDOM option and etc.
4.

Add small document about CMake and Postgres. (README.cmake bad idea for
name but I think now it’s ok)
5.

I have stopped use gendef.pl for MSVC build.
6.

Fix for AIX 7.1

About patches:

cmake_v2_1_main_files.patch

Only CMake files. CMakeFiles.txt is main files with rules. *.cmake is
special modules with macros and functions. *.in is template for generate
headers. (like autoconf)

cmake_v2_2_c_define.patch

Small chages in c.h . At first it is “#pragma fenv_access (off)” it is
necessary if we use /fp:strict for MSVC compiler. Without this pragma we
can’t calc floats for const variables in compiller time (2 * M_PI for
example). Strict mode important if we want to be close with ieee754 float
format on MSVC (1.0 / 0.0 = inf for example). Detail info here:
https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx

Second change is because I find and set HAVE_INT128 directly from CMake.
PG_INT128_TYPE used only for autoconfig scripts.

cmake_v2_3_rijndael.patch

First I added special wraparound because here CMake have circular
dependency (cmake very smart here). Second I removed rijndael.tbl because
it generated during build process every time.

cmake_v2_4_uuid.patch

Another small patch. Right place for uuid.h I find by CMake and not
necessary this ifdef hell.

cmake_v2_5_readme.patch

Small exercise to CMake world for all who want to try.

Questions for discussion:

In generated project by CMake we always have only one enter point. Also
INSTALL macross support only including to “all” targets. It follows that it
is impossible build contrib modules separately only with “all” target. Here
write about this behavior:
https://cmake.org/cmake/help/v3.7/prop_tgt/EXCLUDE_FROM_ALL.html

Interesting note:

Many folks here love only command line but many novices developers from
modern generation love GUI. Also using command line on Windows it's really
big pain. With CMake you can work with Postgres under MSVC more
comfortable. For example my screenshots of regress tests result runing from
MSVC: https://twitter.com/stalkerg/status/814423972263657472

Fin:

I hope this patchset one step closer to merge. In future I see a lot of
work but I suppose it is right and important direction. I'm ready for this
long journey.

Big thanks Peter Eisentraut for advices and the right vector of
development.

Also big thanks Mark Kirkwood for help and discussion.

Happy New Year, everyone!

On 12/30/16 9:10 AM, Yuriy Zhuravlev wrote:
> cmake_v2_2_c_define.patch
>
> Small chages in c.h . At first it is “#pragma fenv_access (off)” it is
> necessary if we use /fp:strict for MSVC compiler. Without this pragma we
> can’t calc floats for const variables in compiller time (2 * M_PI for
> example). Strict mode important if we want to be close with ieee754
> float format on MSVC (1.0 / 0.0 = inf for example). Detail info here:
> https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx

I don't understand what this has to do with cmake. If this is a
worthwhile improvement for the Windows build, then please explain why,
with a "before" and "after" output and a patch for the existing build
system as well.

> Second change is because I find and set HAVE_INT128 directly from CMake.
> PG_INT128_TYPE used only for autoconfig scripts.

It might also be worth refactoring the existing Autoconf code here to
make this consistent.

(My assumption is that if we were to move forward with cmake or any
other build system change, we would have to keep the old one alongside
at least for a little while. So any changes to the C code would need to
be side-ported.)

> cmake_v2_3_rijndael.patch
>
> First I added special wraparound because here CMake have circular
> dependency (cmake very smart here). Second I removed rijndael.tbl
> because it generated during build process every time.

Please explain what the circular dependency is. If there is one, we
should also side-port this change.

> cmake_v2_4_uuid.patch
>
> Another small patch. Right place for uuid.h I find by CMake and not
> necessary this ifdef hell.

This patch removes the uuid.h include but doesn't add it anywhere else.
How does it work?

> Questions for discussion:
>
> In generated project by CMake we always have only one enter point. Also
> INSTALL macross support only including to “all” targets. It follows that
> it is impossible build contrib modules separately only with “all”
> target. Here write about this behavior:
> https://cmake.org/cmake/help/v3.7/prop_tgt/EXCLUDE_FROM_ALL.html

Yeah, I think this is how the MSVC stuff effectively works right now as
well.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Jan 3, 2017 at 11:11 PM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> On 12/30/16 9:10 AM, Yuriy Zhuravlev wrote:
>> cmake_v2_2_c_define.patch
>>
>> Small chages in c.h . At first it is “#pragma fenv_access (off)” it is
>> necessary if we use /fp:strict for MSVC compiler. Without this pragma we
>> can’t calc floats for const variables in compiller time (2 * M_PI for
>> example). Strict mode important if we want to be close with ieee754
>> float format on MSVC (1.0 / 0.0 = inf for example). Detail info here:
>> https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx
>
> I don't understand what this has to do with cmake. If this is a
> worthwhile improvement for the Windows build, then please explain why,
> with a "before" and "after" output and a patch for the existing build
> system as well.
>
>> Second change is because I find and set HAVE_INT128 directly from CMake.
>> PG_INT128_TYPE used only for autoconfig scripts.
>
> It might also be worth refactoring the existing Autoconf code here to
> make this consistent.
>
> (My assumption is that if we were to move forward with cmake or any
> other build system change, we would have to keep the old one alongside
> at least for a little while. So any changes to the C code would need to
> be side-ported.)
>
>> cmake_v2_3_rijndael.patch
>>
>> First I added special wraparound because here CMake have circular
>> dependency (cmake very smart here). Second I removed rijndael.tbl
>> because it generated during build process every time.
>
> Please explain what the circular dependency is. If there is one, we
> should also side-port this change.
>
>> cmake_v2_4_uuid.patch
>>
>> Another small patch. Right place for uuid.h I find by CMake and not
>> necessary this ifdef hell.
>
> This patch removes the uuid.h include but doesn't add it anywhere else.
> How does it work?
>
>> Questions for discussion:
>>
>> In generated project by CMake we always have only one enter point. Also
>> INSTALL macross support only including to “all” targets. It follows that
>> it is impossible build contrib modules separately only with “all”
>> target. Here write about this behavior:
>> https://cmake.org/cmake/help/v3.7/prop_tgt/EXCLUDE_FROM_ALL.html
>
> Yeah, I think this is how the MSVC stuff effectively works right now as
> well.

I am marking this patch as "returned with feedback". That's quite a
heavy change and it looks to be too late in the development cycle of
PG10 to consider it. Peter's commit bits, who is also the reviewer,
are beginning to smoke as well after everything that has happened for
the logical replication changes.
--
Michael

On 2017-01-24 15:50:47 +0900, Michael Paquier wrote:
> I am marking this patch as "returned with feedback". That's quite a
> heavy change and it looks to be too late in the development cycle of
> PG10 to consider it. Peter's commit bits, who is also the reviewer,
> are beginning to smoke as well after everything that has happened for
> the logical replication changes.

I'm doubtful about this being ready in time too, but it seems a might
heavyhanded to make that call on your own. Including the judgement about
Peter's capability to handle more.

Greetings,

Andres Freund

On Tue, Jan 24, 2017 at 3:58 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2017-01-24 15:50:47 +0900, Michael Paquier wrote:
>> I am marking this patch as "returned with feedback". That's quite a
>> heavy change and it looks to be too late in the development cycle of
>> PG10 to consider it. Peter's commit bits, who is also the reviewer,
>> are beginning to smoke as well after everything that has happened for
>> the logical replication changes.
>
> I'm doubtful about this being ready in time too, but it seems a might
> heavyhanded to make that call on your own. Including the judgement about
> Peter's capability to handle more.

Sure, sorry for that. I am switching back the patch, let's see what
happens by the end of the CF.
--
Michael

On 24 Jan. 2017 18:00, "Andres Freund" <andres(at)anarazel(dot)de> wrote:

On 2017-01-24 15:50:47 +0900, Michael Paquier wrote:
> I am marking this patch as "returned with feedback". That's quite a
> heavy change and it looks to be too late in the development cycle of
> PG10 to consider it. Peter's commit bits, who is also the reviewer,
> are beginning to smoke as well after everything that has happened for
> the logical replication changes.

I'm doubtful about this being ready in time too, but it seems a might
heavyhanded to make that call on your own. Including the judgement about
Peter's capability to handle more.

Personally I think we should aim to have this in as a non default build
mode in pg10 if it can be made ready, and aim to make it default in pg11 at
least for Windows.

Craig Ringer <craig(dot)ringer(at)2ndquadrant(dot)com> writes:
> Personally I think we should aim to have this in as a non default build
> mode in pg10 if it can be made ready, and aim to make it default in pg11 at
> least for Windows.

AFAIK we haven't committed to accepting this at all, let alone trying
to do so on a tight schedule. And I believe there was general agreement
that we would not accept it as something to maintain in parallel with
the existing makefiles. If we have to maintain two build systems, we
have that already.

regards, tom lane

On 1/24/17 8:37 AM, Tom Lane wrote:
> Craig Ringer <craig(dot)ringer(at)2ndquadrant(dot)com> writes:
>> Personally I think we should aim to have this in as a non default build
>> mode in pg10 if it can be made ready, and aim to make it default in pg11 at
>> least for Windows.
>
> AFAIK we haven't committed to accepting this at all, let alone trying
> to do so on a tight schedule. And I believe there was general agreement
> that we would not accept it as something to maintain in parallel with
> the existing makefiles. If we have to maintain two build systems, we
> have that already.

My preferred scenario would be to replace the Windows build system by
this first, then refine it, then get rid of Autoconf.

The ideal timeline would be to have a ready patch to commit early in a
development cycle, then get rid of the Windows build system by the end
of it. Naturally, this would need buy-in from Windows developers.

I don't foresee replacing the Autoconf build system by this immediately.

Right now, however, the patch isn't moving at all, and I don't see it
going into PG10, so I'm fine with returning with feedback.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut wrote:

> Right now, however, the patch isn't moving at all, and I don't see it
> going into PG10, so I'm fine with returning with feedback.

There are a bunch of side patches that we should apply separately, such
as the pgcrypto fix. I don't understand why they are part of this patch
series, really, given that they seem largely independent prerequisites.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jan 27, 2017 at 11:09 PM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> On 1/24/17 8:37 AM, Tom Lane wrote:
>> Craig Ringer <craig(dot)ringer(at)2ndquadrant(dot)com> writes:
>>> Personally I think we should aim to have this in as a non default build
>>> mode in pg10 if it can be made ready, and aim to make it default in pg11 at
>>> least for Windows.
>>
>> AFAIK we haven't committed to accepting this at all, let alone trying
>> to do so on a tight schedule. And I believe there was general agreement
>> that we would not accept it as something to maintain in parallel with
>> the existing makefiles. If we have to maintain two build systems, we
>> have that already.
>
> My preferred scenario would be to replace the Windows build system by
> this first, then refine it, then get rid of Autoconf.
>
> The ideal timeline would be to have a ready patch to commit early in a
> development cycle, then get rid of the Windows build system by the end
> of it. Naturally, this would need buy-in from Windows developers.

This looks like a really good plan to me.
--
Michael

On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote:
> My preferred scenario would be to replace the Windows build system by
> this first, then refine it, then get rid of Autoconf.
>
> The ideal timeline would be to have a ready patch to commit early in a
> development cycle, then get rid of the Windows build system by the end
> of it. Naturally, this would need buy-in from Windows developers.
>
> I don't foresee replacing the Autoconf build system by this immediately.

I'm very strongly against this path, it seems way too likely that we'll
end up with yet another fragile thing that nobody from the *nix side
will be able to test.

- Andres

On 1/27/17 6:11 PM, Andres Freund wrote:
> On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote:
>> My preferred scenario would be to replace the Windows build system by
>> this first, then refine it, then get rid of Autoconf.
>>
>> The ideal timeline would be to have a ready patch to commit early in a
>> development cycle, then get rid of the Windows build system by the end
>> of it. Naturally, this would need buy-in from Windows developers.
>>
>> I don't foresee replacing the Autoconf build system by this immediately.
>
> I'm very strongly against this path, it seems way too likely that we'll
> end up with yet another fragile thing that nobody from the *nix side
> will be able to test.

That's a fair concern, but at least with CMake, someone from the *nix
side *can* test it, whereas right now it's completely separate.

What kind of strategy do you have in mind?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Jan 28, 2017 at 12:20 PM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> On 1/27/17 6:11 PM, Andres Freund wrote:
>> On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote:
>>> My preferred scenario would be to replace the Windows build system by
>>> this first, then refine it, then get rid of Autoconf.
>>>
>>> The ideal timeline would be to have a ready patch to commit early in a
>>> development cycle, then get rid of the Windows build system by the end
>>> of it. Naturally, this would need buy-in from Windows developers.
>>>
>>> I don't foresee replacing the Autoconf build system by this immediately.
>>
>> I'm very strongly against this path, it seems way too likely that we'll
>> end up with yet another fragile thing that nobody from the *nix side
>> will be able to test.
>
> That's a fair concern, but at least with CMake, someone from the *nix
> side *can* test it, whereas right now it's completely separate.

And people complain all the time that the MSVC build scripts are hacky
and complicated.. So by beginning from there we switch from one build
to the other, not increasing the number of builds that need to be
maintained. Based on that Peter's strategy looks appealing to me. By
the way, I am marking the patch as returned with feedback for this CF.
--
Michael

On 2017-01-27 22:20:41 -0500, Peter Eisentraut wrote:
> On 1/27/17 6:11 PM, Andres Freund wrote:
> > On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote:
> >> My preferred scenario would be to replace the Windows build system by
> >> this first, then refine it, then get rid of Autoconf.
> >>
> >> The ideal timeline would be to have a ready patch to commit early in a
> >> development cycle, then get rid of the Windows build system by the end
> >> of it. Naturally, this would need buy-in from Windows developers.
> >>
> >> I don't foresee replacing the Autoconf build system by this immediately.
> >
> > I'm very strongly against this path, it seems way too likely that we'll
> > end up with yet another fragile thing that nobody from the *nix side
> > will be able to test.
>
> That's a fair concern, but at least with CMake, someone from the *nix
> side *can* test it, whereas right now it's completely separate.

Given that fact, I just don't buy why it's a good idea to not also
replace autoconf initially. Either we're able to properly test it -
i.e. it runs all tests - on *nix or we're not. There's not a a whole of
effort between those if you also want to do the windows side of things
properly.

> What kind of strategy do you have in mind?

Do all of it. I'm unconvinced that a windows only version buys us
meaningful savings, and I think the dangers of adding more duplication
(msvc stuff after all gets some information from the makefiles) and
long-term coexistence are quite severe.

Greetings,

Andres Freund

On 1/30/17 1:28 AM, Andres Freund wrote:
> Given that fact, I just don't buy why it's a good idea to not also
> replace autoconf initially.

Well, I find it a bit scary. If you do the big switch all at once, then
you will have to dedicate the following 3 months to fixing complaints
from developers and build farmers.

> Either we're able to properly test it - i.e. it runs all tests - on *nix or we're not.

That would work if there were a single entry point into the build
system. But in practice there are many, and every one of them is
someone's favorite. It's unlikely that we will be able to enumerate all
of them during patch review.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>
> I don't understand what this has to do with cmake. If this is a
> worthwhile improvement for the Windows build, then please explain why,
> with a "before" and "after" output and a patch for the existing build
> system as well.

During the porting process, I meet such situations when I should fix
something. It's happening because I build with different way also current
build system is trying to avoid many sharp corners.
If talk about this situation - without strict mode many "floats" checks
don't work correctly. You can read the link above. Besides this option puts
by build system. I think we can make a new thread for this approach. (with
patch for current perl system)

It might also be worth refactoring the existing Autoconf code here to
> make this consistent.

I do it because it's convenient in CMake. I can change this it's not big
deal.

Please explain what the circular dependency is. If there is one, we
> should also side-port this change.

It's an important part. I have a rule for generate rijndael.tbl by
gen-rtab who make from rijndael.c (with special define) who include
rijndael.tbl .
If I generate rijndael.tbl it's to force build gen-rtab and generate
rijndael.tbl again.
CMake knows about "includes" in files but we can make the wraparound macro
to hide include.

This patch removes the uuid.h include but doesn't add it anywhere else.

How does it work?

CMake sends to compiler right place for uuid.h (I mean -I/usr/include and
etc for gcc).

> Yeah, I think this is how the MSVC stuff effectively works right now as
> well.

I glad to hear it.

2017-01-03 17:11 GMT+03:00 Peter Eisentraut <
peter(dot)eisentraut(at)2ndquadrant(dot)com>:

> On 12/30/16 9:10 AM, Yuriy Zhuravlev wrote:
> > cmake_v2_2_c_define.patch
> >
> > Small chages in c.h . At first it is “#pragma fenv_access (off)” it is
> > necessary if we use /fp:strict for MSVC compiler. Without this pragma we
> > can’t calc floats for const variables in compiller time (2 * M_PI for
> > example). Strict mode important if we want to be close with ieee754
> > float format on MSVC (1.0 / 0.0 = inf for example). Detail info here:
> > https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx
>
> I don't understand what this has to do with cmake. If this is a
> worthwhile improvement for the Windows build, then please explain why,
> with a "before" and "after" output and a patch for the existing build
> system as well.
>
> > Second change is because I find and set HAVE_INT128 directly from CMake.
> > PG_INT128_TYPE used only for autoconfig scripts.
>
> It might also be worth refactoring the existing Autoconf code here to
> make this consistent.
>
> (My assumption is that if we were to move forward with cmake or any
> other build system change, we would have to keep the old one alongside
> at least for a little while. So any changes to the C code would need to
> be side-ported.)
>
> > cmake_v2_3_rijndael.patch
> >
> > First I added special wraparound because here CMake have circular
> > dependency (cmake very smart here). Second I removed rijndael.tbl
> > because it generated during build process every time.
>
> Please explain what the circular dependency is. If there is one, we
> should also side-port this change.
>
> > cmake_v2_4_uuid.patch
> >
> > Another small patch. Right place for uuid.h I find by CMake and not
> > necessary this ifdef hell.
>
> This patch removes the uuid.h include but doesn't add it anywhere else.
> How does it work?
>
> > Questions for discussion:
> >
> > In generated project by CMake we always have only one enter point. Also
> > INSTALL macross support only including to “all” targets. It follows that
> > it is impossible build contrib modules separately only with “all”
> > target. Here write about this behavior:
> > https://cmake.org/cmake/help/v3.7/prop_tgt/EXCLUDE_FROM_ALL.html
>
> Yeah, I think this is how the MSVC stuff effectively works right now as
> well.
>
> --
> Peter Eisentraut http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

2017-01-28 1:50 GMT+03:00 Michael Paquier <michael(dot)paquier(at)gmail(dot)com>:

> On Fri, Jan 27, 2017 at 11:09 PM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> > On 1/24/17 8:37 AM, Tom Lane wrote:
> >> Craig Ringer <craig(dot)ringer(at)2ndquadrant(dot)com> writes:
> >>> Personally I think we should aim to have this in as a non default build
> >>> mode in pg10 if it can be made ready, and aim to make it default in
> pg11 at
> >>> least for Windows.
> >>
> >> AFAIK we haven't committed to accepting this at all, let alone trying
> >> to do so on a tight schedule. And I believe there was general agreement
> >> that we would not accept it as something to maintain in parallel with
> >> the existing makefiles. If we have to maintain two build systems, we
> >> have that already.
> >
> > My preferred scenario would be to replace the Windows build system by
> > this first, then refine it, then get rid of Autoconf.
> >
> > The ideal timeline would be to have a ready patch to commit early in a
> > development cycle, then get rid of the Windows build system by the end
> > of it. Naturally, this would need buy-in from Windows developers.
>
> This looks like a really good plan to me.

I think it's best plan because when this patch will be in Postgres guys
from community can test it for Unix systems too.
Support two build systems it's not big deal really. I have been working on
this past year without any big troubles.
Also we have second perl build system...

On 2/8/17 6:21 AM, Yuriy Zhuravlev wrote:
> Support two build systems it's not big deal really. I have been working
> on this past year without any big troubles.
> Also we have second perl build system...

The perl/msvc build system pulls in information from the makefiles. So
when you add a file or something basic like that, you don't have to
update it. So it's really more like 1.5 build systems.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

On 2017-01-30 10:26:18 -0500, Peter Eisentraut wrote:
> On 1/30/17 1:28 AM, Andres Freund wrote:
> > Given that fact, I just don't buy why it's a good idea to not also
> > replace autoconf initially.
>
> Well, I find it a bit scary. If you do the big switch all at once, then
> you will have to dedicate the following 3 months to fixing complaints
> from developers and build farmers.

That'll be the case just as well if we spread it out over two cycles,
except that we'll have it in multiple phases, and we run into the danger
of a half-done conversion.

I'd rather not change systems at all than run into the danger of that.

> > Either we're able to properly test it - i.e. it runs all tests - on *nix or we're not.
>
> That would work if there were a single entry point into the build
> system. But in practice there are many, and every one of them is
> someone's favorite. It's unlikely that we will be able to enumerate all
> of them during patch review.

Not sure what you mean with "entry point"?

- Andres

Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> On 2/8/17 6:21 AM, Yuriy Zhuravlev wrote:
>> Support two build systems it's not big deal really. I have been working
>> on this past year without any big troubles.
>> Also we have second perl build system...

> The perl/msvc build system pulls in information from the makefiles. So
> when you add a file or something basic like that, you don't have to
> update it. So it's really more like 1.5 build systems.

Really it's more like 1.1 build systems, in that the MSVC scripts do that
just well enough that you *usually* don't have to think about them. But
then when they fail, and you have to figure out why, it can be a pain.

For my own purposes, the only thing that I find seriously annoying about
the status quo is the amount of time required to run "configure". For
me, that step is usually comparable to or even more than the time to
do the build proper, because (a) ccache and (b) multiple CPUs.
configure isn't parallelizable, and there's probably nothing that
can be done about that. If CMake offers a substantial improvement
in configuration time then that would be attractive. Otherwise I'd
just as soon see us put the effort into making the MSVC scripts more
robust and able to pull more data from the makefiles.

regards, tom lane

On 2/8/17 3:52 PM, Tom Lane wrote:
> For my own purposes, the only thing that I find seriously annoying about
> the status quo is the amount of time required to run "configure". For
> me, that step is usually comparable to or even more than the time to
> do the build proper, because (a) ccache and (b) multiple CPUs.
> configure isn't parallelizable, and there's probably nothing that
> can be done about that. If CMake offers a substantial improvement
> in configuration time then that would be attractive. Otherwise I'd
> just as soon see us put the effort into making the MSVC scripts more
> robust and able to pull more data from the makefiles.

FWIW, I've had good luck adding -C to configure to cache the output. I'd
guess it's at least 10x faster on my laptop.

Obviously doesn't help if you're doing where you're testing something
that alters config output. In those cases I'll either edit config.cache
and delete the relevant lines or just temporarily move it out of the way
(or just nuke it...).
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

Hi,

On 2017-02-08 16:52:19 -0500, Tom Lane wrote:
> For my own purposes, the only thing that I find seriously annoying about
> the status quo is the amount of time required to run "configure". For
> me, that step is usually comparable to or even more than the time to
> do the build proper, because (a) ccache and (b) multiple CPUs.
> configure isn't parallelizable, and there's probably nothing that
> can be done about that.

I use autoconf caching feature to make that a bit less painful (plus
some scripting about when to scrap the cache file...). I find that
seriously annoying too.

> If CMake offers a substantial improvement
> in configuration time then that would be attractive. Otherwise I'd
> just as soon see us put the effort into making the MSVC scripts more
> robust and able to pull more data from the makefiles.

Some of the build-tooling in cmake is quite nice, I have to admit. I've
e.g. grown to like using ninja instead of make to build the resulting
files. Primarily in projects that take longer than pg to compile - a
clean build in llvm with ninja takes like 0.1 seconds.

Being more able to rely on things working on windows when doing them on
linux does seem like an advantage to me - fiddlin with Mkvcbuild.pm is
quite annoying.

- Andres

On Mon, Jan 30, 2017 at 10:26 AM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> On 1/30/17 1:28 AM, Andres Freund wrote:
>> Given that fact, I just don't buy why it's a good idea to not also
>> replace autoconf initially.
>
> Well, I find it a bit scary. If you do the big switch all at once, then
> you will have to dedicate the following 3 months to fixing complaints
> from developers and build farmers.

I agree with that. I think replacing the Windows build system first
and then the non-Windows build system later is a better plan than
replacing both at the same time.

But also, I'm not really sure whether this conversion makes sense. I
mean, any build system is going to require some work, and accordingly
our present build systems require some work. cmake will require
different work, but not necessarily less. The current system has a
long history; we pretty much know it works. Switching will inevitably
break some things. Maybe we'll end up better off in the long term,
but maybe we won't. Things are pretty good now, so it seems like it
would be easy for them to get worse rather than better. If nothing
else, everybody who has to learn the new system either to use it for
development or because they are doing packaging will have to do some
amount of extra work as a result of any switch.

I do agree that - in theory - one build system is better than two.
But two well-tested, reliable build systems could easily be better
than one system with a bunch of problems. And the points downthread
about our two existing systems being not entirely separate are on
point, too.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2017-02-10 12:07:15 -0500, Robert Haas wrote:
> On Mon, Jan 30, 2017 at 10:26 AM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> > On 1/30/17 1:28 AM, Andres Freund wrote:
> >> Given that fact, I just don't buy why it's a good idea to not also
> >> replace autoconf initially.
> >
> > Well, I find it a bit scary. If you do the big switch all at once, then
> > you will have to dedicate the following 3 months to fixing complaints
> > from developers and build farmers.
>
> I agree with that. I think replacing the Windows build system first
> and then the non-Windows build system later is a better plan than
> replacing both at the same time.

Obviously I disagree ;)

But I'm more replying because of:

> But also, I'm not really sure whether this conversion makes sense. I
> mean, any build system is going to require some work, and accordingly
> our present build systems require some work. cmake will require
> different work, but not necessarily less. The current system has a
> long history; we pretty much know it works. Switching will inevitably
> break some things. Maybe we'll end up better off in the long term,
> but maybe we won't. Things are pretty good now, so it seems like it
> would be easy for them to get worse rather than better.

I do think it's kinda ok for people who've dealt with this for some
time. But for the first few times having to write autoconf macros is
quite intimidating. A somewhat less obscure way to deal with that is
worth something. As is better file dependency management, across
different environments. As is the IDE integration cmake is more and
more getting. As is properly builtin working caching of configure tests
(llvm first cmake run, 7.7s, second 2.54s). As is the fact that a lot of
big projects (llvm, kde, qt, and a lot of others) migrated to it.

For me personally those benefits aren't worth that much. I've learned
autoconf stuff. I've scripting around autoconf caching. But for newer
people that's not true.

Greetings,

Andres Freund

On Fri, Feb 10, 2017 at 6:07 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Mon, Jan 30, 2017 at 10:26 AM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> > On 1/30/17 1:28 AM, Andres Freund wrote:
> >> Given that fact, I just don't buy why it's a good idea to not also
> >> replace autoconf initially.
> >
> > Well, I find it a bit scary. If you do the big switch all at once, then
> > you will have to dedicate the following 3 months to fixing complaints
> > from developers and build farmers.
>
> I agree with that. I think replacing the Windows build system first
> and then the non-Windows build system later is a better plan than
> replacing both at the same time.
>
> But also, I'm not really sure whether this conversion makes sense. I
> mean, any build system is going to require some work, and accordingly
> our present build systems require some work. cmake will require
> different work, but not necessarily less. The current system has a
> long history; we pretty much know it works. Switching will inevitably
> break some things. Maybe we'll end up better off in the long term,
> but maybe we won't. Things are pretty good now, so it seems like it
> would be easy for them to get worse rather than better. If nothing
> else, everybody who has to learn the new system either to use it for
> development or because they are doing packaging will have to do some
> amount of extra work as a result of any switch.
>
>
For me a killer feature would be if/when we can get to a point where we can
have something pgxs-style on cmake that also works on windows.

Our homemade Windows build system works OK for postgres, and while ugly it
is as you say well tested by now. But it doesn't do *anything* to help
people build extensions on Windows.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 02/10/2017 08:27 PM, Magnus Hagander wrote:
> For me a killer feature would be if/when we can get to a point where we can
> have something pgxs-style on cmake that also works on windows.
>
> Our homemade Windows build system works OK for postgres, and while ugly it
> is as you say well tested by now. But it doesn't do *anything* to help
> people build extensions on Windows.

Do we need to build PostgreSQL itself using cmake, to achieve that?
Could we write something like pgxs for cmake, only for extensions?

- Heikki

On 2017-02-10 19:27:55 +0100, Magnus Hagander wrote:
> For me a killer feature would be if/when we can get to a point where we can
> have something pgxs-style on cmake that also works on windows.

That should be quite possible. The ugliest part will be to determine
where to include a cmake file from (since that'll be copied in every
such project), but after that it should be nearly trivial.

- Andres

On Fri, Feb 10, 2017 at 7:31 PM, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:

> On 02/10/2017 08:27 PM, Magnus Hagander wrote:
>
>> For me a killer feature would be if/when we can get to a point where we
>> can
>> have something pgxs-style on cmake that also works on windows.
>>
>> Our homemade Windows build system works OK for postgres, and while ugly it
>> is as you say well tested by now. But it doesn't do *anything* to help
>> people build extensions on Windows.
>>
>
> Do we need to build PostgreSQL itself using cmake, to achieve that? Could
> we write something like pgxs for cmake, only for extensions?
>
>
I guess we wouldn't, but we'd still need the "replacement for autoconf"
part. So then we're back to maintaining multiple buildsystems.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 2017-02-10 20:31:12 +0200, Heikki Linnakangas wrote:
> On 02/10/2017 08:27 PM, Magnus Hagander wrote:
> > For me a killer feature would be if/when we can get to a point where we can
> > have something pgxs-style on cmake that also works on windows.
> >
> > Our homemade Windows build system works OK for postgres, and while ugly it
> > is as you say well tested by now. But it doesn't do *anything* to help
> > people build extensions on Windows.
>
> Do we need to build PostgreSQL itself using cmake, to achieve that? Could we
> write something like pgxs for cmake, only for extensions?

I don't see why it'd need to be done together. The minimal version
would be a simple cmake file that just sets a bunch of variables from
pg_config, provides a few rules for specifying the current pgxs stuff,
and an example stanza how to include that file. We'd have to duplicate
some of the pgxs specific logic, but that's probably not too bad.

- Andres

On 2017-02-10 19:33:18 +0100, Magnus Hagander wrote:
> I guess we wouldn't, but we'd still need the "replacement for autoconf"
> part. So then we're back to maintaining multiple buildsystems.

Hm? Do we really need that? Most of the things in an extension you do
*not* want to determine separately from the backend. It's not like pgxs
atm really allows to differ wildly from autoconf's results. And most of
the relevant determinations made by autoconf are available in headers
and/or we can generate a cmake include file with the results of
autoconf.

- Andres

On Feb 10, 2017 19:41, "Andres Freund" <andres(at)anarazel(dot)de> wrote:

On 2017-02-10 19:33:18 +0100, Magnus Hagander wrote:
> I guess we wouldn't, but we'd still need the "replacement for autoconf"
> part. So then we're back to maintaining multiple buildsystems.

Hm? Do we really need that? Most of the things in an extension you do
*not* want to determine separately from the backend. It's not like pgxs
atm really allows to differ wildly from autoconf's results. And most of
the relevant determinations made by autoconf are available in headers
and/or we can generate a cmake include file with the results of
autoconf.

Yeah, you're right. You need the output from the process, it mot the
process itself.

/Magnus

On 02/10/2017 01:27 PM, Magnus Hagander wrote:
> On Fri, Feb 10, 2017 at 6:07 PM, Robert Haas <robertmhaas(at)gmail(dot)com
> <mailto:robertmhaas(at)gmail(dot)com>> wrote:
>
> On Mon, Jan 30, 2017 at 10:26 AM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com
> <mailto:peter(dot)eisentraut(at)2ndquadrant(dot)com>> wrote:
> > On 1/30/17 1:28 AM, Andres Freund wrote:
> >> Given that fact, I just don't buy why it's a good idea to not also
> >> replace autoconf initially.
> >
> > Well, I find it a bit scary. If you do the big switch all at
> once, then
> > you will have to dedicate the following 3 months to fixing
> complaints
> > from developers and build farmers.
>
> I agree with that. I think replacing the Windows build system first
> and then the non-Windows build system later is a better plan than
> replacing both at the same time.
>
> But also, I'm not really sure whether this conversion makes sense. I
> mean, any build system is going to require some work, and accordingly
> our present build systems require some work. cmake will require
> different work, but not necessarily less. The current system has a
> long history; we pretty much know it works. Switching will inevitably
> break some things. Maybe we'll end up better off in the long term,
> but maybe we won't. Things are pretty good now, so it seems like it
> would be easy for them to get worse rather than better. If nothing
> else, everybody who has to learn the new system either to use it for
> development or because they are doing packaging will have to do some
> amount of extra work as a result of any switch.
>
>
> For me a killer feature would be if/when we can get to a point where
> we can have something pgxs-style on cmake that also works on windows.
>
> Our homemade Windows build system works OK for postgres, and while
> ugly it is as you say well tested by now. But it doesn't do *anything*
> to help people build extensions on Windows.
>
>

Watch this space. There is work being done on building extensions out of
tree using CMake on Windows. It's pretty neat, and I'm hoping to demo it
publicly soon. But it's not reliant on CMake in core.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11 Feb. 2017 08:42, "Andrew Dunstan" <andrew(dot)dunstan(at)2ndquadrant(dot)com>
wrote:

On 02/10/2017 01:27 PM, Magnus Hagander wrote:
> On Fri, Feb 10, 2017 at 6:07 PM, Robert Haas <robertmhaas(at)gmail(dot)com
> <mailto:robertmhaas(at)gmail(dot)com>> wrote:
>
> On Mon, Jan 30, 2017 at 10:26 AM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com
> <mailto:peter(dot)eisentraut(at)2ndquadrant(dot)com>> wrote:
> > On 1/30/17 1:28 AM, Andres Freund wrote:
> >> Given that fact, I just don't buy why it's a good idea to not also
> >> replace autoconf initially.
> >
> > Well, I find it a bit scary. If you do the big switch all at
> once, then
> > you will have to dedicate the following 3 months to fixing
> complaints
> > from developers and build farmers.
>
> I agree with that. I think replacing the Windows build system first
> and then the non-Windows build system later is a better plan than
> replacing both at the same time.
>
> But also, I'm not really sure whether this conversion makes sense. I
> mean, any build system is going to require some work, and accordingly
> our present build systems require some work. cmake will require
> different work, but not necessarily less. The current system has a
> long history; we pretty much know it works. Switching will inevitably
> break some things. Maybe we'll end up better off in the long term,
> but maybe we won't. Things are pretty good now, so it seems like it
> would be easy for them to get worse rather than better. If nothing
> else, everybody who has to learn the new system either to use it for
> development or because they are doing packaging will have to do some
> amount of extra work as a result of any switch.
>
>
> For me a killer feature would be if/when we can get to a point where
> we can have something pgxs-style on cmake that also works on windows.
>
> Our homemade Windows build system works OK for postgres, and while
> ugly it is as you say well tested by now. But it doesn't do *anything*
> to help people build extensions on Windows.
>
>

Watch this space. There is work being done on building extensions out of
tree using CMake on Windows. It's pretty neat, and I'm hoping to demo it
publicly soon. But it's not reliant on CMake in core.

Yeah. In fact it's completely independent of core's build system it and
works with any supported Pg, using pg_config for discovery.

We don't need cmake in core for pgxs-like functionality on Windows.

I'd like to use it instead of our Perl windows stuff msbuild stuff in core
though. I wonder if it's practical to generate the cmake file lists etc
from our Makefiles initialy like we do for the current system, and just
replace the MSVC/msbuild project generators with cmake initially.

On 2017-02-10 10:22:34 -0800, Andres Freund wrote:
> On 2017-02-10 12:07:15 -0500, Robert Haas wrote:
> > But also, I'm not really sure whether this conversion makes sense. I
> > mean, any build system is going to require some work, and accordingly
> > our present build systems require some work. cmake will require
> > different work, but not necessarily less. The current system has a
> > long history; we pretty much know it works. Switching will inevitably
> > break some things. Maybe we'll end up better off in the long term,
> > but maybe we won't. Things are pretty good now, so it seems like it
> > would be easy for them to get worse rather than better.
>
> I do think it's kinda ok for people who've dealt with this for some
> time. But for the first few times having to write autoconf macros is
> quite intimidating. A somewhat less obscure way to deal with that is
> worth something. As is better file dependency management, across
> different environments. As is the IDE integration cmake is more and
> more getting. As is properly builtin working caching of configure tests
> (llvm first cmake run, 7.7s, second 2.54s). As is the fact that a lot of
> big projects (llvm, kde, qt, and a lot of others) migrated to it.
>
> For me personally those benefits aren't worth that much. I've learned
> autoconf stuff. I've scripting around autoconf caching. But for newer
> people that's not true.

Craig's email just now reminded me of another advantage: Using cmake
across the board, would mean we'd use the same ./configure alike logic
on both windows and linux. To me that seems quite and advantage over
managing pg_config.h.win32/config_default.pl manually/separately - we
obviously have screwed up quite badly there in the past
(cf376a4adc0805b0).

On Fri, Feb 10, 2017 at 5:07 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> But also, I'm not really sure whether this conversion makes sense. I
> mean, any build system is going to require some work, and accordingly
> our present build systems require some work. cmake will require
> different work, but not necessarily less. The current system has a
> long history; we pretty much know it works. Switching will inevitably
> break some things. Maybe we'll end up better off in the long term,
> but maybe we won't. Things are pretty good now, so it seems like it
> would be easy for them to get worse rather than better. If nothing
> else, everybody who has to learn the new system either to use it for
> development or because they are doing packaging will have to do some
> amount of extra work as a result of any switch.
>

For what it's worth (even well-maintained) cmake is a usability regression
from well-maintained autotools from syseng/packager perspective.

Two anecdotes:

- ./configure is much nicer from user perspective. Compare:
./configure --prefix=/bla --enable-foo --disable-bar --with-ssl=/opt/myssl
cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BLAH -DCMAKE_FOO
-DCMAKE_WHO_KNOWS_WHAT:PATH=/opt/myssl

- Things like set(_PYTHON3_VERSIONS 3.7 3.6 3.5 3.4 3.3 3.2 3.1 3.0)
<https://github.com/Kitware/CMake/blob/master/Modules/FindPythonInterp.cmake#L43>
Guess what, on older cmake versions this list did not include anything
older that 3.3, so it was failing with in-comprehensive generic "not found"
error even though 3.4 was installed.
With this "fix" somebody somewhere will be banging their head against the
wall again once 3.8 is out.
Overall, when things go wrong debugging cmake requires cmake knowledge,
while autotools mostly require shell knowledge which is much more common
(again, for sysadmins/packagers).

So while cmake is better for developers it is usually worse off for
packagers and advanced users. I'm not saying migration is not worth it, I'm
pointing out costs of such migration.

PS: I personally like Google's internal version of https://bazel.build/ a
lot. I've never used open-source version but I presume it's similar. While
it has many problems (Java, lack of popular IDE support, lack of popularity
and, again, Java) good parts are rules are both machine- and human-
readable and writable and generally easy to debug. I'm not bold enough to
propose PostgreSQL to use it, but I'd be happy to see ideas from it to be
used elsewhere.

--
Vladimir Rusinov
Storage SRE, Google Ireland

Google Ireland Ltd.,Gordon House, Barrow Street, Dublin 4, Ireland
Registered in Dublin, Ireland
Registration Number: 368047

2017-02-12 20:55 GMT+03:00 Vladimir Rusinov <vrusinov(at)google(dot)com>:

> Overall, when things go wrong debugging cmake requires cmake knowledge,
> while autotools mostly require shell knowledge which is much more common
> (again, for sysadmins/packagers).

It's not really true because of CMake scripts much easier than tons of crap
bash (configure) and m4 scripts in Autotools, also please don't forget
Windows MSVC, Xcode and etc usage.

PS: I personally like Google's internal version of https://bazel.build/ a
> lot. I've never used open-source version but I presume it's similar. While
> it has many problems (Java, lack of popular IDE support, lack of popularity
> and, again, Java) good parts are rules are both machine- and human-
> readable and writable and generally easy to debug. I'm not bold enough to
> propose PostgreSQL to use it, but I'd be happy to see ideas from it to be
> used elsewhere.

We have many build systems, for example, another one http://mesonbuild.com/
but CMake the best today as meta build system.

On 2/8/17 4:44 PM, Andres Freund wrote:
>> Well, I find it a bit scary. If you do the big switch all at once, then
>> you will have to dedicate the following 3 months to fixing complaints
>> from developers and build farmers.
>
> That'll be the case just as well if we spread it out over two cycles,
> except that we'll have it in multiple phases, and we run into the danger
> of a half-done conversion.

They key term is "dedicated". I don't think anyone wants to spend 3
months doing nothing but fixing the build system. If we spread it over
several releases and bring more people up to speed along the way, that
looks more manageable.

>> That would work if there were a single entry point into the build
>> system. But in practice there are many, and every one of them is
>> someone's favorite. It's unlikely that we will be able to enumerate all
>> of them during patch review.
>
> Not sure what you mean with "entry point"?

People have all kinds of expectations on how the build system behaves.
It's not just ./configure; make; make install. All kinds of niche and
edge cases have evolved over the years. Variables you can set,
overrides, targets in some subdirectories, building in subdirectories
and having it build another subdirectory first, testing this way and
testing that way, testing with a custom locale or a custom database
name, building before testing or not, testing without reinstalling, and
so on and so on. You'd have to make sure at least some of that
continues to work or be able to explain it away. And most of it is not
really documented.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Feb 13, 2017 at 10:52 AM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> People have all kinds of expectations on how the build system behaves.
> It's not just ./configure; make; make install. All kinds of niche and
> edge cases have evolved over the years. Variables you can set,
> overrides, targets in some subdirectories, building in subdirectories
> and having it build another subdirectory first, testing this way and
> testing that way, testing with a custom locale or a custom database
> name, building before testing or not, testing without reinstalling, and
> so on and so on. You'd have to make sure at least some of that
> continues to work or be able to explain it away. And most of it is not
> really documented.

...which is another argument for just not changing anything. I mean,
again, the current Windows build system is unbeautiful and
occasionally takes some work, but if we switch to cmake, that has to
get enough better to make up for all of the things that get worse.
And it's far from obvious than this will be so, especially when you
consider the fact that many people have detailed knowledge of how to
tweak the current system to do what they want. As you point out here,
a lot of that stuff may stop working if we replace the system
wholesale.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Feb 8, 2017 at 04:52:19PM -0500, Tom Lane wrote:
> Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> > On 2/8/17 6:21 AM, Yuriy Zhuravlev wrote:
> >> Support two build systems it's not big deal really. I have been working
> >> on this past year without any big troubles.
> >> Also we have second perl build system...
>
> > The perl/msvc build system pulls in information from the makefiles. So
> > when you add a file or something basic like that, you don't have to
> > update it. So it's really more like 1.5 build systems.
>
> Really it's more like 1.1 build systems, in that the MSVC scripts do that
> just well enough that you *usually* don't have to think about them. But
> then when they fail, and you have to figure out why, it can be a pain.

If cmake isn't going to be able to query the Makefiles and adjust to
changes we make there, changing our Windows build system from MSVC to
cmake takes us from maintaining 1.1 build systems to two build systems,
and I don't think anyone wants that.

If we go to cmake, I think we need to agree we will eventually _only_
use cmake. I don't think having two build systems we have to maintain
is better than 1.1 build systems where we can mostly ignore 0.1 of that.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +