Meson Build System

[anarazel]

Tasks for meson use in postgres:

• Missing Features
  • libedit support #38 libedit support (easy-medium)
  • generate pkg-config files (easy-medium)
  • fully implement "native language support" (see https://postgr.es/m/bfcd5353-0fb3-a05c-6f62-164d98c5689d%40enterprisedb.com )
  • check vs installcheck (maybe via different test setups?)
  • meson: postgres flex wrapper #50
• Not yet built in meson:
  • check for other bits and pieces
  • ECPG, except for windows
  • pl/tcl (medium) #22
• Not tested yet with meson:
  • ECPG (medium)
  • search for other "manually invoked" tests (easy)
• OS dependent issues
  • building against system ldap fails on macos #9 (unknown difficulty)
  • rpath on netbsd broken for at least perl, there's also a possibly related issue on suse
  • AIX symbol generation needs to be fixed
  • windows icon generation (PGAPPICON and PGFILEDESC in makefiles) is implemented only for one or two tools, needs to be generalized
  • simplify exports.txt handling, use it in all the required places (e.g. ecpg)
• Installation
  • "pkglibdir" logic from src/Makefile.global.in needs to be mirrored to some degree (medium)
  • meson: install headers to the same location as autoconf build does #30 (partially done)
  • Header install with install_subdir() is error prone (think tempfiles)
• Cleanups:
  • Centralize operating-system specific logic. There's too many places querying host_machine.system() right now
  • Merge gendef.pl and gendef2.pl back together (medium)
  • src/tools/testwrap should be cleaned up made unnecessary (path unclear as of yet)
  • src/tools/irlink shouldn't be bash, but python #36 (medium)
  • merge top-level of tap and regress test specifications, shouldn't be two separate arrays with the same information at the top #31 (easy-medium, fairly verbose changes)
• Open questions:
  • figure out how to deal with PGXS / building out-of-tree extension
  • LLVM bitcode generation implementation isn't even close to generalized enough (hard)
• Other Todos:
  • meson: write test for pkg-config based extension builds #58 (easy - medium)
  • installation.sgml etc need a lot of adjustment
  • Write README.md introducing how to build with meson on a number of operating systems #33 (easy)
  • sync up configure tests / meson implementation, add missing ones
• Bugs:
  • Currently all PG_TEST_EXTRA tests are run unconditionally #28
  • test for availability of IPC::Run at configure time, instead of failing tests due to missing it #32 (easy)
• Implement missing options:
  • Synchronize option descriptions from configure.ac to meson_options.txt (easy)
  • Check which other options we want to have that configure has but meson_options doesn't
  • --with-wal-blocksize / XLOG_BLCKSZ (easy) #15
  • --with-segsize / RELSEG_SIZE (easy) #16
  • --with-pgport (easy) #17
  • --enable-tap-tests, currently unconditional, without even checking for it (easy) #18
  • meson: Implement missing directory options #61

Testing Tasks

• automated testing of various linux distributions, via docker containers, at least (should be optional cirrus tasks):
  • RHEL 7 (easy-medium) #19
  • RHEL 8 (easy-medium) #20
  • Fedora Rawhide (easy-medium) #21
  • Suse (See Adding openSUSE Tumbleweed to meson build #7 )
  • Debian on linux arm (supported by cirrus), image exists already (easy)
• automated testing via VMs
  • OpenBSD ( openbsd: Build Image pg-vm-images#6 )
  • NetBSD ( netbsd: Build Image pg-vm-images#7 )
• automated test against all supported meson versions (easy-medium)
• Add test of meson upstream versions #81 (easy-medium)
• Test a few extensions built via PGXS in CI (easy)
• Automate building a lot of extensions via script (medium)

Improvements to the meson integration that likely aren't critical:

• build dependency files for xsltproc et al for more accurate incremental builds (medium-hard)

Questions:

• Do we need to implement "sqlmansectnum" logic for docs?
• Partial installation support needed? Right now everything will be installed. Likely a problem for things like src/test/modules/dummy_index_am - but those currently need to be installed for tests.

Relevant meson improvements / issues:

• caching for library searches - biggest contributor to configure time now
• Framework dependency extracts -I in addition to framework, leading to endless include recursion mesonbuild/meson#10002 - worked around, but still a trap
• Expose Environment::get_build_command() to meson.build mesonbuild/meson#8511 - not critical, but helpful
• arm macos libraries rejected due to expecting aarch64, but seeing aarch64e mesonbuild/meson#9493 - worked around, but still a trap
• vs_module_defs should be allowed for executable(). mesonbuild/meson#9254 - would simplify code
• Invalid XML generated in testlog.junit.xml with \x07 in test output mesonbuild/meson#9894 - not critical, but would allow easier to analyze test results
• mtest TAP parser parses fails on Test::More and node-tap subtests mesonbuild/meson#10032 - should fix so that we eventually can rely on tap subtests (once minimum meson version is high enough)

[samay-sharma]

@anarazel : Can you update the Task list with this issue as well : mesonbuild/meson#9894 . I opened this against meson for the last meson issue.

[anarazel]

@anarazel : Can you update the Task list with this issue as well : mesonbuild/meson#9894 . I opened this against meson for the last meson issue.

Done. You probably now would also have the permissions to do so.

[tristan957]

Could you expand on this?

Header install with install_subdir() is error prone (think tempfiles)

I have submitted mesonbuild/meson#11849 fyi.

[eli-schwartz]

install_subdir() will install an entire directory tree, even if not all the files in that tree are supposed to be installed. This can also include text editor backup files etc.

postgres/src/include/meson.build

Lines 163 to 172 in d1491d3

	# XXX: installing headers this way has the danger of installing editor files
	# etc, unfortunately install_subdir() doesn't allow including / excluding by
	# pattern currently.
	foreach d : header_subdirs
	if d == 'catalog'
	continue
	endif
	install_subdir(d, install_dir: dir_include_server,
	exclude_files: ['.gitignore', 'meson.build'])
	endforeach

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

[anarazel]

@tristan957 Does @eli-schwartz 's explanation answer your question?

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

[tristan957]

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

[anarazel]

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

I am thinking of the opposite. I would like to explicitly specify include patterns, with everything not matching not getting installed. IME it's quite rare that one needs to install all but a few patterns, but very common to install just one type of file.

[tristan957]

Are you thinking that glob syntax would be enough?

[tristan957]

Perhaps a patterns kwarg and then doing the equivalent of what github does would be enough.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore

Then you get negated globs too

[anarazel]

Are you thinking that glob syntax would be enough?

Depends on the complexity of the globs. I think a full regex would be good - without a regex it's hard to exclude temp files. I'd basically want to say something like include_patterns = ['^[a-zA-Z0-9]+.*.h'], so things like ".#filename.h" etc aren't "included".

[tristan957]

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.

Another option would be to expose https://docs.python.org/3/library/glob.html.

[tristan957]

Maybe Meson could just define no positive/negative lookaheads/behinds. That would probably be pretty simple at the end of the day since those seem to be the biggest differentiator between regex impls.

[eli-schwartz]

Yeah there's a few reasons why I'm not a fan of regex here. Layering an exclude pattern on top of an include pattern would probably work better here though.

[tristan957]

Eli, so are you saying that we need include_directories and include_files alongside what already exists for exclude_directories and exclude_files? Or are you advocating for patterns? If so, what pattern syntax?

[eli-schwartz]

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

One possible alternative here is to have the script generate these lists, and write them to a file in the source tree e.g. headerlist.txt, and then use the fs.read() function to read out that list and dump it into the install_headers() function.

Doesn't need to be maintained by hand, the script can run itself in check mode either at configure time or as a test(), and the generator script doesn't need to understand meson syntax well enough to edit a meson.build file.