libtool: -export-symbols-regex breaks things

[dmik]

If -export-symbols-regex is given on the libtool command line, it causes the code path that involves NM to deal with exported symbols instead of EMXEXP. This breaks on OS/2 because NM is unable to handle -OMF files and because it somehow results into wrong constructs due to some missing defines. This may be a 2.4.6 regression since Glib uses this switch on OS/2 and it worked well in libtool 2.4.2.

Currently, this defect prevents us from building pango (which also uses this switch).

[dmik]

I can't test 2.4.2 actually because it fails at some different place now for some unknown reason and I'm not willing to spend time on it; we have to fix 2.4.6 anyway. So far, it seems that on OS/2 libtool uses nm when -export-symbols-regex is given, and emxexp otherwise. It should use emxexp all the time: luckily, this tool understands both A.OUT and OMF files.

A side note: our EMX toolchain is very inconsistent when it comes to object file format. Some tools (e.g. nm) only understand A.OUT, some (e.g. gcc) understand both A.OUT and OMF, but with various deviations (e.g. if -Zomf is given to gcc, it will still output to an .o file instead of .obj; another inconsistency is emxomfar x which will always extract files to .obj even if they were put as .o in it). May be we will sort it all out one day and the world will become a slightly better place...

[dmik]

Another note on this. The described bug only takes place when -Zomf is present in CFLAGS. This matches the above as in -Zomf mode nm can't read object files and hence can't process exports. Until the OMF consistency is somehow gained in the toolchain, we should tend not to use -Zomf. However, sometimes it's not possible (due to various ld bugs, like in case of too many mangled C++ exports). In this case -Zomf is must and as a temporary workaround -export-symbols-regexp is to be removed (emxexp will be used in this case which will export every public symbol though).

[dmik]

Note that after some struggling I made 2.4.2 work with my test project, expat. The place of failure is very strange though. The matter is that an equivalent to the following construct in configure

delay_single_quote_subst='s/'\''/'\'\\\\\\\'\''/g'
lt_prog_compiler_pic="-DDLL_EXPORT"
lt_prog_compiler_pic="$lt_prog_compiler_pic -DPIC"
output=`printf '%s\n' "$lt_prog_compiler_pic" | sed "$delay_single_quote_subst"` 
echo ["$output"]

produces

[-DDLL_EXPORT -DPIC
]

instead of just [-DDLL_EXPORT -DPIC]. I.e. an extra EOL character gets into the result and this breaks the compiler line into two when executing which is of course wrong. It's nonsense since all trailing white space and EOLs should be trimmed by the `` sh command and the real problem is that I can't reproduce this behavior under sh directly (i.e. copy-pasting the above in a new file). it only happens when I run configure and only with configure generated with libtool 2.4.2. In libtool 2.4.6 all works as expected (no extra EOL) although the respective script commands look just the same. I couldn't find an explanation to this other than some mystic sh failure. I'm really not going to go deeper into this as it took too much time already and not our primary problem. This was just for the record. A fix is easy though: you should replace printf '%s\n' with just echo and it will work as it should.

[dmik]

Well, just to do one more test I replaced ash with dash and the extra EOL problem went away! So now it looks much more like a bug in `` command handling of ash. We should fix a few remaining dash problems and switch to it ASAP.

[dmik]

I finally see where it comes from. Version 2.4.2 only had the archive_cmds definition on OS/2 to create DLLs. Looks like this definition was both used when -export-symbols[-regex] is absent and when it is present (it was totally ignored and all public symbols were exported anyway). Note that our glib2 SVN is an exception here because it was using a hard-coded libtool version 2.2.10 that contained OS/2 fixes just in place and that apparently took -export-symbols[-regex] into account (by always using emxexp to generate the exports list, including both OMF and A.OUT modes).

In version 2.4.4 they also added archive_expsym_cmds on OS/2 that gets used when -export-symbols[-regex] is given. And the OS/2 implementation (kindly) provided by upstream (or may be by some secret komh patch) doesn't take OMF mode into account simply assuming that NM works (while with OMF object files it doesn't). As a result, if you use -export-symbols[-regex] and give -Zomf in CFLAGS, the generated DLL ends up exporting nothing (because NM returns no public symbols for object files) and that eventually results into unresolved symbols when linking to this DLL later.

So the fix is rather easy then: switch our SVN libtool to using emxexp instead of nm (just the way the glib2 hard-coded libtool did).

[dmik]

I should also check the libtool docs to see what we need to do if no -export-symbols[-regex] is given: export all or export nothing. I guess we should export all because a DLL with no exports barely makes any sense.

[dmik]

Okay, all symbols should be exported when no -export-symbols[-regex] is given, according to the docs.

I applied the discussed fix in r1280 and it seems to work well. Note that it's now also possible to use -export-symbols exports.def on OS/2 where exports.def is a static file that lists C symbols to export (one per line, may even start with EXPORTS on the first line). This is often used for Windows builds of libtool-based projects to specify a fixed list of exports (instead of using -export-symbols-regex). Given that this Windows file is maintained upstream we may simply use it on OS/2 as well w/o any change to reduce the list of exported symbols to those really intended to be exported (currently we export all public symbols in such cases). The projects using this file include: pango, cairo, fontconfig (and some more). We should switch these projects to -export-symbols upon their next update.

Note that the def file expected by -export-symbols contains C symbols, not DLL symbols. This means that these symbols do NOT start with underscore (it is stripped off from emxexp output and will be automatically added later when needed).

Bitching mode on. It was really complex to accomplish and test this task because libtool is ancient crap full of sick and dirty stuff like multiple nested escaping and so on. Bitching mode off.