Reduce the number of exports in the Qt DLL

[dmik]

Currently, the Qt DLL exports about 16500 symbols. This amount is quite large and will most likely keep increasing slightly with every new Qt release or a new missing feature added. It's not a problem by itself, but sticks us to a concrete version of the linker: IBM ILINK from the distributioin of IBM Visual Age C/C++ 3.08. The general problem with this linker is that it's not free so that you cannot use it unless you have a valid license of Visual Age C/C++, which is obviously not good for such an open source project the Qt library is. Possibly alternatives include:

1. ILINK386 from the IBM OS/2 Developer's Toolkit. I haven't used it for Qt but heard that it has various problems related to linking the GCC code containing C++ classes.
2. ILINK 5.0, freely distributed by IBM (?). Looks like the latest linker for OS/2 made by IBM, but has very strange limitation (seems like a bug actually): the total number of exports cannot exceed 15000 or so. If there are more exports, the DLL just screws up the application using it when the latter gets loaded by the kernel for execution.
3. WCL386 from the Open Watcom compiler package. Currently, this linker is also not able to link the GCC's C++ code properly. Knut St Osmundsen (the current maintainer and the developer of the OS/2 version of GCC) is currently working on this issue, but we don't know anything yet about its limitation on the number of exports.

The last two can be freely used by anyone, so it would be great if they could be used to link the Qt DLL and Qt applications. Taking the limitations of ILINK 5.0 (and possible limitations of WCL386) into account, it's worth a try to reduce the number of exports in the Qt library.

[dmik]

As a simplest way to reduce the number of exports, we can use a filter on the emxexp's output to get rid of public symbols that are nor intended for neither actually used by end-user Qt applications. From what I've found, we can safely exclude the following public symbols:

1. Functions that have names starting with qt_.... These functions are not declared in public headers and used only to share pieces of internal code between different compilation units comprising the Qt library.
2. Classes that have names starting with Q... and ending with ...Private. These classes are not declared in public headers (only forward declarations are visible from there) and are used to hide the implementation details of Qt classes (that helps to maintain binary compatibility a lot).

We could also not export instantiated Qt templates (i.e. template instantiations used inside the Qt library itself). However, I think that it at least should be made optional -- exporting used template instantiations seems to be a good practice for me, because allows the client code to use the same instantiation of the template the Qt library uses.

[dmik]

Well, filtering emxexp's output with sed using the following script (see the attachment below) didn't give the desired effect. For the debug version of the DLL, it filters out just ~1600 symbols (out of total ~21500); for the release version -- only ~900 (out of total ~16200). This is definitely not enough for ILINK50, for example. So I will not use this approach but try a different one.

[dmik]

sed filter for QT.DEF

[dmik]

The other idea was to use objdump/nm and objcopy binutils to do the following:

1. Strip out N_EXT (0x6c) stab entries (created for symbols declared with __declspec(dllexport)) from the symbol table of every a.out-emx object module to a separate file (to prevent emxomf from emitting OMF export records, that would in turn prevent ilink from putting these symbols to the resident symbol table of a DLL).
2. Use files created in the previous step as a source for the .DEF file generation (instead of emxexp's output, and then use the generated .DEF file (with preserved ordinals and with the NONAME keyword for every exported symbol) when linking a DLL.

This idea would definitely work on practice, however I decided not to implement it because:

• It's a bit tricky. The proper way is to teach emxomf and/or weakld to handle N_EXT entires in a more flexible way so that it is possible (via cmd-line switches, for example) to choose from all three types of exports: resident (exported by name), non-resident (exported by name with manually assigned ordinals) and exported by ordinals only. This needs to be descussed with Knut...
• In case of the Qt DLL, if we use __declspec(dllexport), the number of exports reduces to ~12300 for the debug version and to ~11000 for the release version. This is too close to the limit of ILINK50, that makes it not really worth to modify the .DEF file generation approach.
• More over, from what I've recently heard (from Yuri Dario), ILINK50 has been licensed by IBM to build the Mozilla stuff only, that makes even less interesting for Qt.

[dmik]

Some thoughts about making the N_EXP handling more flexible. If we want to export by name only, __declspec(dllexport) already works just fine. But if we want to assign ordinals to exported symbols (either to make them non-resident or to export by ordinal only), it's not so clear where these assignments should be specified.

IBM VAC 3 provides the #pragma export directive to allow to assign ordinals and export names (if they differ from original symbol names) directly in the sources. However, I think GCC doesn't need this kind of support, because if one really needs to manually assign ordinals to particular symbols or to provide an alternative export name, I believe, it can happen only when the amount of symbols to export is relatively small (several dozens or so), and thus this can be easily done using a regular .DEF file (w/o involving the compiler at all).

When it comes to thousands of exports, we're most likely speaking about exported C++ classes. Manual ordinal assigning is definitely not what we want in this case. We actually need ordinals only for two purposes: to have symbols be exported by ordinals only (to get rid of export names that increase the DLL size significantly and slow down export resolution time when somebody tries to resolve them by name) and to guarantee assigned ordinals are persistend across the builds (to simplify maintaininfg of binary DLL compatibility).

One of possible ways to address both of the above needs is an option to emxomf that specifies an external file (exports map file) used to assign ordinals to symbols. It could process this file using the following logic:

while ((symbol = take_next_N_EXP_symbol()))
{
  if (exports_map_file.contains(symbol))
    ordinal = (exports_map_file.ordinal(symbol)    
  else
  {
    ordinal = exports_map_file.next_free_ordinal();
    exports_map_file.add (symbol, ordinal);
  }
}

The other emxomf option could just specify should symbols exported using the exports map file be also placed to the non-resident name table or just remain exported by ordinals only. This way, __declspec(dllexport) will still be the only thing necessary from the compiler in order to export large amounts of symbols with persistent ordinals.

[dmik]

Well, emxomf is not the right place for this because it will require to parse the exports map file on every object module, which is definitely not acceptable. Probably, weakld would be a more suitable place (again, Knut should know it better :)