Make binary builds portable

[Dmitry A. Kuminov]

Currently, Qt binaries (in particular, qmake.exe and QtCore4*.dll) have hard-coded paths in them that refer to various locations (e.g. include directory or plugin directory). This makes it impossible to distribute the Qt library in the binary form (for example, in order to provide a fast-to-deploy development environment that uses the officially distributed Qt libraries, tools and DLLs which makes it possible to distribute Qt DLLs separately from each Qt application and at the same time avoid DLL hell).

In particular, qmake uses these hard-coded paths when generating the Makefiles so if the Qt library root is changed (moved, renamed) an attempt to build an application will fail because include and library files will still be searched at the old location (whose path is built into qmake.exe).

In case of QtCore4.dll and its hard-coded paths, the potential problem is locating the Qt plugins that are by default searched at the hard-coded location that points inside the 'plugins' subdirectory of the Qt library root.

[Dmitry A. Kuminov]

Obviously, we should support overriding hard-coded paths with environment variables and use the location of qmake.exe and/or QtCore4.dll + some heuristics to detect the default paths for everything (tools, includes, libraries, plugins) in case if environment variables are not set.

Until then, there is no sense to distribute official binary builds. What we can do now is to distribute the Qt runtime only (i.e. only DLLs) but this will only work if all the applications using this runtime will be built by the same person to ensure DLL compatibility.

I'd like to make this task as a GA blocker but there are too many blockers for the GA already so we will see.

[Silvan Scherrer]

in my opinion the dll finding should work from the qtcore4.dll, as most ppl will not need qmake and other tools. but the plugins might be used far more.

for qmake and other tools we could eventually use qt.conf

[Dmitry A. Kuminov]

I'm going to implement the following logic for defining paths to the library components (see QLibraryInfo::location() for more information about component locations).

1. qt.conf is searched in the application resource bundle and in the application directory as described here: ​http://qt.nokia.com/doc/4.5/qt-conf.html. If found, it is used to read paths from.
2. qt.conf is searched in the directory that contains the QtCore DLL loaded by the current process (or the application executable's directory if Qt is built as a static library). If found, it is used.
3. qtsys.conf is searched in the %ETC%\qt\ subdirectory. If found, it is used.
4. As the very last step, the directory that contains the QtCore DLL loaded by the current process (or the application executable's directory if Qt is built as a static library) is used as the prefix (base) path for individual component directories.

Steps 3 and 4 will be OS/2-specific and will allow to completely get rid of hard-coded component paths as they don't make any sense on OS/2 anyway.

Steps 1 and 2 are useful for applications that provide their own Qt runtime with them (this practice should be avoided though). Step 3 is useful for the system-wide Qt installation (which we will eventually do). Step 4 is the last resort for cases if all other steps fail (e.g. there is no system-wide Qt installation and the application doesn't provide qt.conf with it).

[Dmitry A. Kuminov]

Actually, steps 2, 3 and 4 are OS/2-specific (other platforms don't do what step 2 describes).

The important addition to the above is that in steps 2, 3 and 4 the Prefix path in qt.conf is relative to the directory that contains this path. This allows to make portable Qt4 runtime bundles with qt.conf in them: if you specify a correct relative path for Prefix, you will be able to move the whole bundle to a separate directory and it will still report the correct paths (taking the new location into account). A typical directory layout of the portable Qt4 runtime bundle may look like this:

.\QtCore4.dll
.\QtGui4.dll
.\Qt*4.dll
.\accessible\<accessible_plugins>
.\codecs\<codec_plugins>
.\<other_plugin_type>\<other_plugins>
.\translations\*.qm
.\qt.conf

with qt.conf containing the following:

[Paths]
Prefix = .
Plugins = .
Settings = $(ETC)/qt

However, in case of step 3 (qtsys.conf) the Prefix path is relative to the *root* directory of the boot drive. This is done because qtsys.conf is located in %ETC%\qt and making it relative to this location makes no sense since the DLLs will never be placed there anyway. A typical system-wide Qt4 installation will then look like this:

C:\OS2\DLL\QtCore4.dll
C:\OS2\DLL\QtGui4.dll
C:\OS2\DLL\Qt*4.dll
C:\OS2\DLL\qt\plugins\accessible\<accessible_plugins>
C:\OS2\DLL\qt\plugins\codecs\<codec_plugins>
C:\OS2\DLL\qt\plugins\<other_plugin_type>\<other_plugins>
C:\OS2\DLL\qt\translations\*.qm
C:\MPTN\ETC\qt\qtsys.conf

where C: is the boot drive and \OS2\DLL\ may be replaced with \ecs\dll\ on eCS systems; and C:\MPTN\ETC is actually where %ETC% points to. The accompanying qtsys.conf in %ETC%\qt will be:

[Paths]
Prefix = /OS2/DLL/qt
Settings = $(ETC)/qt

This way, even if the user later changes the letter of the boot drive, it will not break the paths.

We will even be able to supply pre-built development bundles of Qt4 that in addition to the Qt4 runtime DLLs and plugins contain header files and tools such as qmake.exe and moc.exe. All that is necessary to make it work is to retain a standard Qt4 source tree (with ./bin, ./include, ./lib and so on) and place a qt.conf file in ./bin that will only contain:

[Paths]
Prefix = ..

This will make sure that qmake.exe will find everything it needs when generating Makefiles. This development bundle will be portable as well (wow :), the only thing necessary to do after moving it to another directory is to run qmake to regenerate the Makefiles (as it still uses absolute paths in a few places) but at least not the Qt4 library itself.

[Dmitry A. Kuminov]

Ergh. Another hairy Qt defect. A path like "\some\dir\file.txt" is considered as absolute by Qt which is absolutely wrong on OS/2 (as well as on any other DOS-like OS like Win32) because it doesn't necessarily lead to the same object from all possible locations which breaks the definition of the term "absolute path".

The problem is that there is a dozen of workarounds for this both in Qt itself and I suppose in Qt applications so that I'm afraid correcting this behavior may break these workarounds. Anyway, I'll try to enable it at least on OS/2 and see what happens.

[Dmitry A. Kuminov]

Hey, I didn't change the summary by intention (a frequent thing in trac though).

[Dmitry A. Kuminov]

In r355, I changed the logic of detecting relative and absolute paths. Now, "A:bbb.txt" and "/ccc.dat" are treated as relative paths instead of absolute. The rationale behind that is that the absolute path, by definition, leads to the same object from any location which is not true for paths like these. This conclusion gives us the right to consider the previous behavior it as a vendor bug (as it was copied from the Win32 code where it is still present). But again, I didn't touch the Win32 part for the reasons mentioned above.

[Dmitry A. Kuminov]

I committed the described portability scheme in r357. Please test it by making a clean build in a fresh directory to make sure there are no regressions.

[Dmitry A. Kuminov]

Okay, I found the reason of one r355 regression (see the message from Michael Holzapfel in qt.devel). This is again a compilation of hairy inconsistencies in Qt:

1. qmake uses QDir::isRelativePath() to see if a given file path is relative and if so, it prepends each of the search paths to this file path, for no reason. Probably the guy that wrote that code didn't know that Qt has to support other OSes besides *nix. So, given the path /some/dir/file.pro (which is relative on OS/2 because it depends on the current drive letter), qmake prepends a string like /search/here to it which results in garbage (/seacrh/here/some/dir/file.pro. This is wrong but alone it won't cause a crash.

2. This is more complex. When processing subdirs templates, qmake loads .pro files from the subdirectories the SUBDIRS variable refers to. I don't know exactly why: it doesn't generate Makefiles from these .pro files at this step anyway so it could simply leave them alone with the same result. The thing is that when qmake performs this dummy load of each .pro file from SUBDIRS, it doesn't load .qmake.cache for them. As a result, all vital information from .qmake.cache (like QT_ARCH or QT_SOURCE_TREE) is null which, in turn, leads to very bad results. For example, $$QT_SOURCE_TREE/src/corelib/arch/$$QT_ARCH/arch.pri becomes just /src/corelib/archarch.pri, and when qmake tries to include this file, it prepends the current dir to it, and, according to 1., tries to load D:/Coding/Qt/Qt4//src/corelib/arch//arch.pri. The funnest part is that this load statement I took as an example is located in the D:/Coding/Qt/Qt4//src/corelib/arch/arch.pri file. In other words, the .pri file endlessly tries to load itself. After some time the stack space gets exhausted by this infinitve loop and we see a qmake crash described in the mailing list.

[Dmitry A. Kuminov]

qmake is ill :) Look at r362 where I fixed dozens of incorrect QDir::isRelativePath() usages including the one mentioned above. Note two things:

1. The fixes I applied are r355- and OS-neutral. It means that on OSes other than OS/2 it will work the same way as before (since the fix just does the job in a more universal way) and it will even work as before if we roll back r355 on OS/2.
2. I only fixed common qmake parts as well as some Win32 parts that the GNUMAKE generator used in OS/2 relies on. There is a dozen more incorrect places which need to be fixed for general consistency, but I don't have time for such level of cosmetics.

[Dmitry A. Kuminov]

Okay, there is a whole bunch of other invalid assumptions that relative paths are those that don't start with '/', not only in qmake but in various parts of Qt too (for example, QDir::filePath()). This is all WRONG but I'm not going to do the job for Qt developers and fix the whole Qt for them.

I will revert r355, r362 and relative changesets and will introduce another ugly workaround for OS/2 to solve the original problem I made r355 for. "A:bbb.txt" and "/ccc.txt" will again be treated absolute by Qt. We will have to live wit that, as the only way to get rid of this fundamental bug is to rewrite many parts of corelib and qmake from scratch which is beyond our interests.

[Dmitry A. Kuminov]

So, I reverted r362, r361, r356, r355.

In r365, I resurrected the QDir::absoluteFilePath() functionality present in r355. This is now the only way to have all kinds of relative paths correctly resolved, so that

aaa\bbb.txt    becomes  <cur_drv>:\<cur_dir>\aaa\bbb.txt
A:aaa\bbb.txt  becomes  A:\<cur_dir_on_A>\aaa\bbb.txt
\aaa\bbb.txt   becomes  <cur_drv>\aaa\bbb.txt

Just for the record. All this hassle was originally necessary for QLibraryInfo, to resolve relative paths like \OS2\DLL\qt to C:\OS2\DLL\qt where C: is the drive that contains qt.conf... This is ensured to work using the fixed QDir::absoluteFilePath() in r366.

[Dmitry A. Kuminov]

I think it works now as nobody reports any related problems. Closing this defect. If a new problem encountered, please open a new defect.

PS. Silvan, you may now create a WPI for both the runtime (DLLs + plugins + translations) and the development bundle (we'll discuss online what should go there). I think the latter will significantly simplify the life for people wanting to develop or port Qt apps on OS/2 and attract more of them :) I created #107 for tracking this.