The VBox 'Main' server traps in libc

[Valery V. Sedletski]

The problem is with VBox XPCOM API server (VBoxSVC.exe, aka 'Main'), it is related to another problem in ticket #13, but after I started VBoxXPCOMIPCD.exe manually, then VBoxSVC.exe, and if then start a frontend, or a tstVBoxAPIXPCOM.exe testcase (to test if the VBox XPCOM API is working), the VBoxSVC.exe traps in libc066.dll:

[l:\src\vbox\out\os2.x86\release\bin]VBoxSVC
***************************************************
Oracle VM VirtualBox XPCOM Server Version 5.0.0_OSE
(C) 2004-2015 Oracle Corporation
All rights reserved.

Starting event loop....
[press Ctrl-C to quit]

Killed by SIGSEGV
pid=0x5d0f ppid=0x5d03 tid=0x0003 slot=0x00c8 pri=0x0200 mc=0x0001 ps=0x0010
L:\SRC\VBOX\OUT\OS2.X86\RELEASE\BIN\VBOXSVC.EXE
LIBC066 0:0004ddea
cs:eip=005b:1df4ddea      ss:esp=0053:02b9e44c      ebp=02b9e574
 ds=0053      es=0053      fs=150b      gs=0000     efl=00010212
eax=00000004 ebx=00748dac ecx=0000002d edx=00000004 edi=ffffffff esi=00000004
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

And after this trap, VBoxXPCOMIPCD.exe terminates.

[Valery V. Sedletski]

Looking up the 0x0004ddea address in libc066.map, we'll find the following:

Module: D:\coding\libc\0.6\obj\OS2\RELEASE\emx\omf\libc_s.lib(fclose.obj)
0001:0004ddd8  __std_fclose

So, it seems to trap when calling fclose() on a NULL FILE *

[Valery V. Sedletski]

Also, if adding the trace printf's in ring3 runtime OS/2-related code, we can see that RTThreadSelf() returns NULL pThread prior to trapping in the libc. So, it does not initialize with rtThreadNativeAdopt(). Also, I can see much RTSemEventMulti*() calls here. Also, these calls were added quick and dirty and very likely contain errors. Need to check.

PS: 2_diver: Possibly, we need to add the 'Runtime' (and 'Main' too) to the list of components. Note that the Runtime has 'r0drv' part which is linked with the Host Driver as well as 'r3' part which is linked into Ring3 libraries.

[Valery V. Sedletski]

Yesterday I found the cause of the trap in VBoxSVC. The cause was somewhat unfinished code in src\VBox\Devices\Network\slirp\ regarding /etc/resolv.conf vs. %etc%\resolv2. So, it tried to open the file and failed, then tried to close an invalid file descriptor, so, the trap in fclose(). This is fixed. But then the new problem appears, with the int 3 in the host driver, because of a failed assertion (see the next ticket #15).

[Valery V. Sedletski]

This is fixed long time ago too.