YUM fails after update to Libcx 0.4.1

[David McKenna]

After updating libcx to 0.4.1 (using ANPM), YUM often (but not always) fails like this:

[C:\]yum update
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.4.1
/src/shared.c, line 169

Killed by SIGABRT
pid=0x00fa ppid=0x00f9 tid=0x0001 slot=0x00cb pri=0x0200 mc=0x0001 ps=0x0010
C:\USR\BIN\PYTHON2.7.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

ANPM shows empty repository lists (presumably because YUM LIST fails).

[Silvan Scherrer]

did you reboot after the update?

[David McKenna]

Yes... several times. And each time, YUM works at first, than I start getting the listed failure - always the same, no matter what YUM command I use.

[Silvan Scherrer]

and you don't have some other libcx around? like from samba v4 testing or such?

[dmik]

This assertion is when LIBCx can't get access to its shared memory. This is either a weird crash in some other process that breaks initialisation of LIBCx or a nasty race when one LIBCx process inits itself first and very quickly terminates while another one is trying to reuse that process' mutex and shared mem.

Could you please try it with the debug version of LIBCx by doing the following?

1. Download ​http://rpm.netlabs.org/test/ld.zip and install it in place of the RPM LIBCx (i.e. to /@unixroot/usr/lib)
2. Add SET LIBCX_TRACE=-ALL to config.sys
3. Reboot
4. Open a command prompt and do SET LIBCX_TRACE=+ALL there.
5. Try to reproduce the problem and collect all console output.
6. Post output here.

[David McKenna]

OK... here is the output:

---

[C:\]SET LIBCX_TRACE=+ALL

[C:\]yum list
* 000c93bd [6f:1] shared.c:389:_DLL_InitTerm: ulFlag 0

* 000c93bd [6f:1] shar

ed.c:152:shared_init: DosOpenMutexSem = 0

* 000c93bd [6f:1] shared.c:162:shared

_init: DosRequestMutexSem = 0

* 000c93bd [6f:1] shared.c:168:shared_init: DosGe

tNamedSharedMem = 32794

Assertion failed: arc == NO_ERROR, file D:/Coding/libcx/m

aster/src/shared.c, line 169

Killed by SIGABRT
pid=0x006f ppid=0x006e tid=0x0001 slot=0x009c pri=0x0200 mc=0x0001 ps=0x0010
C:\USR\BIN\PYTHON2.7.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
* 000c93bf [6f:1] shared.c:438:ProcessExit: reason 0

* 000c93bf [6f:1] shared

.c:267:shared_term: gMutex 80010050, gpData 0 (heap 0, refcnt 0)

* 000c93bf [6f

:1] shared.c:376:shared_term: DosReleaseMutexSem = 0

* 000c93bf [6f:1] shared.c

:379:shared_term: DosCloseMutexSem = 0

* 000c93c1 [6f:1] shared.c:389:_DLL_Init

Term: ulFlag 1

---

It failed on the 3rd attempt at 'YUM LIST'. There is definitely no other libcx on this system:

[C:\]dir libcx* /s

The volume label in drive C is OS_2.
The Volume Serial Number is 34B0:55C1.

Directory of C:\usr\bin

1-18-17 3:51p 3,420 124 a--- libcx-stats.dbg

1 file(s) 3,420 bytes used

Directory of C:\usr\lib

1-18-17 3:51p 55,856 124 a--- libcx0.dbg
1-17-17 5:55p 134,191 367 a--- libcx0.dll

2 file(s) 190,047 bytes used

Directory of C:\usr\share\doc

1-18-17 5:07p <DIR> 124 a--- libcx-0.4.1

1 file(s) 0 bytes used

Total files listed:

4 file(s) 193,467 bytes used

54,649,655 K bytes free

[David McKenna]

A couple observations since reporting this...

Went back to libcx 0.4.0 and still have this problem, so it is not specific to 0.4.1, Just didn't notice it until now.

One thing I have been doing for quite a while now is loading dll's high (code only, no data) using the highmem utility. I run 'highmem -c *.dll' on the /usr/lib directory (after unlocking everything), then run 'highmem -u' on all libc*.dll's along with winmm.dll and user32.dll (found they didn't work when loaded high). As a test, I unloaded from high memory all dll's in the /usr/lib directory (and rebooted) and so far have not been able to get YUM to fail (at least 25 attempts). So it seems this problem is somehow related to loading something into high memory, even though libcx0.dll itself is not loaded high.

[dmik]

David, thank you for your feedback. The error from DosGetNamedSharedMem (32794) is ERROR_NO_OBJECT. This looks strange as I never saw this error myself and it is not listed among known DosGetNamedSharedMem errors. OTOH, I wonder how that could be related to loading the code segment high... No ideas so far. Although, when closely looking at the relevant code I did find one potential hole (race condition) in LIBCx initialisation that could lead to a failure from DosGetNamedSharedMem (I would expect a different return code though). I closed this hole in my new test build, specially for you to test. It's here: ​http://rpm.netlabs.org/test/ld2.zip. So please return your faulty env back and try this one in it. It will also generate logs.

BTW, in order to keep plain text files intact here, either enclose the contents in {{{ and }}} (each on new line) or upload the file to pastebin and post a link here. This is an example of plain text with its structure preserved using {{{ and }}}:

line 1
  line 2
    line 3

[David McKenna]

OK, thanks. Here's what I did:

1) loaded everything in \usr\lib high, except as already noted
2) rebooted
3) made sure I still got the error - I did on the 4th 'yum list'
4) replaced libcx0.dll with the one in ld2.zip
5) added SET LIBCX_TRACE=-ALL to CONFIG.SYS
6) rebooted
7) opened a command line window and typed 'SET LIBCX_TRACE=+ALL'
8) typed 'yum list'

It worked 3 times, but on the 4th time, the scrolling text never stopped - I let it go for about 10 minutes before closing the window. Do you want the output of the working case? Also, is there a command I can use to capture the console directly to a file?

[dmik]

Good. No, there is no way to redirect it to a file (this is due to one nasty limitation of fork in kLIBC). So you have to redirect yum list output to a file. And yes, I'm interested in it. But I guess it will be too big so better upload it to pastebin.com.

[David McKenna]

OK, I rebooted then ran 'yum list > yum.log'. This put all the results of 'yum list into a file and the console only showed the trace stuff. Then I copied the console contents exactly as shown into yumtrace.log. Both attached.

[David McKenna]

I should mention, when I do this and I get the never ending scrolling, yum.log never gets anything inserted.

[David McKenna]

FWIW... through trial and error I found that if I do not load mmap.dll high, this problem goes away. So now my list of 'do not load high' dll's in \usr\lib is:

libc066.dll
libcx0.dll
mmap.dll
user32.dll
winmm.dll

[Silvan Scherrer]

as this clearly is not a yum issue I close the ticket. If you think libcx should be high loadable open a issue at the libcx bugtracker at github.

​https://github.com/bitwiseworks/libcx