coreutils: sort fails to close stdin if linked with OpenSSL

[dmik]

If you try to exec a piped sort call (e.g. execlp("sh.exe", "sh.exe", "-c", "sort | cat")) in a process whose STDIN is a socketpair's handle (i.e. was redirected to a pipe), then it will spit out the following error right before termination when it tries to fclose(stdin):

sort: close failed: -: Bad file number

The strange thing is that this failure only happens if sort.exe is built against OpenSSL (-lcrypto) to get fast MD5 calculation. If OpenSSL is disabled in the coreutils build (so that a slower fallback MD5 code is used), all works fine, even with socketpair etc.

[dmik]

So far I have no idea how md5 code from OpenSSL may screw stdin and it requires a lot of testing within both OpenSSL and coreutils to find it out. For this reason I'm postponing it for and simply providing a workaround for this in r2048.

[dmik]

Note that there is another tool with a similar problem: cat. One might experience messages like this

cat: closing standard input: Bad file number

in rpmbuild output when building some packages (e.g libc). This happens when running the os2deps.sh rpmbuild helper script, after cat in this script reads out all stdin sent to the script by rpmbuild.exe and then attempts to close it.

The funny (actually, ugly) thing is that this that this problem with cat is not permanent too. Sometimes it just works (it's the case with my old eCS 2.2 Beta II install, but not with my new ArcaOS 5.0.2 install). Since a local socket pair is involved in this pipeline (to perform stdio redirection), my current guess is that under some circumstances the OS/2 TCP/IP stack reports bad file number if the other end of the socket pair associated with stdin is closed by the time cat wants to close its end. An given that it doesn't occur all the time, it might be a timing issue.

In case of rpmbuild this mesasge is harmless (other than being annoying) but perhaps it makes sense to apply the same workaround as for sort (i.e. r2048) to cat as well — to get rid of this annoying error.

[Silvan Scherrer]

this seems to be a libcx issue, as more and more libcx based programs behave like that. An when build w/o libcx the message is gone

[dmik]

Note that there is ​https://github.com/bitwiseworks/libcx/issues/60 where I fixed the issue globally in kLIBC (329a5ae62cafed944b316853831c7676fd8f0ac5). Perhaps, it makes sense to undo r2048.

[Silvan Scherrer]

in r2310 I removed r2048 again, as done in klibc now.
We consider this fixed.