Poppler exhausts OS2's semaphore handles when opening large pdfs

[Gregg Young]

Poppler has 13 classes that create a new semaphore for each class object. Created in the constructor destroyed in the distructor. With large documents the number can grow to exhaust the 65,535 available semaphore handles.

The fix should be relatively easy. Make the semaphores statics. Create in the constructor only if they don't exist then destroy in the exit code. I can do the fix but what should I checkout vendor/current or the trunk? Anything special needed to build it?

Note qpdfview suffers from the same problem. It is possible poppler is leaking semaphore handle but if it is it is at a very slow rate. The problem seems worse in later versions.

See Lucide ticket 355 for more details. ​http://trac.netlabs.org/lucide/ticket/355

[Silvan Scherrer]

I looked at the code and poppler doesn't use any mutex anymore. It used to do that long ago. see ​http://trac.netlabs.org/ports/changeset?reponame=&new=898%40poppler%2Ftrunk%2Fgoo&old=878%40poppler%2Ftrunk%2Fgoo

But poppler uses pthread since then and this uses mutex iirc.

To build poppler I heavily suggest to look at the spec file. As this explains it all. If you have some doubts please write me a mail and I try to help. Which should be possible as I maintain poppler :)

[Gregg Young]

I download the source and finally got autoconf and friends to create makefiles. I then tried to build it. It fails with a bunch of undefined symbol errors at CXXLD libpoppler.la. It also gives 5 SYS 1003 errors "The syntax of the command is incorrect". I am reasonably sure the errors are while libtool is running. The problem is it doesn't tell me what commands result in the errors. I tried make -d and libtool -v -Wall but I don't get any more information about the errors. Any suggestions as to how to debug this. I think I have all the required packages.

[Silvan Scherrer]

I hope you followed the spec to autoreconf and also used the parameters like in the spec for configure. If that's the case please zip the config.log the makefile and the output of make and attach it in here. I will then take a look. As I'm almost sure you miss something.
And the most important question: you use only rpm installed tools I hope. As only this set up is 100% supported by us.

[Gregg Young]

This is on a clean install of ArcaOS. I used ANPM to add all the tools. I have not run any of my other build environment scripts.
I downloaded vendor/current;
ran autoreconf -fvi in sh
ran ./config (needed to fix the SHELL path to /usr/bin/sh)
ran make
It built everything up until libpoppler.la. It appears that every function in it was seen as an undefined reference. Attached is config.log, poppler/makefile and the make output. Let me know if you need anything else. Thanks

[tellie]

Hi Greg,
I would like Silvan suggest go to look at the spec file..
​http://trac.netlabs.org/rpm/browser/spec/trunk/SPECS/poppler.spec
As i see you miss some dependencies...

[Silvan Scherrer]

Yes please look at the spec file. And vendor/current is of course the wrong one. As this as pure vendor. Trunk has the changes. But as said the spec describes it all.

[Gregg Young]

Hi

I reviewed the spec file and verified I now have all the packages listed. I downloaded the trunk and ran autoreconf -fvi. I then added:
POPPLER_QT4_CFLAGS='-DOS2'
POPPLER_QT4_LIBS='-lQtCore4 -lQtGui4 -lQtNetwork4 -lQtXml4'
POPPLER_QT4_TEST_CFLAGS=$POPPLER_QT4_CFLAGS
POPPLER_QT4_TEST_LIBS=$POPPLER_QT4_LIBS
PATH=$PATH';/@unixroot/usr/lib/qt4/bin'

LDFLAGS=" -Zhigh-mem -Zomf -Zargs-wild -Zargs-resp -lcx"

export LDFLAGS
export POPPLER_QT4_CFLAGS
export POPPLER_QT4_LIBS
export POPPLER_QT4_TEST_CFLAGS
export POPPLER_QT4_TEST_LIBS
export PATH
export VENDOR="%{vendor}"
to configure.

Also added:
CONFIG_SHELL="/usr/bin/sh"
export CONFIG_SHELL

Fixed a failure to run config.sub

I ran ./configure --enable-poppler-qt4=yes --enable-zlib=yes \

--enable-zlib-uncompress=yes --enable-libopenjpeg=none \
--enable-shared --disable-static \
--enable-xpdf-headers

I then ran make.

I get the same syntax and undefined symbol errors with CXXLD libpoppler.la

Note I also tried it using:
aclocal; autoheader; automake; autoreconf -fvi
I did that because make gave warnings about missing autoheader and automake. Running it this way fixed those errors but the build still failed the same as before.

[Silvan Scherrer]

do you have wlink and friends as rpm installed? are you 100% sure you have all rpm at the latest release? as setting a config_shell is not needed when you have all rpm at latest and you did not change any config.sys by hand. As the rpm add some necessary settings. eventually attach the config.sys as well, as I then can spot the missing settings, if any.
Andf change

AM_DEFAULT_VERBOSITY = 0

to

AM_DEFAULT_VERBOSITY = 1

in the makefile. Then you should see a lot more. As I suspect you still miss something.

[Silvan Scherrer]

and btw these are the config.sys settings which are set by some rpm packages:

SET EMXOMFLD_LINKER=wl.exe
SET EMXOMFLD_TYPE=WLINK
SET EMXOMFLD_RC=wrc.exe
SET EMXOMFLD_RC_TYPE=WRC
SET TERM=os2

REM [ Default shell values ]
SET SHELL=C:/usr/bin/sh.exe
SET EMXSHELL=C:/usr/bin/sh.exe
SET CONFIG_SHELL=C:/usr/bin/sh.exe
SET MAKESHELL=C:/usr/bin/sh.exe
SET EXECSHELL=C:/usr/bin/sh.exe

of course adjusted to your drive letters

[Gregg Young]

Replying to diver:

do you have wlink and friends as rpm installed? are you 100% sure you have all rpm at the latest release? as setting a config_shell is not needed when you have all rpm at latest and you did not change any config.sys by hand. As the rpm add some necessary settings. eventually attach the config.sys as well, as I then can spot the missing settings, if any.
Andf change

AM_DEFAULT_VERBOSITY = 0

to

AM_DEFAULT_VERBOSITY = 1

in the makefile. Then you should see a lot more. As I suspect you still miss something.

Changing the verbosity gave me this from libtool (The longest command line is 2941)The files are in the .libs/libpoppler.lax/*.a directories:
libtool: link: (cd .libs/libpoppler.lax/goo.a && ar x "X:/Ptrunk/poppler/../goo/.libs/goo.a")
SYS1003: The syntax of the command is incorrect.
libtool: link: (cd .libs/libpoppler.lax/fofi.a && ar x "X:/Ptrunk/poppler/../fofi/.libs/fofi.a")
SYS1003: The syntax of the command is incorrect.
libtool: link: (cd .libs/libpoppler.lax/splash.a && ar x "X:/Ptrunk/poppler/../splash/.libs/splash.a")
SYS1003: The syntax of the command is incorrect.
libtool: link: printf %s\n "LIBRARY popple66 INITINSTANCE TERMINSTANCE" > .libs/popple66.def
libtool: link: printf %s\n "DESCRIPTION \"@#community:0.52.0#@##1## 06 Aug 2018 11:33:00 ZOBOPEEP::::0::@@poppler\"" >> .libs/popple66.def
libtool: link: printf %s\n "DATA MULTIPLE NONSHARED" >> .libs/popple66.def
libtool: link: printf %s\n EXPORTS >> .libs/popple66.def
libtool: link: emxexp .libs/libpoppler_la-SplashOutputDev.o .libs/libpoppler_la-DCTStream.o .libs/libpoppler_la-FlateEncoder.o .libs/libpoppler_la-FlateStream.o .libs/libpoppler_la-SignatureHandler.o .libs/libpoppler_la-JPXStream.o .libs/libpoppler_la-Annot.o .libs/libpoppler_la-Array.o .libs/libpoppler_la-BuiltinFont.o .libs/libpoppler_la-BuiltinFontTables.o .libs/libpoppler_la-CachedFile.o .libs/libpoppler_la-Catalog.o .libs/libpoppler_la-CharCodeToUnicode.o .libs/libpoppler_la-CMap.o .libs/libpoppler_la-DateInfo.o .libs/libpoppler_la-Decrypt.o .libs/libpoppler_la-Dict.o .libs/libpoppler_la-Error.o .libs/libpoppler_la-FileSpec.o .libs/libpoppler_la-FontEncodingTables.o .libs/libpoppler_la-Form.o .libs/libpoppler_la-FontInfo.o .libs/libpoppler_la-Function.o .libs/libpoppler_la-Gfx.o .libs/libpoppler_la-GfxFont.o .libs/libpoppler_la-GfxState.o .libs/libpoppler_la-GlobalParams.o .libs/libpoppler_la-Hints.o .libs/libpoppler_la-JArithmeticDecoder.o .libs/libpoppler_la-JBIG2Stream.o .libs/libpoppler_la-Lexer.o .libs/libpoppler_la-Linearization.o .libs/libpoppler_la-Link.o .libs/libpoppler_la-LocalPDFDocBuilder.o .libs/libpoppler_la-Movie.o .libs/libpoppler_la-NameToCharCode.o .libs/libpoppler_la-Object.o .libs/libpoppler_la-OptionalContent.o .libs/libpoppler_la-Outline.o .libs/libpoppler_la-OutputDev.o .libs/libpoppler_la-Page.o .libs/libpoppler_la-PageTransition.o .libs/libpoppler_la-Parser.o .libs/libpoppler_la-PDFDoc.o .libs/libpoppler_la-PDFDocEncoding.o .libs/libpoppler_la-PDFDocFactory.o .libs/libpoppler_la-PopplerCache.o .libs/libpoppler_la-ProfileData.o .libs/libpoppler_la-PreScanOutputDev.o .libs/libpoppler_la-PSTokenizer.o .libs/libpoppler_la-Rendition.o .libs/libpoppler_la-SignatureInfo.o .libs/libpoppler_la-StdinCachedFile.o .libs/libpoppler_la-StdinPDFDocBuilder.o .libs/libpoppler_la-Stream.o .libs/libpoppler_la-StructTreeRoot.o .libs/libpoppler_la-StructElement.o .libs/libpoppler_la-strtok_r.o .libs/libpoppler_la-UnicodeMap.o .libs/libpoppler_la-UnicodeTypeTable.o .libs/libpoppler_la-UTF.o .libs/libpoppler_la-ViewerPreferences.o .libs/libpoppler_la-XRef.o .libs/libpoppler_la-PSOutputDev.o .libs/libpoppler_la-TextOutputDev.o .libs/libpoppler_la-MarkedContentOutputDev.o .libs/libpoppler_la-PageLabelInfo.o .libs/libpoppler_la-SecurityHandler.o .libs/libpoppler_la-Sound.o .libs/libpoppler_la-XpdfPluginAPI.o | /@unixroot/usr/bin/sed.exe /"_DLL_InitTerm"/d >> .libs/popple66.def
libtool: link: g++ -Zdll -Zomf -o .libs/popple66.dll .libs/libpoppler_la-SplashOutputDev.o .libs/libpoppler_la-DCTStream.o .libs/libpoppler_la-FlateEncoder.o .libs/libpoppler_la-FlateStream.o .libs/libpoppler_la-SignatureHandler.o .libs/libpoppler_la-JPXStream.o .libs/libpoppler_la-Annot.o .libs/libpoppler_la-Array.o .libs/libpoppler_la-BuiltinFont.o .libs/libpoppler_la-BuiltinFontTables.o .libs/libpoppler_la-CachedFile.o .libs/libpoppler_la-Catalog.o .libs/libpoppler_la-CharCodeToUnicode.o .libs/libpoppler_la-CMap.o .libs/libpoppler_la-DateInfo.o .libs/libpoppler_la-Decrypt.o .libs/libpoppler_la-Dict.o .libs/libpoppler_la-Error.o .libs/libpoppler_la-FileSpec.o .libs/libpoppler_la-FontEncodingTables.o .libs/libpoppler_la-Form.o .libs/libpoppler_la-FontInfo.o .libs/libpoppler_la-Function.o .libs/libpoppler_la-Gfx.o .libs/libpoppler_la-GfxFont.o .libs/libpoppler_la-GfxState.o .libs/libpoppler_la-GlobalParams.o .libs/libpoppler_la-Hints.o .libs/libpoppler_la-JArithmeticDecoder.o .libs/libpoppler_la-JBIG2Stream.o .libs/libpoppler_la-Lexer.o .libs/libpoppler_la-Linearization.o .libs/libpoppler_la-Link.o .libs/libpoppler_la-LocalPDFDocBuilder.o .libs/libpoppler_la-Movie.o .libs/libpoppler_la-NameToCharCode.o .libs/libpoppler_la-Object.o .libs/libpoppler_la-OptionalContent.o .libs/libpoppler_la-Outline.o .libs/libpoppler_la-OutputDev.o .libs/libpoppler_la-Page.o .libs/libpoppler_la-PageTransition.o .libs/libpoppler_la-Parser.o .libs/libpoppler_la-PDFDoc.o .libs/libpoppler_la-PDFDocEncoding.o .libs/libpoppler_la-PDFDocFactory.o .libs/libpoppler_la-PopplerCache.o .libs/libpoppler_la-ProfileData.o .libs/libpoppler_la-PreScanOutputDev.o .libs/libpoppler_la-PSTokenizer.o .libs/libpoppler_la-Rendition.o .libs/libpoppler_la-SignatureInfo.o .libs/libpoppler_la-StdinCachedFile.o .libs/libpoppler_la-StdinPDFDocBuilder.o .libs/libpoppler_la-Stream.o .libs/libpoppler_la-StructTreeRoot.o .libs/libpoppler_la-StructElement.o .libs/libpoppler_la-strtok_r.o .libs/libpoppler_la-UnicodeMap.o .libs/libpoppler_la-UnicodeTypeTable.o .libs/libpoppler_la-UTF.o .libs/libpoppler_la-ViewerPreferences.o .libs/libpoppler_la-XRef.o .libs/libpoppler_la-PSOutputDev.o .libs/libpoppler_la-TextOutputDev.o .libs/libpoppler_la-MarkedContentOutputDev.o .libs/libpoppler_la-PageLabelInfo.o .libs/libpoppler_la-SecurityHandler.o .libs/libpoppler_la-Sound.o .libs/libpoppler_la-XpdfPluginAPI.o -llcms2 -ltiff -ljpeg -lpng16 -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lfontconfig -lfreetype -lpthread -lz -lcx -L/@unixroot/usr/bin/../lib/gcc/i386-pc-os2-emx/4.9.2 -L/@unixroot/usr/bin/../lib/gcc -L/@unixroot/usr/bin/../lib/gcc/i386-pc-os2-emx/4.9.2/../../.. -L/@unixroot/usr/lib -lstdc++ -lm -lgcc_so_d -lc_alias -lc_dll -los2 -lgcc_so_d -lc_alias -lc_dll -los2 -lgcc_so_d -lc_alias -lc_dll -los2 -lgcc_so_d -lc_alias -lc_dll -los2 -g -O2 -Zhigh-mem -Zomf -Zargs-wild -Zargs-resp -Zhigh-mem -Zomf -Zargs-wild -Zargs-resp .libs/popple66.def
weakld: error: Unresolved symbol (UNDEF) 'ZN6Splash11getSoftMaskEv'.

[Gregg Young]

Replying to diver:

and btw these are the config.sys settings which are set by some rpm packages:

SET EMXOMFLD_LINKER=wl.exe
SET EMXOMFLD_TYPE=WLINK
SET EMXOMFLD_RC=wrc.exe
SET EMXOMFLD_RC_TYPE=WRC
SET TERM=os2

REM [ Default shell values ]
SET SHELL=C:/usr/bin/sh.exe
SET EMXSHELL=C:/usr/bin/sh.exe
SET CONFIG_SHELL=C:/usr/bin/sh.exe
SET MAKESHELL=C:/usr/bin/sh.exe
SET EXECSHELL=C:/usr/bin/sh.exe

of course adjusted to your drive letters

The first set (EMXOMFLD/TERM) were present. The shells weren't but interestingly the header (REM [ Default shell values ]) was present. This header was added by one of the RPMs. Adding the shells fixed the need to add CONFIG_SHELL to configure but didn't fix the problem. Could this be a *path issue? Thanks

[Gregg Young]

I added --debug to libtool and found out what was missing. It was find. It now builds. Thanks for your help. The SYS 1003s were caused by the lack of find. It would be nice if it had been a no such file error.

[Silvan Scherrer]

Replying to gyoung:

Replying to diver:

and btw these are the config.sys settings which are set by some rpm packages:

SET EMXOMFLD_LINKER=wl.exe
SET EMXOMFLD_TYPE=WLINK
SET EMXOMFLD_RC=wrc.exe
SET EMXOMFLD_RC_TYPE=WRC
SET TERM=os2

REM [ Default shell values ]
SET SHELL=C:/usr/bin/sh.exe
SET EMXSHELL=C:/usr/bin/sh.exe
SET CONFIG_SHELL=C:/usr/bin/sh.exe
SET MAKESHELL=C:/usr/bin/sh.exe
SET EXECSHELL=C:/usr/bin/sh.exe

of course adjusted to your drive letters

The first set (EMXOMFLD/TERM) were present. The shells weren't but interestingly the header (REM [ Default shell values ]) was present. This header was added by one of the RPMs. Adding the shells fixed the need to add CONFIG_SHELL to configure but didn't fix the problem. Could this be a *path issue? Thanks

unfortunately ArcaOS installs some rpm not right. Means it does not add the config.sys changes. IIRC it's corrected in the last drops. But I'm not sure.

[Silvan Scherrer]

Replying to gyoung:

I added --debug to libtool and found out what was missing. It was find. It now builds. Thanks for your help. The SYS 1003s were caused by the lack of find. It would be nice if it had been a no such file error.

Yes find is used a lot in libtool. If you had installed rpm-build rpm (which is needed to create rpm's itself) findutil would have been installed.

It would be great, if you could note down all needed packages config.sys changes and such to build poppler. So we could create a better wiki page to help newbies to set up their env to have something building.

[Gregg Young]

I have attached Steven's patch which fixes this issue. It would be great if an updated Poppler with this fix could be released. I will put together a list of packages and setting for building Poppler. It will be several weeks before I do this as I leave for vacation tomorrow. Thanks

[Silvan Scherrer]

r2299 fixes it. a new poppler release is around the corner and this will include the fix.