Opened 17 months ago

Last modified 4 months ago

#377 new defect

Lucide is incredibly slow reading PDFs over network (Samba share)

Reported by: darcio Owned by:
Priority: major Milestone: 1.4.1
Component: Plugin: PDF Version: 1.4.0
Keywords: Cc:

Description

I have the latest Lucide test version installed and as I attempted to move my locally stored documents to the LAN NAS box (ZyXel? 325 V2) I discovered that attempting to view any PDFs stored on that NAS box is incredibly slow. Local storage is fast though.

I'm using NetDrive? (3.1.6.876) here with the latest Samba client (samba-client.i686 - 4.9.5-1.oc00). Indeed, reading other files appears to move along pretty quickly, JPG images are fast, as are MP4 files. Only the PDFs appear to suffer.

I opened a thread on this topic here (https://www.os2world.com/forum/index.php/topic,2034.0.html). Paul reviewed the samba client logs and ollowing some troubleshooting suggestions I discovered that Lucide 1.3.5 GA does not appear to suffer from the same problem.

Not sure what additional info/logs to provide at the moment to help troubleshoot further.

Attachments (2)

lucide_dll_list (106.7 KB) - added by darcio 17 months ago.
dynamic DLL load for local storage PDF file
lucide_dll_list_NAS (118.5 KB) - added by darcio 17 months ago.
dynamic DLL load for NAS storage PDF file

Download all attachments as: .zip

Change History (18)

comment:1 Changed 17 months ago by Silvan Scherrer

please try also with qpdfview to see if it's poppler or a lucide issue. as we did some tests and qpdfview was a lot faster over network than lucide. but YMMV

comment:2 Changed 17 months ago by herwigb

Tried with Lucide 1.4.0 using popple66.dll and qpdfview with latest poppler, in addition with bitwise ndpsmb plugin 2.3.0 (based upon Samba 3.6.x). Neither setup shows any "incredible slowness".

qpdfview is faster than Lucide, however Lucide is workable normally.

comment:3 Changed 17 months ago by Gregg Young

I just tried with the Lucide that hopefully will be 1.5.0 GA and don't see this. (Using the smb client 4.9.5 the server is 3.3.16 on OS2) The only changes from the 1.4.1 beta 4 are the addition of a Polish translation and fixing unhighlighting neither of which would effect this. Dariusz can you send me a file that shows this issue for you? Here Lucide is slightly faster than qpdfview but both are fully usable. 1.3.5 is slower than 1.5.0 also.

Last edited 17 months ago by Gregg Young (previous) (diff)

comment:4 Changed 17 months ago by darcio

So guys, the poppler library installed here is: poppler.pentium4 - 0.59.0-3.oc00.

@diver Yes, QPDFView shows the same behaviour, identical throughput if you will, it is using the same poppler library as Lucide.

@gyoung I sent you an email with the PDF file, it's about 2.5Meg.

Per Gregg's suggestion I will try to find Beta2 of QPDFViewer to try it out first and will re-install Lucide RC4 and downlevel the poppler library.

comment:5 Changed 17 months ago by Gregg Young

@Dariusz I got the pdf you sent. I tried it with Lucide 1.5.0, QPDFView 0.4.17 beta 2 and Ghostview. While Ghostview probably loaded the fastest the differences were negligible. The only meaningful difference I saw was the file open dialog in QPDFView took much longer to populate than the other 2.

QPDFView 0.4.17 beta 2 is in the netlabs-rel RPM repository.

You probably don't want to down level. The Poppler I am using 0.59.0-3 which is the latest. I have the i686 version installed. The difference between -2 and -3 is the deprecated internal jpg rendering engine is used for -2 and an external lib is used for jpg rendering in -3.

comment:6 Changed 17 months ago by darcio

OK, so you are definitely seeing something different than I. Alright, sounds to me like something else on my machine is impacting this.

I ran through the dynamic DLL load of 'DLLTree', generated a report for a local PDF load as well as the NAS based load. There are some differences and to me they seem kind of weird, meaning: MMIO stuff is pulled in for the NAS read.

Take a look please at the load report, if you do a compare on these you'll find DLLs like: gifproc.dll, mmparts.dll, mmptmri.dll, os2oor3u.dll and others. Many come from the \mmos2\dll tree, others directly from the \os2\dll tree.

I have no idea why these would be invoked on a NAS load. In addition to this test I confirmed that I am seeing this slowness across multiple network storage locations (trying to exclude just my NAS box as the culprit) such as Win10 and Win7 shares.

One more comment re: actual speed. In the case of NAS PDF read I am seeing rates of 100-750 Kbytes, whereas something like GhostView? will pull a file down at 2-5 Mbytes, this is consistent across all LAN devices. So on average I'm seeing a 10x difference in performance, which should not be the case.

Changed 17 months ago by darcio

Attachment: lucide_dll_list added

dynamic DLL load for local storage PDF file

Changed 17 months ago by darcio

Attachment: lucide_dll_list_NAS added

dynamic DLL load for NAS storage PDF file

comment:7 Changed 17 months ago by darcio

Any ideas what other debugging approaches I could try to further troubleshoot this issue?

Given the loaded DLLs list I thought that maybe OpenDoc? was getting pulled in somehow (see stuff like OS2OM30.DLL and OS2OOOC.DLL which are part of mmparts.dll file). But I checked my system and the old OpenDoc? install I had on here (part of the VisualAge? install I think?) is no longer present.

comment:8 Changed 17 months ago by Gregg Young

A couple of things you can try. Update to libcn0.dll. The other thing would be to try with all the libs coming from the 686 repo instead of Pentium 4. I will diff the 2 dll lists and see if any of it seems relevant.

I will send you the first 1.5.1 build if you like. However, it requires libcn0.dll and libcx 0.6.5-1

Couple more questions. You said jpgs load quickly. Is that using Lucide? This issue is with files loading in Lucide not the populating of the file dialog correct? Does Lucide try to make thumbnails on loading? If so have you tried turning thumbnails off?

comment:9 Changed 16 months ago by darcio

@gyoung I have libcn0.dll already installed, specifically I have the following:
libc.i686 1:0.1.1-1.oc00
libc-devel.i686 1:0.1.1-1.oc00
libcurl.pentium4 7.37.0-2.oc00
libcx.pentium4 0.6.5-1.oc00
libcx-devel.pentium4 0.6.5-1.oc00

I do not have thumbnails enabled, I disabled when I was attempting to debug the slow-load issue.

The JPEG view through Lucide is FAST...so typically I see a 5-6 Meg throughput as opposed to the 400k with PDFs. It matches the rates I'm seeing with using PMView directly.

Last edited 16 months ago by darcio (previous) (diff)

comment:10 Changed 16 months ago by darcio

...ahhh, sorry missed a couple more questions:

1) load time - only applicable to actual loading of PDFs, I am not seeing this with File Dialog population
2) 1.5.1 - sure thing, fire away, happy to test, the pre-req DLLs are already on my machine I believe

comment:11 Changed 6 months ago by Lewis Rosenthal

Sorry for the delay in following up...

Is this still an issue?

comment:12 Changed 4 months ago by darcio

@lewis Yeah, this is precisely the issue I am trying to troubleshoot today (ref. the AOS Testers email post on the subject of NetDrive? Samba plug-in source).

I am reluctant to open a new ticket as you had asked b/c this appears to be some sort of PDF library (poppler maybe) issue as opposed to a pure Samba plugin/client problem.

Currently I am re-testing all of it here, so far I can re-confirm the following:

1) slow reads with Lucide and QPDFView, 20-35Kbyte

2) faster reads with GSView, 150-250Kbyte

3) smbclient 'get' command shows great throughput

smb: \> get Masonite_Fiberglass_Steel-2010.pdf

getting file \Masonite_Fiberglass_Steel-2010.pdf of size 31804971 as Masonite_Fiberglass_Steel-2010.pdf (102506.7 KiloBytes?/sec) (average 89573.3 KiloBytes?/sec)

4) file manager copy (Larsen Commander) shows a steady 30-45Mbyte throughput across numerous files (tested with multiple files not just PDFs to rule out any sort of weirdness going on).

So in the end I'm inclined to say not a Samba client/NetDrive plug-in problem perhaps, rather some weird library cross interaction maybe?

comment:13 Changed 4 months ago by Lewis Rosenthal

Summary: 1_41_RC4 is incredibly slow reading PDFs over networkLucide is incredibly slow reading PDFs over network (Samba share)

(Updated ticket summary to remove reference to 1.41 RC4, as this seems more systemic.)

I completely forgot that this was the issue at hand, Dariusz. Thanks for staying on it.

Just for the sake of completeness:

  1. Is EA support enabled on the client side?
  2. Is EA support enabled on the server side?
  3. What is on the server side (OS, filesystem)?
  4. Samba dialect?
  5. R/W or RO share?
  6. Is Lucide configured for thumbnail creation?
  7. If thumbnails already exist, are they rendered slowly? (I would guess)
  8. If thumbnails have been turned off, does this make a difference?

Please confirm that this (currently) is with Lucide 1.5.0 GA, as delivered with ArcaOS 5.0.5 GA, using lupplr.dll also 1.5.0 GA (bldlevel for each is your friend).

Here's what I can confirm:

Test document contains graphics and has an image on the front page. It is a PDF v1.5, created from MS Word 2010, is 978K and has a long filename, with no spaces, but with several dots (SLES11SP3Supplement-SPP2013.02.0ReleaseNotes.pdf).

SMB3 / openSUSE 13.2 x64 host / Samba 4.4 / encryption:

  • with or without thumbnails, with or without EAs, loading is slow
  • when mounted RO, loading is somewhat better (progress meter goes away sooner, though it still takes some time to render the first page of my test document)

SMB1 / NetWare 6.5 SP8 / NetWare CIFS / no encryption

  • with or without thumbnails, without EAs (unsupported in NetWare CIFS), loading is slow
  • when mounted RO, loading is somewhat better (same as above)

NetWare Requester / NetWare 6.5 SP8 / NWIFS (same server and volume as above)

  • with or without thumbnails, with EAs, loading is very fast
  • can't test RO mapping but I can flag the file RO; no real difference

The last two tests are somewhat telling, as these were two different protocols used for the same server (granted, the NetWare requester uses IPX and not TCP).

In contrast, opening the same 3MB BMP (Samba vs NWIFS) was nearly identical between the two. This does narrow the suspect list to poppler, IMO.

comment:14 Changed 4 months ago by darcio

Here are the details you asked for:

1) Is EA support enabled on the client side?
Yes, EA is enabled on the client side, local IFS is JFS

2) Is EA support enabled on the server side?
No, EA is not enabled on the remote share

3) What is on the server side (OS, filesystem)?
Server side OS is 'Linux 2.6.31.8 armv5tel', FS is ext4

4) Samba dialect?
3.5.6

5) R/W or RO share?
By default it is R/W, tried both to see impact, no change.

6) Is Lucide configured for thumbnail creation?
No, tried both to rule one or the other out, no change.

7) If thumbnails already exist, are they rendered slowly? (I would guess)
Yes, the rendering speed appears to be predicated by the overall PDF read speed, which seems to make sense.

8) If thumbnails have been turned off, does this make a difference?
No, makes no difference, or at least the PDF read speed is so slow that if there is a difference one can not tell.

Yes, I am using Lucide 1.5.0 GA:
Vendor: Arca Noae
Revision: 1.05.0_GA
Date/Time?: 2019-03-20 01:50:00
Build Machine: ZOBOPEEP
Language Code: EN
Country Code: US
File Version: 1.5

...and using lupplr.dll also 1.5.0 GA:
Vendor: Arca Noae
Revision: 1.05.0_GA
Date/Time?: 2019-03-20 01:50:00
Build Machine: ZOBOPEEP
Language Code: EN
Country Code: US
File Version: 1.5

comment:15 Changed 4 months ago by Gregg Young

Based om the information provided I would conclude this is a poppler problem. There is a new poppler and updated qpdflib package in netlabs-exp. It is not for the faint of heart. I tried installing it and got a fork trap in one of the cups packages that cascaded into a trap 0008. If one of you can get it installed let me know if the slow transfers still exist for qpdfview. Just so you know the Linux boys in their relentless drive to stamp out backward compatibility have changed the interface such that upgrading would require a major rewrite of Lucide. However if this issue is fixed my interest in looking at updating will at least no longer be 0.

comment:16 Changed 4 months ago by darcio

Guys!

As I troubleshooted this I did eventually grab the exp release of poppler, specifically I now have the following installed here:

poppler-legacy-65-0.49.0-2.oc00.L.pentium4
poppler-legacy-72-0.61.1-1.oc00.L.pentium4
poppler-legacy-70-0.59.0-3.oc00.L.pentium4
poppler-qt-0.84.0-1.oc00.pentium4
poppler-0.84.0-1.oc00.pentium4

The legacy-65 can be ignored, I only tossed that into the mix when I attempted to go back to Lucide 1.35 (since that was the last known release with good LAN performance). Problem there is that having installed that release I now get a trap (SIGTERM I think) in one of the STD libs, so I'm thinking given it's age it's related to the updated gcc stuff as the SIG was thrown in libstdc library if I recall (anyways, that's a separate issue though since I went back to the only viable option, that being 1.50GA).

Gregg, I do not know if that's the version of poppler you are thinking of, it if is, I'm afraid it's the same story as the previous one, that was 0.59.0-3 (oc00) here.

Let me clarify something though, I do not have qpdf-libs installed now. In fact my qpdfview is a separate app (non-rpm) because the RPM has CUPS as a pre-req and I haven't installed that yet since my Postscript drivers are working out pretty well for the Brother HL5470DW printer. Anyways, the stand-alone qpdfview does not appear to require that qpdf-libs stuff. Or it if does, most likely they are actually part of the application install itself, they DLLs I have here are:

7-12-18 6:20a 29,325 346 a--- qpdfdjvu.dll
7-12-18 6:20a 17,372 345 a--- qpdfimg.dll
7-12-18 6:20a 73,209 345 a--- qpdfpdf.dll
7-12-18 6:20a 18,047 344 a--- qpdfps.dll

One last point from my testing here. The PDF view performance between my NAS and Win7 based shares is the same, so I think it is another pointer towards the poppler issue as the underlying OS fs, share server settings, etc. of course are different between the NAS box (Linux) and my Win7Pro box.

I have specific log.ndpsmb entries that show the size of the xfer window where I literally see 256 byte chunks for the PDF reads while things like JPEG, MP4, etc. literally go in 32K or 64K chunks. I haven't quite figured out why these themselves differ given my settings in smb.conf, but again that's a separate tunning concern.

Note: See TracTickets for help on using tickets.