ndpdav.dll from 1.2.0 GA package destroys large files during upload

[andib]

Although upload seems to work ok the file is corrupted afterwards.

{0}[t:\download] comp t60_test.zip e:t60_test.zip

Compare file T:t60_test.zip and file e:t60_test.zip.

A COMPARE error occurred at OFFSET 1B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 75

A COMPARE error occurred at OFFSET 2B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 61

A COMPARE error occurred at OFFSET 3B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 8

A COMPARE error occurred at OFFSET 4B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 58

A COMPARE error occurred at OFFSET 5B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 11

A COMPARE error occurred at OFFSET 6B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 13

A COMPARE error occurred at OFFSET 7B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 9D

A COMPARE error occurred at OFFSET 8B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = B2

A COMPARE error occurred at OFFSET 9B3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = B1

A COMPARE error occurred at OFFSET AB3FF
Mismatching byte of file 1 = 0

Mismatching byte of file 2 = 9B

There were 10 or more mismatches in comparing
the files.  The system is ending the COMPARE command.

Do you want to compare more files (Y/N)? n

The ndpdav.dll which comes with 1_2_0-GA (both .wpi and .zip) has bldlevel -

{0}[p:\util\ndfs\ndplugs] bldlevel ndpdav.dll.nok.1.2.0
Build Level Display Facility Version 6.12.675 Sep 25 2001
(C) Copyright IBM Corporation 1993-2001
Signature:       @#Vitali:1.0.0.0#@NetDrive for OS/2 DAV plugin, based on NEON library http://www.w
bdav.org/neon
Vendor:          Vitali
Revision:        1.00.0.0
File Version:    1.0
Description:     NetDrive for OS/2 DAV plugin, based on NEON library http://www.webdav.org/neon

Changing to older one works. bldlevel from working dll -

{0}[p:\util\ndfs\ndplugs] bldlevel ndpdav.dll.ok
Build Level Display Facility Version 6.12.675 Sep 25 2001
(C) Copyright IBM Corporation 1993-2001
Signature:       @#Netlabs.org:1.1.1#@##1## 16 Jul 2009 13:28:38     paperino::en::::@@NDPDAV - DAV
NetDrive External Plugin Build GA
Vendor:          Netlabs.org
Revision:        1.01.1.1
Date/Time:       16 Jul 2009 13:28:38
Build Machine:   paperino
Language Code:   en
File Version:    1.1
Description:     NDPDAV - DAV NetDrive External Plugin Build GA

It does not matter if I use the openssl/expat/pthread/neon dlls from ndpdav package or the ones which came with rpm. Btw. a bad habit to distribute the same dlls with different dates (rpm/ndpdav)

Another bad habit is to name a newer file as revision 1.00.0.0 where a five year older one exist with revision 1.01.1.1. Or is there simple a wrong dll packed with the 1_2_0-GA files?

[diver]

r77 fixed the signature part.

for the rest I have completely no idea what it could be, as I use it also and I never got any problems with it so far.

[andib]

Unfortunately no difference with 1.2.1. No matter if using dlls from rpm nor the included one (which should be the same anyway).

The problem starts at 0x1B3FF and repeats every 0x10000 until 0x5B3FF. Then after another 0x1E000 bytes it repeats two times every 0x10000... I can only speculate that there is a problem in one of the crypto/ssl/... dlls in combination with my internet connection. Which does not show in any other application and is the same as I use since years. Between my workstations there is a router build of a T40 with latest Injoy firewall (integrated Intel NIC for LAN and PCMCIA 3com NIC for WAN) and a ST585 ADSL router. No clue what that all can have to do with this problem.

But, as an older npdav.dll from Yuri works it seems there is a problem in the neon/ssl/crypto/... dlls. Yuris build seems to included these dlls (static linked?) whereas with newer builds the rpm ones are used.

Now I tracked the problem down to be a difference between 1.1.1 and 1.1.2. Yuris

{0}[p:\util\ndfs\ndplugs] bldlevel ndpdav.dll
Build Level Display Facility Version 6.12.675 Sep 25 2001
(C) Copyright IBM Corporation 1993-2001
Signature:       @#Netlabs.org:1.1.2#@##1## 4 Sep 2009 10:53:59      paperino::en
::::@@NDPDAV - DAV NetDrive External Plugin Build GA
Vendor:          Netlabs.org
Revision:        1.01.1.2
Date/Time:       4 Sep 2009 10:53:59
Build Machine:   paperino
Language Code:   en
File Version:    1.1
Description:     NDPDAV - DAV NetDrive External Plugin Build GA

does NOT work here whereas the following one works -

{0}[p:\util\ndfs\ndplugs] bldlevel ndpdav.dll
Build Level Display Facility Version 6.12.675 Sep 25 2001
(C) Copyright IBM Corporation 1993-2001
Signature:       @#Netlabs.org:1.1.1#@##1## 16 Jul 2009 13:28:38     paperino::en
::::@@NDPDAV - DAV NetDrive External Plugin Build GA
Vendor:          Netlabs.org
Revision:        1.01.1.1
Date/Time:       16 Jul 2009 13:28:38
Build Machine:   paperino
Language Code:   en
File Version:    1.1
Description:     NDPDAV - DAV NetDrive External Plugin Build GA

I think the failing one is the first version where the password entry in ndfs was changed from clear text to *. Btw. I'm using Netdrive3.1.1 with File System Driver 3.026 (11.08.08 23.04 35.232 28 _ nd.exe)

[diver]

So you have to stick with old ones.i can't reproduce here and I have neither time nor will to invest in such a complete weird problem.

[Yuri Dario]

All rpm packages on netlabs are uploaded using plugin 1.2.0, no corruption for me.

[andib]

Just made another test with a fresh eCS2.2beta2 installation on drive Y:. Choice mostly default values. Even put all the rpm stuff at Y:. So this time really all is new. Made the yum installs necessary for ndpdav plugin and a few others. Put the \usr directories in path and libpath in front of all others (as usual).

Uploaded 2.4MByte file and downloaded. Same problem as described above. Only difference, first zeroed out byte is at 1C3FF instead 1B3FF.

I can not think of any problem with my installation as all is new.

[andib]

I made one interesting observation now. Although I've tried in the past without success I tested again copy from command line. And now it did work. At least 2 times. Have to test more.

How do you guys copy to dav shares? Command line? I usually take LarsenCommander? for a lot of copying without problems - ftp, netbios, samba all work okay. Tried FM3 but this tooks ages on my F: drive with ftp, samba and dav shares. Seems FM3 tries to read some info even if the Netdrive connected server (mostly ftp here) does not respond. So FM3 is hardly usable for me.

Now my guess is Lcmd sends data in bigger chunks while 4os2/cmd must do it a bit different. First chuck something like 0x1B400 and following 0x10000 pieces. Maybe this triggers the problem. But that's a guess only. As Lcmd does have no problem with any other share there must be something different with ndpdav or netdrive or....

[andib]

The problem seems to be ndpdav (or netdrive?) can not write smootly 64k blocks. Standard copy with 4os2 sends data over dav connection in 61440 (0xF000) byte chucks. LarsenCommander? seems to want transfer 64k blocks. But seems this is not possible. NdpFileWrite? alternating writes 65535 bytes (1 byte less than 64k) and 1 byte. This looks strange and maybe the reason why transfer speed is not that good too.

Although there is no indication for a write error in the log this perfectly matches my observation where 1 byte is wrong every 64k block.

Needs more investigation especially if this is a netdrive or ndpdav limitation.

[Yuri Dario]

does 4os2 copy work correctly?

[andib]

I now think the problem in in netdrive. Netdrive is not able to transfere 64kByte (65536 bytes) blocks correctly. Waiting for Vitali to comment on this.

Another explanation for this problem maybe dav can not handle 1 byte transfers. Netdrive splits 64k blocks in 2 consecutive blocks (one is 65535, the other one is 1 byte). It may be possible but I do not think that's the case here.

Reason why problem is only visible with LarsenCommander? is cause Lcmd has implemented a copy strategy with dynamic buffer sizes adoption to match what's best for current channel. For this Lcmd dynamically increases block sizes until an internal limit. For small files block size will not go beyond 64k - 1 so it does not trigger the netdrive problem. But with larger files block sizes can go bigger depending on actual transfer speed. This is the reason why the problem is that hard to reproduce.

Every other file system / transfer channel I tested including usb does not have a problem even with 256MB block size. But netdrive does.

Standard cmd.exe and 4os2.exe seems to limit block size to 0xF000. Is there any desing limit in OS/2 which prohibit larger blocks? If yes, then Lcmd did violate that. If no, then netdrive is buggy.

As far as I can see this is not a problem of the ndpdav plugin but in base netdrive code we unfortunately do not have.

[andib]

Debugging further I've found the real problem in ndpdav plugin now. In ne_nd.cpp the code block

    // ticket:21 adjust size for 0-len writes
    size_t size = range->end - range->start + 1;
    if (range->end == range->start)
        size = 0;

sets size to 0 for 1 byte blocks and so do request a zero length buffer afterwards although it would need 1 byte.

If I comment out the 'if' block netdrive/ndpdav plugin works correct with big buffers too.

Yuri introduced this problem with http://trac.netlabs.org/ndpdav/changeset/61. I think we can savely revert this piece of code back to orginal (delete above lines) and problem would be solved. Yuri can you comment on this?

[andib]

Tested again with standard 4os2 copy - without my changes (commenting out the if statement above) a file will be corrupted if it is 0xF001 in size. 4os2 copy uses max. 0xF000 buffer size and so after that request 1 byte buffer to be copied. This fails as 1 byte transfers are not possible with current code.

Every 4os2 copy command of files sized (n x 0xF000) + 1 fails.

I tested standard os/2 copy command too. It uses 0xFE00 as max. buffer size instead. So with standard copy command every file sized (n x 0xFE00) + 1 fails.

It depends on copy program which is used which file sizes exacly fails but there are sizes which fails with every of them. Most probably it wasn't seen so often cause 1) the file size must be very special and 2) no other program than (old) LCMD I know off uses 0x10000 buffer size 3) most times it does not matter if the very last character of a file is zeroed out. Nonethless this is a bug in ndpdav.

Btw. I tested with my code 1 byte sized file transfer too and it works fine. Unfortunately Yuri did not respond to this ticket nor to email. So I'm giving up trying to help improving ndpdav for now and keep using my own fixed version from now.

[diver]

this issue is fixed since long in Netdrive. since version 3.1.5 it's fixed.