Perfomance issue - ne_request_dispatch()

[andib]

This ticket is to do discuss performace issues with ndpdav. Here my findings hope others share there experience too.

Copying to a local webdav share on another OS/2 box is rather slow. Transferring a 40MB file needs between 30 seconds and 3 minutes. No matter if I use cmd.exe, 4os2 or LarsenCommander. Using LarsenCommander you can see very well that some parts of the file are transferred very fast, other parts are transferred very slow. I see the same behavior with cmd.exe in the pmprintf log. Copying to netlabs.org is even slower but that may partly depend on my crappy ADSL connection.

Strange thing is part of transfer is very fast, other part is very slow. When it is slow then ndpdav needs about 2 seconds for ne_request_dispatch(req) to complete. Sometimes ne_request_dispatch(req) returns immediately, sometimes it needs 2s. Here part of pmprintf log from special ndpdav version with additional logging messages

10:37:37  E:/_work/ndpdav/trunk/source/ne_nd.cpp: start 0x2BDAAC end 0x2CCAAB
10:37:37  request buffer sized 0xF000
10:37:37  request buffer sized 0xF000
10:37:37  ne_set_request_body_buffer done
10:37:39  ne_request_dispatch done
10:37:39  ne_request_destroy done
10:37:39  increase file ptr (2874028) with 61440

The relevant source code in ne_nd.cpp -

    ne_print_request_header(req, "Content-Range",
                            "bytes %" NE_FMT_OFF_T "-%" NE_FMT_OFF_T "/*",
                            range->start, range->end);

		LOG((ND_LL_DEBUG, "%s: start 0x%0llX end 0x%0llX\r\n", __FILE__, range->start, range->end));
    // ticket:21 adjust size for 0-len writes
    size_t size = range->end - range->start + 1;
		LOG((ND_LL_DEBUG, "request buffer sized 0x%0X\r\n", size));
//    if (range->end == range->start)
//        size = 0;
		LOG((ND_LL_DEBUG, "request buffer sized 0x%0X\r\n", size));
    ne_set_request_body_buffer(req, (const char *)pbuffer, size);
		LOG((ND_LL_DEBUG, "ne_set_request_body_buffer done\r\n", size));

    ret = ne_request_dispatch(req);
 		LOG((ND_LL_DEBUG, "ne_request_dispatch done\r\n", size));

    status = ne_get_status(req);

What does ne_request_dispatch(req) do and why does it take 2s sometimes while other times it returns immediately?

Using current ndpdav svn code with some additional logging messages, Neon library 0.30.0, OpenSSL 1.0.0.n, EXPAT 2.1.0.

[Yuri Dario]

The slowness is due to the two-step request API: first you need to authorize API call, then you can perform real call. This makes global I/O slow when you have to deal with parsing directories or file data. When the real transfer starts, things are better because the plugin tries to use a big buffer to reduce API overhead. But I/O from dropbox site is not so fast, never seen more than 150/200 KB/s.