git: Support USE_CURL_MULTI

[dmik]

After fixing #19 I found out that our git is very slow when cloning over HTTPS with USE_CURL_MULTI defined. This define is automatically enabled for all recent versions of CURL and causes git to use several HTTP streams for talking to the server (which in theory should improve the overall performance).

While working on #19 I disabled USE_CURL_MULTI to simplify debugging HTTP transfers. However, now, when I enabled it back, I noticed that the clone speed significantly decreased. For instance, cloning ​https://github.com/psmedley/gcc with *no* USE_CURL_MULTI gives us around 200 KiB/s second. When USE_CURL_MULTI is enabled, the speed drops to 25 KiB/s (i.e. almost 10 times slower). This is not acceptable.

I'm not 100% sure if it's a git or curl problem. At least, when I try to use CURL.EXE to directly download a ZIP from github it is as fast as it should be (around 200 KiB/s) but I'm not sure if CURL.EXE uses multiple streams internally or not (and there are no command line options to control this).

Note that this applies to git 2.0.0 (built from the dmik branch ATM).

[dmik]

As we can't invest too much time in it right now, one option is to force disable USE_CURL_MULTI on OS/2 but some parts of the code show that it may limit some functionality. I.e. git push over HTTP may stop working. I need to check this. Also I have a couple of OS/2-specific places related to USE_CURL_MULTI that may affect its work which I'm going to check now too.

[dmik]

BTW, not only git is slow with USE_CURL_MULTI, it also refuses to work with some sites at all. E.g. it is hanging forever while trying to clone something from bitbucket.org.

[dmik]

The command that doesn't work when USE_CURL_MULTI is not set is git-http-push.exe (the source is in http-push.c and there is a check that aborts the command if USE_CURL_MULTI is not defined). However, I couldn't get git to a situation where it would use this command. When I do git push in a HTTPS repository, all works. I even created a test repo and successfully pushed a commit there.

The http-push docs mention this, however:

*NOTE*: This command is temporarily disabled if your libcurl
is older than 7.16, as the combination has been reported
not to work and sometimes corrupts repository.

Our curl version is 7.37 so it shouldn't be disabled but it seems it still is and the git-remote-http command seems to be used for doing the http push job instead. At least git doesn't start anything else besides git-remote-http when I ask it to push.

I will omit this problem for now assuming that all works until we face a real case when git-http-push.exe (which certainly won't work with USE_CURL_MULTI due to an explicit check in http-push.c) is used.

[dmik]

Okay, I found the reason why it doesn't work. In r797 I replaced select() with nanosleep blindly following this bauxite's change to git 1.7.3.2: http://trac.netlabs.org/ports/browser/git/branches/komh/http.c?rev=609#L660. However, in recent git versions the MULTI mode was significantly improved and this code got an important change: now select is used there not only as a dummy sleep method as before but they also monitor some sockets while waiting. Replacing this select with nanosleep made it not react on socket state changes which in turn slowed the speed down in some cases (github.com) and completely broke the connection in other cases (bitbucket.org).

I reverted the wrong commit in r847. Now all seems to be as fast as it should be. Closing this defect.