git: clone of big repo crashes

[dmik]

When trying to clone ​https://github.com/psmedley/gcc with the 2.0 branch after applying r861 and r862, I get the following crash:

SYS1808: out files:   4% (3240/76483)
The process has stopped.  The software diagnostic
code (exception code) is  0005.

It happens after successfully fetching all the repo data (and building the index) during checking out a working copy from the index.

[dmik]

I will try to build r860 to see if it solves the problem.

[dmik]

These are the entries from POPUPLOG.OS2 btw:

09-09-2014  02:30:19  SYS3171  PID 9703  TID 0001  Slot 00b8
D:\CODING\GIT\BRANCH-2.0-RUN\BIN\GIT.EXE
0aa40d8b
02fe8189
P1=ec8b5500  P2=011cec81  P3=8b530000  P4=8b571445
EAX=002db6c0  EBX=00000004  ECX=08f19d3d  EDX=00000000
ESI=000ceadf  EDI=08f19d55
DS=0000  DSACC=****  DSLIM=********
ES=0000  ESACC=****  ESLIM=********
FS=0000  FSACC=****  FSLIM=********
GS=0000  GSACC=****  GSLIM=********
CS:EIP=b690:0000ffff  CSACC=****  CSLIM=********
SS:ESP=0000:2003013c  SSACC=****  SSLIM=********
EBP=002db6f8  FLG=00010212

------------------------------------------------------------

09-09-2014  04:59:43  SYS3171  PID adfe  TID 0001  Slot 00bb
D:\CODING\GIT\BRANCH-2.0-RUN\BIN\GIT.EXE
c0000005
ffffffff
P1=00000000  P2=ffffffff  P3=XXXXXXXX  P4=XXXXXXXX
EAX=002db6c0  EBX=00000004  ECX=0a2ba585  EDX=00000000
ESI=000ceb2f  EDI=0a2ba5f0
DS=0000  DSACC=****  DSLIM=********
ES=0000  ESACC=****  ESLIM=********
FS=0000  FSACC=****  FSLIM=********
GS=0000  GSACC=****  GSLIM=********
CS:EIP=0000:0000001c  CSACC=****  CSLIM=********
SS:ESP=0000:2003013c  SSACC=****  SSLIM=********
EBP=002db6f8  FLG=00010212

Looks like memory corruption. It can be also related to our new mmap code.

[dmik]

Disabling memory mapped files fixes the checkout. So it looks like mmap is the guilty person. Note that all earlier git releases didn't use mmap at all — there is a fallback option that uses regular memory buffers limited to 1M for file operations (with mmap enabled this buffer is as big as 32M). Adding mmap to git required to implement mmap offset support (see r816 an also relevant WPSTK changes) however it doesn't mean that this new code is guilty (as our mmap was simply never used in this scenario before) — there may be some other flaw in WPSTK which makes it not work as expected by git. Anyway, this requires more investigation (in particular, exceptq trap report generation and analysis) which is beyond our current time frame.

For now, I simply disabled mmap usage in r863. We will come to this problem later.

[Silvan Scherrer]

this might be worth to try with latest libcx again

[dmik]

With the latest LIBCx it doesn't fully work either since LIBCx doesn't support overlapped mappings which git uses, see ​https://github.com/bitwiseworks/libcx/issues/29. Once it's fixed, git should be rechecked against it. For now, mmap usage is still disabled in git.

However, it turns out that the original problem with huge repos is not mmap but the stack size (1Mb by default). Increasing it to 8Mb in r1883 completely fixed the problem with cloning the gcc repo here (git 2.11.0). So I'm closing this issue.