Splitting large files to 4 GB pieces

[Valery V. Sedletski]

It is suggested a new feature, to support files larger than 4 GB, on fat32. Because the file size in FAT is limited to 32 bits, the size is limited to 4 GB (2 GB if signed. The file offset is signed 32-bit integer, so, it is limited to +-2 GB). So, the file size cannot be >= 4GB, by design. But we see a nice feature of archivers to split files to some size, or VirtualPC can split a *.vhd file to multiple 2 GB pieces. So, why not add a feature of splitting files to 2 or 4 GB pieces into the fat32 FS driver itself? All pieces could be listed in a special *.dsc file, a simple text file with a filename per line. So that, the IFS will hide all pieces, but show a full concatenated file instead, plus the file with the same name and a ".dsc" suffix added. If you delete the *.dsc file, the file becomes a set of separate pieces again, and if you recreate it, the file is shown concatenated. The *.dsc file can be easily recreated by a simple command "dir somefile.* >somefile.dsc". Also, if you copying a large file to a FAT32 partition, it gets automatically splitted to a set of 4 GB pieces with default names, plus a *.dsc file created. But the file names are not limited to default ones. So, you can rename them as you wish, and change the *.dsc file accordingly.

What do you think? I would be very happy with such feature, having possibility to copy the VM files and big movies to my 1 TB USB FAT32 hard disk, which I use for backup purposes. Also, I'll be able to use these files directly from this disk, without the need to copy them elsewhere. And, if I carried this harddisk to users of other OS'es, I can simply copy pieces to a 64-bit file system, like NTFS, and concatenate them with a simple command, like copy /b ...

[Valery V. Sedletski]

Of course, this should be enabled if a /split command line switch is specified for fat32.ifs, and disabled by default.

[erdmann]

Is the FAT32 filesystem itself is limited to 32-bits or is this just the limited implementation under OS/2 ?

[Valery V. Sedletski]

FAT is itself a 32-bit FS (excluding exFAT). So, in FAT12/FAT16/FAT32, file size field is limited to a 32-bit number. So, max file size is 4 GB. But 16-bit IFS-es, like FAT32, HPFS, CDFS are even limited to 2 GB, because file pointer is 32-bit signed. But currently, fat32.ifs supports 64-bit file pointers. Files of size up to 4 GB can be copied to/from fat32. But FS_CHGFILEPTRL is not called. FS_CHGFILEPTR is called for some reason, and pointer is truncated to 32 bits (signed). So, random seek is not working atm for big files. If we'll try breaking files to 4 GB pieces, 64-bit seek will not work either, but carrying big files on FAT32 disk would be possible.

[Valery V. Sedletski]

I decided to use FAT+ (​http://www.fdos.org/kernel/fatplus.txt) spec instead. It uses some reserved bits of a directory entry for storing additional file size bits. These bits are in the "EA mark byte", which used for fat32.ifs for indicating that the file has EA's, or it has critical EA's. These are 6th and 7th bits. Also, bits 3-4 are reserved by WinNT for indicating that the file has lowercase name and/or extension. So, 4 bits are free. If file size is not more than 32 GB, 0-2nd bits are used for storing the upper three bits of file size. If file size is bigger, then a special FAT_PLUS_FSZ EA is using for storing the 64-bit file size. And the presence of this EA is marked by the 5th bit of the "EA mark byte".

The FAT+ support is enabled with the /plus command line switch for fat32.ifs. FAT+ is enabled in fat32.ifs and CHKDSK since r303. Please test.