Dirty flag is not respected when mounting FAT32 media

[Lewis Rosenthal]

When the dirty flag is set - for whatever reason - media should not be mounted and available for writing until checked. This is expected behavior on OS/2.

While autocheck may work for always-connected-at-boot-time media, it does not work for removable media.

Currently, USB media inserted after boot time with the dirty flag set is simply mounted. This may lead to further data corruption unless chkdsk is performed against the volume.

[Valery V. Sedletski]

2lewisr: This is known. There are (and have been, since older versions) some problems with dirty flag setting. Also, #27 already exists -- why create a new ticket?

Media is mounted readonly if not checked. This behaviour has always been with fat32. So that, data cannot be corrupted. And, that behaviour is undesirable to be changed this time, because CHKDSK on fat32 is not fully functional. It can be changed later, when someone will enhance CHKDSK, but it is beyond current tasks. So, just do "f32stat d: /clean" for now, to clean the dirty flag (or do a CHKDSK/scandisk on windows).

Autocheck cannot work with removable media as it is mounted at a later stage, than FS_INIT, and a drive letter does not exist that time (so, it is impossible to check removables on FS_INIT).

[Lewis Rosenthal]

Also, #27 already exists -- why create a new ticket?

#27 specifically deals with not clearing the dirty flag when checking. This ticket relates to ignoring the dirty flag when mounting, that's all. I'm not familiar enough with this code to know what may be the same cause/solution.

Running f32stat d: /clean is a little beyond the scope when the user has no idea that the dirty flag has been set. How would one know unless looking with a utility like DFSee, in which case, the flag can be cleared there, but again, not knowing why it was set in the first place could lead to unpredictable results. The best scenario is chkdsk, but again, unless the flag is noticed and acted upon appropriately, one would not know to do this.

100% agreed about autocheck, which was the reason for my mentioning it in the ticket summary. This issue specifically deals with hot-plugged media.

Thanks!

[Valery V. Sedletski]

The user will see the dirty flag is set immediately, as it notices that the disk is mounted read only. No need to use dfsee or whatever. Yes, I agree that it's better to not mount the media if dirty bit is not set. But that requires CHKDISK to fix all errors present. But CHKDSK is not functional enough to fix all errors, and requirement to fix all errors before it can be mounted means that we'll need to boot windows to fix that, which is not always acceptable. Moreover, on a bootable FAT32 removable media (and even, on HPFS/JFS media for my current boot with locatecd.sys), it is impossible to clear the dirty flag automatically on boot (because locatecd.sys starts after fat32.ifs init). Also, on my 128 GB fat32 flash or my 1 TB fat32 harddisk, it will take hours to be checked (fat32 is not journalled file system. Even with 64 GB HPFS drive, CHKDSK takes half an hour to be checked).

[Lewis Rosenthal]

The user will see the dirty flag is set immediately, as it notices that the disk is mounted read only.

Hmmm... AFAICT, the user will only see a failure when attempting to write, and nothing indicating the reason for the failure is that the disk has been mounted read-only due to the dirty flag being set. Now, you and I may be able to infer what is likely happening, but I can guaranty you that my brother - who uses OS/2 every day - will not have a clue. Instead, he will contact me as the sysadmin, and tell me that he received a USB stick from a client, he was able to mount it, read from it, but could not save data back to it. He will not know that the dirty flag is set, only that he cannot write to what he would assume is writable media.

I understand your point about CHKDSK not fixing all errors, but this is always the case (CHKDSK can't be expected to fix all errors - under any OS), and surely, the answer is not to go hunting for a Windows machine (I don't have a physical Windows box handy, myself, only VMs running under Linux).

If I insert an HPFS-formatted stick with the dirty flag set, it is not mounted for reading or writing until CHKDSK is run against it. That's the behavior I would expect.

Finally, the amount of time it takes is a separate issue entirely, and does not negate the fact that dirty media should not be trusted, and clearing the flag simply to gain write access is not a sound approach to dealing with media with the dirty flag set. Whether it takes 30 seconds or 30 hours, if a CHKDSK pass is needed, it is needed.

If we need to hold this off until the next release, I understand, and we can keep this open and move on. However, I really think that this is important for any filesystem driver. If we need to have someone else look at CHKDSK, I understand that, as well. I saw no ticket documenting the fact that the dirty flag was not being handled as expected, so I opened this ticket, that's all. I don't mean to sound difficult, so please don't take my comments as such.

[Valery V. Sedletski]

2lewisr: So, what can you suggest instead? Make CHKDSK clear the dirty flag even if it cannot fix errors? Then what are the advantages over just doing "f32stat d: /clean"? The latter is just easier. If I need to run CHKDSK explicitly, I'll do it. Theoretically, I can agree with you, but practically...

PS: It may be a good idea to change the behaviour of fat32.ifs when dirty bit is set, by a command line switch: a) Mount it readonly b) do not mount until the errors are fixed. The 1st variant is more preferrable if we are booting from a fat32 USB disk (otherwise it will fail booting, because autocheck is not working with locatecd.sys).

The problem is that HPFS or JFS cannot work if the disk is readonly, but FAT32 supports that. So, AFAIK, HPFS will just trap if disk is readonly. It just doesn't expect this.

[Lewis Rosenthal]

After conferring with colleagues, we would suggest the following behavior:

1. If a volume is clean, mount it R/W.
2. If a volume is dirty, do not mount it until chkdsk has been run. Even a minimally functional chkdsk may indeed "clean up" 99% of the possible corruption.
3. If we can determine that the volume is readable without errors (time is not a factor; safety is), then clear the dirty flag.
4. If we cannot determine the status of the volume even after a chkdsk pass, leave the dirty bit set and do not mount it.

This approach would align with your option "b," above.

In no event would I suggest simply clearing the dirty flag without at least attempting to correct what errors may be correctable.

The f32stat utility is still available for those who don't want to wait(!) for a CHKDSK pass (I'm trying to weigh the potential amount of data on a large volume and the desire to protect that against even a 2-hour chkdsk in the event of the dirty flag being set - I can't imagine why someone would want to risk that).

A separate ticket, then, would be to identify the shortcomings of UFAT32 regarding disk checking and to address those so that ultimately, chkdsk on FAT32 from OS/2 is every bit as thorough as under Windows.

[Valery V. Sedletski]

Starting from r172, the disk access is denied (ERROR_VOLUME_DIRTY is returned) on DosOpen and DosFindFirst. So, the IFS won't allow to access disk if it is dirty. A command line option "/readonly" is introduced. If it is specified, the IFS behaves the old way. The dirty flag can still be toggled by the f32stat utility.

PS: I uploaded the new alpha8 of fat32.ifs: ZIP archive: ​ftp://osfree.org/upload/fat32/fat32-0.10.a8-r172.zip, WarpIN package: ​ftp://osfree.org/upload/fat32/fat32-0.10.a8-r172.wpi. Note than it contains more changes: 1) preliminary > 2 Gb file support. Files of size up to 4 GB are now supported. But seek beyond 2 GB is still not working, because FS_CHGFILEPTR is called instead of FS_CHGFILEPTRL (so, the file pointer is truncated to 32-bits) 2) support for 64 KB clusters is introduced. But there are some bugs introduced, so ReadSector may sometimes read zero bytes.

[Lewis Rosenthal]

Okay, r175 seems to behave closer to what I would expect.

I did force a condition which put the FATs out of sync. Normally (and as FAT32's chkdsk advised) a chkdsk from Windows would be required. Instead, I used DFSee 14.2's FATSIM and FATWRIM to copy the first FAT over the second, exited DFSee, ejected the stick, reinserted, and ran CHKDSK against it. The result was a clean stick. That was an extreme case, of course.

I have tested it just picking up lost clusters, and fixing them.

Toggling the dirty flag does indeed render the volume inaccessible, and toggling it back allows access, without specifically remounting. Very nice.

Thanks!