fcntl lock API doesn't match POSIX

[dmik]

The file segment lock functionality provided by fcntl() is implemented with the DosSetFileLocks?[L] API. However, this API has several serious limitations that make it not compatible with POSIX (​http://pubs.opengroup.org/onlinepubs/009695399/functions/fcntl.html, ​http://man7.org/linux/man-pages/man2/fcntl.2.html):

1. No deadlock detection. If process A holds a lock of byte 1 and attempts to lock byte 2 in wait mode while process B holds a lock of byte 2 and attempts to lock byte 1 in wait mode, both will wait forever.
2. No support for overlapping ranges, automatic range split/join with atomic lock upgrade etc.

This makes fcntl() file locks barely usable in real POSIX applications. One of the examples is Samba where these locks are used heavily to implement TDB (database) locking and multi-process access. Samba is a critical piece of software hence proper fcntl() lock support is such as well.

[dmik]

An and there is also:

3. When the file lock is set on a file, it's impossible to write into it even from the lock owning process (well, this looks like a part of a generic difference between advisory locking on Linux and mandatory locking on OS/2 for file operations, but still - it makes problems for Samba too).

Note that although that joining adjacent lock ranges could be theoretically implemented with DosSetFIleLocks using its support for atomic unlock/lock, practice shows that if the unlock and lock regions don't match, the operation doesn't happen atomically (and the docs also honestly warn about that).

What needs to be done is a separate implementation of file segment locks in LIBC that uses a table of file names and locks regions in shared memory and manages it accordingly. Deadlock support can be added there rather easily, as well as support for region join/split and automatic and atomic lock upgrade. This will also vanish the file write operation failure problem when a lock is held.

[dmik]

Forgot to mention that fcntl() lock misbehavior in Samba is a source of constant problems with our port. Under heavy load (many clients, many open files) users get either frequent TDB file corruption with server process termination or denial of service (OS/2 shares stop responding, most likely because of deadlocks) which requires a restart. I tried to work around some of the issues here ​http://trac.netlabs.org/samba/changeset/895, but since deadlocks are still there and atomic lock upgrade doesn't work, this isn't expected to help much.