linux-kernel - edge case in posix file lock deadlock detection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <8B2B011D-6E2E-4FEA-8FEE-B69D437B8547@icloud.com>
Date:   Wed, 30 Jan 2019 19:32:37 -0800
From:   Theodore Dubois <tblodt@...oud.com>
To:     Theodore Dubois <tblodt@...oud.com>
Cc:     linux-kernel@...r.kernel.org
Subject: edge case in posix file lock deadlock detection

I'm having trouble figuring out how the kernel handles a particular case in deadlock detection on posix file locks. Here's the scenario:

PID 1: locks byte 2
PID 3: locks byte 0
PID 2: locks byte 10
PID 1: locks byte 10
PID 2: locks bytes 0-2 inclusive

The last step fails with EDEADLK, but I'm not sure how that is detected. The specified range conflicts with two different locks, and the first loop in posix_lock_inode would find whichever one comes first in the linked list, and pass that to the deadlock detector.

If the lock on byte 2 comes first in the list, a cycle would be found between the lock on byte 2 and the lock on byte 10. But if the lock on byte 0 comes first, the deadlock detector would return NULL from what_owner_is_waiting_for on that lock since PID 3 has no other locks, and immediately succeed.

What am I missing?

Thanks,
~Theodore