linux-kernel - Re: mlockall(MCL_CURRENT) blocking infinitely

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ad13f479-3fda-b55a-d311-ef3914fbe649@suse.cz>
Date:   Tue, 5 Nov 2019 16:28:21 +0100
From:   Vlastimil Babka <vbabka@...e.cz>
To:     snazy@...zy.de, Michal Hocko <mhocko@...nel.org>,
        Josef Bacik <josef@...icpanda.com>,
        Johannes Weiner <hannes@...xchg.org>, Jan Kara <jack@...e.cz>,
        "Kirill A. Shutemov" <kirill@...temov.name>
Cc:     Randy Dunlap <rdunlap@...radead.org>, linux-kernel@...r.kernel.org,
        Linux MM <linux-mm@...ck.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "Potyra, Stefan" <Stefan.Potyra@...ktrobit.com>
Subject: Re: mlockall(MCL_CURRENT) blocking infinitely

On 11/5/19 2:23 PM, Robert Stupp wrote:
> "git bisect" led to a result.
> 
> The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
> "Merge branch 'akpm' (rest of patches from Andrew)".
> 
> The first bad commit in the merged series of commits is
> https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> . a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
> is good.

Ah, great you could bisect this. CCing people from the commit
6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

First mail in thread:
https://lore.kernel.org/linux-mm/b8ff71f5-2d9c-7ebb-d621-017d4b9bc932@infradead.org/

> I've also verified 5.1.21 and 5.3.8 (from
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/)
> without 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11 and both builds are
> good.
> (The 5.1.21 and 5.3.7 builds from Ubuntu were bad, so I haven't cross-
> checked "vanilla" 5.1.21 and 5.3.8 kernel builds.)
> 
> 
> 
> Recap symptoms:
> - `mlockall(MCL_CURRENT)` hangs
> - shutdown/reboot hangs when it reaches "shutdown->reboot"
> - `cat /proc/$(pidof test)/smaps` shows "Locked" w/ odd values, which
> are equal to "Pss"
> 
> Affected:
> - `cryptsetup luksOpen` hangs (when it tries to lock memory)
> - Apache Cassandra hangs during startup (when it performs an
> `mlockall(MCL_CURRENT)`)
> 
> 
> 
> git checkout v5.1.21
> # revert the "comment-only" commit (no need to test this one)
> # "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
> git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # "filemap: drop the mmap_sem for all blocking operations"
> git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> --> GOOD
> 
> git checkout v5.3.8
> # revert the "comment-only" commit (no need to test this one)
> # "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
> git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # "filemap: drop the mmap_sem for all blocking operations"
> git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> --> GOOD
> 
> 
> 
> Bisect log:
> git bisect start
> # bad: [9e98c678c2d6ae3a17cb2de55d17f69dddaa231b] Linux 5.1-rc1
> git bisect bad 9e98c678c2d6ae3a17cb2de55d17f69dddaa231b
> # good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
> git bisect good 1c163f4c7b3f621efff9b28a47abb36f7378d783
> # good: [e266ca36da7de45b64b05698e98e04b578a88888] Merge tag 'staging-
> 5.1-rc1' of
> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect good e266ca36da7de45b64b05698e98e04b578a88888
> # good: [36011ddc78395b59a8a418c37f20bcc18828f1ef] Merge tag 'gfs2-
> 5.1.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-
> gfs2
> git bisect good 36011ddc78395b59a8a418c37f20bcc18828f1ef
> # good: [6bc3fe8e7e172d5584e529a04cf9eec946428768] tools: mark
> 'test_vmalloc.sh' executable
> git bisect good 6bc3fe8e7e172d5584e529a04cf9eec946428768
> # good: [dc2535be1fd547fbd56aff091370280007b0a1af] Merge tag 'clk-for-
> linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
> git bisect good dc2535be1fd547fbd56aff091370280007b0a1af
> # bad: [2b9c272cf5cd81708e51b4ce3e432ce9566cfa47] Merge tag 'fbdev-
> v5.1' of git://github.com/bzolnier/linux
> git bisect bad 2b9c272cf5cd81708e51b4ce3e432ce9566cfa47
> # good: [9bc446100334dbbc14eb3757274ef08746c3f9bd] Merge tag
> 'microblaze-v5.1-rc1' of git://git.monstr.eu/linux-2.6-microblaze
> git bisect good 9bc446100334dbbc14eb3757274ef08746c3f9bd
> # bad: [5160bcce5c3c80de7d8722511c144d3041409657] Merge tag 'f2fs-for-
> 5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
> git bisect bad 5160bcce5c3c80de7d8722511c144d3041409657
> # good: [240a59156d9bcfabceddb66be449e7b32fb5dc4a] f2fs: fix to add
> refcount once page is tagged PG_private
> git bisect good 240a59156d9bcfabceddb66be449e7b32fb5dc4a
> # good: [9352ca585b2ac7b67d2119b9386573b2a4c0ef4b] Merge tag 'pm-5.1-
> rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> git bisect good 9352ca585b2ac7b67d2119b9386573b2a4c0ef4b
> # good: [f261c4e529dac5608a604d3dd3ae1cd2adf23c89] Merge branch 'akpm'
> (patches from Andrew)
> git bisect good f261c4e529dac5608a604d3dd3ae1cd2adf23c89
> # good: [aadcef64b22f668c1a107b86d3521d9cac915c24] f2fs: fix to avoid
> deadlock in f2fs_read_inline_dir()
> git bisect good aadcef64b22f668c1a107b86d3521d9cac915c24
> # bad: [8b0f9fa2e02dc95216577c3387b0707c5f60fbaf] filemap: add a
> comment about FAULT_FLAG_RETRY_NOWAIT behavior
> git bisect bad 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # bad: [6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11] filemap: drop the
> mmap_sem for all blocking operations
> git bisect bad 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> # bad: [a75d4c33377277b6034dd1e2663bce444f952c14] filemap: kill
> page_cache_read usage in filemap_fault
> git bisect good a75d4c33377277b6034dd1e2663bce444f952c14
> 
> 
> All kernels built with
> make oldconfig # accept the defaults
> make bindeb-pkg
> 
> 
> 
> On Fri, 2019-10-25 at 17:58 +0200, Robert Stupp wrote:
>> On Fri, 2019-10-25 at 16:00 +0200, Michal Hocko wrote:
>>> And one more thing. Considering that you are able to reproduce and
>>> you
>>> have a working kernel, could you try to bisect this?
>>
>> Yikes - running self-built kernels brings back a lot of memories ;)
>>
>> Anyway, going this route (using the `config` from Ubuntu 5.1.x as a
>> base and accepting the defaults for `make oldconfig`):
>>
>> git checkout v5.1-rc1
>> git bisect start
>> git bisect bad
>> git bisect good v5.0
>>
>> ... first try @ e266ca36da7de45b64b05698e98e04b578a88888 is a `git
>> bisect good`
>>
>> Will report back, when I've got a result...
>>
> 
>