Message-ID: <fa6599459300c61da6348cdfd0cfda79e1c17a7a.camel@gmx.de>
Date:   Tue, 05 Nov 2019 14:23:25 +0100
From:   Robert Stupp <snazy@....de>
To:     Michal Hocko <mhocko@...nel.org>
Cc:     Randy Dunlap <rdunlap@...radead.org>, linux-kernel@...r.kernel.org,
        Linux MM <linux-mm@...ck.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "Potyra, Stefan" <Stefan.Potyra@...ktrobit.com>
Subject: Re: mlockall(MCL_CURRENT) blocking infinitely

"git bisect" led to a result.

The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
"Merge branch 'akpm' (rest of patches from Andrew)".

The first bad commit in the merged series of commits is
https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
. a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
is good.

I've also verified 5.1.21 and 5.3.8 (from
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/)
without 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11 and both builds are
good.
(The 5.1.21 and 5.3.7 builds from Ubuntu were bad, so I haven't cross-
checked "vanilla" 5.1.21 and 5.3.8 kernel builds.)



Recap symptoms:
- `mlockall(MCL_CURRENT)` hangs
- shutdown/reboot hangs when it reaches "shutdown->reboot"
- `cat /proc/$(pidof test)/smaps` shows "Locked" w/ odd values, which
are equal to "Pss"

Affected:
- `cryptsetup luksOpen` hangs (when it tries to lock memory)
- Apache Cassandra hangs during startup (when it performs an
`mlockall(MCL_CURRENT)`)



git checkout v5.1.21
# revert the "comment-only" commit (no need to test this one)
# "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# "filemap: drop the mmap_sem for all blocking operations"
git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
--> GOOD

git checkout v5.3.8
# revert the "comment-only" commit (no need to test this one)
# "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# "filemap: drop the mmap_sem for all blocking operations"
git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
--> GOOD



Bisect log:
git bisect start
# bad: [9e98c678c2d6ae3a17cb2de55d17f69dddaa231b] Linux 5.1-rc1
git bisect bad 9e98c678c2d6ae3a17cb2de55d17f69dddaa231b
# good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
git bisect good 1c163f4c7b3f621efff9b28a47abb36f7378d783
# good: [e266ca36da7de45b64b05698e98e04b578a88888] Merge tag 'staging-
5.1-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good e266ca36da7de45b64b05698e98e04b578a88888
# good: [36011ddc78395b59a8a418c37f20bcc18828f1ef] Merge tag 'gfs2-
5.1.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-
gfs2
git bisect good 36011ddc78395b59a8a418c37f20bcc18828f1ef
# good: [6bc3fe8e7e172d5584e529a04cf9eec946428768] tools: mark
'test_vmalloc.sh' executable
git bisect good 6bc3fe8e7e172d5584e529a04cf9eec946428768
# good: [dc2535be1fd547fbd56aff091370280007b0a1af] Merge tag 'clk-for-
linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
git bisect good dc2535be1fd547fbd56aff091370280007b0a1af
# bad: [2b9c272cf5cd81708e51b4ce3e432ce9566cfa47] Merge tag 'fbdev-
v5.1' of git://github.com/bzolnier/linux
git bisect bad 2b9c272cf5cd81708e51b4ce3e432ce9566cfa47
# good: [9bc446100334dbbc14eb3757274ef08746c3f9bd] Merge tag
'microblaze-v5.1-rc1' of git://git.monstr.eu/linux-2.6-microblaze
git bisect good 9bc446100334dbbc14eb3757274ef08746c3f9bd
# bad: [5160bcce5c3c80de7d8722511c144d3041409657] Merge tag 'f2fs-for-
5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
git bisect bad 5160bcce5c3c80de7d8722511c144d3041409657
# good: [240a59156d9bcfabceddb66be449e7b32fb5dc4a] f2fs: fix to add
refcount once page is tagged PG_private
git bisect good 240a59156d9bcfabceddb66be449e7b32fb5dc4a
# good: [9352ca585b2ac7b67d2119b9386573b2a4c0ef4b] Merge tag 'pm-5.1-
rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect good 9352ca585b2ac7b67d2119b9386573b2a4c0ef4b
# good: [f261c4e529dac5608a604d3dd3ae1cd2adf23c89] Merge branch 'akpm'
(patches from Andrew)
git bisect good f261c4e529dac5608a604d3dd3ae1cd2adf23c89
# good: [aadcef64b22f668c1a107b86d3521d9cac915c24] f2fs: fix to avoid
deadlock in f2fs_read_inline_dir()
git bisect good aadcef64b22f668c1a107b86d3521d9cac915c24
# bad: [8b0f9fa2e02dc95216577c3387b0707c5f60fbaf] filemap: add a
comment about FAULT_FLAG_RETRY_NOWAIT behavior
git bisect bad 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# bad: [6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11] filemap: drop the
mmap_sem for all blocking operations
git bisect bad 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
# bad: [a75d4c33377277b6034dd1e2663bce444f952c14] filemap: kill
page_cache_read usage in filemap_fault
git bisect good a75d4c33377277b6034dd1e2663bce444f952c14


All kernels built with
make oldconfig # accept the defaults
make bindeb-pkg



On Fri, 2019-10-25 at 17:58 +0200, Robert Stupp wrote:
> On Fri, 2019-10-25 at 16:00 +0200, Michal Hocko wrote:
> > And one more thing. Considering that you are able to reproduce and
> > you
> > have a working kernel, could you try to bisect this?
>
> Yikes - running self-built kernels brings back a lot of memories ;)
>
> Anyway, going this route (using the `config` from Ubuntu 5.1.x as a
> base and accepting the defaults for `make oldconfig`):
>
> git checkout v5.1-rc1
> git bisect start
> git bisect bad
> git bisect good v5.0
>
> ... first try @ e266ca36da7de45b64b05698e98e04b578a88888 is a `git
> bisect good`
>
> Will report back, when I've got a result...
>

Powered by blists - more mailing lists