Message-ID: <e791c078-a821-4636-b44d-e02c22c046cc@roeck-us.net>
Date: Tue, 30 Jul 2024 16:29:50 -0700
From: Guenter Roeck <linux@...ck-us.net>
To: Peter Zijlstra <peterz@...radead.org>
Cc: Jens Axboe <axboe@...nel.dk>,
 Linus Torvalds <torvalds@...ux-foundation.org>,
 Andy Lutomirski <luto@...nel.org>, Ingo Molnar <mingo@...hat.com>,
 Peter Anvin <hpa@...or.com>,
 Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
 the arch/x86 maintainers <x86@...nel.org>
Subject: Re: Linux 6.11-rc1

On 7/30/24 13:09, Peter Zijlstra wrote:
> On Tue, Jul 30, 2024 at 01:04:49PM -0700, Guenter Roeck wrote:
>> On 7/30/24 12:38, Peter Zijlstra wrote:
>>> On Tue, Jul 30, 2024 at 01:31:18PM -0600, Jens Axboe wrote:
>>>> On 7/30/24 1:22 PM, Peter Zijlstra wrote:
>>>>> On Tue, Jul 30, 2024 at 11:53:31AM -0700, Linus Torvalds wrote:
>>>>>
>>>>>> Which makes me think it's asm_exc_int3 just recursively failing.
>>>>>
>>>>> Sounds like text_poke() going sideways, there's a jump_label fail out
>>>>> there:
>>>>>
>>>>>    https://lkml.kernel.org/r/20240730132626.GV26599@noisy.programming.kicks-ass.net
>>>>
>>>> No change with this applied...
>>>>
>>>> Also not sure if you read my link, but a few things to note:
>>>>
>>>> - It only happens with gcc-11 here. I tried 12/13/14 and those
>>>>     are fine, don't have anything older
>>>
>>> One of my test boxes has 4.4 4.6 4.8 4.9 5 6 8 9 10 11 12 13
>>>
>>> (now I gotta go figure out wth 7 went :-) And yeah, we don't support
>>> most of those version anymore (phew).
>>>
>>> So if its easy to setup, I could try older GCCs.
>>>
>>
>> WFM with gcc 9.4, 10.3, 12.4, and 13.3. gcc 11.4 and 11.5 both fail.
> 
> 10.5 and 13.2 worked for me, and I can confirm 11.4 makes it go boom.
> 
>> Maybe I should just switch to a more recent version of gcc and call it a day,
>> in the hope that it is a compiler (or qemu) problem and doesn't just hide
>> the problem.
>>
>> Thoughts ?
> 
> Tempting, but I think it would be good to figure out what in GCC-11
> makes it sad, gcc-11 is still well within the supported range of GCCs
> afaik.
> 
> Lets see if its something that wants to be bisected.

I tried bisecting several ways, but it always ends up at commit 0256994887d7
("Merge tag 'for-6.11/block-post-20240722' of git://git.kernel.dk/linux").
Manual build confirmed that 0256994887d7 fails but 0256994887d7~1,
which is commit dd018c238b84 ("Merge tag 'bcachefs-2024-07-22' of
https://evilpiepirate.org/git/bcachefs") is fine, at least for me.

I then rebased 'for-6.11/block-post-20240722' on top of
dd018c238b84 and tried again. Result is below.

However, reverting this patch as well as the subsequent patches does not
fix the problem, and reverting the entire merge from the mainline kernel
doesn't fix it either.

The next step was to bisect starting from 0256994887d7, reverting the block merges
at each step. That points to the io_uring merge (second set of bisect results).
Hoever, reverting that merge doesn't help, and neither does reverting both
the block and the io_uring merges.

On the other side, reverting nothing but enabling CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
makes the problem disappear. But that doesn't really help, especially since reverting
the patches touching CONFIG_CRYPTO_MANAGER_DISABLE_TESTS does _not_ help.

Baffled. Is it possible that the crashing code catches some page boundary ?

Guenter

---
# bad: [a9dd34ab77277f0fb7fa41a3edb8f0a71f7d791f] block: don't free the integrity payload in bio_integrity_unmap_free_user
# good: [dd018c238b8489b6dd8c06f6b962ea75d79115ff] Merge tag 'bcachefs-2024-07-22' of https://evilpiepirate.org/git/bcachefs
git bisect start 'HEAD' 'dd018c238b84'
# bad: [113799f9042573ba197de7a78a1e450cb40573ac] block: don't call bio_uninit from bio_endio
git bisect bad 113799f9042573ba197de7a78a1e450cb40573ac
# good: [473252aab8bf1a86e4266cb65f7baac1c10a70d9] block: also return bio_integrity_payload * from stubs
git bisect good 473252aab8bf1a86e4266cb65f7baac1c10a70d9
# first bad commit: [113799f9042573ba197de7a78a1e450cb40573ac] block: don't call bio_uninit from bio_endio

---
# bad: [8400291e289ee6b2bf9779ff1c83a291501f017b] Linux 6.11-rc1
# good: [0256994887d7c89c2a41d872aac67605bda8f115] Merge tag 'for-6.11/block-post-20240722' of git://git.kernel.dk/linux
git bisect start 'v6.11-rc1' '0256994887d7'
# good: [b2eed73360dffea91ea64e8f19330c950dd42ebb] Merge tag 'linux-watchdog-6.11-rc1' of git://www.linux-watchdog.org/g
git bisect good b2eed73360dffea91ea64e8f19330c950dd42ebb
# good: [0ba9b1551185a8b42003b708b6a9c25a9808701e] Merge tag 'drm-next-2024-07-26' of https://gitlab.freedesktop.org/drl
git bisect good 0ba9b1551185a8b42003b708b6a9c25a9808701e
# good: [8e333791d4605dbce611c22f71a86721c9afc336] Merge tag 'gpio-fixes-for-v6.11-rc1' of git://git.kernel.org/pub/scmx
git bisect good 8e333791d4605dbce611c22f71a86721c9afc336
# bad: [5437f30d3458ad36e83ab96088d490ebfee844d8] Merge tag '6.11-rc-smb-client-fixes-part2' of git://git.samba.org/sfr6
git bisect bad 5437f30d3458ad36e83ab96088d490ebfee844d8
# good: [910bfc26d16d07df5a2bfcbc63f0aa9d1397e2ef] Merge tag 'rust-6.11' of https://github.com/Rust-for-Linux/linux
git bisect good 910bfc26d16d07df5a2bfcbc63f0aa9d1397e2ef
# bad: [8c9307474333d8d100870b45af00bfeb1872c836] Merge tag 'io_uring-6.11-20240726' of git://git.kernel.dk/linux
git bisect bad 8c9307474333d8d100870b45af00bfeb1872c836
# good: [29d63b94036e561a016ec8878b44aad6650d23e2] io_uring: align iowq and task request error handling
git bisect good 29d63b94036e561a016ec8878b44aad6650d23e2
# good: [358169617602f6f71b31e5c9532a09b95a34b043] io_uring/napi: pass ktime to io_napi_adjust_timeout
git bisect good 358169617602f6f71b31e5c9532a09b95a34b043
# good: [ef9ca17ca458ac7253ae71b552e601e49311fc48] hostfs: fix the host directory parse when mounting.
git bisect good ef9ca17ca458ac7253ae71b552e601e49311fc48
# good: [bc4eee85ca6ce5335efe314215841712b5531449] Merge tag 'vfs-6.11-rc1.fixes.3' of git://git.kernel.org/pub/scm/lins
git bisect good bc4eee85ca6ce5335efe314215841712b5531449
# first bad commit: [8c9307474333d8d100870b45af00bfeb1872c836] Merge tag 'io_uring-6.11-20240726' of git://git.kernel.dx

Powered by blists - more mailing lists