linux-kernel - Re: [PATCH 5.15 000/159] 5.15.116-rc1 review

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sun, 11 Jun 2023 18:12:27 -0700
From:   Guenter Roeck <linux@...ck-us.net>
To:     Pavel Machek <pavel@...x.de>
Cc:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        stable@...r.kernel.org, patches@...ts.linux.dev,
        linux-kernel@...r.kernel.org, torvalds@...ux-foundation.org,
        akpm@...ux-foundation.org, shuah@...nel.org, patches@...nelci.org,
        lkft-triage@...ts.linaro.org, jonathanh@...dia.com,
        f.fainelli@...il.com, sudipm.mukherjee@...il.com,
        srw@...dewatkins.net, rwarsow@....de,
        Thomas Gleixner <tglx@...utronix.de>,
        Ido Schimmel <idosch@...dia.com>
Subject: Re: [PATCH 5.15 000/159] 5.15.116-rc1 review

On 6/11/23 08:14, Guenter Roeck wrote:
> On 6/10/23 14:14, Guenter Roeck wrote:
>> Hi,
>>
>> On 6/10/23 12:23, Pavel Machek wrote:
>>> Hi!
>>>
>>>>> Build results:
>>>>>     total: 155 pass: 155 fail: 0
>>>>> Qemu test results:
>>>>>     total: 499 pass: 498 fail: 1
>>>>> Failed tests:
>>>>>     arm:kudo-bmc:multi_v7_defconfig:npcm:usb0.1:nuvoton-npcm730-kudo:rootfs
>>>>>
>>>>> The test failure is spurious and not new. I observe it randomly on
>>>>> multi_v7_defconfig builds, primarily on npcm platforms. There is no error
>>>>> message, just a stalled boot. I have been trying to bisect for a while,
>>>>> but I have not been successful so far. No immediate concern; I just wanted
>>>>> to mention it in case someone else hits the same or a similar problem.
>>>>>
>>>>
>>>> I managed to revise my bisect script sufficiently enough to get reliable
>>>> results. It looks like the culprit is commit 503e554782c9 (" debugobject:
>>>> Ensure pool refill (again)"); see bisect log below. Bisect on four
>>>> different systems all have the same result. After reverting this patch,
>>>> I do not see the problem anymore (again, confirmed on four different
>>>> systems). If anyone has an idea how to debug this, please let me know.
>>>> I'll be happy to give it a try.
>>>
>>> You may want to comment out debug_objects_fill_pool() in
>>> debug_object_activate or debug_object_assert_init to see which one is
>>> causing the failure...
>>>
>>> CONFIG_PREEMPT_RT is disabled for you, right? (Should 5.15 even have
>>> that option?)
>>>
>>
>> CONFIG_PREEMPT_RT is disabled (it depends on ARCH_SUPPORTS_RT which is not
>> enabled by any architecture in v5.15.y).
>>
>> The added call in debug_object_activate() triggers the problem.
>> Any idea what to do about it or how to debug it further ?
>>
> 
> I did some more debugging. The call to debug_object_activate()
> from debug_hrtimer_activate() causes the immediate problem, and the
> call from debug_timer_activate() causes a second (less likely) problem,
> where the stall is seen during reboot.
> 
> In other words, the problem is (only) seen if DEBUG_OBJECTS_TIMERS
> is enabled.
> 

Bisect log between v5.15 and v6.1 below. The fix is all but impossible to backport,
and I still have no idea what is actually going on. I think I'll just disable
DEBUG_OBJECTS_TIMERS in affected tests of v5.15.y.

Guenter

---
# fixed: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
# broken: [8bb7eca972ad531c9b149c0a51ab43a417385813] Linux 5.15
git bisect start 'v6.1' 'v5.15'
# broken: [7fa2e481ff2fee20e0338d98489eb9f513ada45f] Merge branch 'big-tcp'
git bisect broken 7fa2e481ff2fee20e0338d98489eb9f513ada45f
# fixed: [9e2e5ea3b28f81512c792f30729edb1db0c21f6a] Merge tag 'usb-6.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
git bisect fixed 9e2e5ea3b28f81512c792f30729edb1db0c21f6a
# fixed: [4ad680f083ec360e0991c453e18a38ed9ae500d7] Merge tag 'staging-5.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect fixed 4ad680f083ec360e0991c453e18a38ed9ae500d7
# fixed: [2518f226c60d8e04d18ba4295500a5b0b8ac7659] Merge tag 'drm-next-2022-05-25' of git://anongit.freedesktop.org/drm/drm
git bisect fixed 2518f226c60d8e04d18ba4295500a5b0b8ac7659
# broken: [fea3043314f30a87ca04fd1219661810600e256f] Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect broken fea3043314f30a87ca04fd1219661810600e256f
# broken: [f8122500a039abeabfff41b0ad8b6a2c94c1107d] Merge branch 'etnaviv/next' of https://git.pengutronix.de/git/lst/linux into drm-next
git bisect broken f8122500a039abeabfff41b0ad8b6a2c94c1107d
# fixed: [7e062cda7d90543ac8c7700fc7c5527d0c0f22ad] Merge tag 'net-next-5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect fixed 7e062cda7d90543ac8c7700fc7c5527d0c0f22ad
# broken: [9fa87dd23251574a29cf948fd16cf39075762f3e] Merge tag 'linux-can-next-for-5.19-20220523' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next
git bisect broken 9fa87dd23251574a29cf948fd16cf39075762f3e
# fixed: [88a618920e9baabc1780479e2fbb68e5551d0563] Merge tag 'docs-5.19' of git://git.lwn.net/linux
git bisect fixed 88a618920e9baabc1780479e2fbb68e5551d0563
# broken: [fdaf9a5840acaab18694a19e0eb0aa51162eeeed] Merge tag 'folio-5.19' of git://git.infradead.org/users/willy/pagecache
git bisect broken fdaf9a5840acaab18694a19e0eb0aa51162eeeed
# broken: [164f9fcb21cc9a144ca9ebcf85b00c49537f6be2] docs/ja_JP/SubmittingPatches: Suggest the use of scripts/get_maintainer.pl
git bisect broken 164f9fcb21cc9a144ca9ebcf85b00c49537f6be2
# broken: [2e17ce1106e04a7f3a83796ec623881487f75dd3] Merge tag 'slab-for-5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
git bisect broken 2e17ce1106e04a7f3a83796ec623881487f75dd3
# fixed: [701850dc0c31bfadf75a0a74af7d2c97859945ec] printk, tracing: fix console tracepoint
git bisect fixed 701850dc0c31bfadf75a0a74af7d2c97859945ec
# broken: [1fc0ca9e0db61882208650b3603071e9f4b5cfee] printk: add con_printk() macro for console details
git bisect broken 1fc0ca9e0db61882208650b3603071e9f4b5cfee
# broken: [2bb2b7b57f81255c13f4395ea911d6bdc70c9fe2] printk: add functions to prefer direct printing
git bisect broken 2bb2b7b57f81255c13f4395ea911d6bdc70c9fe2
# fixed: [8e274732115f63c1d09136284431b3555bd5cc56] printk: extend console_lock for per-console locking
git bisect fixed 8e274732115f63c1d09136284431b3555bd5cc56
# fixed: [09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7] printk: add kthread console printers
git bisect fixed 09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7
# first fixed commit: [09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7] printk: add kthread console printers