netdev - Re: ARM board lockups/hangs triggered by locks and mutexes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <2a6f6a2e-532b-4c25-ba5b-86bb0022e58b@gmail.com>
Date: Fri, 18 Aug 2023 22:24:54 +0200
From: Rafał Miłecki <zajec5@...il.com>
To: Geert Uytterhoeven <geert@...ux-m68k.org>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>,
 Will Deacon <will@...nel.org>, Waiman Long <longman@...hat.com>,
 Boqun Feng <boqun.feng@...il.com>, Russell King <linux@...linux.org.uk>,
 Daniel Lezcano <daniel.lezcano@...aro.org>,
 Thomas Gleixner <tglx@...utronix.de>, Florian Fainelli
 <f.fainelli@...il.com>, linux-clk@...r.kernel.org,
 "linux-arm-kernel@...ts.infradead.org"
 <linux-arm-kernel@...ts.infradead.org>,
 Network Development <netdev@...r.kernel.org>,
 Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
 OpenWrt Development List <openwrt-devel@...ts.openwrt.org>,
 bcm-kernel-feedback-list <bcm-kernel-feedback-list@...adcom.com>
Subject: Re: ARM board lockups/hangs triggered by locks and mutexes

On 18.08.2023 22:23, Rafał Miłecki wrote:
> On 14.08.2023 11:04, Geert Uytterhoeven wrote:
>> Hi Rafal,
>>
>> On Mon, Aug 7, 2023 at 1:11 PM Rafał Miłecki <zajec5@...il.com> wrote:
>>> On 4.08.2023 13:07, Rafał Miłecki wrote:
>>>> I triple checked that. Dropping a single unused function breaks kernel /
>>>> device stability on BCM53573!
>>>>
>>>> AFAIK the only thing below diff actually affects is location of symbols
>>>> (I actually verified that by comparing System.map before and after -
>>>> over 22'000 of relocated symbols).
>>>>
>>>> Can some unfortunate location of symbols cause those hangs/lockups?
>>>
>>> I performed another experiment. First I dropped mtd_check_of_node() to
>>> bring kernel back to the stable state.
>>>
>>> Then I started adding useless code to the mtdchar_unlocked_ioctl(). I
>>> ended up adding just enough to make sure all post-mtd symbols in
>>> System.map got the same offset as in case of backporting
>>> mtd_check_of_node().
>>>
>>> I started experiencing lockups/hangs again.
>>>
>>> I repeated the same test with adding dumb code to the brcm_nvram_probe()
>>> and verifying symbols offsets following brcm_nvram_probe one.
>>>
>>> I believe this confirms that this problem is about offset or alignment
>>> of some specific symbol(s). The remaining question is what symbols and
>>> how to fix or workaround that.
>>
>> I had similar experiences on other ARM platforms many years ago:
>> bisection lead to something completely bogus, and it turned out
>> adding a single line of innocent code made the system lock-up or crash
>> unexpectedly.  It was definitely related to alignment, as adding the
>> right extra amount of innocent code would fix the problem. Until some
>> later change changing alignment again...
>> I never found the real cause, but the problems went away over time.
>> I am not sure I did enable all required errata config options, so I
>> may have missed some...
> 
> I already experiented some weird performance variations on Broadcom's
> Northstar platform that was related to symbols layout & cache hit/miss
> ratio. For that reason I use -falign-functions=32 for that whole
> OpenWrt's "bcm53xx" target (it covers Northstar and BCM53573). So
> this aspect should be ruled out already in my case.

Relevant OpenWrt commit with some description and links: b54ef39e0b91 ("bcm53xx: use -falign-functions=32 for kernel compilation"):

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=b54ef39e0b910a4b8eaca0497fe9b63e8392262a