netdev - Re: [Regression] stress-ng udp-flood causes kernel panic on Ampere Altra

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Sat, 2 Jul 2022 12:08:46 +0100
From:   Kajetan Puchalski <kajetan.puchalski@....com>
To:     Florian Westphal <fw@...len.de>
Cc:     Pablo Neira Ayuso <pablo@...filter.org>,
        Jozsef Kadlecsik <kadlec@...filter.org>,
        Florian Westphal <fw@...len.de>,
        "David S. Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>, Mel Gorman <mgorman@...e.de>,
        lukasz.luba@....com, dietmar.eggemann@....com,
        netfilter-devel@...r.kernel.org, coreteam@...filter.org,
        netdev@...r.kernel.org, stable@...r.kernel.org,
        regressions@...ts.linux.dev, linux-kernel@...r.kernel.org
Subject: Re: [Regression] stress-ng udp-flood causes kernel panic on Ampere
 Altra

On Fri, Jul 01, 2022 at 10:01:10PM +0200, Florian Westphal wrote:
> Kajetan Puchalski <kajetan.puchalski@....com> wrote:
> > While running the udp-flood test from stress-ng on Ampere Altra (Mt.
> > Jade platform) I encountered a kernel panic caused by NULL pointer
> > dereference within nf_conntrack.
> > 
> > The issue is present in the latest mainline (5.19-rc4), latest stable
> > (5.18.8), as well as multiple older stable versions. The last working
> > stable version I found was 5.15.40.
> 
> Do I need a special setup for conntrack?

I don't think there was any special setup involved, the config I started
from was a generic distribution config and I didn't change any
networking-specific options. In case that's helpful here's the .config I
used.

https://pastebin.com/Bb2wttdx

> 
> No crashes after more than one hour of stress-ng on
> 1. 4 core amd64 Fedora 5.17 kernel
> 2. 16 core amd64, linux stable 5.17.15
> 3. 12 core intel, Fedora 5.18 kernel
> 4. 3 core aarch64 vm, 5.18.7-200.fc36.aarch64
> 

That would make sense, from further experiments I ran it somehow seems
to be related to the number of workers being spawned by stress-ng along
with the CPUs/cores involved.

For instance, running the test with <=25 workers (--udp-flood 25 etc.)
results in the test running fine for at least 15 minutes.
Running the test with 30 workers results in a panic sometime before it
hits the 15 minute mark.
Based on observations there seems to be a corellation between the number
of workers and how quickly the panic occurs, ie with 30 it takes a few
minutes, with 160 it consistently happens almost immediately. That also
holds for various numbers of workers in between.

On the CPU/core side of things, the machine in question has two CPU
sockets with 80 identical cores each. All the panics I've encountered
happened when stress-ng was ran directly and unbound.
When I tried using hwloc-bind to bind the process to one of the CPU
sockets, the test ran for 15 mins with 80 and 160 workers with no issues,
no matter which CPU it was bound to.

Ie the specific circumstances under which it seems to occur are when the
test is able to run across multiple CPU sockets with a large number
of workers being spawned.

> I used standard firewalld ruleset for all of these and manually tuned
> conntrack settings to make sure the early evict path (as per backtrace)
> gets exercised.