netdev - pegged softirq and NAPI race (?)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <A782704A-DF97-4E85-B10A-D2268A67DFD7@fb.com>
Date:   Tue, 18 Sep 2018 08:41:36 +0000
From:   Song Liu <songliubraving@...com>
To:     Networking <netdev@...r.kernel.org>
CC:     "edumazet@...gle.com" <edumazet@...gle.com>,
        "jeffrey.t.kirsher@...el.com" <jeffrey.t.kirsher@...el.com>,
        "alexander.h.duyck@...el.com" <alexander.h.duyck@...el.com>,
        Michael Chan <michael.chan@...adcom.com>,
        Kernel Team <Kernel-team@...com>
Subject: pegged softirq and NAPI race (?) 

We are debugging this issue that netconsole message triggers pegged softirq
(ksoftirqd taking 100% CPU for many seconds). We found this issue in
production with both bnxt and ixgbe, on a 4.11 based kernel. This is easily
reproducible with ixgbe on 4.11, and latest net/net-next (see [1] for more
detail).

After debugging for some time, we found that this issue is likely related
to 39e6c8208d7b ("net: solve a NAPI race"). After reverting this commit,
the steps described in [1] cannot reproduce the issue on ixgbe. Reverting
this commit also reduces the chances we hit the issue with bnxt (it still
happens with a lower rate).

I tried to fix this issue with relaxed variant (or older version) of
napi_schedule_prep() in netpoll, just like the one on napi_watchdog().
However, my tests do not always go as expected.

Please share your comments/suggestions on which direction shall we try
to fix this.

Thanks in advance!
Song

[1] https://www.spinics.net/lists/netdev/msg522328.html