netdev - Re: [Intel-wired-lan] [BUG] ice: Temporary packet processing overload causes permanent RX drops

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [<thread-prev] [day] [month] [year] [list]

Message-ID: <0dac6185-83e3-4963-b38b-48681affb929@intel.com>
Date: Thu, 11 Dec 2025 14:17:52 -0800
From: Jacob Keller <jacob.e.keller@...el.com>
To: Marcus Wichelmann <marcus.wichelmann@...zner-cloud.de>, Tony Nguyen
	<anthony.l.nguyen@...el.com>, Przemek Kitszel <przemyslaw.kitszel@...el.com>,
	Andrew Lunn <andrew+netdev@...n.ch>, "David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>, "Paolo
 Abeni" <pabeni@...hat.com>, <intel-wired-lan@...ts.osuosl.org>, Netdev
	<netdev@...r.kernel.org>, <linux-kernel@...r.kernel.org>
CC: <sdn@...zner-cloud.de>
Subject: Re: [Intel-wired-lan] [BUG] ice: Temporary packet processing overload
 causes permanent RX drops



On 12/11/2025 10:00 AM, Marcus Wichelmann wrote:
> Am 09.12.25 um 01:05 schrieb Jacob Keller:
>> On 12/5/2025 6:01 AM, Marcus Wichelmann wrote:
>>> Hi there, I broke some network cards again. This time I noticed continuous RX packet drops with an Intel E810-XXV.
>>>>> We have reproduced this with:
>>>   Linux 6.8.0-88-generic (Ubuntu 24.04)
>>>   Linux 6.14.0-36-generic (Ubuntu 24.04 HWE)
>>>   Linux 6.18.0-061800-generic (Ubuntu Mainline PPA)
>>
>> I think we recently merged a bunch of work on the Rx path as part of our
>> conversion to page pool. It would be interesting to see if those changes
>> impact this. Clearly the issue goes back some time since v6.8 at least..
> Hi Jacob,
> 
> I guess you mean 93f53db9f9dc ("ice: switch to Page Pool")?
> 
> I have now repeated all tests with a kernel built from latest net-next
> branch and can still reproduce it, even though I needed way higher packet
> rates (15 instead of 4 Mpps when using 256 channels). Something about the
> packet processing on our test system seems to have gotten way more
> efficient with this kernel update.
> 

Good info. I'm not certain if the refactors to switch to page pool would
fully explain this, but we did have some big improvements for certain
workloads with the switch.

> The symptoms are the same. The following IO_PAGE_FAULTs appear in the
> kernel log and after that, there is a permanent packet loss of 1-10%
> even at very low packet rates.
> 
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
>   kernel: ice 0000:c7:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002b address=0x4000180000 flags=0x0020]
>   kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND
>   kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
>   [...]
>   kernel: ice 0000:c7:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x4000180000 flags=0x0020]
>   kernel: amd_iommu_report_page_fault: 10 callbacks suppressed
>   [...]
> 
> I experimented with a few different channel counts and noticed that
> the issue only occurs with a combined channel count >128. So on
> systems with less many CPU cores, this bug probably never occurs.
> 

Interesting that it only triggers if you get IO_PAGE_FAULT.

>   256: reproduced.
>   254: reproduced.
>   200: reproduced.
>   129: reproduced.
>   128: stable.
>    64: stable.
> 
> Tested using "ethtool -L eth{0,1} combined XXX".
> 
> With <=128 channels, only the "... hogged CPU ..." warnings appear
> but no IO_PAGE_FAULTs. There is also no permanent packet loss after
> stopping the traffic generator.
> 
>>> [...]
>>>
>>> 3. Stop the traffic generator and re-run it with a way lower packet rate, e.g. 10.000 pps. Now it can be seen that
>>> a good part of these packets is being dropped, even though the kernel could easily keep up with this small packet rate.
>>
>> I assume the rx_dropped counter still incrementing here?
> 
> Yes. After the NIC is in this broken state, a few percent of all
> packets is being dropped and the rx_dropped counter increases
> with each of them.
> 

Right. That's quite strange.

>>> [...]
> 
> I also looked into why the packet processing load on this system
> is so high and `perf top` shows that it almost completely
> originates from native_queued_spin_lock_slowpath.
> 
> When digging deeper using `perf lock contention -Y spinlock`:
> 
>  contended   total wait     max wait     avg wait         type   caller
>    1724043      4.36 m     198.66 us    151.66 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
>      35960      2.51 s     112.57 ms     69.51 us     spinlock   __netif_receive_skb_core.constprop.0+0x832
>        620    103.79 ms    189.87 us    167.40 us     spinlock   do_sys_poll+0x26f
> 
> I'm not yet sure what is causing this.
> I don't think it's related to this issue, but maybe that's part of
> what brings this bug to daylight, so probably still worth a mention.
> 
> I hope you can make some sense of all that.
> 
> Thanks,
> Marcus



Download attachment "OpenPGP_signature.asc" of type "application/pgp-signature" (237 bytes)