netdev - Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAK8fFZ6+BNjNdemB+P=SuwU6X9a9CmtkR8Nux-XG7QHdcswvQQ@mail.gmail.com>
Date: Wed, 21 May 2025 12:50:01 +0200
From: Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
To: "Keller, Jacob E" <jacob.e.keller@...el.com>, Jakub Kicinski <kuba@...nel.org>, 
	"Kitszel, Przemyslaw" <przemyslaw.kitszel@...el.com>, "Damato, Joe" <jdamato@...tly.com>, 
	"intel-wired-lan@...ts.osuosl.org" <intel-wired-lan@...ts.osuosl.org>, 
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>, "Nguyen, Anthony L" <anthony.l.nguyen@...el.com>, 
	Michal Swiatkowski <michal.swiatkowski@...ux.intel.com>, 
	"Czapnik, Lukasz" <lukasz.czapnik@...el.com>, "Dumazet, Eric" <edumazet@...gle.com>, 
	"Zaki, Ahmed" <ahmed.zaki@...el.com>, Martin Karsten <mkarsten@...terloo.ca>
Cc: Igor Raits <igor@...ddata.com>, Daniel Secik <daniel.secik@...ddata.com>, 
	Zdenek Pesek <zdenek.pesek@...ddata.com>
Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE
 driver after upgrade to 6.13.y (regression in commit 492a044508ad)

čt 17. 4. 2025 v 19:52 odesílatel Keller, Jacob E
<jacob.e.keller@...el.com> napsal:
>
>
>
> > -----Original Message-----
> > From: Jakub Kicinski <kuba@...nel.org>
> > Sent: Wednesday, April 16, 2025 5:13 PM
> > To: Keller, Jacob E <jacob.e.keller@...el.com>
> > Cc: Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>; Kitszel, Przemyslaw
> > <przemyslaw.kitszel@...el.com>; Damato, Joe <jdamato@...tly.com>; intel-wired-
> > lan@...ts.osuosl.org; netdev@...r.kernel.org; Nguyen, Anthony L
> > <anthony.l.nguyen@...el.com>; Igor Raits <igor@...ddata.com>; Daniel Secik
> > <daniel.secik@...ddata.com>; Zdenek Pesek <zdenek.pesek@...ddata.com>;
> > Dumazet, Eric <edumazet@...gle.com>; Martin Karsten
> > <mkarsten@...terloo.ca>; Zaki, Ahmed <ahmed.zaki@...el.com>; Czapnik,
> > Lukasz <lukasz.czapnik@...el.com>; Michal Swiatkowski
> > <michal.swiatkowski@...ux.intel.com>
> > Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE
> > driver after upgrade to 6.13.y (regression in commit 492a044508ad)
> >
> > On Wed, 16 Apr 2025 22:57:10 +0000 Keller, Jacob E wrote:
> > > > > And you're reverting just and exactly 492a044508ad13 ?
> > > > > The memory for persistent config is allocated in alloc_netdev_mqs()
> > > > > unconditionally. I'm lost as to how this commit could make any
> > > > > difference :(
> > > >
> > > > Yes, reverted the 492a044508ad13.
> > >
> > > Struct napi_config *is* 1056 bytes
> >
> > You're probably looking at 6.15-rcX kernels. Yes, the affinity mask
> > can be large depending on the kernel config. But report is for 6.13,
> > AFAIU. In 6.13 and 6.14 napi_config was tiny.
>
> Regardless, it should still be ~64KB even in that case which is a far cry from eating all available memory. Something else must be going on....
>
> Thanks,
> Jake

Hello

Some observation, this "problem" still exists with the latest 6.14.y
and there must be multiple issues, the memory utilization is slowly
going down, from 3GB to 100MB in 10-20days. at home NUMA nodes where
intel x810 NIC are (looks like some memory leak related to
networking).

So without the revert the kawadX usage is observed asap like till
1-2d, with revert of mentioned commit kswadX starts to consume
resources later like in ~10d-20d later. It is almost impossible to use
servers with Intel X810 cards (ice driver) with recent linux kernels.

Were you able to reproduce the memory problems in your testbed?

Best,
Jaroslav