netdev - RE: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE driver after upgrade to 6.13.y (regression in commit 492a044508ad)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CO1PR11MB5089365F31BCD97E59CCFA83D6BD2@CO1PR11MB5089.namprd11.prod.outlook.com>
Date: Wed, 16 Apr 2025 22:57:10 +0000
From: "Keller, Jacob E" <jacob.e.keller@...el.com>
To: Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>, Jakub Kicinski
	<kuba@...nel.org>
CC: "Kitszel, Przemyslaw" <przemyslaw.kitszel@...el.com>, "Damato, Joe"
	<jdamato@...tly.com>, "intel-wired-lan@...ts.osuosl.org"
	<intel-wired-lan@...ts.osuosl.org>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>, "Nguyen, Anthony L" <anthony.l.nguyen@...el.com>,
	Igor Raits <igor@...ddata.com>, Daniel Secik <daniel.secik@...ddata.com>,
	Zdenek Pesek <zdenek.pesek@...ddata.com>, "Dumazet, Eric"
	<edumazet@...gle.com>, Martin Karsten <mkarsten@...terloo.ca>, "Zaki, Ahmed"
	<ahmed.zaki@...el.com>, "Czapnik, Lukasz" <lukasz.czapnik@...el.com>, "Michal
 Swiatkowski" <michal.swiatkowski@...ux.intel.com>
Subject: RE: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE
 driver after upgrade to 6.13.y (regression in commit 492a044508ad)



> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@...osl.org> On Behalf Of Jaroslav
> Pulchart
> Sent: Wednesday, April 16, 2025 9:04 AM
> To: Jakub Kicinski <kuba@...nel.org>
> Cc: Kitszel, Przemyslaw <przemyslaw.kitszel@...el.com>; Damato, Joe
> <jdamato@...tly.com>; intel-wired-lan@...ts.osuosl.org; netdev@...r.kernel.org;
> Nguyen, Anthony L <anthony.l.nguyen@...el.com>; Igor Raits
> <igor@...ddata.com>; Daniel Secik <daniel.secik@...ddata.com>; Zdenek Pesek
> <zdenek.pesek@...ddata.com>; Dumazet, Eric <edumazet@...gle.com>; Martin
> Karsten <mkarsten@...terloo.ca>; Zaki, Ahmed <ahmed.zaki@...el.com>;
> Czapnik, Lukasz <lukasz.czapnik@...el.com>; Michal Swiatkowski
> <michal.swiatkowski@...ux.intel.com>
> Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE
> driver after upgrade to 6.13.y (regression in commit 492a044508ad)
> 
> >
> > On Wed, 16 Apr 2025 09:13:23 +0200 Jaroslav Pulchart wrote:
> > > By "traced" I mean using the kernel and checking memory situation on
> > > numa nodes with and without production load.  Numa nodes, with X810
> > > NIC, showing a quite less available memory with default queue length
> > > (num of all cpus) and it needs to be lowered to 1-2 (for unused
> > > interfaces) and up-to-count of numa node cores on used interfaces to
> > > make the memory allocation reasonable and server avoiding "kswapd"...
> > >
> > > See "MemFree" on numa 0 + 1 on different/smaller but utilized (running
> > > VMs + using network) host server with 8 numa nodes (32GB RAM each, 28G
> > > in Hugepase for VMs and 4GB for host os):
> >
> > FWIW you can also try the tools/net/ynl/samples/page-pool
> > application, not sure if Intel NICs init page pools appropriately
> > but this will show you exactly how much memory is sitting on Rx rings
> > of the driver (and in net socket buffers).
> 
> I'm not familiar with the page-pool tool, I try to build it, run it
> and nothing is shown. Any hint/menual how to use it?
> 
> >
> > > 6.13.y vanilla (lot of kswapd0 in background):
> > >     NUMA nodes:     0       1       2       3       4       5       6       7
> > >     HPTotalGiB:     28      28      28      28      28      28      28      28
> > >     HPFreeGiB:      0       0       0       0       0       0       0       0
> > >     MemTotal:       32220   32701   32701   32686   32701   32701
> > > 32701   32696
> > >     MemFree:        274     254     1327    1928    1949    2683    2624    2769
> > > 6.13.y + Revert (no memory issues at all):
> > >     NUMA nodes: 0 1 2 3 4 5 6 7
> > >     HPTotalGiB: 28 28 28 28 28 28 28 28
> > >     HPFreeGiB: 0 0 0 0 0 0 0 0
> > >     MemTotal: 32220 32701 32701 32686 32701 32701 32701 32696
> > >     MemFree: 2213 2438 3402 3108 2846 2672 2592 3063
> > >
> > > We need to lower the queue on all X810 interfaces from default (64 in
> > > this case), to ensure we have memory available for host OS services.
> > >     ethtool -L em2 combined 1
> > >     ethtool -L p3p2 combined 1
> > >     ethtool -L em1 combined 6
> > >     ethtool -L p3p1 combined 6
> > > This trick "does not work" without the revert.
> >
> > And you're reverting just and exactly 492a044508ad13 ?
> > The memory for persistent config is allocated in alloc_netdev_mqs()
> > unconditionally. I'm lost as to how this commit could make any
> > difference :(
> 
> Yes, reverted the 492a044508ad13.

Struct napi_config *is* 1056 bytes, or about 1Kb, and we will allocate one per max queue with this change, resulting in 1KB per CPU.. if there is a 64 CPU system this should be at most 64KB... That seems unlikely to be the root cause of memory outage like this is just the napi_config structure....

Perhaps something that netif_napi_restore_config is somehow causing us to end up with more allocated memory? Or some interaction with our ethtool callback to reduce the number of rings is not working properly..?