Message-ID: <CAK8fFZ6PPw1nshtSp+QZ_2VVWVrsCKZDdsxdPF9Tjc0=_gi=Wg@mail.gmail.com>
Date: Tue, 1 Jul 2025 08:48:04 +0200
From: Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
To: Jacob Keller <jacob.e.keller@...el.com>
Cc: Maciej Fijalkowski <maciej.fijalkowski@...el.com>, Jakub Kicinski <kuba@...nel.org>, 
	Przemek Kitszel <przemyslaw.kitszel@...el.com>, 
	"intel-wired-lan@...ts.osuosl.org" <intel-wired-lan@...ts.osuosl.org>, "Damato, Joe" <jdamato@...tly.com>, 
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>, "Nguyen, Anthony L" <anthony.l.nguyen@...el.com>, 
	Michal Swiatkowski <michal.swiatkowski@...ux.intel.com>, 
	"Czapnik, Lukasz" <lukasz.czapnik@...el.com>, "Dumazet, Eric" <edumazet@...gle.com>, 
	"Zaki, Ahmed" <ahmed.zaki@...el.com>, Martin Karsten <mkarsten@...terloo.ca>, 
	Igor Raits <igor@...ddata.com>, Daniel Secik <daniel.secik@...ddata.com>, 
	Zdenek Pesek <zdenek.pesek@...ddata.com>
Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA nodes with ICE
 driver after upgrade to 6.13.y (regression in commit 492a044508ad)

> On 6/30/2025 2:56 PM, Jacob Keller wrote:
> > Unfortunately it looks like the fix I mentioned has landed in 6.14, so
> > its not a fix for your issue (since you mentioned 6.14 has failed
> > testing in your system)
> >
> > $ git describe --first-parent --contains --match=v* --exclude=*rc*
> > 743bbd93cf29f653fae0e1416a31f03231689911
> > v6.14~251^2~15^2~2
> >
> > I don't see any other relevant changes since v6.14. I can try to see if
> > I see similar issues with CONFIG_MEM_ALLOC_PROFILING on some test
> > systems here.
>
> On my system I see this at boot after loading the ice module from
>
> $ grep -F "/ice/" /proc/allocinfo | sort -g | tail | numfmt --to=iec>
>       26K      230 drivers/net/ethernet/intel/ice/ice_irq.c:84 [ice]
> func:ice_get_irq_res
> >          48K        2 drivers/net/ethernet/intel/ice/ice_arfs.c:565 [ice] func:ice_init_arfs
> >          57K      226 drivers/net/ethernet/intel/ice/ice_lib.c:397 [ice] func:ice_vsi_alloc_ring_stats
> >          57K      226 drivers/net/ethernet/intel/ice/ice_lib.c:416 [ice] func:ice_vsi_alloc_ring_stats
> >          85K      226 drivers/net/ethernet/intel/ice/ice_lib.c:1398 [ice] func:ice_vsi_alloc_rings
> >         339K      226 drivers/net/ethernet/intel/ice/ice_lib.c:1422 [ice] func:ice_vsi_alloc_rings
> >         678K      226 drivers/net/ethernet/intel/ice/ice_base.c:109 [ice] func:ice_vsi_alloc_q_vector
> >         1.1M      257 drivers/net/ethernet/intel/ice/ice_fwlog.c:40 [ice] func:ice_fwlog_alloc_ring_buffs
> >         7.2M      114 drivers/net/ethernet/intel/ice/ice_txrx.c:493 [ice] func:ice_setup_rx_ring
> >         896M   229264 drivers/net/ethernet/intel/ice/ice_txrx.c:680 [ice] func:ice_alloc_mapped_page
>
> Its about 1GB for the mapped pages. I don't see any increase moment to
> moment. I've started an iperf session to simulate some traffic, and I'll
> leave this running to see if anything changes overnight.
>
> Is there anything else that you can share about the traffic setup or
> otherwise that I could look into?  Your system seems to use ~2.5 x the
> buffer size as mine, but that might just be a smaller number of CPUs.
>
> Hopefully I'll get some more results overnight.

The traffic is random production workloads from VMs, using standard
Linux or OVS bridges. There is no specific pattern to it. I haven’t
had any luck reproducing (or was not patient enough) this with iperf3
myself. The two active (UP) interfaces are in an LACP bonding setup.
Here are our ethtool settings for the two member ports (em1 and p3p1)

# ethtool -l em1
Channel parameters for em1:
Pre-set maximums:
RX: 64
TX: 64
Other: 1
Combined: 64
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 8

# ethtool -g em1
Ring parameters for em1:
Pre-set maximums:
RX: 8160
RX Mini: n/a
RX Jumbo: n/a
TX: 8160
TX push buff len: n/a
Current hardware settings:
RX: 8160
RX Mini: n/a
RX Jumbo: n/a
TX: 8160
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
RX Push: off
TX push buff len: n/a
TCP data split: n/a

# ethtool -c em1
Coalesce parameters for em1:
Adaptive RX: off  TX: off
stats-block-usecs: n/a
sample-interval: n/a
pkt-rate-low: n/a
pkt-rate-high: n/a

rx-usecs: 12
rx-frames: n/a
rx-usecs-irq: n/a
rx-frames-irq: n/a

tx-usecs: 28
tx-frames: n/a
tx-usecs-irq: n/a
tx-frames-irq: n/a

rx-usecs-low: n/a
rx-frame-low: n/a
tx-usecs-low: n/a
tx-frame-low: n/a

rx-usecs-high: 0
rx-frame-high: n/a
tx-usecs-high: n/a
tx-frame-high: n/a

CQE mode RX: n/a  TX: n/a

tx-aggr-max-bytes: n/a
tx-aggr-max-frames: n/a
tx-aggr-time-usecs: n/a

# ethtool -k em1
Features for em1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: on
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

# ethtool -i em1
driver: ice
version: 6.15.3-3.gdc.el9.x86_64
firmware-version: 4.51 0x8001e501 23.0.8
expansion-rom-version:
bus-info: 0000:63:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool em1
Settings for em1:
Supported ports: [ FIBRE ]
Supported link modes:   1000baseT/Full
                25000baseCR/Full
                25000baseSR/Full
                1000baseX/Full
                10000baseCR/Full
                10000baseSR/Full
                10000baseLR/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes:  25000baseCR/Full
                10000baseCR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: None RS BASER
Speed: 25000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000007 (7)
                       drv probe link
Link detected: yes


>
> Thanks,
> Jake

Powered by blists - more mailing lists