[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <aIuepME92Q9iR22Z@861G6M3>
Date: Thu, 31 Jul 2025 11:49:40 -0500
From: Chris Arges <carges@...udflare.com>
To: Jesper Dangaard Brouer <hawk@...nel.org>
Cc: Dragos Tatulea <dtatulea@...dia.com>,
Dipayaan Roy <dipayanroy@...ux.microsoft.com>, horms@...nel.org,
kuba@...nel.org, kys@...rosoft.com, haiyangz@...rosoft.com,
wei.liu@...nel.org, decui@...rosoft.com, andrew+netdev@...n.ch,
davem@...emloft.net, edumazet@...gle.com, pabeni@...hat.com,
longli@...rosoft.com, kotaranov@...rosoft.com, ast@...nel.org,
daniel@...earbox.net, john.fastabend@...il.com, sdf@...ichev.me,
lorenzo@...nel.org, michal.kubiak@...el.com,
ernis@...ux.microsoft.com, shradhagupta@...ux.microsoft.com,
shirazsaleem@...rosoft.com, rosenp@...il.com,
netdev@...r.kernel.org, linux-hyperv@...r.kernel.org,
linux-rdma@...r.kernel.org, bpf@...r.kernel.org,
linux-kernel@...r.kernel.org, ssengar@...ux.microsoft.com,
dipayanroy@...rosoft.com, kernel-team <kernel-team@...udflare.com>,
Tariq Toukan <tariqt@...dia.com>,
Saeed Mahameed <saeedm@...dia.com>,
Yunsheng Lin <linyunsheng@...wei.com>
Subject: Re: [PATCH v2] net: mana: Use page pool fragments for RX buffers
instead of full pages to improve memory efficiency.
On 2025-07-31 18:36:04, Jesper Dangaard Brouer wrote:
>
>
> On 30/07/2025 09.31, Dragos Tatulea wrote:
> > On Tue, Jul 29, 2025 at 01:20:07PM -0700, Dipayaan Roy wrote:
> > > On Tue, Jul 29, 2025 at 12:15:23PM +0200, Jesper Dangaard Brouer wrote:
> > > >
> > > >
> > > > On 23/07/2025 21.07, Dipayaan Roy wrote:
> > > > > This patch enhances RX buffer handling in the mana driver by allocating
> > > > > pages from a page pool and slicing them into MTU-sized fragments, rather
> > > > > than dedicating a full page per packet. This approach is especially
> > > > > beneficial on systems with large page sizes like 64KB.
> > > > >
> > > > > Key improvements:
> > > > >
> > > > > - Proper integration of page pool for RX buffer allocations.
> > > > > - MTU-sized buffer slicing to improve memory utilization.
> > > > > - Reduce overall per Rx queue memory footprint.
> > > > > - Automatic fallback to full-page buffers when:
> > > > > * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> > > > > * The XDP path is active, to avoid complexities with fragment reuse.
> > > > > - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> > > > > changes, ensuring consistency in RX buffer allocation.
> > > > >
> > > > > Testing on VMs with 64KB pages shows around 200% throughput improvement.
> > > > > Memory efficiency is significantly improved due to reduced wastage in page
> > > > > allocations. Example: We are now able to fit 35 rx buffers in a single 64kb
> > > > > page for MTU size of 1500, instead of 1 rx buffer per page previously.
> > > > >
> > > > > Tested:
> > > > >
> > > > > - iperf3, iperf2, and nttcp benchmarks.
> > > > > - Jumbo frames with MTU 9000.
> > > > > - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
> > > > > testing the XDP path in driver.
> > > > > - Page leak detection (kmemleak).
> > > > > - Driver load/unload, reboot, and stress scenarios.
> > > >
> > > > Chris (Cc) discovered a crash/bug[1] with page pool fragments used
> > > > from the mlx5 driver.
> > > > He put together a BPF program that reproduces the issue here:
> > > > - [2] https://github.com/arges/xdp-redirector
> > > >
> > > > Can I ask you to test that your driver against this reproducer?
> > > >
> > > >
> > > > [1] https://lore.kernel.org/all/aIEuZy6fUj_4wtQ6@861G6M3/
> > > >
> > > > --Jesper
> > > >
> > >
> > > Hi Jesper,
> > >
> > > I was unable to reproduce this issue on mana driver.
> > >
> > Please note that I had to make a few adjustments to get reprodduction on
> > mlx5:
> >
> > - Make sure that the veth MACs are recognized by the device. Otherwise
> > traffic might be dropped by the device.
> >
> > - Enable GRO on the veth device. Otherwise packets get dropped before
> > they reach the devmap BPF program.
> >
> > Try starting the test program with one thread and see if you see packets
> > coming through veth1-ns1 end of the veth pair.
> >
>
> Hi Dipayaan,
>
> Enabling GRO on the veth device is quite important for the test to be valid.
>
> I've asked Chris to fix this in the reproducer. He can report back when
> he have done this, so you can re-run the test. It is also good advice
> from Dragos that you should check packets are coming through the veth
> pair, to make sure the test is working.
>
> The setup.sh script also need to be modified, as it is loading xdp on a
> net_device called "ext0" [0], which is specific to our systems (which
> default also have GRO enabled for veth).
>
> [0] https://github.com/arges/xdp-redirector/blob/main/setup.sh#L28
>
> --Jesper
I pushed some updates to the setup script to make it easier to use. If you have
issues running the script, please share the output.
--chris
Powered by blists - more mailing lists