Message-ID: <f445633e-b72c-4b5d-bb18-acda1c1d4de6@kernel.org>
Date: Wed, 18 Jun 2025 14:46:43 +0200
From: Jesper Dangaard Brouer <hawk@...nel.org>
To: Mina Almasry <almasrymina@...gle.com>
Cc: netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
 linux-kselftest@...r.kernel.org, "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>,
 Shuah Khan <shuah@...nel.org>, Ilias Apalodimas
 <ilias.apalodimas@...aro.org>, Toke Høiland-Jørgensen
 <toke@...e.dk>, Ignat Korchagin <ignat@...udflare.com>
Subject: Re: [PATCH net-next v4] page_pool: import Jesper's page_pool
 benchmark



On 16/06/2025 23.11, Mina Almasry wrote:
> On Mon, Jun 16, 2025 at 2:29 AM Jesper Dangaard Brouer <hawk@...nel.org> wrote:
>> On 15/06/2025 22.59, Mina Almasry wrote:
>>> From: Jesper Dangaard Brouer <hawk@...nel.org>
>>>
>>> We frequently consult with Jesper's out-of-tree page_pool benchmark to
>>> evaluate page_pool changes.
>>>
>>> Import the benchmark into the upstream linux kernel tree so that (a)
>>> we're all running the same version, (b) pave the way for shared
>>> improvements, and (c) maybe one day integrate it with nipa, if possible.
>>>
>>> Import bench_page_pool_simple from commit 35b1716d0c30 ("Add
>>> page_bench06_walk_all"), from this repository:
>>> https://github.com/netoptimizer/prototype-kernel.git
>>>
>>> Changes done during upstreaming:
>>> - Fix checkpatch issues.
>>> - Remove the tasklet logic not needed.
>>> - Move under tools/testing
>>> - Create ksft for the benchmark.
>>> - Changed slightly how the benchmark gets build. Out of tree, time_bench
>>>     is built as an independent .ko. Here it is included in
>>>     bench_page_pool.ko
>>>
>>> Steps to run:
>>>
>>> ```
>>> mkdir -p /tmp/run-pp-bench
>>> make -C ./tools/testing/selftests/net/bench
>>> make -C ./tools/testing/selftests/net/bench install INSTALL_PATH=/tmp/run-pp-bench
>>> rsync --delete -avz --progress /tmp/run-pp-bench mina@...RVER:~/
>>> ssh mina@...RVER << EOF
>>>     cd ~/run-pp-bench && sudo ./test_bench_page_pool.sh
>>> EOF
>>> ```
>>>
>>> Output:
>>>
>>> ```
>>> (benchmrk dmesg logs)
>>>
>>
>> Something is off with benchmark numbers compared to the OOT version.
>>
> 
> I assume you're comparing my results (my kernel config + my hardware +
> upstream benchmark) with your results (your kernel config + your
> hardware + OOT version). The problem may be in OOT vs upstream but it
> may be just different code/config/hardware.

True I used OOT version.

Just applied this patch, but I get compile error. Because Makefile tries
to get kernel headers (net/page_pool/helpers.h) from local Linux
installation instead of git tree.  This need to be adjusted for patch,
such that it builds with src-local/git tree provided headers.

$ make -C ./tools/testing/selftests/net/bench/page_pool/
make: Entering directory 
'/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool'
make[1]: Entering directory '/usr/src/kernels/6.5.12-100.fc37.x86_64'
   CC [M] 
/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool/bench_page_pool_simple.o
/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool/bench_page_pool_simple.c:12:10: 
fatal error: net/page_pool/helpers.h: No such file or directory
    12 | #include <net/page_pool/helpers.h>
       |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [scripts/Makefile.build:243: 
/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool/bench_page_pool_simple.o] 
Error 1
make[2]: *** [/usr/src/kernels/6.5.12-100.fc37.x86_64/Makefile:2046: 
/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool] 
Error 2
make[1]: *** [Makefile:246: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/kernels/6.5.12-100.fc37.x86_64'
make: *** [Makefile:14: all] Error 2
make: Leaving directory 
'/home/jbrouer/git/kernel/net/tools/testing/selftests/net/bench/page_pool'


>> Adding my numbers below, they were run on my testlab with:
>>    - CPU E5-1650 v4 @ 3.60GHz
>>    - kernel: net.git v6.15-12438-gd9816ec74e6d
>>
>>> Fast path results:
>>> no-softirq-page_pool01 Per elem: 11 cycles(tsc) 4.368 ns
>>>
>>
>> Fast-path on your CPU is faster (22 cycles(tsc) 6.128 ns) than my CPU.
>> What CPU is this?
> 
> My test setup is a Gcloud A3 VM (so virtualized). The CPU is:
> 
> cat /proc/cpuinfo
> ...
> model name      : Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz
> 
>>
>> Type:no-softirq-page_pool01 Per elem: 22 cycles(tsc) 6.128 ns (step:0)
>>    - (measurement period time:0.061282924 sec time_interval:61282924)
>>    - (invoke count:10000000 tsc_interval:220619745)
>>
>>> ptr_ring results:
>>> no-softirq-page_pool02 Per elem: 527 cycles(tsc) 195.187 ns
>>
>> I'm surprised that ptr_ring benchmark is very slow, compared to my
>> result (below) 60 cycles(tsc) 16.853 ns.
>>
>> Type:no-softirq-page_pool02 Per elem: 60 cycles(tsc) 16.853 ns (step:0)
>>    - (measurement period time:0.168535760 sec time_interval:168535760)
>>    - (invoke count:10000000 tsc_interval:606734160)
>>
>> Maybe your kernel is compiled with some CONFIG debug thing that makes it
>> slower?
>>
> 
> Yeah, I actually just checked and I have CONFIG_DEBUG_NET on in my
> build, and a lot of other debug configs are turned on.
> 

The CONFIG_DEBUG_NET should be low overhead, so I don't expect this to
be the root-cause.  Other CONFIG options are more likely the issue.

> Let me investigate here. Maybe trimming the debug configs and double
> checking my tree for debug logs I added would point to the difference.
>

The perf record trick below will help you identify overhead more rabidly 
IMHO.


> I could also try to put both the OOT version and upstream version in
> my tree and do a proper A/B comparison that way.
> 
> If you do get chance to run this upstream version from your exact tree
> and config, that would be a good A/B comparison as well.
> 
>> You can troubleshoot like this:
>>    - select the `no-softirq-page_pool02` test via run_flags=$((2#100)).
>>
>>    # perf record -g modprobe bench_page_pool_simple run_flags=$((2#100))
>> loops=$((100*10**6))
>>    # perf report --no-children
>>
> 
> Thanks, will do.

With a micro-benchmark like this, the perf report should help you to
quickly indentify the kernel debug CONFIG that adds the overhead.

--Jesper

Powered by blists - more mailing lists