| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CACGkMEte=LwvkxPh-tesJHLVYQV1YZF4is1Yamokhkzaf5GOWw@mail.gmail.com>
Date: Fri, 6 Feb 2026 11:21:43 +0800
From: Jason Wang <jasowang@...hat.com>
To: Simon Schippers <simon.schippers@...dortmund.de>
Cc: willemdebruijn.kernel@...il.com, andrew+netdev@...n.ch,
davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org, pabeni@...hat.com,
mst@...hat.com, eperezma@...hat.com, leiyang@...hat.com,
stephen@...workplumber.org, jon@...anix.com, tim.gebauer@...dortmund.de,
netdev@...r.kernel.org, linux-kernel@...r.kernel.org, kvm@...r.kernel.org,
virtualization@...ts.linux.dev
Subject: Re: [PATCH net-next v7 3/9] tun/tap: add ptr_ring consume helper with
netdev queue wakeup
On Fri, Feb 6, 2026 at 6:28 AM Simon Schippers
<simon.schippers@...dortmund.de> wrote:
>
> On 2/5/26 04:59, Jason Wang wrote:
> > On Wed, Feb 4, 2026 at 11:44 PM Simon Schippers
> > <simon.schippers@...dortmund.de> wrote:
> >>
> >> On 2/3/26 04:48, Jason Wang wrote:
> >>> On Mon, Feb 2, 2026 at 4:19 AM Simon Schippers
> >>> <simon.schippers@...dortmund.de> wrote:
> >>>>
> >>>> On 1/30/26 02:51, Jason Wang wrote:
> >>>>> On Thu, Jan 29, 2026 at 5:25 PM Simon Schippers
> >>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>
> >>>>>> On 1/29/26 02:14, Jason Wang wrote:
> >>>>>>> On Wed, Jan 28, 2026 at 3:54 PM Simon Schippers
> >>>>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>>>
> >>>>>>>> On 1/28/26 08:03, Jason Wang wrote:
> >>>>>>>>> On Wed, Jan 28, 2026 at 12:48 AM Simon Schippers
> >>>>>>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On 1/23/26 10:54, Simon Schippers wrote:
> >>>>>>>>>>> On 1/23/26 04:05, Jason Wang wrote:
> >>>>>>>>>>>> On Thu, Jan 22, 2026 at 1:35 PM Jason Wang <jasowang@...hat.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jan 21, 2026 at 5:33 PM Simon Schippers
> >>>>>>>>>>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 1/9/26 07:02, Jason Wang wrote:
> >>>>>>>>>>>>>>> On Thu, Jan 8, 2026 at 3:41 PM Simon Schippers
> >>>>>>>>>>>>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 1/8/26 04:38, Jason Wang wrote:
> >>>>>>>>>>>>>>>>> On Thu, Jan 8, 2026 at 5:06 AM Simon Schippers
> >>>>>>>>>>>>>>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Introduce {tun,tap}_ring_consume() helpers that wrap __ptr_ring_consume()
> >>>>>>>>>>>>>>>>>> and wake the corresponding netdev subqueue when consuming an entry frees
> >>>>>>>>>>>>>>>>>> space in the underlying ptr_ring.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Stopping of the netdev queue when the ptr_ring is full will be introduced
> >>>>>>>>>>>>>>>>>> in an upcoming commit.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Co-developed-by: Tim Gebauer <tim.gebauer@...dortmund.de>
> >>>>>>>>>>>>>>>>>> Signed-off-by: Tim Gebauer <tim.gebauer@...dortmund.de>
> >>>>>>>>>>>>>>>>>> Signed-off-by: Simon Schippers <simon.schippers@...dortmund.de>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>> drivers/net/tap.c | 23 ++++++++++++++++++++++-
> >>>>>>>>>>>>>>>>>> drivers/net/tun.c | 25 +++++++++++++++++++++++--
> >>>>>>>>>>>>>>>>>> 2 files changed, 45 insertions(+), 3 deletions(-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/net/tap.c b/drivers/net/tap.c
> >>>>>>>>>>>>>>>>>> index 1197f245e873..2442cf7ac385 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/net/tap.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/net/tap.c
> >>>>>>>>>>>>>>>>>> @@ -753,6 +753,27 @@ static ssize_t tap_put_user(struct tap_queue *q,
> >>>>>>>>>>>>>>>>>> return ret ? ret : total;
> >>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +static void *tap_ring_consume(struct tap_queue *q)
> >>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>> + struct ptr_ring *ring = &q->ring;
> >>>>>>>>>>>>>>>>>> + struct net_device *dev;
> >>>>>>>>>>>>>>>>>> + void *ptr;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + spin_lock(&ring->consumer_lock);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + ptr = __ptr_ring_consume(ring);
> >>>>>>>>>>>>>>>>>> + if (unlikely(ptr && __ptr_ring_consume_created_space(ring, 1))) {
> >>>>>>>>>>>>>>>>>> + rcu_read_lock();
> >>>>>>>>>>>>>>>>>> + dev = rcu_dereference(q->tap)->dev;
> >>>>>>>>>>>>>>>>>> + netif_wake_subqueue(dev, q->queue_index);
> >>>>>>>>>>>>>>>>>> + rcu_read_unlock();
> >>>>>>>>>>>>>>>>>> + }
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + spin_unlock(&ring->consumer_lock);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + return ptr;
> >>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> static ssize_t tap_do_read(struct tap_queue *q,
> >>>>>>>>>>>>>>>>>> struct iov_iter *to,
> >>>>>>>>>>>>>>>>>> int noblock, struct sk_buff *skb)
> >>>>>>>>>>>>>>>>>> @@ -774,7 +795,7 @@ static ssize_t tap_do_read(struct tap_queue *q,
> >>>>>>>>>>>>>>>>>> TASK_INTERRUPTIBLE);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> /* Read frames from the queue */
> >>>>>>>>>>>>>>>>>> - skb = ptr_ring_consume(&q->ring);
> >>>>>>>>>>>>>>>>>> + skb = tap_ring_consume(q);
> >>>>>>>>>>>>>>>>>> if (skb)
> >>>>>>>>>>>>>>>>>> break;
> >>>>>>>>>>>>>>>>>> if (noblock) {
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/net/tun.c b/drivers/net/tun.c
> >>>>>>>>>>>>>>>>>> index 8192740357a0..7148f9a844a4 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/net/tun.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/net/tun.c
> >>>>>>>>>>>>>>>>>> @@ -2113,13 +2113,34 @@ static ssize_t tun_put_user(struct tun_struct *tun,
> >>>>>>>>>>>>>>>>>> return total;
> >>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +static void *tun_ring_consume(struct tun_file *tfile)
> >>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>> + struct ptr_ring *ring = &tfile->tx_ring;
> >>>>>>>>>>>>>>>>>> + struct net_device *dev;
> >>>>>>>>>>>>>>>>>> + void *ptr;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + spin_lock(&ring->consumer_lock);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> + ptr = __ptr_ring_consume(ring);
> >>>>>>>>>>>>>>>>>> + if (unlikely(ptr && __ptr_ring_consume_created_space(ring, 1))) {
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I guess it's the "bug" I mentioned in the previous patch that leads to
> >>>>>>>>>>>>>>>>> the check of __ptr_ring_consume_created_space() here. If it's true,
> >>>>>>>>>>>>>>>>> another call to tweak the current API.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> + rcu_read_lock();
> >>>>>>>>>>>>>>>>>> + dev = rcu_dereference(tfile->tun)->dev;
> >>>>>>>>>>>>>>>>>> + netif_wake_subqueue(dev, tfile->queue_index);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> This would cause the producer TX_SOFTIRQ to run on the same cpu which
> >>>>>>>>>>>>>>>>> I'm not sure is what we want.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> What else would you suggest calling to wake the queue?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't have a good method in my mind, just want to point out its implications.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have to admit I'm a bit stuck at this point, particularly with this
> >>>>>>>>>>>>>> aspect.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What is the correct way to pass the producer CPU ID to the consumer?
> >>>>>>>>>>>>>> Would it make sense to store smp_processor_id() in the tfile inside
> >>>>>>>>>>>>>> tun_net_xmit(), or should it instead be stored in the skb (similar to the
> >>>>>>>>>>>>>> XDP bit)? In the latter case, my concern is that this information may
> >>>>>>>>>>>>>> already be significantly outdated by the time it is used.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Based on that, my idea would be for the consumer to wake the producer by
> >>>>>>>>>>>>>> invoking a new function (e.g., tun_wake_queue()) on the producer CPU via
> >>>>>>>>>>>>>> smp_call_function_single().
> >>>>>>>>>>>>>> Is this a reasonable approach?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm not sure but it would introduce costs like IPI.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> More generally, would triggering TX_SOFTIRQ on the consumer CPU be
> >>>>>>>>>>>>>> considered a deal-breaker for the patch set?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It depends on whether or not it has effects on the performance.
> >>>>>>>>>>>>> Especially when vhost is pinned.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I meant we can benchmark to see the impact. For example, pin vhost to
> >>>>>>>>>>>> a specific CPU and the try to see the impact of the TX_SOFTIRQ.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I ran benchmarks with vhost pinned to CPU 0 using taskset -p -c 0 ...
> >>>>>>>>>>> for both the stock and patched versions. The benchmarks were run with
> >>>>>>>>>>> the full patch series applied, since testing only patches 1-3 would not
> >>>>>>>>>>> be meaningful - the queue is never stopped in that case, so no
> >>>>>>>>>>> TX_SOFTIRQ is triggered.
> >>>>>>>>>>>
> >>>>>>>>>>> Compared to the non-pinned CPU benchmarks in the cover letter,
> >>>>>>>>>>> performance is lower for pktgen with a single thread but higher with
> >>>>>>>>>>> four threads. The results show no regression for the patched version,
> >>>>>>>>>>> with even slight performance improvements observed:
> >>>>>>>>>>>
> >>>>>>>>>>> +-------------------------+-----------+----------------+
> >>>>>>>>>>> | pktgen benchmarks to | Stock | Patched with |
> >>>>>>>>>>> | Debian VM, i5 6300HQ, | | fq_codel qdisc |
> >>>>>>>>>>> | 100M packets | | |
> >>>>>>>>>>> | vhost pinned to core 0 | | |
> >>>>>>>>>>> +-----------+-------------+-----------+----------------+
> >>>>>>>>>>> | TAP | Transmitted | 452 Kpps | 454 Kpps |
> >>>>>>>>>>> | + +-------------+-----------+----------------+
> >>>>>>>>>>> | vhost-net | Lost | 1154 Kpps | 0 |
> >>>>>>>>>>> +-----------+-------------+-----------+----------------+
> >>>>>>>>>>>
> >>>>>>>>>>> +-------------------------+-----------+----------------+
> >>>>>>>>>>> | pktgen benchmarks to | Stock | Patched with |
> >>>>>>>>>>> | Debian VM, i5 6300HQ, | | fq_codel qdisc |
> >>>>>>>>>>> | 100M packets | | |
> >>>>>>>>>>> | vhost pinned to core 0 | | |
> >>>>>>>>>>> | *4 threads* | | |
> >>>>>>>>>>> +-----------+-------------+-----------+----------------+
> >>>>>>>>>>> | TAP | Transmitted | 71 Kpps | 79 Kpps |
> >>>>>>>>>>> | + +-------------+-----------+----------------+
> >>>>>>>>>>> | vhost-net | Lost | 1527 Kpps | 0 |
> >>>>>>>>>>> +-----------+-------------+-----------+----------------+
> >>>>>>>>>
> >>>>>>>>> The PPS seems to be low. I'd suggest using testpmd (rxonly) mode in
> >>>>>>>>> the guest or an xdp program that did XDP_DROP in the guest.
> >>>>>>>>
> >>>>>>>> I forgot to mention that these PPS values are per thread.
> >>>>>>>> So overall we have 71 Kpps * 4 = 284 Kpps and 79 Kpps * 4 = 326 Kpps,
> >>>>>>>> respectively. For packet loss, that comes out to 1154 Kpps * 4 =
> >>>>>>>> 4616 Kpps and 0, respectively.
> >>>>>>>>
> >>>>>>>> Sorry about that!
> >>>>>>>>
> >>>>>>>> The pktgen benchmarks with a single thread look fine, right?
> >>>>>>>
> >>>>>>> Still looks very low. E.g I just have a run of pktgen (using
> >>>>>>> pktgen_sample03_burst_single_flow.sh) without a XDP_DROP in the guest,
> >>>>>>> I can get 1Mpps.
> >>>>>>
> >>>>>> Keep in mind that I am using an older CPU (i5-6300HQ). For the
> >>>>>> single-threaded tests I always used pktgen_sample01_simple.sh, and for
> >>>>>> the multi-threaded tests I always used pktgen_sample02_multiqueue.sh.
> >>>>>>
> >>>>>> Using pktgen_sample03_burst_single_flow.sh as you did fails for me (even
> >>>>>> though the same parameters work fine for sample01 and sample02):
> >>>>>>
> >>>>>> samples/pktgen/pktgen_sample03_burst_single_flow.sh -i tap0 -m
> >>>>>> 52:54:00:12:34:56 -d 10.0.0.2 -n 100000000
> >>>>>> /samples/pktgen/functions.sh: line 79: echo: write error: Operation not
> >>>>>> supported
> >>>>>> ERROR: Write error(1) occurred
> >>>>>> cmd: "burst 32 > /proc/net/pktgen/tap0@0"
> >>>>>>
> >>>>>> ...and I do not know what I am doing wrong, even after looking at
> >>>>>> Documentation/networking/pktgen.rst. Every burst size except 1 fails.
> >>>>>> Any clues?
> >>>>>
> >>>>> Please use -b 0, and I'm Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz.
> >>>>
> >>>> I tried using "-b 0", and while it worked, there was no noticeable
> >>>> performance improvement.
> >>>>
> >>>>>
> >>>>> Another thing I can think of is to disable
> >>>>>
> >>>>> 1) mitigations in both guest and host
> >>>>> 2) any kernel debug features in both host and guest
> >>>>
> >>>> I also rebuilt the kernel with everything disabled under
> >>>> "Kernel hacking", but that didn’t make any difference either.
> >>>>
> >>>> Because of this, I ran "pktgen_sample01_simple.sh" and
> >>>> "pktgen_sample02_multiqueue.sh" on my AMD Ryzen 5 5600X system. The
> >>>> results were about 374 Kpps with TAP and 1192 Kpps with TAP+vhost_net,
> >>>> with very similar performance between the stock and patched kernels.
> >>>>
> >>>> Personally, I think the low performance is to blame on the hardware.
> >>>
> >>> Let's double confirm this by:
> >>>
> >>> 1) make sure pktgen is using 100% CPU
> >>> 2) Perf doesn't show anything strange for pktgen thread
> >>>
> >>> Thanks
> >>>
> >>
> >> I ran pktgen using pktgen_sample01_simple.sh and, in parallel, started a
> >> 100 second perf stat measurement covering all kpktgend threads.
> >>
> >> Across all configurations, a single CPU was fully utilized.
> >>
> >> Apart from that, the patched variants show a higher branch frequency and
> >> a slightly increased number of context switches.
> >>
> >>
> >> The detailed results are provided below:
> >>
> >> Processor: Ryzen 5 5600X
> >>
> >> pktgen command:
> >> sudo perf stat samples/pktgen/pktgen_sample01_simple.sh -i tap0 -m
> >> 52:54:00:12:34:56 -d 10.0.0.2 -n 10000000000
> >>
> >> perf stat command:
> >> sudo perf stat --timeout 100000 -p $(pgrep kpktgend | tr '\n' ,) -o X.txt
> >>
> >>
> >> Results:
> >> Stock TAP:
> >> 46.997 context-switches # 467,2 cs/sec cs_per_second
> >> 0 cpu-migrations # 0,0 migrations/sec migrations_per_second
> >> 0 page-faults # 0,0 faults/sec page_faults_per_second
> >> 100.587,69 msec task-clock # 1,0 CPUs CPUs_utilized
> >> 8.491.586.483 branch-misses # 10,9 % branch_miss_rate (50,24%)
> >> 77.734.761.406 branches # 772,8 M/sec branch_frequency (66,85%)
> >> 382.420.291.585 cpu-cycles # 3,8 GHz cycles_frequency (66,85%)
> >> 377.612.185.141 instructions # 1,0 instructions insn_per_cycle (66,85%)
> >> 84.012.185.936 stalled-cycles-frontend # 0,22 frontend_cycles_idle (66,35%)
> >>
> >> 100,100414494 seconds time elapsed
> >>
> >>
> >> Stock TAP+vhost-net:
> >> 47.087 context-switches # 468,1 cs/sec cs_per_second
> >> 0 cpu-migrations # 0,0 migrations/sec migrations_per_second
> >> 0 page-faults # 0,0 faults/sec page_faults_per_second
> >> 100.594,09 msec task-clock # 1,0 CPUs CPUs_utilized
> >> 8.034.703.613 branch-misses # 11,1 % branch_miss_rate (50,24%)
> >> 72.477.989.922 branches # 720,5 M/sec branch_frequency (66,86%)
> >> 382.218.276.832 cpu-cycles # 3,8 GHz cycles_frequency (66,85%)
> >> 349.555.577.281 instructions # 0,9 instructions insn_per_cycle (66,85%)
> >> 83.917.644.262 stalled-cycles-frontend # 0,22 frontend_cycles_idle (66,35%)
> >>
> >> 100,100520402 seconds time elapsed
> >>
> >>
> >> Patched TAP:
> >> 47.862 context-switches # 475,8 cs/sec cs_per_second
> >> 0 cpu-migrations # 0,0 migrations/sec migrations_per_second
> >> 0 page-faults # 0,0 faults/sec page_faults_per_second
> >> 100.589,30 msec task-clock # 1,0 CPUs CPUs_utilized
> >> 9.337.258.794 branch-misses # 9,4 % branch_miss_rate (50,19%)
> >> 99.518.421.676 branches # 989,4 M/sec branch_frequency (66,85%)
> >> 382.508.244.894 cpu-cycles # 3,8 GHz cycles_frequency (66,85%)
> >> 312.582.270.975 instructions # 0,8 instructions insn_per_cycle (66,85%)
> >> 76.338.503.984 stalled-cycles-frontend # 0,20 frontend_cycles_idle (66,39%)
> >>
> >> 100,101262454 seconds time elapsed
> >>
> >>
> >> Patched TAP+vhost-net:
> >> 47.892 context-switches # 476,1 cs/sec cs_per_second
> >> 0 cpu-migrations # 0,0 migrations/sec migrations_per_second
> >> 0 page-faults # 0,0 faults/sec page_faults_per_second
> >> 100.581,95 msec task-clock # 1,0 CPUs CPUs_utilized
> >> 9.083.588.313 branch-misses # 10,1 % branch_miss_rate (50,28%)
> >> 90.300.124.712 branches # 897,8 M/sec branch_frequency (66,85%)
> >> 382.374.510.376 cpu-cycles # 3,8 GHz cycles_frequency (66,85%)
> >> 340.089.181.199 instructions # 0,9 instructions insn_per_cycle (66,85%)
> >> 78.151.408.955 stalled-cycles-frontend # 0,20 frontend_cycles_idle (66,31%)
> >>
> >> 100,101212911 seconds time elapsed
> >
> > Thanks for sharing. I have more questions:
> >
> > 1) The number of CPU and vCPUs
>
> qemu runs with a single core. And my host system is now a Ryzen 5 5600x
> with 6 cores, 12 threads.
> This is my command for TAP+vhost-net:
>
> sudo qemu-system-x86_64 -hda debian.qcow2
> -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no,vhost=on
> -device virtio-net-pci,netdev=mynet0 -m 1024 -enable-kvm
>
> For TAP only it is the same but without vhost=on.
>
> > 2) If you pin vhost or vCPU threads
>
> Not in the previous shown benchmark. I pinned vhost in other benchmarks
> but since there is only minor PPS difference I omitted for the sake of
> simplicity.
>
> > 3) what does perf top looks like or perf top -p $pid_of_vhost
>
> The perf reports for the pid_of_vhost from pktgen_sample01_simple.sh
> with TAP+vhost-net (not pinned, pktgen single queue, fq_codel) are shown
> below. I can not see a huge difference between stock and patched.
>
> Also I included perf reports from the pktgen_pids. I find them more
> intersting because tun_net_xmit shows less overhead for the patched.
> I assume that is due to the stopped netdev queue.
>
> I have now benchmarked pretty much all possible combinations (with a
> script) of TAP/TAP+vhost-net, single/multi-queue pktgen, vhost
> pinned/not pinned, with/without -b 0, fq_codel/noqueue... All of that
> with perf records..
> I could share them if you want but I feel this is getting out of hand.
>
>
> Stock:
> sudo perf record -p "$vhost_pid"
> ...
> # Overhead Command Shared Object Symbol
> # ........ ............... .......................... ..........................................
> #
> 5.97% vhost-4874 [kernel.kallsyms] [k] _copy_to_iter
> 2.68% vhost-4874 [kernel.kallsyms] [k] tun_do_read
> 2.23% vhost-4874 [kernel.kallsyms] [k] native_write_msr
> 1.93% vhost-4874 [kernel.kallsyms] [k] __check_object_size
Let's disable CONFIG_HARDENED_USERCOPY and retry.
> 1.61% vhost-4874 [kernel.kallsyms] [k] __slab_free.isra.0
> 1.56% vhost-4874 [kernel.kallsyms] [k] __get_user_nocheck_2
> 1.54% vhost-4874 [kernel.kallsyms] [k] iov_iter_zero
> 1.45% vhost-4874 [kernel.kallsyms] [k] kmem_cache_free
> 1.43% vhost-4874 [kernel.kallsyms] [k] tun_recvmsg
> 1.24% vhost-4874 [kernel.kallsyms] [k] sk_skb_reason_drop
> 1.12% vhost-4874 [kernel.kallsyms] [k] srso_alias_safe_ret
> 1.07% vhost-4874 [kernel.kallsyms] [k] native_read_msr
> 0.76% vhost-4874 [kernel.kallsyms] [k] simple_copy_to_iter
> 0.75% vhost-4874 [kernel.kallsyms] [k] srso_alias_return_thunk
> 0.69% vhost-4874 [vhost] [k] 0x0000000000002e70
> 0.59% vhost-4874 [kernel.kallsyms] [k] skb_release_data
> 0.59% vhost-4874 [kernel.kallsyms] [k] __skb_datagram_iter
> 0.53% vhost-4874 [vhost] [k] 0x0000000000002e5f
> 0.51% vhost-4874 [kernel.kallsyms] [k] slab_update_freelist.isra.0
> 0.46% vhost-4874 [kernel.kallsyms] [k] kfree_skbmem
> 0.44% vhost-4874 [kernel.kallsyms] [k] skb_copy_datagram_iter
> 0.43% vhost-4874 [kernel.kallsyms] [k] skb_free_head
> 0.37% qemu-system-x86 [unknown] [k] 0xffffffffba898b1b
> 0.35% vhost-4874 [vhost] [k] 0x0000000000002e6b
> 0.33% vhost-4874 [vhost_net] [k] 0x000000000000357d
> 0.28% vhost-4874 [kernel.kallsyms] [k] __check_heap_object
> 0.27% vhost-4874 [vhost_net] [k] 0x00000000000035f3
> 0.26% vhost-4874 [vhost_net] [k] 0x00000000000030f6
> 0.26% vhost-4874 [kernel.kallsyms] [k] __virt_addr_valid
> 0.24% vhost-4874 [kernel.kallsyms] [k] iov_iter_advance
> 0.22% vhost-4874 [kernel.kallsyms] [k] perf_event_update_userpage
> 0.22% vhost-4874 [kernel.kallsyms] [k] check_stack_object
> 0.19% qemu-system-x86 [unknown] [k] 0xffffffffba2a68cd
> 0.19% vhost-4874 [kernel.kallsyms] [k] dequeue_entities
> 0.19% vhost-4874 [vhost_net] [k] 0x0000000000003237
> 0.18% vhost-4874 [vhost_net] [k] 0x0000000000003550
> 0.18% vhost-4874 [kernel.kallsyms] [k] x86_pmu_del
> 0.18% vhost-4874 [vhost_net] [k] 0x00000000000034a0
> 0.17% vhost-4874 [kernel.kallsyms] [k] x86_pmu_disable_all
> 0.16% vhost-4874 [vhost_net] [k] 0x0000000000003523
> 0.16% vhost-4874 [kernel.kallsyms] [k] amd_pmu_addr_offset
> ...
>
>
> sudo perf record -p "$kpktgend_pids":
> ...
> # Overhead Command Shared Object Symbol
> # ........ ........... ................. ...............................................
> #
> 10.98% kpktgend_0 [kernel.kallsyms] [k] tun_net_xmit
> 10.45% kpktgend_0 [kernel.kallsyms] [k] memset
> 8.40% kpktgend_0 [kernel.kallsyms] [k] __alloc_skb
> 6.31% kpktgend_0 [kernel.kallsyms] [k] kmem_cache_alloc_node_noprof
> 3.13% kpktgend_0 [kernel.kallsyms] [k] srso_alias_safe_ret
> 2.40% kpktgend_0 [kernel.kallsyms] [k] sk_skb_reason_drop
> 2.11% kpktgend_0 [kernel.kallsyms] [k] srso_alias_return_thunk
This is a hint that SRSO migitaion is enabled.
Have you disabled CPU_MITIGATIONS via either Kconfig or kernel command
line (mitigations = off) for both host and guest?
Thanks
Powered by blists - more mailing lists