netdev - Re: [PATCH net-next v2 12/18] openvswitch: Move ovs_frag_data_storage into the struct ovs_pcpu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250416164509.FOo_r2m1@linutronix.de>
Date: Wed, 16 Apr 2025 18:45:09 +0200
From: Sebastian Andrzej Siewior <bigeasy@...utronix.de>
To: Aaron Conole <aconole@...hat.com>
Cc: netdev@...r.kernel.org, linux-rt-devel@...ts.linux.dev,
	"David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
	Simon Horman <horms@...nel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Eelco Chaudron <echaudro@...hat.com>,
	Ilya Maximets <i.maximets@....org>, dev@...nvswitch.org
Subject: Re: [PATCH net-next v2 12/18] openvswitch: Move
 ovs_frag_data_storage into the struct ovs_pcpu_storage

On 2025-04-15 12:26:13 [-0400], Aaron Conole wrote:
> I'm going to reply here, but I need to bisect a bit more (though I
> suspect the results below are due to 11/18).  When I tested with this
> patch there were lots of "unexplained" latency spikes during processing
> (note, I'm not doing PREEMPT_RT in my testing, but I guess it would
> smooth the spikes out at the cost of max performance).
> 
> With the series:
> [SUM]   0.00-300.00 sec  3.28 TBytes  96.1 Gbits/sec  9417             sender
> [SUM]   0.00-300.00 sec  3.28 TBytes  96.1 Gbits/sec                  receiver
> 
> Without the series:
> [SUM]   0.00-300.00 sec  3.26 TBytes  95.5 Gbits/sec  149             sender
> [SUM]   0.00-300.00 sec  3.26 TBytes  95.5 Gbits/sec                  receiver
> 
> And while the 'final' numbers might look acceptable, one thing I'll note
> is I saw multiple stalls as:
> 
> [  5]  57.00-58.00  sec   128 KBytes   903 Kbits/sec    0   4.02 MBytes
> 
> But without the patch, I didn't see such stalls.  My testing:
> 
> 1. Install openvswitch userspace and ipcalc
> 2. start userspace.
> 3. Setup two netns and connect them (I have a more complicated script to
>    set up the flows, and I can send that to you)
> 4. Use iperf3 to test (-P5 -t 300)
> 
> As I wrote I suspect the locking in 11 is leading to these stalls, as
> the data I'm sending shouldn't be hitting the frag path.
> 
> Do these results seem expected to you?

You have slightly better throughput but way more retries. I wouldn't
expect that. And then the stall.

Patch 10 & 12 move per-CPU variables around and makes them "static"
rather than allocating them at module init time. I would not expect this
to have a negative impact.
Patch #11 assigns the current thread to a variable and clears it again.
The remaining lockdep code disappears. The whole thing runs with BH
disabled so no preemption.

I can't explain what you observe here. Unless it is a random glitch
please send the script and I try to take a look.

Sebastian