netdev - Re: "meaningful" spinlock contention when bound to non-intr CPU?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 02 Feb 2007 11:54:14 -0800
From:	Rick Jones <rick.jones2@...com>
To:	Andi Kleen <ak@...e.de>
Cc:	Linux Network Development list <netdev@...r.kernel.org>
Subject: Re: "meaningful" spinlock contention when bound to non-intr CPU?

Andi Kleen wrote:
>>The meta question behind all that would seem to be whether the scheduler 
>>should be telling us where to perform the network processing, or should 
>>the network processing be telling the scheduler what to do? (eg all my 
>>old blathering about IPS vs TOPS in HP-UX...)
> 
> 
> That's an unsolved problem.  But past experiments suggest that giving
> the scheduler more imperatives than just "use CPUs well" are often net-losses.

I wasn't thinking about giving the scheduler more imperitives really 
(?), just letting "networking" know more about where threads executed 
accessing given connections. (eg TOPS)

> I suspect it cannot be completely solved in the general case. 

Not unless the NIC can peer into the connection table and see where each 
connection was last accessed by user-space.

>>Well, yes and no.  If I drop the "burst" and instead have N times more 
>>netperf's going, I see the same lock contention situation.  I wasn't 
>>expecting to - thinking that if there were then N different processes on 
>>each CPU the likelihood of there being a contention on any one socket 
>>was low, but it was there just the same.
>>
>>That is part of what makes me wonder if there is a race between wakeup 
> 
> 
> A race?

Perhaps a poor choice of words on my part - something along the lines of:

hold_lock();
wake_up_someone();
release_lock();

where the someone being awoken can try to grab the lock before the path 
doing the waking manages to release it.

> 
> 
>>and release of a lock.
> 
> 
> You could try with echo 1 > /proc/sys/net/ipv4/tcp_low_latency.
> That should change RX locking behaviour significantly.

Running the same 8 netperf's with TCP_RR and burst bound to different 
CPU than the NIC interrupt, the lockmeter output looks virtually 
unchanged.  Still release_sock, tcp_v4_rcv, lock_sock_nested at their 
same offsets.

However, if I run the multiple-connection-per-thread code, and have each 
service 32 concurrent connections, and bind to a CPU other than the 
interrupt CPU, the lock contention in this case does appear to go away.

rick jones
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html