netdev - Re: Understanding lock contention in __udp4_lib_mcast

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJZOPZLmbMsOAPhZ6GkY8EHReNjiHbRFw8XF8J5eCdVhb5jFFA@mail.gmail.com>
Date:	Fri, 28 Jun 2013 00:39:38 +0300
From:	Or Gerlitz <or.gerlitz@...il.com>
To:	Shawn Bohrer <sbohrer@...advisors.com>
Cc:	netdev@...r.kernel.org
Subject: Re: Understanding lock contention in __udp4_lib_mcast_deliver

On Thu, Jun 27, 2013 at 10:22 PM, Shawn Bohrer <sbohrer@...advisors.com> wrote:
> I'm looking for opportunities to improve the multicast receive
> performance for our application, and I thought I'd spend some time
> trying to understand what I thought might be a small/simple
> improvement.  Profiling with perf I see that there is spin_lock
> contention in __udp4_lib_mcast_deliver:
>
> 0.68%  swapper  [kernel.kallsyms]               [k] _raw_spin_lock
>        |
>        --- _raw_spin_lock
>           |
>           |--24.13%-- perf_adjust_freq_unthr_context.part.21
>           |
>           |--22.40%-- scheduler_tick
>           |
>           |--14.96%-- __udp4_lib_mcast_deliver
>
> The lock appears to be the udp_hslot lock protecting sockets that have
> been hashed based on the local port.  My application contains multiple
> processes and each process listens on multiple multicast sockets.  We
> open one socket per multicast addr:port per machine.  All of the local
> UDP ports are unique.  The UDP hash table has 65536 entries and it
> appears the hash function is simply the port number plus a
> contribution from the network namespace (I believe the namespace part
> should be constant since we're not not using network namespaces.
> Perhaps I should disable CONFIG_NET_NS).
>
> $ dmesg | grep "UDP hash"
> [    0.606472] UDP hash table entries: 65536 (order: 9, 2097152 bytes)
>
> At this point I'm confused why there is any contention over this
> spin_lock.  We should have enough hash table entries to avoid
> collisions and all of our ports are unique.  Assuming that part is
> true I could imagine contention if the kernel is processing two
> packets for the same multicast addr:port in parallel but is that
> possible?  I'm using a Mellanox ConnectX-3 card that has multiple
> receive queues but I've always thought that all packets for a given
> srcip:srcport:destip:destport would all be delivered to a single queue
> and thus not processed in parallel.

But for a given dest mcast ip / udp port you can have multiple senders
where each has different source ucast ip / udp port, isn't that?
anyway if you disable the udp_rss module param you can be sure that
all traffic to dest mcast group goes to one ring, still this ring will
be the only one active, so I am not this is wise.

Or.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html