netdev - Re: big picture UDP/IP performance question re 2.6.18 -> 2.6.32

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Fri, 07 Oct 2011 21:07:03 +0100
From:	Ben Hutchings <bhutchings@...arflare.com>
To:	starlight@...nacle.cx
Cc:	chetan loke <loke.chetan@...il.com>,
	Eric Dumazet <eric.dumazet@...il.com>,
	linux-kernel@...r.kernel.org, netdev <netdev@...r.kernel.org>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Christoph Lameter <cl@...two.org>, Willy Tarreau <w@....eu>,
	Ingo Molnar <mingo@...e.hu>,
	Stephen Hemminger <stephen.hemminger@...tta.com>,
	Benjamin LaHaise <bcrl@...ck.org>,
	Joe Perches <joe@...ches.com>, lokechetan@...il.com,
	Con Kolivas <conman@...ivas.org>,
	Serge Belyshev <belyshev@...ni.sinp.msu.ru>
Subject: Re: big picture UDP/IP performance question re 2.6.18 -> 2.6.32

On Fri, 2011-10-07 at 14:37 -0400, starlight@...nacle.cx wrote:
> At 02:09 PM 10/7/2011 -0400, chetan loke wrote:
[...]
> b) Use non-commodity(?) NICs(from vendors
> >you mentioned): where it might have some on-board
> >memory(cushion) and so it can absorb the spikes
> >and can also smoothen out too many
> >PCI-transactions for bursty (and small payload -
> >as in 64 byte traffic). But wait, when you use the
> >libs provided by these vendors, then their
> >driver(especially the Rx path) is not so much
> >working in inline mode as NIC drivers in case a)
> >above. This driver with a special Rx-path purely
> >exists for managing your mmap'd queues.So
> >of-course it's going to be faster that the
> >traditional inline drivers. In this partial-inline
> >mode, the adapter might i) batch the packets and
> >ii) send a single notification to the
> >host-side. With that single event you are now
> >processing 1+ packets.

Solarflare's user-mode queues have their own wakeup moderation timers,
similar to interrupt moderation timers.

Note also that NICs designed for user-mode networking may allow DMA
descriptor/event rings and doorbell registers to be mapped into user
processes.  This allows packets to be passed continuously without any
need to call into the kernel.

> Kernel bypass is probably the best answer for
> what we do.  Problem has been lack of maturity
> in their driver software.  Looks like it's reaching
> a point where they cover our use case.  As mentioned
> earlier, Solarflare could not match the Intel
> 82599 + ixgbe for this app last year.  Was a
> disaster.

Ouch.  What's the application?

> Myricom is focused on UDP (better
> for us), but only just added multi-core IRQ
> doorbell wakeups in recent months.  Previously
> one had to accept all IRQs on a single core or
> poll, neither of which works for us.
[...]

Our hardware does support spreading wakeups for user-mode queues across
multiple kernel-mode event queues (hence multiple CPUs), although not in
a very flexible way.  I believe most Onload users prefer to keep their
threads polling for events rather than waiting for wakeups, though, so
it doesn't yet use this feature.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html