[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <470ADF15.2090100@garzik.org>
Date: Mon, 08 Oct 2007 21:53:25 -0400
From: Jeff Garzik <jeff@...zik.org>
To: David Miller <davem@...emloft.net>
CC: hadi@...erus.ca, peter.p.waskiewicz.jr@...el.com,
krkumar2@...ibm.com, johnpol@....mipt.ru,
herbert@...dor.apana.org.au, kaber@...sh.net,
shemminger@...ux-foundation.org, jagana@...ibm.com,
Robert.Olsson@...a.slu.se, rick.jones2@...com, xma@...ibm.com,
gaagaan@...il.com, netdev@...r.kernel.org, rdreier@...co.com,
mingo@...e.hu, mchan@...adcom.com, general@...ts.openfabrics.org,
kumarkr@...ux.ibm.com, tgraf@...g.ch, randy.dunlap@...cle.com,
sri@...ibm.com, linux-kernel@...r.kernel.org
Subject: Re: parallel networking
David Miller wrote:
> From: Jeff Garzik <jeff@...zik.org>
> Date: Mon, 08 Oct 2007 10:22:28 -0400
>
>> In terms of overall parallelization, both for TX as well as RX, my gut
>> feeling is that we want to move towards an MSI-X, multi-core friendly
>> model where packets are LIKELY to be sent and received by the same set
>> of [cpus | cores | packages | nodes] that the [userland] processes
>> dealing with the data.
>
> The problem is that the packet schedulers want global guarantees
> on packet ordering, not flow centric ones.
>
> That is the issue Jamal is concerned about.
Oh, absolutely.
I think, fundamentally, any amount of cross-flow resource management
done in software is an obstacle to concurrency.
That's not a value judgement, just a statement of fact.
"traffic cops" are intentional bottlenecks we add to the process, to
enable features like priority flows, filtering, or even simple socket
fairness guarantees. Each of those bottlenecks serves a valid purpose,
but at the end of the day, it's still a bottleneck.
So, improving concurrency may require turning off useful features that
nonetheless hurt concurrency.
> The more I think about it, the more inevitable it seems that we really
> might need multiple qdiscs, one for each TX queue, to pull this full
> parallelization off.
>
> But the semantics of that don't smell so nice either. If the user
> attaches a new qdisc to "ethN", does it go to all the TX queues, or
> what?
>
> All of the traffic shaping technology deals with the device as a unary
> object. It doesn't fit to multi-queue at all.
Well the easy solutions to networking concurrency are
* use virtualization to carve up the machine into chunks
* use multiple net devices
Since new NIC hardware is actively trying to be friendly to
multi-channel/virt scenarios, either of these is reasonably
straightforward given the current state of the Linux net stack. Using
multiple net devices is especially attractive because it works very well
with the existing packet scheduling.
Both unfortunately impose a burden on the developer and admin, to force
their apps to distribute flows across multiple [VMs | net devs].
The third alternative is to use a single net device, with SMP-friendly
packet scheduling. Here you run into the problems you described "device
as a unary object" etc. with the current infrastructure.
With multiple TX rings, consider that we are pushing the packet
scheduling from software to hardware... which implies
* hardware-specific packet scheduling
* some TC/shaping features not available, because hardware doesn't
support it
Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists