lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <470A3D24.3050803@garzik.org> Date: Mon, 08 Oct 2007 10:22:28 -0400 From: Jeff Garzik <jeff@...zik.org> To: hadi@...erus.ca CC: David Miller <davem@...emloft.net>, peter.p.waskiewicz.jr@...el.com, krkumar2@...ibm.com, johnpol@....mipt.ru, herbert@...dor.apana.org.au, kaber@...sh.net, shemminger@...ux-foundation.org, jagana@...ibm.com, Robert.Olsson@...a.slu.se, rick.jones2@...com, xma@...ibm.com, gaagaan@...il.com, netdev@...r.kernel.org, rdreier@...co.com, Ingo Molnar <mingo@...e.hu>, mchan@...adcom.com, general@...ts.openfabrics.org, kumarkr@...ux.ibm.com, tgraf@...g.ch, randy.dunlap@...cle.com, sri@...ibm.com, Linux Kernel Mailing List <linux-kernel@...r.kernel.org> Subject: parallel networking (was Re: [PATCH 1/4] [NET_SCHED] explict hold dev tx lock) jamal wrote: > On Sun, 2007-07-10 at 21:51 -0700, David Miller wrote: > >> For these high performance 10Gbit cards it's a load balancing >> function, really, as all of the transmit queues go out to the same >> physical port so you could: >> >> 1) Load balance on CPU number. >> 2) Load balance on "flow" >> 3) Load balance on destination MAC >> >> etc. etc. etc. > > The brain-block i am having is the parallelization aspect of it. > Whatever scheme it is - it needs to ensure the scheduler works as > expected. For example, if it was a strict prio scheduler i would expect > that whatever goes out is always high priority first and never ever > allow a low prio packet out at any time theres something high prio > needing to go out. If i have the two priorities running on two cpus, > then i cant guarantee that effect. Any chance the NIC hardware could provide that guarantee? 8139cp, for example, has two TX DMA rings, with hardcoded characteristics: one is a high prio q, and one a low prio q. The logic is pretty simple: empty the high prio q first (potentially starving low prio q, in worst case). In terms of overall parallelization, both for TX as well as RX, my gut feeling is that we want to move towards an MSI-X, multi-core friendly model where packets are LIKELY to be sent and received by the same set of [cpus | cores | packages | nodes] that the [userland] processes dealing with the data. There are already some primitive NUMA bits in skbuff allocation, but with modern MSI-X and RX/TX flow hashing we could do a whole lot more, along the lines of better CPU scheduling decisions, directing flows to clusters of cpus, and generally doing a better job of maximizing cache efficiency in a modern multi-thread environment. IMO the current model where each NIC's TX completion and RX processes are both locked to the same CPU is outmoded in a multi-core world with modern NICs. :) But I readily admit general ignorance about the kernel process scheduling stuff, so my only idea about a starting point was to see how far to go with the concept of "skb affinity" -- a mask in sk_buff that is a hint about which cpu(s) on which the NIC should attempt to send and receive packets. When going through bonding or netfilter, it is trivial to 'or' together affinity masks. All the various layers of net stack should attempt to honor the skb affinity, where feasible (requires interaction with CFS scheduler?). Or maybe skb affinity is a dumb idea. I wanted to get people thinking on the bigger picture. Parallelization starts at the user process. Jeff - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists