netdev - Re: SMP code / network stack

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20080110154548.4b78ec7c.dada1@cosmosbay.com>
Date:	Thu, 10 Jan 2008 15:45:48 +0100
From:	Eric Dumazet <dada1@...mosbay.com>
To:	Jeba Anandhan <jeba.anandhan@...oni.com>
Cc:	netdev@...r.kernel.org, matthew.hattersley@...oni.com
Subject: Re: SMP code / network stack

On Thu, 10 Jan 2008 14:05:46 +0000
Jeba Anandhan <jeba.anandhan@...oni.com> wrote:

> Hi All,
> 
> If a server has multiple processors and N number of ethernet cards, is
> it possible to handle transmission by each processor separately? .In
> other words, each processor will be responsible for tx of few ethernet
> cards?.
> 
> 
> 
> Example: Server has 4 processors and 8 ethernet cards. is it possible
> for each processor for transmission using 2 ethernet cards only?. So
> that, at a instant , data will be send out from 8 ethernet cards.

Hi Jeba

Modern ethernet cards have a big TX queue, so that even one CPU is enough
to keep several cards busy in //

You can check /proc/interrupts and change /proc/irq/*/smp_affinities to direct IRQ to 
particular cpus, but transmit is usually trigered by processes that might run on different
cpus.

If all ethernet cards are on the same IRQ, then you might have a problem...

Example on a dual processor :
# cat /proc/interrupts 
           CPU0       CPU1       
  0:   11472559   74291833    IO-APIC-edge  timer
  2:          0          0          XT-PIC  cascade
  8:          0          1    IO-APIC-edge  rtc
 81:          0          0   IO-APIC-level  ohci_hcd
 97: 1830022231        847   IO-APIC-level  ehci_hcd, eth0
121:  163095662  166443627   IO-APIC-level  libata
NMI:          0          0 
LOC:   85887285   85887193 
ERR:          0
MIS:          0

You can see eth0 is on IRQ 97
Then :
# cat /proc/irq/97/smp_affinity 
00000001
# echo 2 >/proc/irq/97/smp_affinity
# grep 97 /proc/interrupts
 97: 1830035216       2259   IO-APIC-level  ehci_hcd, eth0
# sleep 10
# grep 97 /proc/interrupts
 97: 1830035216       5482   IO-APIC-level  ehci_hcd, eth0

You can see only CPU1 is now handling IRQ 97 (but CPU0 is allowed to give to eth0 some transmit work)

You might want to check /proc/net/softnet_stat too.

If your server is doing something very special (network trafic, no disk accesses or number crunching),
 you might need to bind application processes to cpus, not only network irqs.

process A, using nic eth0 & eth1, bound to CPU 0 (process and IRQs)
process B, using nic eth2 & eth3, bound to CPU 1
process C, using nic eth4 & eth5, bound to CPU 2
process D, using nic eth6 & eth7, bound to CPU 3


Also, take a look at "ethtool -c ethX" command
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html