netdev - Re: ixgbe question

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Pine.WNT.4.64.0911231525180.10056@ppwaskie-MOBL2.amr.corp.intel.com>
Date:	Mon, 23 Nov 2009 15:28:18 -0800 (Pacific Standard Time)
From:	"Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@...el.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
cc:	"robert@...julf.net" <robert@...julf.net>,
	Jesper Dangaard Brouer <hawk@...u.dk>,
	"Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@...el.com>,
	Linux Netdev List <netdev@...r.kernel.org>
Subject: Re: ixgbe question

On Mon, 23 Nov 2009, Eric Dumazet wrote:

> robert@...julf.net a écrit :
> > Eric Dumazet writes:
> > 
> >  > Jesper Dangaard Brouer a écrit :
> >  > 
> >  > > How is your smp_affinity mask's set?
> >  > > 
> >  > > grep . /proc/irq/*/fiber1-*/../smp_affinity
> >  > 
> > 
> >  Weird... set clone_skb to 1 to be sure and vary dst or something so 
> >  the HW classifier selects different queues and with proper RX affinty. 
> >  
> >  You should see in /proc/net/softnet_stat something like:
> > 
> > 012a7bb9 00000000 000000ae 00000000 00000000 00000000 00000000 00000000 00000000
> > 01288d4c 00000000 00000049 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128fe28 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> > 01295387 00000000 00000047 00000000 00000000 00000000 00000000 00000000 00000000
> > 0129a722 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128c5e4 00000000 00000046 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128f718 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> > 012993e3 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
> > 
> 
> slone_skb set to 1, this changes nothing but slows down pktgen (obviously)
> 
> Result: OK: 117614452(c117608705+d5746) nsec, 100000000 (60byte,0frags)
>   850235pps 408Mb/sec (408112800bps) errors: 0
> 
> All RX processing of 16 RX queues done by CPU 1 only.

Ok, I was confused earlier.  I thought you were saying that all packets 
were headed into a single Rx queue.  This is different.

Do you know what version of irqbalance you're running, or if it's running 
at all?  We've seen issues with irqbalance where it won't recognize the 
ethernet device if the driver has been reloaded.  In that case, it won't 
balance the interrupts at all.  If the default affinity was set to one 
CPU, then well, you're screwed.

My suggestion in this case is after you reload ixgbe and start your tests, 
see if it all goes to one CPU.  If it does, then restart irqbalance 
(service irqbalance restart - or just kill it and restart by hand).  Then 
start running your test, and in 10 seconds you should see the interrupts 
move and spread out.

Let me know if this helps,
-PJ