netdev - RE: Question on "net: allocate skbs on local node"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1302157012.2701.73.camel@edumazet-laptop>
Date:	Thu, 07 Apr 2011 08:16:52 +0200
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Wei Gu <wei.gu@...csson.com>
Cc:	netdev <netdev@...r.kernel.org>,
	Alexander Duyck <alexander.h.duyck@...el.com>,
	Jeff Kirsher <jeffrey.t.kirsher@...el.com>
Subject: RE: Question on "net: allocate skbs on local node"

Le jeudi 07 avril 2011 à 07:16 +0200, Eric Dumazet a écrit :
> Le jeudi 07 avril 2011 à 06:58 +0200, Eric Dumazet a écrit :
> > Le jeudi 07 avril 2011 à 10:16 +0800, Wei Gu a écrit :
> > > Hi Eric,
> > > Testing with ixgbe Linux 2.6.38 driver:
> > > We have a little better thruput figure with this driver, but it looks
> > > not scalling at all, I always stressed one CPU core/24.
> > > And when look the perf report for ksoftirqd/24, the most cost function
> > > is still "_raw_spin_unlock_irqstore" and the IRQ/s is huge, it's
> > > somehow conflicts with desgin of NAPI. On linux 2.6.32 while the CPU
> > > was stressed the IRQ will descreased while the NAPI will running much
> > > on the polling mode. I don't know why on 2.6.38 the IRQ was keep
> > > increasing.
> > 
> > 
> > CC netdev and Intel guys, since they said it should not happen (TM)
> > 
> > IF you dont use DCA (make sure ioatdma module is not loaded), how comes
> > alloc_iova() is called at all ?
> > 
> > IF you use DCA, how comes its called, since the same CPU serves a given
> > interrupt ?
> > 
> > 
> 
> But then, maybe you forgot to cpu affine IRQS ?
> 
> High performance routing setup is tricky, since you probably want to
> disable many features that are ON by default : Most machines act as a
> end host.
> 
> 

Please dont send me anymore private mails, I do think the issue you have
is on a setup, not a particular optimization done in network stack.


Copy of your private mail :

> On 2.6.38, I got a lot of "rx_missed_errors" on NIC, which means the
> rx loop was really busy to get packet from the receiving ring. Usually
> in this case it shouldn't exit the softirqs and keep polling in order
> to decrease the initrs.
> 
> On 2.6.32, I can Rx and Tx 2.3Mpps with no packet lost(error on NIC),
> but on 2.6.38 I can only reach 50kpps with a lot of
> "rx_missed_errors", and all the binding cpu core was 100% in SI. I
> don't think there was any optimizations on it.

I hope you understand there is something wrong with your setup ?

50.000 pps on a 64 cpu machine is a bad joke.

We can reach +10.000.000 on a 16 cpus one.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html