lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Tue, 22 Feb 2011 07:13:02 +0100
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Jon Zhou <Jon.Zhou@...u.com>
Cc:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: RE: why all packets have same queue no when rps enabled?

Le lundi 21 février 2011 à 21:56 -0800, Jon Zhou a écrit :
> 
> > -----Original Message-----
> > From: Eric Dumazet [mailto:eric.dumazet@...il.com]
> > Sent: Tuesday, February 22, 2011 1:41 PM
> > To: Jon Zhou
> > Cc: netdev@...r.kernel.org
> > Subject: Re: why all packets have same queue no when rps enabled?
> > 
> > Le lundi 21 février 2011 à 20:07 -0800, Jon Zhou a écrit :
> > > Hi
> > >
> > > I expect each incoming packet will have a different queue no. when I
> > enabled RPS on kernel 2.6.36.4
> > >
> > > cat /sys/class/net/eth4/queues/rx-0/rps_cpus
> > > 00000000,000000ff
> > >
> > > CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
> > CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12
> > CPU13      CPU14      CPU15      CPU16
> > >       CPU17      CPU18      CPU19      CPU20      CPU21      CPU22
> > CPU23      CPU24      CPU25      CPU26      CPU27      CPU28      CPU29
> > CPU30      CPU31
> > >      HI:          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >    TIMER:    6027512     710165    2623243     542768     427807
> > 217424     192940     217043          0          0          0
> > 0          0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >   NET_TX:    1365741         59     750957          0        171
> > 0          3          0          0          0          0          0
> > 0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >   NET_RX:   40465750   11140803    8545859   14417762    8913471
> > 12298691   14216845    3348431   < ---- indeed spread across cpus
> > >
> > >
> > > I manually disable RSS on the intel X520 multiqueue supported NIC.
> > > Cat /proc/interrupts
> > >
> > >   87:   21348294          0          0          0          0
> > 0          0          0   PCI-MSI-edge      eth4-rx-0
> > >   88:      38394          0          0          0          0
> > 0          0          0   PCI-MSI-edge      eth4-tx-0
> > >
> > >
> > >
> > > When I tried the below program to filter packet by queue no.I got
> > these results:
> > >
> > > struct sock_filter BPF_code[]= {
> > >     { 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE},
> > >     { 0x15, 0, 1, 0x00000001 },
> > >     { 0x6, 0, 0, 0x0000ffff },
> > >     { 0x6, 0, 0, 0x00000000 }
> > >   };
> > >
> > >   struct sock_fprog Filter;
> > >
> > >   Filter.len = 4;//15;
> > >   Filter.filter = BPF_code;
> > >
> > >   if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) {
> > >     perror("socket");
> > >     exit(1);
> > >   }
> > >
> > >   /* Set the network card in promiscuos mode */
> > >   strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ);
> > >   if (ioctl(sock,SIOCGIFFLAGS,&ethreq)==-1) {
> > >     perror("ioctl");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >   ethreq.ifr_flags|=IFF_PROMISC;
> > >   if (ioctl(sock,SIOCSIFFLAGS,&ethreq)==-1) {
> > >     perror("ioctl");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >
> > >   /* Attach the filter to the socket */
> > >   if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter,
> > sizeof(Filter))<0){
> > >     perror("setsockopt");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >   static int count = 0;
> > >   while (1) {
> > >     printf("#%d----------\n",count++);
> > >     n = recvfrom(sock,buffer,2048,0,NULL,NULL);
> > >     printf("%d bytes read\n",n);
> > > ...
> > > }
> > >
> > >
> > > Looks almost all packets fall at same queue?
> > > Will RPS allocate queue no for each packet? and what hash algorithm
> > rps used? (is it Toeplitz hash algorithm?)
> > >
> > 
> > I believe you are mistaken.
> > 
> > RPS is not there to spread load on _all_ cpus, but to use a hash
> > function so that all packets of a given flow are directed to a given
> > cpu.
> > 
> > If you receive 1.000.000 packets of the same flow, they all are
> > delivered to one CPU.
> > 
> > 
> With RSS supported NIC, I saw packet#0,1,2,3,~packet#7 will be delivered to different queue, but after RSS disabled and RPS turn on, all these packet will be allocated same queue no(I used the above filter program to find that)
> Is there any fault in the filter program?  
> 

I dont know...

The easy way to find out is to actually do "cat /proc/net/softnet_stat"

Since the last column is the number of time a cpu was given a packet by
another cpu (RPS or RFS action)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists