lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Mon, 24 Oct 2011 15:09:13 -0300 From: Luciano Ruete <lruete@...ure.com.ar> To: Eric Dumazet <eric.dumazet@...il.com> Cc: netdev@...r.kernel.org Subject: Re: Kernel Panic every 2 weeks on ISP server (NULL pointer dereference) On Sunday, October 23, 2011 02:16:29 am Eric Dumazet wrote: > Le samedi 22 octobre 2011 à 22:18 -0300, Luciano Ruete a écrit : > > Hi, > > > > I'm the sysadmin at a 3500 customers ISP, wich runs an iptables+tc > > solution for load balancing and QoS. > > > > Every 2 or 3 weeks the server panics with a "NULL pointer dereference" > > and with IP at "dev_queue_xmit" > > > > It is curious that if i disable MSI on the network card driver this > > panics seems to disapear, does this ring a bell? > > > > The server is an IBM, previously with Broadcom NetXtreme II BCM5709 nics > > and now with Intel 82576. I change the nics thinking that maybe the bug > > was in Broadcom Driver but it seems to affect MSI in general. > > > > The tc+iptables rules are auto-generated with sequreisp[1] an ISP > > solution that i wrote and is open sourced under AGPLv3. > > > > Tell me if you need any further information, and plz CC because I'm not > > suscribed. > > > > > > root@...ver:~# uname -a > > Linux server 2.6.35-30-server #60~lucid1-Ubuntu SMP Tue Sep 20 22:28:40 > > UTC 2011 x86_64 GNU/Linux > > > > > > [1]https://github.com/sequre/sequreisp > > Hi Luciano Hi Eric! Thanks for your answer... > > [694250.472081] Code: f6 > 49 c1 e6 07 shl $0x7,%r14 > 66 89 93 ac 00 00 00 mov %dx,0xac(%rbx) >[...] > This looks like a dev_pick_tx() bug, using an out of bound > queue_index number and returning a txq pointing after > the device allocated array. Clear explanation, is there a tool to map the trace to kernel code, or you did this by hand? > With recent kernels, this cannot happen anymore because > we added fixes in this area. > > You could try Ubuntu 11.10 (based on linux 3.0) kernel > on your server, or apply following patch : > > commit df32cc193ad88f7b1326b90af799c927b27f7654 > Author: Tom Herbert <therbert@...gle.com> > Date: Mon Nov 1 12:55:52 2010 -0700 > > net: check queue_index from sock is valid for device > > In dev_pick_tx recompute the queue index if the value stored in the > socket is greater than or equal to the number of real queues for the > device. The saved index in the sock structure is not guaranteed to > be appropriate for the egress device (this could happen on a route > change or in presence of tunnelling). The result of the queue index > being bad would be to return a bogus queue (crash could prersumably > follow). Lot of ruote changes in this server, there are 30 upstream providers(15 are dynamic IP ADSLs) load balanced using VLANs and a VLAN switch. Thanks again i will try the kernel upgrade and post results in this thread. Regards! -- Luciano Ruete Sequre - Sys Admin Mitre 617, piso 7, of. 1 +54 261 4254894 Mendoza - Argentina http://www.sequreisp.com/ http://www.sequre.com.ar/ -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists