netdev - Re: Kernel Panic every 2 weeks on ISP server (NULL pointer dereference)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <201110241509.14027.lruete@sequre.com.ar>
Date:	Mon, 24 Oct 2011 15:09:13 -0300
From:	Luciano Ruete <lruete@...ure.com.ar>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	netdev@...r.kernel.org
Subject: Re: Kernel Panic every 2 weeks on ISP server (NULL pointer dereference)

On Sunday, October 23, 2011 02:16:29 am Eric Dumazet wrote:
> Le samedi 22 octobre 2011 à 22:18 -0300, Luciano Ruete a écrit :
> > Hi,
> > 
> > I'm the sysadmin at a 3500 customers ISP, wich runs an iptables+tc
> > solution for load balancing and QoS.
> > 
> > Every 2 or 3 weeks the server panics with a "NULL pointer dereference"
> > and with IP at "dev_queue_xmit"
> > 
> > It is curious that if i disable MSI on the network card driver this
> > panics seems to disapear, does this ring a bell?
> > 
> > The server is an IBM, previously with Broadcom NetXtreme II BCM5709 nics
> > and now with Intel 82576. I change the nics thinking that maybe the bug
> > was in Broadcom Driver but it seems to affect MSI in general.
> > 
> > The tc+iptables rules are auto-generated with sequreisp[1] an ISP
> > solution that i wrote and is open sourced under AGPLv3.
> > 
> > Tell me if you need any further information, and plz CC because I'm not
> > suscribed.
> > 
> > 
> > root@...ver:~# uname -a
> > Linux server 2.6.35-30-server #60~lucid1-Ubuntu SMP Tue Sep 20 22:28:40
> > UTC 2011 x86_64 GNU/Linux
> > 
> > 
> > [1]https://github.com/sequre/sequreisp
> 
> Hi Luciano

Hi Eric!

Thanks for your answer...

> 
> [694250.472081] Code: f6
> 49 c1 e6 07          shl    $0x7,%r14
> 66 89 93 ac 00 00 00 mov    %dx,0xac(%rbx)
>[...]
> This looks like a dev_pick_tx() bug, using an out of bound
> queue_index number and returning a txq pointing after
> the device allocated array.

Clear explanation, is there a tool to map the trace to kernel code, or you did 
this by hand? 

> With recent kernels, this cannot happen anymore because
> we added fixes in this area.
> 
> You could try Ubuntu 11.10 (based on linux 3.0) kernel
> on your server, or apply following patch :
> 
> commit df32cc193ad88f7b1326b90af799c927b27f7654
> Author: Tom Herbert <therbert@...gle.com>
> Date:   Mon Nov 1 12:55:52 2010 -0700
> 
>     net: check queue_index from sock is valid for device
> 
>     In dev_pick_tx recompute the queue index if the value stored in the
>     socket is greater than or equal to the number of real queues for the
>     device.  The saved index in the sock structure is not guaranteed to
>     be appropriate for the egress device (this could happen on a route
>     change or in presence of tunnelling).  The result of the queue index
>     being bad would be to return a bogus queue (crash could prersumably
>     follow).

Lot of ruote changes in this server, there are 30 upstream providers(15 are 
dynamic IP ADSLs) load balanced using VLANs and a VLAN switch.

Thanks again i will try the kernel upgrade and post results in this thread.

Regards!
-- 
Luciano Ruete
Sequre - Sys Admin
Mitre 617, piso 7, of. 1 
+54 261 4254894
Mendoza - Argentina
http://www.sequreisp.com/
http://www.sequre.com.ar/
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html