linux-kernel - oops in tcp_xmit_retransmit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 08 Jul 2010 10:22:02 +0200
From:	Tejun Heo <tj@...nel.org>
To:	"David S. Miller" <davem@...emloft.net>,
	lkml <linux-kernel@...r.kernel.org>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:	"Fehrmann, Henning" <henning.fehrmann@....mpg.de>,
	Carsten Aulbert <carsten.aulbert@....mpg.de>
Subject: oops in tcp_xmit_retransmit_queue() w/ v2.6.32.15

Hello,

We've been seeing oops in tcp_xmit_retransmit_queue() w/ 2.6.32.15.
Please see the attached photoshoot.  This is happening on a HPC
cluster and very interestingly caused by one particular job.  How long
it takes isn't clear yet (at least more than a day) but when it
happens it happens on a lot of machines in relatively short time.

With a bit of disassemblying, I've found that the oops is happening
during tcp_for_write_queue_from() because the skb->next points to
NULL.

 void tcp_xmit_retransmit_queue(struct sock *sk)
 {
 ...
	if (tp->retransmit_skb_hint) {
		skb = tp->retransmit_skb_hint;
		last_lost = TCP_SKB_CB(skb)->end_seq;
		if (after(last_lost, tp->retransmit_high))
			last_lost = tp->retransmit_high;
	} else {
		skb = tcp_write_queue_head(sk);
		last_lost = tp->snd_una;
	}

 =>	tcp_for_write_queue_from(skb, sk) {
		 __u8 sacked = TCP_SKB_CB(skb)->sacked;

		 if (skb == tcp_send_head(sk))
			 break;
		 /* we could do better than to assign each time */
		 if (hole == NULL)

This can happen for one of the following reasons,

1. tp->retransmit_skb_hint is NULL and tcp_write_queue_head() is NULL
   too.  ie. tcp_xmit_retransmit_queue() is called on an empty write
   queue for some reason.

2. tp->retransmit_skb_hint is pointing to a skb which is not on the
   write_queue.  ie. somebody forgot to update hint while removing the
   skb from the write queue.

3. The hint is pointing to a skb on the list but the list itself is
   corrupt.

I added some debug code and the crash is happening when
tp->retransmit_skb_hint is not NULL but tp->retransmit_skb_hint->next
is NULL.  So, #1 is out; unfortunately, I didn't have debug code in
place to discern between #2 and #3.

Does anything ring a bell?  This is a production system and debugging
affects quite a number of people.  I can put debug code in to discern
between #2 and #3 but I'm basically shooting in the dark and it would
be great if someone has a better idea.

Thanks.

-- 
tejun

Download attachment "oops.jpg" of type "image/jpeg" (92585 bytes)