netdev - Re: [REGRESSION] tcp/ipv4: kernel panic because of (possible) division by zero

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 09 Jan 2016 19:34:47 +0200
From:	Oleksandr Natalenko <oleksandr@...alenko.name>
To:	Yuchung Cheng <ycheng@...gle.com>
Cc:	netdev <netdev@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Patrick McHardy <kaber@...sh.net>,
	Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
	James Morris <jmorris@...ei.org>,
	Alexey Kuznetsov <kuznet@....inr.ac.ru>,
	"David S. Miller" <davem@...emloft.net>
Subject: Re: [REGRESSION] tcp/ipv4: kernel panic because of (possible) division by zero

Here is stacktrace got via netconsole with ECN enabled and first patch 
applied:

https://gist.github.com/57fd5e2795b86c40eb80

Now will recompile kernel with patch you've sent to upstream and test it with 
ECN enabled again.

On середа, 6 січня 2016 р. 10:43:45 EET Yuchung Cheng wrote:
> On Wed, Jan 6, 2016 at 10:19 AM, Yuchung Cheng <ycheng@...gle.com> wrote:
> > On Wed, Jan 6, 2016 at 8:50 AM, Oleksandr Natalenko
> > 
> > <oleksandr@...alenko.name> wrote:
> >> Unfortunately, the patch didn't help -- I've got the same stacktrace with
> >> slightly different offset (+3) within the function.
> >> 
> >> Now trying to get full stacktrace via netconsole. Need more time.
> >> 
> >> Meanwhile, any other ideas on what went wrong?
> > 
> > That's odd b/c the patch already checks and avoids div0. Can post me
> > the stacktrace and kernel warnings if any ...
> > 
> > One possibility is that tcp_cwnd_reduction() may set a cwnd of 0,
> > which then gets used to start another recovery phase. This may or may
> > not be the culprit of this div0 issue because I wasn't able to
> > reproduce exactly your issue on our servers. But I will post the fix
> > today and CC you.
> 
> Could you turn off ecn (sysctl net.ipv4.tcp_ecn=0) to see if this still
> happen?
> >> On December 22, 2015 4:10:32 AM EET, Yuchung Cheng <ycheng@...gle.com> 
wrote:
> >> >On Mon, Dec 21, 2015 at 12:25 PM, Oleksandr Natalenko
> >> >
> >> ><oleksandr@...alenko.name> wrote:
> >> >> Commit 3759824da87b30ce7a35b4873b62b0ba38905ef5 (tcp: PRR uses CRB
> >> >
> >> >mode by
> >> >
> >> >> default and SS mode conditionally) introduced changes to
> >> >
> >> >net/ipv4/tcp_input.c
> >> >
> >> >> tcp_cwnd_reduction() that, possibly, cause division by zero, and
> >> >
> >> >therefore,
> >> >
> >> >> kernel panic in interrupt handler [1].
> >> >> 
> >> >> Reverting 3759824da87b30ce7a35b4873b62b0ba38905ef5 seems to fix the
> >> >
> >> >issue.
> >> >
> >> >> I'm able to reproduce the issue on 4.3.0–4.3.3 once per several day
> >> >> (occasionally).
> >> >> 
> >> >> What could be done to help in debugging this issue?
> >> >
> >> >Do you have ECN enabled (i.e. sysctl net.ipv4.tcp_ecn > 0)?
> >> >
> >> >If so I suspect an ACK carrying ECE during CA_Loss causes entering CWR
> >> >state w/o calling tcp_init_cwnd_reduct() to set tp->prior_cwnd. Can
> >> >you try this debug / quick-fix patch and send me the error message if
> >> >any?
> >> >
> >> >> Regards,
> >> >> 
> >> >>   Oleksandr.
> >> >> 
> >> >> [1] http://i.piccy.info/
> >> >
> >> >i9/6f5cb187c4ff282d189f78c63f95af43/1450729403/283985/951663/panic.jpg
> >> 
> >> --
> >> Sent from my Android device with K-9 Mail. Please excuse my brevity.