netdev - Re: [RFC] random SYN drops causing connect() delays

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4BC5A8ED.6040908@nets.rwth-aachen.de>
Date:	Wed, 14 Apr 2010 13:37:17 +0200
From:	Lennart Schulte <lennart.schulte@...s.rwth-aachen.de>
To:	<tgraf@...radead.org>
CC:	<netdev@...r.kernel.org>
Subject: Re: [RFC] random SYN drops causing connect() delays

Hi,
this is very similar to what i have noticed, but up to now I couldn't figure out where it came from. 
Thanks very much for clearing it up!

> I have been tracking down an issue commonly referred to as the 3-sec
> connect() delay. It exists since recent 2.6.x kernels and has never
> been fixed even though it disappeared in recent releases unless
> sched_child_runs_first is set to 1 again.
>
> What happens is that if a client attemps to open many connections to
> a socket with only minimal delay inbetween attemps some SYNs are
> randomly dropped on the server side causing the client to resend after
> the 3 sec TCP timeout and thus causing connect()s to be randomly delayed.
>
> Facts:
>  - Issue can be reproduced over loopback or real networks.
>  - Enabling SO_LINGER on the client side will make the issue disappear!!
>  - While the issue is appearing, the acceptq seems to be overflowing. Both
>    LISTENOVERFLOWS and LISTENDROPS are increasing although not by the exact
>    number of delay occurences. inetdiag reports sk_max_ack_backlog to be 0
>    therefore one possibility that comes to mind is that sk_ack_backlog
>    underflows due to a race.
>  - The issue disappeared in recent kernels, I bisected it down to the following
>    commit:
> 	commit 2bba22c50b06abe9fd0d23933b1e64d35b419262
> 	Author: Mike Galbraith <efault@....de>
> 	Date:   Wed Sep 9 15:41:37 2009 +0200
>
> 	    sched: Turn off child_runs_first
> 	    
> 	    Set child_runs_first default to off.
>
>    Setting kernel.sched_child_runs_first=1 makes the isssue reappear in recent
>    kernels.  This hardens the theory of a race condition.
>  - It looks like that the issue can only be reproduced if the server
>    socket sends out data immediately after the connection has been established
>    but I cannot proof this theory.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html