netdev - Re: After many hours all outbound connections get stuck in SYN

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Pine.LNX.4.64.0712192016280.24742@kivilampi-30.cs.helsinki.fi>
Date:	Wed, 19 Dec 2007 20:32:59 +0200 (EET)
From:	"Ilpo Järvinen" <ilpo.jarvinen@...sinki.fi>
To:	James Nichols <jamesnichols3@...il.com>
cc:	Netdev <netdev@...r.kernel.org>
Subject: Re: After many hours all outbound connections get stuck in SYN_SENT

On Wed, 19 Dec 2007, James Nichols wrote:

> > > And the problem almost instanteaously resolved itself and outbound
> > > connection attempts were succesful.
> >
> > New or the pending ones?
> 
> I'm fairly sure that sockets that were already open in SYN_SENT state
> when I turned tcp_sack off started to work as the count of sockets in
> SYN_SENT state drops very rapidly.

Heh, "very rapidly" :-/, considering that you have 200 x SYN_SENT flows 
and if tcp_syn_retries is set to default, you will see something happening 
quite quickly for sure and the whole situation is over in ~3 mins if I 
counted correctly.

> > > Is there a kernel buffer or some data structure that tcp_sack uses
> > > that gets filled up after an extended period of operation?
> >
> > SACK has pretty little meaning in context of SYNs, there's only the
> > sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK.
> >
> > The SACK scoreboard is currently included to the skbs (has been like
> > this for very long time), so no additional data structures should be
> > there because of SACK...
> 
> I've been seeing this problem for about 4 years, so could it be
> related to the scoreboard implementation somehow?

Scoreboard has no meaning in this context, it is used while _input_ 
packets are processed. If you don't get SYN-ACKs at all, it doesn't
have any meaning.

> > /proc/net/tcp couple of times in a row, try something something like
> > this:
> >
> > for i in (seq 1 40); do cat /proc/net/tcp; echo "-----"; sleep 10; done
> 
> I can set up to do this the next time the problem occurs.

for i in $(seq 1 40); ... is the right way to do the loop. :-)

> > > I'm running kernel 2.6.18 on RedHat, but have had this problem occur
> > > on earlier kernel versions (all 2.4 and 2.6).
> >
> > I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat
> > has backported them). Though they're not that critical nor anything in
> > them should affect in SYN_SENT state.
> 
> Ok, unless there is direct evidence that there is a fix to this
> problem in a later kernel I won't be able to upgrade.  If there is a
> redhat provided patch I can probably do that.

...They won't affect in SYN_SENT.

-- 
 i.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html