netdev - [RFC] EADDRINUSE from bind() on application restart after killing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <5099dc39-c6d9-115a-855b-6aa98d17eb4b@collabora.com>
Date:   Tue, 24 May 2022 13:18:55 +0500
From:   Muhammad Usama Anjum <usama.anjum@...labora.com>
To:     Eric Dumazet <edumazet@...gle.com>,
        "David S. Miller" <davem@...emloft.net>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        David Ahern <dsahern@...nel.org>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>,
        "open list:NETWORKING [TCP]" <netdev@...r.kernel.org>
Cc:     usama.anjum@...labora.com,
        Gabriel Krisman Bertazi <krisman@...labora.com>,
        LKML <linux-kernel@...r.kernel.org>,
        open list <linux-kernel@...r.kernel.org>
Subject: [RFC] EADDRINUSE from bind() on application restart after killing

Hello,

We have a set of processes which talk with each other through a local
TCP socket. If the process(es) are killed (through SIGKILL) and
restarted at once, the bind() fails with EADDRINUSE error. This error
only appears if application is restarted at once without waiting for 60
seconds or more. It seems that there is some timeout of 60 seconds for
which the previous TCP connection remains alive waiting to get closed
completely. In that duration if we try to connect again, we get the error.

We are able to avoid this error by adding SO_REUSEADDR attribute to the
socket in a hack. But this hack cannot be added to the application
process as we don't own it.

I've looked at the TCP connection states after killing processes in
different ways. The TCP connection ends up in 2 different states with
timeouts:

(1) Timeout associated with FIN_WAIT_1 state which is set through
`tcp_fin_timeout` in procfs (60 seconds by default)

(2) Timeout associated with TIME_WAIT state which cannot be changed. It
seems like this timeout has come from RFC 1337.

The timeout in (1) can be changed. Timeout in (2) cannot be changed. It
also doesn't seem feasible to change the timeout of TIME_WAIT state as
the RFC mentions several hazards. But we are talking about a local TCP
connection where maybe those hazards aren't applicable directly? Is it
possible to change timeout for TIME_WAIT state for only local
connections without any hazards?

We have tested a hack where we replace timeout of TIME_WAIT state from a
value in procfs for local connections. This solves our problem and
application starts to work without any modifications to it.

The question is that what can be the best possible solution here? Any
thoughts will be very helpful.

Regards,

-- 
Muhammad Usama Anjum