netdev - Re: [RFC] EADDRINUSE from bind() on application restart after killing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iLOdgExV3ydkg0r2iNwavSp5Zu9hskf34TTqmCZQCfUdA@mail.gmail.com>
Date:   Fri, 30 Sep 2022 08:16:48 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     Muhammad Usama Anjum <usama.anjum@...labora.com>
Cc:     "open list:NETWORKING [TCP]" <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        David Ahern <dsahern@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Paul Gofman <pgofman@...eweavers.com>
Subject: Re: [RFC] EADDRINUSE from bind() on application restart after killing

On Fri, Sep 30, 2022 at 6:24 AM Muhammad Usama Anjum
<usama.anjum@...labora.com> wrote:
>
> Hi Eric,
>
> RFC 1337 describes the TIME-WAIT Assassination Hazards in TCP. Because
> of this hazard we have 60 seconds timeout in TIME_WAIT state if
> connection isn't closed properly. From RFC 1337:
> > The TIME-WAIT delay allows all old duplicate segments time
> enough to die in the Internet before the connection is reopened.
>
> As on localhost there is virtually no delay. I think the TIME-WAIT delay
> must be zero for localhost connections. I'm no expert here. On localhost
> there is no delay. So why should we wait for 60 seconds to mitigate a
> hazard which isn't there?

Because we do not specialize TCP stack for loopback.

It is easy to force delays even for loopback (tc qdisc add dev lo root
netem ...)

You can avoid TCP complexity (cpu costs) over loopback using AF_UNIX instead.

TIME_WAIT sockets are optional.
If you do not like them, simply set /proc/sys/net/ipv4/tcp_max_tw_buckets to 0 ?

>
> Zapping the sockets in TIME_WAIT and FIN_WAIT_2 does removes them. But
> zap is required from privileged (CAP_NET_ADMIN) process. We are having
> hard time finding a privileged process to do this.

Really, we are not going to add kludges in TCP stacks because of this reason.

>
> Thanks,
> Usama
>
>
> On 5/24/22 1:18 PM, Muhammad Usama Anjum wrote:
> > Hello,
> >
> > We have a set of processes which talk with each other through a local
> > TCP socket. If the process(es) are killed (through SIGKILL) and
> > restarted at once, the bind() fails with EADDRINUSE error. This error
> > only appears if application is restarted at once without waiting for 60
> > seconds or more. It seems that there is some timeout of 60 seconds for
> > which the previous TCP connection remains alive waiting to get closed
> > completely. In that duration if we try to connect again, we get the error.
> >
> > We are able to avoid this error by adding SO_REUSEADDR attribute to the
> > socket in a hack. But this hack cannot be added to the application
> > process as we don't own it.
> >
> > I've looked at the TCP connection states after killing processes in
> > different ways. The TCP connection ends up in 2 different states with
> > timeouts:
> >
> > (1) Timeout associated with FIN_WAIT_1 state which is set through
> > `tcp_fin_timeout` in procfs (60 seconds by default)
> >
> > (2) Timeout associated with TIME_WAIT state which cannot be changed. It
> > seems like this timeout has come from RFC 1337.
> >
> > The timeout in (1) can be changed. Timeout in (2) cannot be changed. It
> > also doesn't seem feasible to change the timeout of TIME_WAIT state as
> > the RFC mentions several hazards. But we are talking about a local TCP
> > connection where maybe those hazards aren't applicable directly? Is it
> > possible to change timeout for TIME_WAIT state for only local
> > connections without any hazards?
> >
> > We have tested a hack where we replace timeout of TIME_WAIT state from a
> > value in procfs for local connections. This solves our problem and
> > application starts to work without any modifications to it.
> >
> > The question is that what can be the best possible solution here? Any
> > thoughts will be very helpful.
> >
> > Regards,
> >
>
> --
> Muhammad Usama Anjum