netdev - Re: [PATCH] ipvs: Fix reuse connection if RS weight is 0

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 27 Oct 2021 09:43:31 +0800
From:   yangxingwu <xingwu.yang@...il.com>
To:     Julian Anastasov <ja@....bg>
Cc:     Simon Horman <horms@...ge.net.au>,
        Pablo Neira Ayuso <pablo@...filter.org>,
        kadlec@...filter.org, fw@...len.de,
        "David S. Miller" <davem@...emloft.net>, kuba@...nel.org,
        netdev@...r.kernel.org, lvs-devel@...r.kernel.org,
        netfilter-devel@...r.kernel.org, coreteam@...filter.org,
        linux-kernel <linux-kernel@...r.kernel.org>,
        linux-doc@...r.kernel.org, corbet@....net
Subject: Re: [PATCH] ipvs: Fix reuse connection if RS weight is 0

Julian

what we want is if RS weight is 0, then no new connections should be
served even if conn_reuse_mode is 0, just as commit dc7b3eb900aa
("ipvs: Fix reuse connection if real server is
dead") trying to do

Pls let me know if there are any other issues of concern

On Tue, Oct 26, 2021 at 2:13 PM yangxingwu <xingwu.yang@...il.com> wrote:
>
> thanks Julian
>
> yes, I know that the one-second delay issue has been fixed by commit
> f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f if we set conn_reuse_mode to
> 1
>
> BUT  it's still NOT what we expected with sysctl settings
> (conn_reuse_mode == 0 && expire_nodest_conn == 1).
>
> We run kubernetes in extremely diverse environments and this issue
> happens a lot.
>
> On Tue, Oct 26, 2021 at 1:44 PM Julian Anastasov <ja@....bg> wrote:
> >
> >
> >         Hello,
> >
> > On Tue, 26 Oct 2021, yangxingwu wrote:
> >
> > > thanks julian
> > >
> > > What happens in this situation is that if we set the wait of the
> > > realserver to 0 and do NOT remove the weight zero realserver with
> > > sysctl settings (conn_reuse_mode == 0 && expire_nodest_conn == 1), and
> > > the client reuses its source ports, the kernel will constantly
> > > reuse connections and send the traffic to the weight 0 realserver.
> >
> >         Yes, this is expected when conn_reuse_mode=0.
> >
> > > you may check the details from
> > > https://github.com/kubernetes/kubernetes/issues/81775
> >
> >         What happens if you try conn_reuse_mode=1? The
> > one-second delay in previous kernels should be corrected with
> >
> > commit f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f
> > Date:   Wed Jul 1 18:17:19 2020 +0300
> >
> >     ipvs: allow connection reuse for unconfirmed conntrack
> >
> > > On Tue, Oct 26, 2021 at 2:12 AM Julian Anastasov <ja@....bg> wrote:
> > > >
> > > > On Mon, 25 Oct 2021, yangxingwu wrote:
> > > >
> > > > > Since commit dc7b3eb900aa ("ipvs: Fix reuse connection if real server is
> > > > > dead"), new connections to dead servers are redistributed immediately to
> > > > > new servers.
> > > > >
> > > > > Then commit d752c3645717 ("ipvs: allow rescheduling of new connections when
> > > > > port reuse is detected") disable expire_nodest_conn if conn_reuse_mode is
> > > > > 0. And new connection may be distributed to a real server with weight 0.
> > > >
> > > >         Your change does not look correct to me. At the time
> > > > expire_nodest_conn was created, it was not checked when
> > > > weight is 0. At different places different terms are used
> > > > but in short, we have two independent states for real server:
> > > >
> > > > - inhibited: weight=0 and no new connections should be served,
> > > >         packets for existing connections can be routed to server
> > > >         if it is still available and packets are not dropped
> > > >         by expire_nodest_conn.
> > > >         The new feature is that port reuse detection can
> > > >         redirect the new TCP connection into a new IPVS conn and
> > > >         to expire the existing cp/ct.
> > > >
> > > > - unavailable (!IP_VS_DEST_F_AVAILABLE): server is removed,
> > > >         can be temporary, drop traffic for existing connections
> > > >         but on expire_nodest_conn we can select different server
> > > >
> > > >         The new conn_reuse_mode flag allows port reuse to
> > > > be detected. Only then expire_nodest_conn has the
> > > > opportunity with commit dc7b3eb900aa to check weight=0
> > > > and to consider the old traffic as finished. If a new
> > > > server is selected, any retrans from previous connection
> > > > would be considered as part from the new connection. It
> > > > is a rapid way to switch server without checking with
> > > > is_new_conn_expected() because we can not have many
> > > > conns/conntracks to different servers.
> >
> > Regards
> >
> > --
> > Julian Anastasov <ja@....bg>