netdev - Re: [PATCH v1 net-next 00/13] tcp/udp: Introduce optional per-netns hash table.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 26 Aug 2022 09:51:44 -0700
From:   Kuniyuki Iwashima <kuniyu@...zon.com>
To:     <edumazet@...gle.com>
CC:     <chuck.lever@...cle.com>, <davem@...emloft.net>,
        <jlayton@...nel.org>, <keescook@...omium.org>, <kuba@...nel.org>,
        <kuni1840@...il.com>, <kuniyu@...zon.com>,
        <linux-fsdevel@...r.kernel.org>, <mcgrof@...nel.org>,
        <netdev@...r.kernel.org>, <pabeni@...hat.com>, <yzaikin@...gle.com>
Subject: Re: [PATCH v1 net-next 00/13] tcp/udp: Introduce optional per-netns hash table.

From:   Eric Dumazet <edumazet@...gle.com>
Date:   Fri, 26 Aug 2022 08:17:25 -0700
> On Thu, Aug 25, 2022 at 5:05 PM Kuniyuki Iwashima <kuniyu@...zon.com> wrote:
> >
> > The more sockets we have in the hash table, the more time we spend
> > looking up the socket.  While running a number of small workloads on
> > the same host, they penalise each other and cause performance degradation.
> >
> > Also, the root cause might be a single workload that consumes much more
> > resources than the others.  It often happens on a cloud service where
> > different workloads share the same computing resource.
> >
> > On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash
> > entries), after running iperf3 in different netns, creating 24Mi sockets
> > without data transfer in the root netns causes about 10% performance
> > regression for the iperf3's connection.
> >
> >  thash_entries          sockets         length          Gbps
> >         524288                1              1          50.7
> >                            24Mi             48          45.1
> >
> > It is basically related to the length of the list of each hash bucket.
> > For testing purposes to see how performance drops along the length,
> > I set 131072 (1Mi / 8) to thash_entries, and here's the result.
> >
> >  thash_entries          sockets         length          Gbps
> >         131072                1              1          50.7
> >                             1Mi              8          49.9
> >                             2Mi             16          48.9
> >                             4Mi             32          47.3
> >                             8Mi             64          44.6
> >                            16Mi            128          40.6
> >                            24Mi            192          36.3
> >                            32Mi            256          32.5
> >                            40Mi            320          27.0
> >                            48Mi            384          25.0
> >
> > To resolve the socket lookup degradation, we introduce an optional
> > per-netns hash table for TCP and UDP.  With a smaller hash table, we
> > can look up sockets faster and isolate noisy neighbours.  Also, we can
> > reduce lock contention.
> >
> > We can control and check the hash size via sysctl knobs.  It requires
> > some tuning based on workloads, so the per-netns hash table is disabled
> > by default.
> >
> >   # dmesg | cut -d ' ' -f 5- | grep "established hash"
> >   TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
> >
> >   # sysctl net.ipv4.tcp_ehash_entries
> >   net.ipv4.tcp_ehash_entries = 524288  # can be changed by thash_entries
> >
> >   # sysctl net.ipv4.tcp_child_ehash_entries
> >   net.ipv4.tcp_child_ehash_entries = 0  # disabled by default
> >
> >   # ip netns add test1
> >   # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries
> >   net.ipv4.tcp_ehash_entries = -524288  # share the global ehash
> >
> >   # sysctl -w net.ipv4.tcp_child_ehash_entries=100
> >   net.ipv4.tcp_child_ehash_entries = 100
> >
> >   # sysctl net.ipv4.tcp_child_ehash_entries
> >   net.ipv4.tcp_child_ehash_entries = 128  # rounded up to 2^n
> >
> >   # ip netns add test2
> >   # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries
> >   net.ipv4.tcp_ehash_entries = 128  # own per-netns ehash
> >
> >   [ UDP has the same interface as udp_hash_entries and
> >     udp_child_hash_entries. ]
> >
> > When creating per-netns concurrently with different sizes, we can
> > guarantee the size by doing one of these ways.
> >
> >   1) Share the global hash table and create per-netns one
> >
> >   First, unshare() with tcp_child_ehash_entries==0.  It creates dedicated
> >   netns sysctl knobs where we can safely change tcp_child_ehash_entries
> >   and clone()/unshare() to create a per-netns hash table.
> >
> >   2) Lock the sysctl knob
> >
> 
> This is orthogonal.
> 
> Your series should have been split in three really.
> 
> I do not want to discuss the merit of re-instating LOCK_MAND :/

I see.
I'll drop the flock() part at once and respin TCP part only in v2.