netdev - Re: Increased UDP socket memory on Linux 6.1

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iLU+OJR4pvFxM0akOLLSV2yCbR9Kb8ap3u3UOxh2Xy1Bw@mail.gmail.com>
Date:   Wed, 22 Mar 2023 15:09:41 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     Ignat Korchagin <ignat@...udflare.com>
Cc:     netdev <netdev@...r.kernel.org>,
        kernel-team <kernel-team@...udflare.com>
Subject: Re: Increased UDP socket memory on Linux 6.1

On Wed, Mar 22, 2023 at 2:57 PM Ignat Korchagin <ignat@...udflare.com> wrote:
>
> Hello,
>
> We were investigating unusual packet drops on our systems potentially
> related to our recent migration to the 6.1 kernel. We have noticed a
> substantial increase in UDP socket memory for the same workload. Below
> are two servers in the same datacentre doing the same workload.
>
> On 5.15.90 (our previous kernel):
> $ cat /proc/net/sockstat
> sockets: used 174831
> TCP: inuse 112301 orphan 145 tw 23829 alloc 135086 mem 313582
> UDP: inuse 7613 mem 1667
> UDPLITE: inuse 0
> RAW: inuse 7
> FRAG: inuse 0 memory 0
>
> But on 6.1.20:
> $ cat /proc/net/sockstat
> sockets: used 168911
> TCP: inuse 108857 orphan 124 tw 23674 alloc 130096 mem 235530
> UDP: inuse 7555 mem 10514

10514 pages 'forward allocated' for 7555 UDP sockets seems ok to me.

UDP sockets have their own notion of 'forward_deficit'  and
forward_threshold based on SO_RCVBUF values.

Do you have the following commit yet in your kernel ?

commit 8a3854c7b8e4532063b14bed34115079b7d0cb36
Author: Paolo Abeni <pabeni@...hat.com>
Date:   Thu Oct 20 19:48:52 2022 +0200

    udp: track the forward memory release threshold in an hot cacheline

    When the receiver process and the BH runs on different cores,
    udp_rmem_release() experience a cache miss while accessing sk_rcvbuf,
    as the latter shares the same cacheline with sk_forward_alloc, written
    by the BH.

    With this patch, UDP tracks the rcvbuf value and its update via custom
    SOL_SOCKET socket options, and copies the forward memory threshold value
    used by udp_rmem_release() in a different cacheline, already accessed by
    the above function and uncontended.

    Since the UDP socket init operation grown a bit, factor out the common
    code between v4 and v6 in a shared helper.

    Overall the above give a 10% peek throughput increase under UDP flood.

    Signed-off-by: Paolo Abeni <pabeni@...hat.com>
    Reviewed-by: Eric Dumazet <edumazet@...gle.com>
    Acked-by: Kuniyuki Iwashima <kuniyu@...zon.com>
    Signed-off-by: David S. Miller <davem@...emloft.net>


> UDPLITE: inuse 0
> RAW: inuse 7
> FRAG: inuse 0 memory 0
>
> For roughly the same amount of UDP sockets the UDP memory is much
> higher. TCP memory looks different above as well, but according to our
> longer-term metrics overall it is the same, but UDP is substantially
> bigger.
>
> Here's the snapshot of the same metric from the same servers in
> graphical form [1]. The server indicated by a blue line was rebooted
> into 6.1.20 and you can see the UDP memory jumped compared to the
> green server (on 5.15.90). We're not sure yet, but perhaps it is an
> artifact of [2], namely commit 4890b686f4088c90 ("net: keep
> sk->sk_forward_alloc as small as possible") and commit
> 3cd3399dd7a84ada ("net: implement per-cpu reserves for
> memory_allocated")
>
> We don't know at this point if it is related to our unusual rate of
> packet drops, but just wanted to point this out and see if the UDP
> memory increase is expected.

>
> Thanks,
> Ignat
>
> [1]: https://pub-ddb0f42c43e74ce4a1424bc33f965f9a.r2.dev/udp-mem.jpg
> [2]: https://lore.kernel.org/netdev/20220609063412.2205738-1-eric.dumazet@gmail.com/