netdev - Increased UDP socket memory on Linux 6.1

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CALrw=nHWdZA=nGizO4hd1xineoNuhN7hh5nGrHFiint1m72afQ@mail.gmail.com>
Date:   Wed, 22 Mar 2023 21:57:14 +0000
From:   Ignat Korchagin <ignat@...udflare.com>
To:     netdev <netdev@...r.kernel.org>, Eric Dumazet <edumazet@...gle.com>
Cc:     kernel-team <kernel-team@...udflare.com>
Subject: Increased UDP socket memory on Linux 6.1

Hello,

We were investigating unusual packet drops on our systems potentially
related to our recent migration to the 6.1 kernel. We have noticed a
substantial increase in UDP socket memory for the same workload. Below
are two servers in the same datacentre doing the same workload.

On 5.15.90 (our previous kernel):
$ cat /proc/net/sockstat
sockets: used 174831
TCP: inuse 112301 orphan 145 tw 23829 alloc 135086 mem 313582
UDP: inuse 7613 mem 1667
UDPLITE: inuse 0
RAW: inuse 7
FRAG: inuse 0 memory 0

But on 6.1.20:
$ cat /proc/net/sockstat
sockets: used 168911
TCP: inuse 108857 orphan 124 tw 23674 alloc 130096 mem 235530
UDP: inuse 7555 mem 10514
UDPLITE: inuse 0
RAW: inuse 7
FRAG: inuse 0 memory 0

For roughly the same amount of UDP sockets the UDP memory is much
higher. TCP memory looks different above as well, but according to our
longer-term metrics overall it is the same, but UDP is substantially
bigger.

Here's the snapshot of the same metric from the same servers in
graphical form [1]. The server indicated by a blue line was rebooted
into 6.1.20 and you can see the UDP memory jumped compared to the
green server (on 5.15.90). We're not sure yet, but perhaps it is an
artifact of [2], namely commit 4890b686f4088c90 ("net: keep
sk->sk_forward_alloc as small as possible") and commit
3cd3399dd7a84ada ("net: implement per-cpu reserves for
memory_allocated")

We don't know at this point if it is related to our unusual rate of
packet drops, but just wanted to point this out and see if the UDP
memory increase is expected.

Thanks,
Ignat

[1]: https://pub-ddb0f42c43e74ce4a1424bc33f965f9a.r2.dev/udp-mem.jpg
[2]: https://lore.kernel.org/netdev/20220609063412.2205738-1-eric.dumazet@gmail.com/