netdev - Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20250815-herons-fair-c5f3b931@mheyne-amazon>
Date: Fri, 15 Aug 2025 10:00:29 +0000
From: "Heyne, Maximilian" <mheyne@...zon.de>
To: Paul Moore <paul@...l-moore.com>
CC: Kuniyuki Iwashima <kuniyu@...gle.com>, "David S. Miller"
	<davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski
	<kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, Simon Horman
	<horms@...nel.org>, Kuniyuki Iwashima <kuni1840@...il.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>, Jason Baron
	<jbaron@...mai.com>, "Ahmed, Aaron" <aarnahmd@...zon.com>, "Kumar, Praveen"
	<pravkmr@...zon.de>, Eric Paris <eparis@...hat.com>, "linux-audit@...hat.com"
	<linux-audit@...hat.com>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

On Wed, Aug 13, 2025 at 03:00:29PM -0400, Paul Moore wrote:
> On Fri, Aug 8, 2025 at 11:54???AM Kuniyuki Iwashima <kuniyu@...gle.com> wrote:
> > On Fri, Aug 8, 2025 at 6:59???AM Heyne, Maximilian <mheyne@...zon.de> wrote:
> > > On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> > > > Netlink has this pattern in some places
> > > >
> > > >   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
> > > >       atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> > > >
> > > > , which has the same problem fixed by commit 5a465a0da13e ("udp:
> > > > Fix multiple wraparounds of sk->sk_rmem_alloc.").
> > > >
> > > > For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> > > > is always false as the two operands are of int.
> > > >
> > > > Then, a single socket can eat as many skb as possible until OOM
> > > > happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> > > >
> > > > Let's fix it by using atomic_add_return() and comparing the two
> > > > variables as unsigned int.
> > > >
> > > > Before:
> > > >   [root@...ora ~]# ss -f netlink
> > > >   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
> > > >   -1668710080 0               rtnl:nl_wraparound/293               *
> > > >
> > > > After:
> > > >   [root@...ora ~]# ss -f netlink
> > > >   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
> > > >   2147483072 0               rtnl:nl_wraparound/290               *
> > > >   ^
> > > >   `--- INT_MAX - 576
> > > >
> > > > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > > > Reported-by: Jason Baron <jbaron@...mai.com>
> > > > Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> > > > Signed-off-by: Kuniyuki Iwashima <kuniyu@...gle.com>
> > >
> > > Hi Kuniyuki,
> > >
> > > We're seeing soft lockups with this patch in a variety of (stable)
> > > kernel versions.
> > >
> > > I was able to reproduce it on a couple of different EC2 instances also
> > > with the latest linux kernel 6.16-rc7 using the following steps:
> > >
> > > systemctl start auditd
> > > sudo auditctl -D
> > > sudo auditctl -b 512
> > > sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
> > > sudo auditctl -e 1
> > >
> > > Then execute some programs that call some of these memory functions,
> > > such as repeated calls of "sudo auditctl -s" or logging in via SSH.
> > >
> > > These settings are set in a way to create a lot audit messages. Once the
> > > backlog (see auditctl -s) overshoots the backlog_limit, the system soft
> > > lockups:
> > >
> > > [  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
> > > [  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
> > > [  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
> > > [  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
> > > [  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
> > > [  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
> > > [  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
> > > [  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
> > > [  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
> > > [  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
> > > [  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
> > > [  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
> > > [  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
> > > [  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
> > > [  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
> > > [  460.056298] PKRU: 55555554
> > > [  460.056299] Call Trace:
> > > [  460.056300]  <TASK>
> > > [  460.056302]  netlink_attachskb+0xb7/0x2f0
> > > [  460.056308]  ? __pfx_default_wake_function+0x10/0x10
> > > [  460.056313]  netlink_unicast+0xea/0x3b0
> > ...
> > >
> > > We've bug reports from many users, so the issue is rather wide-spread.
> > >
> > > So far I don't know why the commit is causing this issue and will keep
> > > investigating. However, when reverted (together with its 2 follow-up
> > > patches), the issue goes away and host do not lock up.
> >
> > Thanks for the report, Max!
> >
> > Does your tree have this commit ?  This is the 3rd follow-up patch.
> >
> > commit 759dfc7d04bab1b0b86113f1164dc1fec192b859
> > Author: Fedor Pchelkin <pchelkin@...ras.ru>
> > Date:   Mon Jul 28 08:06:47 2025
> >
> >     netlink: avoid infinite retry looping in netlink_unicast()
> 

Hi Paul,

> Hopefully that resolves the problem, Maximilian?

sorry for the late reply. Just tested the commit yesterday and I can
confirm that this fixes our issues.

> Normally the audit subsystem is reasonably robust when faced with
> significant audit loads.  An example I use for testing is to enable
> logging for *every* syscall (from the command line, don't make this
> persist via the config file!) and then shutdown the system; the system
> will obviously slow quite a bit under the absurd load, but it should
> shutdown gracefully without any lockups.
> 

Thank you for suggesting this. Will add something like this to our
internal testing. Do you know whether there is already some stress test
that covers the audit subsystem or would have any selftest found this
issue?

Regards,
Maximilian



Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597