linux-kernel - Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAAVpQUAi6sQ+=S-5oYOPkuPEFk68g2zG81YOA3MYVnTSvTxcjg@mail.gmail.com>
Date: Fri, 8 Aug 2025 08:54:33 -0700
From: Kuniyuki Iwashima <kuniyu@...gle.com>
To: "Heyne, Maximilian" <mheyne@...zon.de>
Cc: "David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>, 
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>, 
	Kuniyuki Iwashima <kuni1840@...il.com>, "netdev@...r.kernel.org" <netdev@...r.kernel.org>, 
	Jason Baron <jbaron@...mai.com>, "Ahmed, Aaron" <aarnahmd@...zon.com>, 
	"Kumar, Praveen" <pravkmr@...zon.de>, Paul Moore <paul@...l-moore.com>, Eric Paris <eparis@...hat.com>, 
	"linux-audit@...hat.com" <linux-audit@...hat.com>, 
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v1 net] netlink: Fix wraparounds of sk->sk_rmem_alloc.

On Fri, Aug 8, 2025 at 6:59 AM Heyne, Maximilian <mheyne@...zon.de> wrote:
>
> On Fri, Jul 04, 2025 at 05:48:18AM +0000, Kuniyuki Iwashima wrote:
> > Netlink has this pattern in some places
> >
> >   if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
> >       atomic_add(skb->truesize, &sk->sk_rmem_alloc);
> >
> > , which has the same problem fixed by commit 5a465a0da13e ("udp:
> > Fix multiple wraparounds of sk->sk_rmem_alloc.").
> >
> > For example, if we set INT_MAX to SO_RCVBUFFORCE, the condition
> > is always false as the two operands are of int.
> >
> > Then, a single socket can eat as many skb as possible until OOM
> > happens, and we can see multiple wraparounds of sk->sk_rmem_alloc.
> >
> > Let's fix it by using atomic_add_return() and comparing the two
> > variables as unsigned int.
> >
> > Before:
> >   [root@...ora ~]# ss -f netlink
> >   Recv-Q      Send-Q Local Address:Port                Peer Address:Port
> >   -1668710080 0               rtnl:nl_wraparound/293               *
> >
> > After:
> >   [root@...ora ~]# ss -f netlink
> >   Recv-Q     Send-Q Local Address:Port                Peer Address:Port
> >   2147483072 0               rtnl:nl_wraparound/290               *
> >   ^
> >   `--- INT_MAX - 576
> >
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Reported-by: Jason Baron <jbaron@...mai.com>
> > Closes: https://lore.kernel.org/netdev/cover.1750285100.git.jbaron@akamai.com/
> > Signed-off-by: Kuniyuki Iwashima <kuniyu@...gle.com>
>
> Hi Kuniyuki,
>
> We're seeing soft lockups with this patch in a variety of (stable)
> kernel versions.
>
> I was able to reproduce it on a couple of different EC2 instances also
> with the latest linux kernel 6.16-rc7 using the following steps:
>
> systemctl start auditd
> sudo auditctl -D
> sudo auditctl -b 512
> sudo auditctl -a always,exit -F arch=b64 -S mmap,munmap,mprotect,brk -k memory_ops
> sudo auditctl -e 1
>
> Then execute some programs that call some of these memory functions,
> such as repeated calls of "sudo auditctl -s" or logging in via SSH.
>
> These settings are set in a way to create a lot audit messages. Once the
> backlog (see auditctl -s) overshoots the backlog_limit, the system soft
> lockups:
>
> [  460.056244] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kauditd:32]
> [  460.056249] Modules linked in: mousedev(E) nls_ascii(E) nls_cp437(E) sunrpc(E) vfat(E) fat(E) psmouse(E) atkbd(E) libps2(E) vivaldi_fmap(E) i8042(E) serio(E) skx_edac_common(E) button(E) ena(E) ghash_clmulni_intel(E) sch_fq_codel(E) drm(E) i2c_core(E) dm_mod(E) drm_panel_orientation_quirks(E) backlight(E) fuse(E) loop(E) dax(E) configfs(E) dmi_sysfs(E) efivarfs(E)
> [  460.056272] CPU: 1 UID: 0 PID: 32 Comm: kauditd Tainted: G            EL      6.16.0-rc7+ #3 PREEMPT(none)
> [  460.056275] Tainted: [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
> [  460.056276] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
> [  460.056277] RIP: 0010:_raw_spin_unlock_irqrestore+0x1b/0x30
> [  460.056284] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 16 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d b5 23 b6 01 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 e9 14 23 00 00 0f 1f 40 00 90
> [  460.056285] RSP: 0018:ffffb762c0123d70 EFLAGS: 00000246
> [  460.056287] RAX: 0000000000000001 RBX: ffff9b14c08eafc0 RCX: ffff9b14c3337348
> [  460.056288] RDX: ffff9b14c3337348 RSI: 0000000000000246 RDI: ffff9b14c3337340
> [  460.056289] RBP: ffff9b14c3337000 R08: ffffffff93cea880 R09: 0000000000000001
> [  460.056290] R10: 0000000000000001 R11: 0000000000000080 R12: ffff9b14c1b72500
> [  460.056291] R13: ffffb762c0123de0 R14: ffff9b14c3337340 R15: ffff9b14c3337080
> [  460.056294] FS:  0000000000000000(0000) GS:ffff9b1563788000(0000) knlGS:0000000000000000
> [  460.056296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  460.056297] CR2: 00007f36fd5d21b4 CR3: 000000010241a002 CR4: 00000000007706f0
> [  460.056298] PKRU: 55555554
> [  460.056299] Call Trace:
> [  460.056300]  <TASK>
> [  460.056302]  netlink_attachskb+0xb7/0x2f0
> [  460.056308]  ? __pfx_default_wake_function+0x10/0x10
> [  460.056313]  netlink_unicast+0xea/0x3b0
...
>
> We've bug reports from many users, so the issue is rather wide-spread.
>
> So far I don't know why the commit is causing this issue and will keep
> investigating. However, when reverted (together with its 2 follow-up
> patches), the issue goes away and host do not lock up.

Thanks for the report, Max!

Does your tree have this commit ?  This is the 3rd follow-up patch.

commit 759dfc7d04bab1b0b86113f1164dc1fec192b859
Author: Fedor Pchelkin <pchelkin@...ras.ru>
Date:   Mon Jul 28 08:06:47 2025

    netlink: avoid infinite retry looping in netlink_unicast()