netdev - Re: Fw: [Bug 220774] New: netem is broken in 6.18

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aR/qwlyEWm/pFAfM@pop-os.localdomain>
Date: Thu, 20 Nov 2025 20:29:54 -0800
From: Cong Wang <xiyou.wangcong@...il.com>
To: Stephen Hemminger <stephen@...workplumber.org>
Cc: netdev@...r.kernel.org, jhs@...atatu.com, kuba@...nel.org,
	linux-kernel@...r.kernel.org, will@...lsroot.io, jschung2@...ton.me,
	savy@...t3mfailure.io
Subject: Re: Fw: [Bug 220774] New: netem is broken in 6.18

Hi Will, Jamal and Jakub,

I already warned you many times before you applied it. Now we have users
complaining, please let me know if you still respect users.

Also, Jamal, if I remember correctly, you said you will work on a long
term solution, now after 4 months, please let us know what your plan is.

Regards,
Cong


On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> Regression caused by:
> 
> commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> Author: William Liu <will@...lsroot.io>
> Date:   Tue Jul 8 16:43:26 2025 +0000
> 
>     net/sched: Restrict conditions for adding duplicating netems to qdisc tree
>     
>     netem_enqueue's duplication prevention logic breaks when a netem
>     resides in a qdisc tree with other netems - this can lead to a
>     soft lockup and OOM loop in netem_dequeue, as seen in [1].
>     Ensure that a duplicating netem cannot exist in a tree with other
>     netems.
>     
>     Previous approaches suggested in discussions in chronological order:
>     
>     1) Track duplication status or ttl in the sk_buff struct. Considered
>     too specific a use case to extend such a struct, though this would
>     be a resilient fix and address other previous and potential future
>     DOS bugs like the one described in loopy fun [2].
>     
>     2) Restrict netem_enqueue recursion depth like in act_mirred with a
>     per cpu variable. However, netem_dequeue can call enqueue on its
>     child, and the depth restriction could be bypassed if the child is a
>     netem.
>     
>     3) Use the same approach as in 2, but add metadata in netem_skb_cb
>     to handle the netem_dequeue case and track a packet's involvement
>     in duplication. This is an overly complex approach, and Jamal
>     notes that the skb cb can be overwritten to circumvent this
>     safeguard.
>     
>     4) Prevent the addition of a netem to a qdisc tree if its ancestral
>     path contains a netem. However, filters and actions can cause a
>     packet to change paths when re-enqueued to the root from netem
>     duplication, leading us to the current solution: prevent a
>     duplicating netem from inhabiting the same tree as other netems.
>     
>     [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
>     [2] https://lwn.net/Articles/719297/
>     
>     Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
>     Reported-by: William Liu <will@...lsroot.io>
>     Reported-by: Savino Dicanosa <savy@...t3mfailure.io>
>     Signed-off-by: William Liu <will@...lsroot.io>
>     Signed-off-by: Savino Dicanosa <savy@...t3mfailure.io>
>     Acked-by: Jamal Hadi Salim <jhs@...atatu.com>
>     Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
>     Signed-off-by: Jakub Kicinski <kuba@...nel.org>
> 
> 
> Begin forwarded message:
> 
> Date: Mon, 10 Nov 2025 19:13:57 +0000
> From: bugzilla-daemon@...nel.org
> To: stephen@...workplumber.org
> Subject: [Bug 220774] New: netem is broken in 6.18
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=220774
> 
>             Bug ID: 220774
>            Summary: netem is broken in 6.18
>            Product: Networking
>            Version: 2.5
>           Hardware: All
>                 OS: Linux
>             Status: NEW
>           Severity: high
>           Priority: P3
>          Component: Other
>           Assignee: stephen@...workplumber.org
>           Reporter: jschung2@...ton.me
>         Regression: No
> 
> [jschung@...alhost ~]$ cat test.sh 
> #!/bin/bash
> 
> DEV="eth0"
> NUM_QUEUES=32
> DUPLICATE_PERCENT="5%"
> 
> tc qdisc del dev $DEV root > /dev/null 2>&1
> tc qdisc add dev $DEV root handle 1: mq
> 
> for i in $(seq 1 $NUM_QUEUES); do
>     HANDLE_ID=$((i * 10))
>     PARENT_ID="1:$i"
>     tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> duplicate $DUPLICATE_PERCENT
> done
> 
> [jschung@...alhost ~]$ sudo ./test.sh 
> [  2976.073299] netem: change failed
> Error: netem: cannot mix duplicating netems with other netems in tree.
> 
> [jschung@...alhost ~]$ uname -r
> 6.18.0-rc4
> 
> -- 
> You may reply to this email to add a comment.
> 
> You are receiving this mail because:
> You are the assignee for the bug.