netdev - Re: Fw: [Bug 220774] New: netem is broken in 6.18

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAM0EoMkPdyqEMa0f4msEveGJxxd9oYaV4f3NatVXR9Fb=iCTcw@mail.gmail.com>
Date: Fri, 21 Nov 2025 07:52:37 -0500
From: Jamal Hadi Salim <jhs@...atatu.com>
To: Cong Wang <xiyou.wangcong@...il.com>
Cc: Stephen Hemminger <stephen@...workplumber.org>, netdev@...r.kernel.org, kuba@...nel.org, 
	linux-kernel@...r.kernel.org, will@...lsroot.io, jschung2@...ton.me, 
	savy@...t3mfailure.io
Subject: Re: Fw: [Bug 220774] New: netem is broken in 6.18

On Thu, Nov 20, 2025 at 11:29 PM Cong Wang <xiyou.wangcong@...il.com> wrote:
>
> Hi Will, Jamal and Jakub,
>
> I already warned you many times before you applied it. Now we have users
> complaining, please let me know if you still respect users.
>
> Also, Jamal, if I remember correctly, you said you will work on a long
> term solution, now after 4 months, please let us know what your plan is.
>
> Regards,
> Cong
>
>
> On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> > Regression caused by:
> >
> > commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> > Author: William Liu <will@...lsroot.io>
> > Date:   Tue Jul 8 16:43:26 2025 +0000
> >
> >     net/sched: Restrict conditions for adding duplicating netems to qdisc tree
> >
> >     netem_enqueue's duplication prevention logic breaks when a netem
> >     resides in a qdisc tree with other netems - this can lead to a
> >     soft lockup and OOM loop in netem_dequeue, as seen in [1].
> >     Ensure that a duplicating netem cannot exist in a tree with other
> >     netems.
> >
> >     Previous approaches suggested in discussions in chronological order:
> >
> >     1) Track duplication status or ttl in the sk_buff struct. Considered
> >     too specific a use case to extend such a struct, though this would
> >     be a resilient fix and address other previous and potential future
> >     DOS bugs like the one described in loopy fun [2].
> >
> >     2) Restrict netem_enqueue recursion depth like in act_mirred with a
> >     per cpu variable. However, netem_dequeue can call enqueue on its
> >     child, and the depth restriction could be bypassed if the child is a
> >     netem.
> >
> >     3) Use the same approach as in 2, but add metadata in netem_skb_cb
> >     to handle the netem_dequeue case and track a packet's involvement
> >     in duplication. This is an overly complex approach, and Jamal
> >     notes that the skb cb can be overwritten to circumvent this
> >     safeguard.
> >
> >     4) Prevent the addition of a netem to a qdisc tree if its ancestral
> >     path contains a netem. However, filters and actions can cause a
> >     packet to change paths when re-enqueued to the root from netem
> >     duplication, leading us to the current solution: prevent a
> >     duplicating netem from inhabiting the same tree as other netems.
> >
> >     [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
> >     [2] https://lwn.net/Articles/719297/
> >
> >     Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
> >     Reported-by: William Liu <will@...lsroot.io>
> >     Reported-by: Savino Dicanosa <savy@...t3mfailure.io>
> >     Signed-off-by: William Liu <will@...lsroot.io>
> >     Signed-off-by: Savino Dicanosa <savy@...t3mfailure.io>
> >     Acked-by: Jamal Hadi Salim <jhs@...atatu.com>
> >     Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
> >     Signed-off-by: Jakub Kicinski <kuba@...nel.org>
> >
> >
> > Begin forwarded message:
> >
> > Date: Mon, 10 Nov 2025 19:13:57 +0000
> > From: bugzilla-daemon@...nel.org
> > To: stephen@...workplumber.org
> > Subject: [Bug 220774] New: netem is broken in 6.18
> >
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=220774
> >
> >             Bug ID: 220774
> >            Summary: netem is broken in 6.18
> >            Product: Networking
> >            Version: 2.5
> >           Hardware: All
> >                 OS: Linux
> >             Status: NEW
> >           Severity: high
> >           Priority: P3
> >          Component: Other
> >           Assignee: stephen@...workplumber.org
> >           Reporter: jschung2@...ton.me
> >         Regression: No
> >
> > [jschung@...alhost ~]$ cat test.sh
> > #!/bin/bash
> >
> > DEV="eth0"
> > NUM_QUEUES=32
> > DUPLICATE_PERCENT="5%"
> >
> > tc qdisc del dev $DEV root > /dev/null 2>&1
> > tc qdisc add dev $DEV root handle 1: mq
> >
> > for i in $(seq 1 $NUM_QUEUES); do
> >     HANDLE_ID=$((i * 10))
> >     PARENT_ID="1:$i"
> >     tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> > duplicate $DUPLICATE_PERCENT
> > done
> >

jschung2@...ton.me: Can you please provide more details about what you
are trying to do so we can see if a different approach can be
prescribed?

cheers,
jamal

> > [jschung@...alhost ~]$ sudo ./test.sh
> > [  2976.073299] netem: change failed
> > Error: netem: cannot mix duplicating netems with other netems in tree.
> >
> > [jschung@...alhost ~]$ uname -r
> > 6.18.0-rc4
> >
> > --
> > You may reply to this email to add a comment.
> >
> > You are receiving this mail because:
> > You are the assignee for the bug.