Message-ID: <oXMTlZ5OaURBe0X3RZCO7zyNf6JJFPYvDW0AiXEg0bXJwXXYJutLhhjmUbetBUD_pGChlN7hDCCx9xFOtj8Hke5G7SM3-u5FQFC5e4T1wPY=@proton.me>
Date: Sat, 22 Nov 2025 06:56:18 +0000
From: 정지수 <jschung2@...ton.me>
To: Jamal Hadi Salim <jhs@...atatu.com>
Cc: Cong Wang <xiyou.wangcong@...il.com>, Stephen Hemminger <stephen@...workplumber.org>, netdev@...r.kernel.org, kuba@...nel.org, linux-kernel@...r.kernel.org, will@...lsroot.io, savy@...t3mfailure.io
Subject: Re: Fw: [Bug 220774] New: netem is broken in 6.18


#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; }
while [[ $# -gt 0 ]]; do
  case "$1" in
    -d) DEV="$2"; shift 2;;
    -q) QUEUE="$2"; shift 2;;
    -P) RTP_DST_PORT="$2"; shift 2;;
    *) usage;;
  endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }


if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
  echo "Setting root qdisc to mq.."
  tc qdisc replace dev "$DEV" root handle 1: mq
fi


echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..."
ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
  echo "some flows will still land on :$QUEUE"
}


echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.."
tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
  netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true


if command -v tshark >/dev/null 2>&1; then
  echo "tshark live RTP view :"
  timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):"
tc -s qdisc show dev "$DEV" | sed -n '1,200p'




Sent with Proton Mail secure email.

On Friday, November 21st, 2025 at 12:52, Jamal Hadi Salim <jhs@...atatu.com> wrote:

> On Thu, Nov 20, 2025 at 11:29 PM Cong Wang xiyou.wangcong@...il.com wrote:
> 
> > Hi Will, Jamal and Jakub,
> > 
> > I already warned you many times before you applied it. Now we have users
> > complaining, please let me know if you still respect users.
> > 
> > Also, Jamal, if I remember correctly, you said you will work on a long
> > term solution, now after 4 months, please let us know what your plan is.
> > 
> > Regards,
> > Cong
> > 
> > On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> > 
> > > Regression caused by:
> > > 
> > > commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> > > Author: William Liu will@...lsroot.io
> > > Date: Tue Jul 8 16:43:26 2025 +0000
> > > 
> > > net/sched: Restrict conditions for adding duplicating netems to qdisc tree
> > > 
> > > netem_enqueue's duplication prevention logic breaks when a netem
> > > resides in a qdisc tree with other netems - this can lead to a
> > > soft lockup and OOM loop in netem_dequeue, as seen in [1].
> > > Ensure that a duplicating netem cannot exist in a tree with other
> > > netems.
> > > 
> > > Previous approaches suggested in discussions in chronological order:
> > > 
> > > 1) Track duplication status or ttl in the sk_buff struct. Considered
> > > too specific a use case to extend such a struct, though this would
> > > be a resilient fix and address other previous and potential future
> > > DOS bugs like the one described in loopy fun [2].
> > > 
> > > 2) Restrict netem_enqueue recursion depth like in act_mirred with a
> > > per cpu variable. However, netem_dequeue can call enqueue on its
> > > child, and the depth restriction could be bypassed if the child is a
> > > netem.
> > > 
> > > 3) Use the same approach as in 2, but add metadata in netem_skb_cb
> > > to handle the netem_dequeue case and track a packet's involvement
> > > in duplication. This is an overly complex approach, and Jamal
> > > notes that the skb cb can be overwritten to circumvent this
> > > safeguard.
> > > 
> > > 4) Prevent the addition of a netem to a qdisc tree if its ancestral
> > > path contains a netem. However, filters and actions can cause a
> > > packet to change paths when re-enqueued to the root from netem
> > > duplication, leading us to the current solution: prevent a
> > > duplicating netem from inhabiting the same tree as other netems.
> > > 
> > > [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
> > > [2] https://lwn.net/Articles/719297/
> > > 
> > > Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
> > > Reported-by: William Liu will@...lsroot.io
> > > Reported-by: Savino Dicanosa savy@...t3mfailure.io
> > > Signed-off-by: William Liu will@...lsroot.io
> > > Signed-off-by: Savino Dicanosa savy@...t3mfailure.io
> > > Acked-by: Jamal Hadi Salim jhs@...atatu.com
> > > Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
> > > Signed-off-by: Jakub Kicinski kuba@...nel.org
> > > 
> > > Begin forwarded message:
> > > 
> > > Date: Mon, 10 Nov 2025 19:13:57 +0000
> > > From: bugzilla-daemon@...nel.org
> > > To: stephen@...workplumber.org
> > > Subject: [Bug 220774] New: netem is broken in 6.18
> > > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=220774
> > > 
> > > Bug ID: 220774
> > > Summary: netem is broken in 6.18
> > > Product: Networking
> > > Version: 2.5
> > > Hardware: All
> > > OS: Linux
> > > Status: NEW
> > > Severity: high
> > > Priority: P3
> > > Component: Other
> > > Assignee: stephen@...workplumber.org
> > > Reporter: jschung2@...ton.me
> > > Regression: No
> > > 
> > > [jschung@...alhost ~]$ cat test.sh
> > > #!/bin/bash
> > > 
> > > DEV="eth0"
> > > NUM_QUEUES=32
> > > DUPLICATE_PERCENT="5%"
> > > 
> > > tc qdisc del dev $DEV root > /dev/null 2>&1
> > > tc qdisc add dev $DEV root handle 1: mq
> > > 
> > > for i in $(seq 1 $NUM_QUEUES); do
> > > HANDLE_ID=$((i * 10))
> > > PARENT_ID="1:$i"
> > > tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> > > duplicate $DUPLICATE_PERCENT
> > > done
> 
> 
> jschung2@...ton.me: Can you please provide more details about what you
> are trying to do so we can see if a different approach can be
> prescribed?
> 
> cheers,
> jamal
> 
> > > [jschung@...alhost ~]$ sudo ./test.sh
> > > [ 2976.073299] netem: change failed
> > > Error: netem: cannot mix duplicating netems with other netems in tree.
> > > 
> > > [jschung@...alhost ~]$ uname -r
> > > 6.18.0-rc4
> > > 
> > > --
> > > You may reply to this email to add a comment.
> > > 
> > > You are receiving this mail because:
> > > You are the assignee for the bug.

Powered by blists - more mailing lists