lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 15 Dec 2022 13:50:15 -0600
From:   Alex Forster <>
To:     Magnus Karlsson <>
Cc:     Shawn Bohrer <>,,,,,
Subject: Re: Possible race with xsk_flush

Hi Magnus,

> Could you please share how you set up the two AF_XDP sockets?

Our architecture is pretty unique:

   outside of │ inside of
    namespace │ namespace
    ┌───────┐ │ ┌───────┐
    │ outer │ │ │ inner │
    │  veth │ │ │ veth  │
    └──┬─▲──┘ │ └──┬─▲──┘
       │ │    │    │ │
    │    shared umem    │

The goal is to position ourselves in the middle of a veth pair so that
we can perform bidirectional traffic inspection and manipulation. To do
this, we attach AF_XDP to both veth interfaces and share a umem between
them. This allows us to forward packets between the veth interfaces
without copying in userspace.

These interfaces are both multi-queue, with AF_XDP sockets attached to
each queue. The queues are each managed on their own (unpinned) threads
and have their own rx/tx/fill/completion rings. We also enable
threaded NAPI on both of these interfaces, which may or may not be an
important detail to note, since the problem appears much harder (though
not impossible) to reproduce with threaded NAPI enabled.

Here’s a script that configures a namespace and veth pair that closely
resembles production, except for enabling threaded NAPI:


set -e -u -x -o pipefail

QUEUES=${QUEUES:=$(($(grep -c ^processor /proc/cpuinfo)))}


ip netns exec $CUSTOMER_NAMESPACE bash <<EOF
  set -e -u -x -o pipefail
  ip addr add dev lo
  ip link set dev lo up

ip link add \
  numrxqueues $QUEUES numtxqueues $QUEUES type veth \
  numrxqueues $QUEUES numtxqueues $QUEUES

  gro off gso off tso off tx off rxvlan off txvlan off
ip link set dev $OUTER_CUSTOMER_VETH up
ip addr add dev $OUTER_CUSTOMER_VETH

ip netns exec $CUSTOMER_NAMESPACE bash <<EOF
  set -e -u -x -o pipefail
    gro off gso off tso off tx off rxvlan off txvlan off
  ip link set dev $INNER_CUSTOMER_VETH up
  ip addr add dev $INNER_CUSTOMER_VETH

> Are you using XDP_DRV mode in your tests?


Powered by blists - more mailing lists