netdev - Re: [RFC PATCH net] net: dsa: flush switchdev workqueue before tearing down CPU/DSA ports

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20210913113126.34th7ubolxwngi3c@skbuf>
Date:   Mon, 13 Sep 2021 14:31:26 +0300
From:   Vladimir Oltean <olteanv@...il.com>
To:     Florian Fainelli <f.fainelli@...il.com>
Cc:     Vladimir Oltean <vladimir.oltean@....com>, netdev@...r.kernel.org,
        Andrew Lunn <andrew@...n.ch>,
        Vivien Didelot <vivien.didelot@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>
Subject: Re: [RFC PATCH net] net: dsa: flush switchdev workqueue before
 tearing down CPU/DSA ports

On Sun, Sep 12, 2021 at 07:20:23PM -0700, Florian Fainelli wrote:
> On 9/12/2021 7:12 PM, Vladimir Oltean wrote:
> > On Sun, Sep 12, 2021 at 07:06:25PM -0700, Florian Fainelli wrote:
> > > On 9/12/2021 9:33 AM, Vladimir Oltean wrote:
> > > > On Sun, Sep 12, 2021 at 09:24:53AM -0700, Florian Fainelli wrote:
> > > > > On 9/12/2021 9:19 AM, Vladimir Oltean wrote:
> > > > > > On Sun, Sep 12, 2021 at 09:13:36AM -0700, Florian Fainelli wrote:
> > > > > > > Reviewed-by: Florian Fainelli <f.fainelli@...il.com>
> > > > > > >
> > > > > > > Did you post this as a RFC for a particular reason, or just to give
> > > > > > > reviewers some time?
> > > > > >
> > > > > > Both.
> > > > > >
> > > > > > In principle there's nothing wrong with what this patch does, only
> > > > > > perhaps maybe something with what it doesn't do.
> > > > > >
> > > > > > We keep saying that a network interface should be ready to pass traffic
> > > > > > as soon as it's registered, but that "walk dst->ports linearly when
> > > > > > calling dsa_port_setup" might not really live up to that promise.
> > > > >
> > > > > That promise most definitively existed back when Lennert wrote this code and
> > > > > we had an array of ports and the switch drivers brought up their port in
> > > > > their ->setup() method, nowadays, not so sure anymore because of the
> > > > > .port_enable() as much as the list.
> > > > >
> > > > > This is making me wonder whether the occasional messages I am seeing on
> > > > > system suspend from __dev_queue_xmit: Virtual device %s asks to queue
> > > > > packet! might have something to do with that and/or the inappropriate
> > > > > ordering between suspending the switch and the DSA master.
> > > >
> > > > Sorry, I have never tested the suspend/resume code path, mostly because
> > > > I don't know what would the easiest way be to wake up my systems from
> > > > suspend. If you could give me some pointers there I would be glad to
> > > > look into it.
> > >
> > > If your systems support suspend/resume just do:
> > >
> > > echo mem > /sys/power/state
> > > or
> > > echo standby > /sys/power/state
> > >
> > > if they don't, then maybe a x86 VM with dsa_loop may precipitate the
> > > problem, but since it uses DSA_TAG_PROTO_NONE, I doubt it, we would need to
> > > pass traffic on the DSA devices for this warning to show up.
> >
> > I figured out a working combination in the meanwhile, I even found a bug
> > in the process:
> > https://patchwork.kernel.org/project/netdevbpf/patch/20210912192805.1394305-1-vladimir.oltean@nxp.com/
> >
> > However I did not see those messages getting printed while pinging after
> > system resume (note that none of the DSA switch drivers I tested with
> > did implement .suspend or .resume), with net-next or with linux-stable/linux-5.14.y.
> >
> > Is there more to your setup to reproduce this issue?
>
> All switch ports are brought up with a DHCP client, the issue is
> definitively intermittent and not frequent, I don't have suspend/resume
> working on 5.14.y yet, but going as far back as 5.10.y should let you see
> the same kind of messages.

Nope, I don't see the message on linux-5.10.y either, but even if I look
at the code I don't understand what goes on.

Here are some facts:

- __dev_queue_xmit only prints "Virtual device %s asks to queue packet!"
  for devices which have a NULL dev->qdisc->enqueue, but return
  NETDEV_TX_BUSY in dev_hard_start_xmit

- only the noqueue qdisc manages that performance, normally register_qdisc
  sets up a default of ->enqueue pointer of noop_qdisc_ops.enqueue, but
  there is a hack in noqueue_init which subverts that.

- The default qdiscs are assigned by attach_default_qdiscs(). There are
  two ways in which an interface can get the noqueue_qdisc_ops by default:
  (a) it declares the IFF_NO_QUEUE feature
  (b) the init of its other default qdisc (for example mq) failed for
      some reason, the kernel prints a warning and falls back to noqueue.

- DSA declares IFF_NO_QUEUE, so it uses the noqueue qdisc by default

- DSA never returns anything other than NETDEV_TX_OK in ndo_start_xmit.

So my conclusion is:
you have the noqueue qdisc on the DSA master, which got there either by
accident due to a failure of the mq qdisc initialization, or
intentionally by:

tc qdisc add dev eth0 root noqueue

When you use the noqueue qdisc and your DSA master device driver returns
NETDEV_TX_BUSY, the packet won't get queued => the system pretty much is
in its own right to complain.