netdev - consistency for statistics with XDP mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <1d9a6548-4d1d-6624-e808-6ab0460a8655@gmail.com>
Date:   Wed, 21 Nov 2018 14:06:49 -0700
From:   David Ahern <dsahern@...il.com>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Cc:     Paweł Staszewski <pstaszewski@...are.pl>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Saeed Mahameed <saeedm@...lanox.com>,
        Jason Wang <jasowang@...hat.com>,
        "Michael S. Tsirkin" <mst@...hat.com>,
        David Miller <davem@...emloft.net>
Subject: consistency for statistics with XDP mode

Paweł ran some more XDP tests yesterday and from it found a couple of
issues. One is a panic in the mlx5 driver unloading the bpf program
(mlx5e_xdp_xmit); he will send a send a separate email for that problem.

The problem I wanted to discuss here is statistics for XDP context. The
short of it is that we need consistency in the counters across NIC
drivers and virtual devices. Right now stats are specific to a driver
with no clear accounting for the packets and bytes handled in XDP.

For example virtio has some stats as device private data extracted via
ethtool:
$ ethtool -S eth2 | grep xdp
    ...
     rx_queue_3_xdp_packets: 5291
     rx_queue_3_xdp_tx: 0
     rx_queue_3_xdp_redirects: 5163
     rx_queue_3_xdp_drops: 0
    ...
     tx_queue_3_xdp_tx: 5163
     tx_queue_3_xdp_tx_drops: 0

And the standard counters appear to track bytes and packets for Rx, but
not Tx if the packet is forwarded in XDP.

Similarly, mlx5 has some counters (thanks to Jesper and Toke for helping
out here):

$ ethtool -S mlx5p1 | grep xdp
     rx_xdp_drop: 86468350180
     rx_xdp_redirect: 18860584
     rx_xdp_tx_xmit: 0
     rx_xdp_tx_full: 0
     rx_xdp_tx_err: 0
     rx_xdp_tx_cqe: 0
     tx_xdp_xmit: 0
     tx_xdp_full: 0
     tx_xdp_err: 0
     tx_xdp_cqes: 0
    ...
     rx3_xdp_drop: 86468350180
     rx3_xdp_redirect: 18860556
     rx3_xdp_tx_xmit: 0
     rx3_xdp_tx_full: 0
     rx3_xdp_tx_err: 0
     rx3_xdp_tx_cqes: 0
    ...
     tx0_xdp_xmit: 0
     tx0_xdp_full: 0
     tx0_xdp_err: 0
     tx0_xdp_cqes: 0
    ...

And no accounting in standard stats for packets handled in XDP.

And then if I understand Jesper's data correctly, the i40e driver does
not have device specific data:

$ ethtool -S i40e1  | grep xdp
[NOTHING]

But rather bumps the standard counters:

sudo ./xdp_rxq_info --dev i40e1 --action XDP_DROP

Running XDP on dev:i40e1 (ifindex:3) action:XDP_DROP options:no_touch
XDP stats       CPU     pps         issue-pps
XDP-RX CPU      1       36,156,872  0
XDP-RX CPU      total   36,156,872

RXQ stats       RXQ:CPU pps         issue-pps
rx_queue_index    1:1   36,156,878  0
rx_queue_index    1:sum 36,156,878

$ ethtool_stats.pl --dev i40e1

Show adapter(s) (i40e1) statistics (ONLY that changed!)
Ethtool(i40e1   ) stat:   2711292859 (  2,711,292,859) <= port.rx_bytes /sec
Ethtool(i40e1   ) stat:      6274204 (      6,274,204) <=
port.rx_dropped /sec
Ethtool(i40e1   ) stat:     42363867 (     42,363,867) <=
port.rx_size_64 /sec
Ethtool(i40e1   ) stat:     42363950 (     42,363,950) <=
port.rx_unicast /sec
Ethtool(i40e1   ) stat:   2165051990 (  2,165,051,990) <= rx-1.bytes /sec
Ethtool(i40e1   ) stat:     36084200 (     36,084,200) <= rx-1.packets /sec
Ethtool(i40e1   ) stat:         5385 (          5,385) <= rx_dropped /sec
Ethtool(i40e1   ) stat:     36089727 (     36,089,727) <= rx_unicast /sec

We really need consistency in the counters and at a minimum, users
should be able to track packet and byte counters for both Rx and Tx
including XDP.

It seems to me the Rx and Tx packet, byte and dropped counters returned
for the standard device stats (/proc/net/dev, ip -s li show, ...) should
include all packets managed by the driver regardless of whether they are
forwarded / dropped in XDP or go up the Linux stack. This also aligns
with mlxsw and the stats it shows which are packets handled by the hardware.

>From there the private stats can include XDP specifics as desired --
like the drops and redirects but that those should be add-ons and even
here some consistency makes life easier for users.

The same standards should be also be applied to virtual devices built on
top of the ports -- e.g,  vlans. I have an API now that allows bumping
stats for vlan devices.

Keeping the basic xdp packets in the standard counters allows Paweł, for
example, to continue to monitor /proc/net/dev.

Can we get agreement on this? And from there, get updates to the mlx5
and virtio drivers?

David