netdev - RE: XDP seeking input from NIC hardware vendors

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Thu, 7 Jul 2016 15:18:11 +0000
From:	"Fastabend, John R" <john.r.fastabend@...el.com>
To:	Jesper Dangaard Brouer <brouer@...hat.com>,
	"iovisor-dev@...ts.iovisor.org" <iovisor-dev@...ts.iovisor.org>
CC:	Brenden Blanco <bblanco@...mgrid.com>,
	Alexei Starovoitov <alexei.starovoitov@...il.com>,
	Rana Shahout <ranas@...lanox.com>, Ari Saha <as754m@....com>,
	Tariq Toukan <tariqt@...lanox.com>,
	Or Gerlitz <ogerlitz@...lanox.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	Simon Horman <horms@...ge.net.au>,
	Simon Horman <simon.horman@...ronome.com>,
	Jakub Kicinski <jakub.kicinski@...ronome.com>,
	Edward Cree <ecree@...arflare.com>
Subject: RE: XDP seeking input from NIC hardware vendors

Hi Jesper,

I have done some previous work on proprietary systems where we used hardware to do the classification/parsing then passed a cookie to the software which used the cookie to lookup a program to run on the packet. When your programs are structured as a bunch of parsing followed by some actions this can provide real performance benefits. Also a lot of existing hardware supports this today assuming you use headers the hardware "knows" about. It's a natural model for hardware that uses a parser followed by tcam/cam/sram/etc lookup tables.

If the goal is to just separate XDP traffic from non-XDP traffic you could accomplish this with a combination of SR-IOV/macvlan to separate the device queues into multiple netdevs and then run XDP on just one of the netdevs. Then use flow director (ethtool) or 'tc cls_u32/flower' to steer traffic to the netdev. This is how we support multiple networking stacks on one device by the way it is called the bifurcated driver. Its not too far of a stretch to think we could offload some simple XDP programs to program the splitting of traffic instead of cls_u32/flower/flow_director and then you would have a stack of XDP programs. One running in hardware and a set running on the queues in software.

The other interesting thing would be to do more than just packet steering but actually run a more complete XDP program. Netronome supports this right. The question I have though is this a stacked of XDP programs one or more designated for hardware and some running in software perhaps with some annotation in the program so the hardware JIT knows where to place programs or do we expect the JIT itself to try and decide what is best to offload. I think the easiest to start with is to annotate the programs.

Also as far as I know a lot of hardware can stick extra data to the front or end of a packet so you could push metadata calculated by the program here in a generic way without having to extend XDP defined metadata structures. Another option is to DMA the metadata to a specified address. With this metadata the consumer/producer XDP programs have to agree on the format but no one else.

FWIW I was hoping to get some data to show performance overhead vs how deep we parse into the packets. I just wont have time to get to it for awhile but that could tell us how much perf gain the hardware could provide.

Thanks,
John

-----Original Message-----
From: Jesper Dangaard Brouer [mailto:brouer@...hat.com] 
Sent: Thursday, July 7, 2016 3:43 AM
To: iovisor-dev@...ts.iovisor.org
Cc: brouer@...hat.com; Brenden Blanco <bblanco@...mgrid.com>; Alexei Starovoitov <alexei.starovoitov@...il.com>; Rana Shahout <ranas@...lanox.com>; Ari Saha <as754m@....com>; Tariq Toukan <tariqt@...lanox.com>; Or Gerlitz <ogerlitz@...lanox.com>; netdev@...r.kernel.org; Simon Horman <horms@...ge.net.au>; Simon Horman <simon.horman@...ronome.com>; Jakub Kicinski <jakub.kicinski@...ronome.com>; Edward Cree <ecree@...arflare.com>; Fastabend, John R <john.r.fastabend@...el.com>
Subject: XDP seeking input from NIC hardware vendors


Would it make sense from a hardware point of view, to split the XDP eBPF program into two stages.

 Stage-1: Filter (restricted eBPF / no-helper calls)
 Stage-2: Program

Then the HW can choose to offload stage-1 "filter", and keep the likely more advanced stage-2 on the kernel side.  Do HW vendors see a benefit of this approach?


The generic problem I'm trying to solve is parsing. E.g. that the first step in every XDP program will be to parse the packet-data, in-order to determine if this is a packet the XDP program should process.

Actions from stage-1 "filter" program:
 - DROP (like XDP_DROP, early drop)
 - PASS (like XDP_PASS, normal netstack)
 - MATCH (call stage-2, likely carry-over opaque return code)

The MATCH action should likely carry-over an opaque return code, that makes sense for the stage-2 program. E.g. proto id and/or data offset.

--
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.