Message-ID: <20150304011817.GA29502@vergenet.net>
Date:	Wed, 4 Mar 2015 10:18:21 +0900
From:	Simon Horman <simon.horman@...ronome.com>
To:	dev@...nvswitch.org
Cc:	netdev@...r.kernel.org
Subject: OVS Offload Decision Proposal

[ CCed netdev as although this is primarily about Open vSwitch userspace
  I believe there are some interested parties not on the Open vSwitch
  dev mailing list ]

Hi,

The purpose of this email is to describe a rough design for driving Open
vSwitch flow offload from user-space. But before getting to that I would
like to provide some background information.

The proposed design is for "OVS Offload Decision": a proposed component of
ovs-vswitchd. In short the top-most red box in the first figure in the
"OVS HW Offload Architecture" document edited by Thomas Graf[1].

[1] https://docs.google.com/document/d/195waUliu7G5YYVuXHmLmHgJ38DFSte321WPq0oaFhyU/edit#heading=h.116je16s8xzw

Assumptions
-----------

There is currently a lively debate on various aspects of flow offloads
within the Linux networking community. As of writing the latest discussion
centers around the "Flows! Offload them." thread[2] on the netdev mailing
list.

[2] http://thread.gmane.org/gmane.linux.network/351860

My aim is not to preempt the outcome of those discussions. But rather to
investigate what offloads might look like in ovs-vswitchd. In order to make
that investigation concrete I have made some assumptions about facilities
that may be provided by the kernel in future. Clearly if the discussions
within the Linux networking community end in a solution that differs from
my assumptions then this work will need to be revisited. Indeed, I entirely
expect this work to be revised and refined and possibly even radically
rethought as time goes on.

That said, my working assumptions are:

* That Open vSwitch may manage flow offloads from user-space. This is as
  opposed to them being transparently handled in the datapath. This does
  not preclude the existence of transparent offloading in the datapath.
  But rather limits this discussion to a mode where offloads are managed
  from user-space.

* That Open vSwitch may add flows to hardware via an API provided by the
  kernel. In particular my working assumption is that the Flow API proposed
  by John Fastabend[3] may be used to add flows to hardware. While the
  existing netlink API may be used to add flows to the kernel datapath.

* That there will be an API provided by the kernel to allow the discovery
  of hardware offload capabilities by user-space. Again my working
  assumption is that the Flow API proposed by John Fastabend[3] may be used
  for this purpose.

[3] http://thread.gmane.org/gmane.linux.network/347188

Rough Design
------------

* Modify flow translation so that the switch parent id[4] of the flow is
  recorded as part of its translation context. The switch parent id was
  recently added to the Linux kernel and provides a common identifier for
  all netdevices that are backed by the same underlying switch hardware for
  some very loose definition of switch. In this scheme if the input and all
  output ports of a flow belong to the same switch hardware then the switch
  id of the translation context would be set accordingly, indicating
  offload of the flow may occur to that switch.

  [4] https://github.com/torvalds/linux/blob/master/Documentation/networking/switchdev.txt

  At this time this excludes both flows that either span multiple switch
  devices or use vports that are not backed directly by netdevices, for
  example tunnel vports. While important I believe these are topics for
  further work.

* At the point where a flow is to be added to the datapath ovs-vswitchd
  should determine if it should be offloaded and if so translate it to a
  flow for the hardware offload API and queue this translated flow up to be
  added to hardware as well as the datapath.

  The translation to hardware flows could be performed along with the
  translation that already occurs from OpenFlow to ODP flows. However, that
  translation is already quite complex and called for a variety of reasons
  other than to prepare flows to be added to the datapath. So I think it
  makes some sense to keep the new translation separate from the existing
  one.

  The determination mentioned above could first check if the switch id is
  set and then may make further checks: for example that there is space in
  the hardware for a new flow, that all the matches and actions of the flow
  may be offloaded.

  There seems to be ample scope for complex logic to determine which flows
  should be offloaded. And I believe that one motivation for handling
  offloads in user-space for such complex logic to live in user-space.
  However, in order to keep things simple in the beginning I propose some
  very simple logic: offload all flows that the hardware supports up until
  the hardware runs out of space.

  This seems like a reasonable start keeping in mind that all flows will
  also be added to the datapath and that ovs-vswitchd constructs flows such
  that they do not overlap.

  A more conservative version of this simple rule would be to remove all
  flows from hardware if a flow is encountered that is not to be added to
  hardware. That is, ensure either all flows that are in hardware are also
  in software or no flows are in hardware at all. This is the approach
  being initially taken for L3 offloads in the Linux kernel[5].

  [5] http://thread.gmane.org/gmane.linux.network/352481/focus=352658

* It seems to me that somewhat tricky problem is how to manage flows in
  hardware. As things stand ovs-vswitchd generally manages flows in the
  datapath by dumping flows, inspecting the dumped flows to see how
  recently they have been used and removing idle flows from the datapath.
  Unfortunately this approach may not be well suited to flows offloaded to
  hardware as dumping flows may be prohibitively expensive. As such I would
  like some consideration given to three approaches. Perhaps in the end all
  will need to be supported. And perhaps there are others:

  1. Dump Flows
     This is the approach currently taken to managing datapath flows. As
     stated above my feeling is that this will not be well suited much
     hardware. However, for simplicity it may be a good place to start.

  2. Notifications
     In this approach flows are added to hardware with a soft timeout and
     hardware removes flows when they timeout sending a notification when
     that occurs. Notifications would be relayed up to user space from the
     driver in the kernel. Some effort may be required to mitigate
     notification storms if many flows are removed in a short space of
     time. It is also of note that there is likely to be hardware that
     can't generate notifications on flow removal.

  3. Aging in hardware
     In this approach flows are added to hardware with a soft timeout and
     hardware removes the flows when they timeout. However no notification
     is generated. And thus ovs-vswitchd has no way of knowing if a flow is
     still present in hardware or not. From a hardware point of view this
     seems to be the simplest to support. But I suspect that it would
     present some significant challenges to ovs-vswitchd in the context of
     its current implementation of flow management. Especially if flows are
     also to be present in the datapath as proposed above.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists