lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Wed, 8 Apr 2020 06:10:19 +0000
From:   Parav Pandit <parav@...lanox.com>
To:     Parav Pandit <parav@...lanox.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Saeed Mahameed <saeedm@...lanox.com>
CC:     "sridhar.samudrala@...el.com" <sridhar.samudrala@...el.com>,
        Aya Levin <ayal@...lanox.com>,
        "andrew.gospodarek@...adcom.com" <andrew.gospodarek@...adcom.com>,
        "sburla@...vell.com" <sburla@...vell.com>,
        "jiri@...nulli.us" <jiri@...nulli.us>,
        Tariq Toukan <tariqt@...lanox.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Vlad Buslov <vladbu@...lanox.com>,
        "lihong.yang@...el.com" <lihong.yang@...el.com>,
        Ido Schimmel <idosch@...lanox.com>,
        "jgg@...pe.ca" <jgg@...pe.ca>,
        "fmanlunas@...vell.com" <fmanlunas@...vell.com>,
        "oss-drivers@...ronome.com" <oss-drivers@...ronome.com>,
        "leon@...nel.org" <leon@...nel.org>,
        "grygorii.strashko@...com" <grygorii.strashko@...com>,
        "michael.chan@...adcom.com" <michael.chan@...adcom.com>,
        Alex Vesker <valex@...lanox.com>,
        "snelson@...sando.io" <snelson@...sando.io>,
        "linyunsheng@...wei.com" <linyunsheng@...wei.com>,
        "magnus.karlsson@...el.com" <magnus.karlsson@...el.com>,
        "dchickles@...vell.com" <dchickles@...vell.com>,
        "jacob.e.keller@...el.com" <jacob.e.keller@...el.com>,
        Moshe Shemesh <moshe@...lanox.com>,
        Mark Zhang <markz@...lanox.com>,
        "aelior@...vell.com" <aelior@...vell.com>,
        Yuval Avnery <yuvalav@...lanox.com>,
        "drivers@...sando.io" <drivers@...sando.io>,
        mlxsw <mlxsw@...lanox.com>,
        "GR-everest-linux-l2@...vell.com" <GR-everest-linux-l2@...vell.com>,
        Yevgeny Kliteynik <kliteyn@...lanox.com>,
        "vikas.gupta@...adcom.com" <vikas.gupta@...adcom.com>,
        Eran Ben Elisha <eranbe@...lanox.com>
Subject: RE: [RFC] current devlink extension plan for NICs

Hi Saeed,

> From: netdev-owner@...r.kernel.org <netdev-owner@...r.kernel.org> On
> Behalf Of Parav Pandit
> On 3/28/2020 2:12 AM, Jakub Kicinski wrote:
> > On Fri, 27 Mar 2020 19:45:53 +0000 Saeed Mahameed wrote:
> 
> >> from what i understand, a real slice is a full isolated HW pipeline
> >> with its own HW resources and HW based isolation, a slice rings/hw
> >> resources can never be shared between different slices, just like a
> >> vf, but without the pcie virtual function back-end..
> 
> >> We need a clear-cut definition of what a Sub-function slice is.. this
> >> RFC doesn't seem to address that clearly.
> >

Did you get a chance to review below content?
Can you please confirm if below description describes, what you were looking for?



> > Definitely. I'd say we need a clear definition of (a) what a
> > sub-functions is, and (b) what a slice is.
> >
> 
> Below is the extended contents for Jiri's RFC that addresses Saeed point on
> clear definition of slice, sub-function and their plumbing.
> Few things already defined by Jiri's RFC such as diagrams, examples, but
> putting it all together below for completeness.
> 

> This RFC extension covers overall requirements, design, alternatives
> considered and rationale for current design choices to support sub-functions
> using slices.
> 
> Overview:
> ---------
> A user wants to use multiple netdev and rdma devices of a PCI device
> without using PCI SR-IOV. This is done by creating slices of a PCI device using
> devlink tool.
> 
> Requirements:
> -------------
> 1. Able to make multiple slices of a device 2. Able to provision such slice to
> use in-kernel 3. Able to have persistent name of netdev and rdma device of a
> slice 4. Reuse current power management (suspend/resume) kernel infra for
>    slice devices; not reinvent locking etc in each driver or devlink 5. Able to
> control, steer, offload network traffic of slices using
>    existing devlink eswitch switchdev support 6. Suddenly not change existing
> pci pf/vf naming scheme by introducing
>    slice
> 7. Support slice of a device in an iommu enabled kernel 8. Dynamically
> create/delete a slice 9. Ability to configure a slice before deploying it 10.
> Get/Set slice attributes like PCI VF slice attributes 11. Reuse/extend virtbus
> for carrying SF slice devices 12. Hot-plug a slice device in host system from
> the NIC
>     (eswitch system) side without running any agent in the
>     host system
> 13. Have unified interface for slice management regardless of
>     deploying on eswitch system or on host system.
> 14. User must be able to create a portion of the device from eswitch
>     system and attach to the untrusted host system. This host system
>     is not accessible to eswitch system for purpose of device
>     life-cycle and initial configuration.
> 
> Slice:
> ------
> A slice represents a portion of the device. A slice is a generic object that
> represents either a PCI VF or PCI PF or PCI sub function (SF) described below.
> 
> Sub-function (SF):
> ------------------
> - An sub-function is a portion of the PCI device which supports multiple
>   class of devices such as netdev, rdma and more.
> - An SF netdev has its own dedicated queues(txq, rxq).
> - An SF rdma device has its own QP1, GID table and rdma resources.
>   An SF rdma resources has its own resource namespace.
> - An SF supports eswitch representation and full offload support
>   when it is running with eswitch support.
> - User must configure eswitch to send/receive packets for an SF.
> - An SF shares PCI level resources with other SFs and/or with its
>   parent PCI function.
>   For example, an SF shares IRQ vectors with other SFs and its
>   PCI function.
>   In future it may have dedicated IRQ vector per SF.
>   An SF has dedicated window in PCI BAR space that is not shared
>   with other SFs or PF. This ensures that when a SF is assigned to
>   an application, only that application can access device resources.
> 
> Overall design:
> ---------------
> A new flavour of slice is created that represents a portion of the device. It is
> equivalent to a PCI VF slice, but new slice exists without PCI SR-IOV. This can
> scale to possibly more than number of SR-IOV VFs. A new slice flavour 'pcisf'
> is introduced which is explained later in this document.
> 
> devlink subsystem is extended to create, delete and deploy a slice using a
> devlink instance.
> 
> Slice life cycle is done using devlink commands explained below.
> 
> (a) Lifecycle command consist of 3 main commands i.e. add, delete and
>     state change.
> (b) Add/delete commands create or delete the slice of a device
>     respectively.
> (c) A slice undergoes one or more configuration before it is
>     activated. This may include,
>      (a) slice hardware address configuration
>      (b) representor netdevice configuration
>      (c) Network policy, steering configuration through representor
> (d) Once a slice is fully configured, user activates it.
>     Slice activation triggers device enumeration and binding to the
>     driver.
> (e) Each slice goes through a state transition during its life cycle.
> (f) Each slice's admin state is controlled by the user. Slice
>     operational is updated based on driver attach/detach tasks on the
>     host system.
> 
>     such as,
>     admin state            description
>     -----------            ------------
>     1. inactive    Slice enter this state when it is newly created.
>                    User typically does most of the configuration in
>                    this state before activating the slice.
> 
>     2. active      State when slice is just activated by user.
> 
>     operational            description
>        state
>     -----------            ------------
>     1. attached    State when slice device is bound to the host
>                    driver. When the slice device is unbound from the
>                    host driver, slice device exits this state and
>                    enters detaching state.
> 
>     2. detaching   State when host is notified to deactivate the
>                    slice device and slice device may be undergoing
>                    detachment from host driver. When slice device is
>                    fully detached from the host driver, slice exits
>                    this state and enters detached state.
> 
>     3. detached    State when driver is fully unbound, it enters
>                    into detached state.
> 
> slice state machine:
> --------------------
>                                slice state set inactive
>                               ----<------------------<---
>                               | or  slice delete        |
>                               |                         |
>   __________              ____|_______              ____|_______
>  |          | slice add  |            |slice state |            |
>  |          |-------->---|            |------>-----|            |
>  | invalid  |            |  inactive  | set active |   active   |
>  |          | slice del  |            |            |            |
>  |__________|--<---------|____________|            |____________|
> 
> slice device operational state machine:
> ---------------------------------------
>   __________                ____________              ___________
>  |          | slice state  |            |driver bus  |           |
>  | invalid  |-------->-----|  detached  |------>-----| attached  |
>  |          | set active   |            | probe()    |           |
>  |__________|              |____________|            |___________|
>                                  |                        |
>                                  ^                    slice set
>                                  |                    set inactive
>                             successful detach             |
>                               or pf reset                 |
>                              ____|_______                 |
>                             |            | driver bus     |
>                  -----------| detaching  |---<-------------
>                  |          |            | remove()
>                  ^          |____________|
>                  |   timeout      |
>                  --<---------------
> 
> More on the slice states:
> (a) Primary reason to run slice life cycle this way is: a user must
>     be able to create and configure a slice on eswitch system, before
>     host system discovers the slice device. Hence, a slice
>     device will not be accessible from the host system until it is
>     explicitly activated by the user. Typically this is done after
>     all necessary slice attributes are configured.
> (b) A user wants to create and configure the slice on eswitch system
>     when host system is power-down state.
> (c) A slice interface for sub-functions should be uniform regardless
>     of slice devices are deployed in eswitch system or host system.
> (d) If a host system software where slice is deployed is compromised,
>     which doesn't detach the slice, such slice remains in detaching
>     state until PF driver is reset on the host. Such slice won't be
>     usable until it is detached gracefully by host software.
>     Forcefully changing its state and reusing it can lead to
>     unexpected behavior and access of slice resources. Hence when
>     opstate = detaching and state = inactive, such slice cannot be
>     activated.
> 
> SF sysfs, virtbus and devlink plumbing:
> ---------------------------------------
> (a) Given that a sub-function device may have PCI BAR resource(s), power
>     management, IRQ configuration, persistence naming etc, a clear sysfs
>     view like existing PCI device is desired.
> 
>     Therefore, an SF resides on a new bus called virtbus.
>     This virtbus holds one or more SFs of a parent PCI device.
> 
>     Whenever user activates a slice, corresponding sub-function device
>     is created on the virtbus and attached to the driver.
> 
> (b) Each SF has user defined unique number associated with it,
>     called 'sfnum'. This sfnum is provided during SF slice creation
>     time. Multiple uses of the sfnum is explained in detail below.
> (c) An active SF slice has a unique 'struct device' anchored on
>     virtbus. An SF is identified using unique name on the virtbus.
> (d) An SF's device (sysfs) name is created using ida assigned by the
>     virtbus core. such as,
>     /sys/bus/virtbus/devices/mlx5_sf.100
>     /sys/bus/virtbus/devices/mlx5_sf.2
> (e) Scope of a sfnum is within the devlink instance who supports SF
>     lifecycle and SF devlink port lifecycle.
>     This sfnum is populated in /sys/bus/virtbus/devices/100/sfnum
> (f) Persistent name of Netdevice and RDMA device of a virtbus SF
>     device is prepared using parent device of SF and sfnum of an SF.
>     Such as,
>     /sys/bus/virtbus/devices/mlx5_sf.100 ->
> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/100
>     Netdev name for such SF = enp6s0f0s1
>     Where <en><parent_dev naming scheme> <s for sf> <sfnum_in_decimal>
>     This generates unique netdev name because parent is involved in the
>     device naming.
>     RDMA device persistent name will be done similarly such as
>     'rocep6s0f0s1'.
> (g) SF devlink instance name is prepared using SF's parent bus/device
>     and sfnum. Such as, pci/0000.06.00.0%sf1.
>     This scheme ensures that SF's devlink device name remains
>     predictable using sfnum regardless of dynamic virtbus device
>     name/index.
> (h) Each virtbus SF device driver id is defined by virtbus core.
>     This driver id is used to bind virtbus SF device with the SF
>     driver which has matching driver-id provided during driver
>     registration time.
>     This is further visible via modpost tools too.
> (i) A new devlink eswitch port flavour named 'pcisf' is introduced for
>     PCI SF devices similar to existing flavour 'pcivf' for PCI VFs.
> (j) devlink eswitch port for PCI SF is added/deleted whenever SF slice
>     is added/deleted.
> (k) SF representor netdev phys_port_name=pf0sf1
>     Format is: pf<pf_number>sf<user_assigned_sfnum>
> 
> Examples:
> ---------
> - Add a slice of flavour SF with sf identifier 4 on eswitch system:
> 
> $ devlink slice add pci/0000.06.00.0 flavour pcisf pfnum 0 sfnum 4 index 100
> 
> - Show a newly added slice on eswitch system:
> $ devlink slice show pci/0000:06:00.0/100 -jp {
>     "slice": {
>         "pci/0000:06:00.0/100": {
>             "flavour": "pcisf",
>             "pfnum": 0,
>             "sfnum": 4,
>             "state" : "inactive",
>             "opstate" : "detached",
>         }
>     }
>   }
> 
> - Show eswitch port configuration for SF on eswitch system:
> $ devlink port show
> pci/0000:06:00.0/65535: type eth netdev ens2f0 flavour physical port 0
> pci/0000:06:00.0/8000: type eth netdev ens2f0sf1 flavour pcisf pfnum 0
> sfnum 4 Here eswitch port index 8000 is assigned by the vendor driver.
> 
> - Activate SF slice to trigger creating a virtbus SF device in host system:
> $ devlink slice set pci/0000.06.00.0/100 state active
> 
> - Query SF slice state on eswitch system:
> $ devlink slice show pci/0000:06:00.0/100 -jp {
>     "slice": {
>         "pci/0000:06:00.0/100": {
>             "flavour": "pcisf",
>             "pfnum": 0,
>             "sfnum": 4,
>             "state": "active",
>             "opstate": "detached",
>         }
>     }
>   }
> 
> - Query SF slice state on eswitch system once driver is loaded on SF:
> $ devlink slice show pci/0000:06:00.0/100 -jp {
>     "slice": {
>         "pci/0000:06:00.0/100": {
>             "flavour": "pcisf",
>             "pfnum": 0,
>             "sfnum": 4,
>             "state": "active",
>             "opstate": "attached",
>         }
>     }
>   }
> 
> - Show devlink instances on host system:
> $ devlink dev show
> pci/0000:06:00.0
> pci/0000:06:00.0%sf4
> 
> - Show devlink ports on host system:
> $ devlink port show
> pci/0000:06:00.0/0: flavour physical type eth netdev netdev enp6s0f0
> pci/0000:06:00.0%sf4/0: flavour virtual type eth netdev netdev enp6s0f0s10
> 
> - Mark a slice inactive, when slice may be in active state:
> $ devlink slice set pci/0000.06.00.0/100 state inactive
> 
> - Query SF slice state on eswitch system once inactivation is triggered:
> Output when it is in detaching state:
> 
> $ devlink slice show pci/0000:06:00.0/100 -jp {
>     "slice": {
>         "pci/0000:06:00.0/100": {
>             "flavour": "pcisf",
>             "pfnum": 0,
>             "sfnum": 4,
>             "state": "inactive",
>             "opstate": "detaching",
>         }
>     }
>   }
> 
> Output once detached:
> $ devlink slice show pci/0000:06:00.0/100 -jp {
>     "slice": {
>         "pci/0000:06:00.0/100": {
>             "flavour": "pcisf",
>             "pfnum": 0,
>             "sfnum": 4,
>             "active": "inactive",
>             "opstate": "detached",
>         }
>     }
>   }
> 
> - Delete the slice which is recently inactivated.
> $ devlink slice del pci/0000.06.00.0/100
> 
> FAQs:
> ----
> 1. What are the differences between PCI VF slice and PCI SF slice?
> Ans:
> (a) PCI SF slice lifecycle is driven by the user at individual slice
>     level.
>     PCI VF slice(s) are created and destroyed by the vendor driver
>     such as mlx5_core. Typically, this is done when SR-IOV is
>     enabled/disabled.
> (b) PCI VF slices state cannot be changed by user as it is currently
>     not support by PCI core and vendor devices. They are always
>     created in active state.
>     PCI SF slice state is controlled by the user.
> 
> 2. What are the similarities between PCI VF and PCI SF slice?
> (a) Both slices have similar config attributes.
> (b) Both slices have eswitch devlink port and representor netdevice.
> 
> 3. What about handling VF slice similar to SF slice?
> Ans:
>    Yes this is desired. When VF vendor device supports this mode,
>    There will be at least one difference between SF and VF
>    slice handling, i.e. SRIOV enablement will continue using sysfs on
>    host system. Once SRIOV is enabled, VF slice commands should
>    function in similar way as SF slice.
> 
> 4. What are the similaries with SF slice and PF slice?
> Ans:
> (a) Both slices have similar config attributes.
> (b) Both slices have eswitch devlink port and representor netdevice.
> 
> 5. Can slice be used to have dynamic PF as slice?
> Ans:
>    Yes. Whenever a vendor device support dynamic PF, same lifecycle,
>    APIs, attributes can be used for PF slice.
> 
> 6. Can slice exist without a eswitch?
> Ans: Yes
> 
> 7. Why not overload devlink port instead of creating new object slice?
> Ans:
>    In one implementation a devlink port can be overloaded/retrofit
>    what slice object wants to achieve.
>    Same reasoning can be applied to overload and retrofit netdevice to
>    achieve what a slice object wants to achieve.
>    However, it is more natural to create a new object (slice) that
>    represents a device for below rationale.
>    (a) Even though a devlink slice has devlink port attached to it,
>        it is narrow model to always have such association. It limits
>        the devlink to be used only with eswitch.
>    (b) slice undergoes state transitions from
>        create->config->activate->inactivate->delete.
>        It is weird to have few port flavours follow state transition
>        and few don't.
> 
> 8. Why a bus is needed?
> Ans:
> (a) To get unique, persistent and deterministic names of netdev,
>     rdma dev of slice/sf.
> (b) device lifecycle operates using similar pci bus and current
>     driver model. No need to invent new lifecycle scheme.
> (c) To follow uniform device config model for VF and SFs, where
>     user must be able to configure the attributes before binding a
>     VF/SF to driver.
> (d) In future, if needed a virtbus SF device can be bound to some
>     other driver similar to how a PCI PF/VF device can be bound to
>     mlx5_core or vfio_pci device.
> (e) Reuse kernel's existing power management framework to
>     suspend/resume SF devices.
> (f) When using SFs in smartnic based system where SF eswitch port and
>     SF slice are located on two different systems, user desire to
>     configure eswitch before activating the SF slice device.
> Bus allows to follow existing driver model for above needs as,
> create->configure_multiple_attributes->deploy.
> 
> 9. Why not further extend (or abuse) mdev bus?
> Ans: Because
> (a) mdev and vfio are coupled together from iommu perspective
> (b) mdev shouldn't be further abuse to use mdev in current state
>     as suggested by Greg-KH, Jason Gunthrope.
> (c) One bus for "sw_api" purpose, hence the new virtbus.
> (d) Few don't like weird guid/uuid of mdev for sub function purpose
> (e) If needed to map a SF to VM using mdev, new mdev slice flavour
>     should be created for lifecycling via devlink or may be continue
>     life cycle via mdev sysfs; but do remaining slice handling via
>     devlink similar to how PCI VF slices are managed.
> 
> 10. Is it ok to reuse virtbus used for matching service?
> Ans: Yes.
> Greg-KH guided in [1] that its ok for virtbus to hold devices with different
> attributes. Such as matching services vs SF devices where SF device will have
> (a) PCI BAR resource info
> (b) IRQ info
> (c) sfnum
> 
> Greg-KH also guided in [1] that its ok to anchor netdev and rdma dev of a SF
> device to SF virtbus device while rdma and netdev of matching service
> virtbus device to anchored at the parent PCI device.
> 
> [1] https://www.spinics.net/lists/linux-rdma/msg87124.html
> 
> 11. Why platform and mfd framework is not used for creating slices?
> Ans:
> (a) As platform documentation clearly describes, platform devices
>     are for a specific platform. They are autonomous devices, unlike
>     user created slices.
> (b) slices are dynamic in nature where each individual slice is
>     created/destroyed independently.
> (c) MFD (multi-function) devices are for a device that comprise
>     more than one non-unique yet varying hardware functionality.
>     While, here each slice is of same type as that of parent device,
>     with less capabilities than the parent device. Given that mfd
>     devices are built on top of platform devices, it inherits similar
>     limitations as that of platform device.
> (d) In few other threads Greg-KH said to not (ab)use platform
>     devices for such purpose.
> (e) Given that for networking related slices, slice is linked to
>     eswitch which is managed by devlink, having lifecycle through
>     devlink, improves locking/synchronization to single subsystem.
> 
> 12. A given slice supports multiple class of devices, such as RDMA,
>    netdev, vdpa device. How can I disable/enable such attributes of
>    a slice?
> Ans:
>    Slice attributes should be extended in vendor neutral and also
>    vendor specific way to enable/disable such attributes depending
>    on attribute type. This should be handled in future patches.
> 
> virtbus limitations:
> --------------------
> 1. virtbus bus will not support IOMMU like how pci bus does.
>    Hence, all protocol devices (such as netdev, rdmadev) must use its
>    parent's DMA device.
>    RDMA subsystem will use ib_device->dma_device with existing ib_dma_
>    wrapper.
>    netdev will use mlx5_core_dev->pci_dev for dma purposes.
>    This is suggested/hinted by Christoph and Jason.
> 
>    Why is it this way?
>    (a) Because currently only rdma and netdev intend to use it.
>    (b) In future if more use case arise, virtbus device can share DMAR
>        and group of same parent PCI device for in-kernel usecase.
> 
> 2. Dedicated or shared irq vector(s) assignment per sub function and
>    its exposure in sysfs will not be supported in initial version.
>    It will be supported in future series.
> 
> 3. PCI BAR resource information as resource files in sysfs will not be
>    supported in initial version.
>    It will be supported in future series.
> 
> Post initial series:
> --------------------
> Following plumbing will be done post initial series.
> 
> 1. Max number of SF slices resource at devlink level 2. Max IRQ vectors
> resource config at devlink level 3. sf lifecycle for kernel vDPA support, still
> under discussion 4. systemd/udev user space patches for persistent naming
> for
>    netdev and rdma
> 
> Out-of-scope:
> -------------
> 1. Creating mdev/vhost backend/vdpa devices using devlink slice APIs.
>    To support vhost backend 'struct device' creation, SF slice
>    create/set API can be extended to enable/disable specific
>    capabilities of the slice. Such as enable/disable kernel vDPA
>    feature or enable/disable RDMA device for a SF slice.
> 2. Similar extension is applicable for a VF slice.
> 3. Multi host support is orthogonal to this and it can be extended
>    in future.
> 
> Example software/system view:
> -----------------------------
>        _______
>       | admin |
>       | user  |----------
>       |_______|         |
>           |             |
>       ____|____       __|______            _____________
>      |         |     |         |          |             |
>      | devlink |     |   ovs   |          |    user     |
>      | tool    |     |_________|          | application |
>      |_________|         |                |_____________|
>            |             |                   |       |
> -----------|-------------|-------------------|-------|-----------
>            |             |           +----------+   +----------+
>            |             |           |  netdev  |   | rdma dev |
>            |             |           +----------+   +----------+
>       (slice cmds,       |              ^             ^
>        add/del/set)      |              |             |
>            |             |              +-------------|
>       _____|___          |              |         ____|________
>      |         |         |              |        |             |
>      | devlink |   +------------+       |        | mlx5_core/ib|
>      | kernel  |   | rep netdev |       |        | drivers     |
>      |_________|   +------------+       |        |_____________|
>            |             |              |             ^
>      (slice cmds)        |              |        (probe/remove)
>       _____|____         |              |         ____|_____
>      |          |        |    +--------------+   |          |
>      | mlx5_core|---------    | virtbus dev  |---|  virtbus |
>      | driver   |             +--------------+   |  driver  |
>      |__________|                                |__________|
>            |                                          ^
>       (sf add/del, events)                            |
>            |                                   (device add/del)
>       _____|____                                  ____|_____
>      |          |                                |          |
>      |  PCI NIC |---- admin activate/deactive    | mlx5_core|
>      |__________|           deactive events ---->| driver   |
>                                                  |__________|
> 

Powered by blists - more mailing lists