netdev - Re: [PATCH/RFC net-next 1/2] devlink: expose port function commands to assign VFs to multiple netdevs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Y+QWBFoz66KrsU7V@x130>
Date:   Wed, 8 Feb 2023 13:37:08 -0800
From:   Saeed Mahameed <saeed@...nel.org>
To:     Simon Horman <simon.horman@...igine.com>
Cc:     Leon Romanovsky <leon@...nel.org>,
        Jakub Kicinski <kuba@...nel.org>,
        David Miller <davem@...emloft.net>,
        Paolo Abeni <pabeni@...hat.com>,
        Michael Chan <michael.chan@...adcom.com>,
        Andy Gospodarek <andy@...yhouse.net>,
        Gal Pressman <gal@...dia.com>,
        Jesse Brandeburg <jesse.brandeburg@...el.com>,
        Tony Nguyen <anthony.l.nguyen@...el.com>,
        Edward Cree <ecree.xilinx@...il.com>,
        Vladimir Oltean <vladimir.oltean@....com>,
        Andrew Lunn <andrew@...n.ch>, Fei Qin <fei.qin@...igine.com>,
        netdev@...r.kernel.org, oss-drivers@...igine.com
Subject: Re: [PATCH/RFC net-next 1/2] devlink: expose port function commands
 to assign VFs to multiple netdevs

On 08 Feb 13:05, Simon Horman wrote:
>On Wed, Feb 08, 2023 at 01:53:48PM +0200, Leon Romanovsky wrote:
>> On Wed, Feb 08, 2023 at 12:36:53PM +0100, Simon Horman wrote:
>> > On Wed, Feb 08, 2023 at 01:21:22PM +0200, Leon Romanovsky wrote:
>> > > On Mon, Feb 06, 2023 at 06:42:27PM -0800, Jakub Kicinski wrote:
>> > > > On Mon,  6 Feb 2023 16:36:02 +0100 Simon Horman wrote:
>> > > > > +VF assignment setup
>> > > > > +---------------------------
>> > > > > +In some cases, NICs could have multiple physical ports per PF. Users can assign VFs to
>> > > > > +different ports.
>> > > >
>> > > > Please make sure you run make htmldocs when changing docs,
>> > > > this will warn.
>> > > >
>> > > > > +- Get count of VFs assigned to physical port::
>> > > > > +
>> > > > > +    $ devlink port show pci/0000:82:00.0/0
>> > > > > +    pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4
>> > > >
>> > > > Physical port has VFs? My knee jerk reaction is that allocating
>> > > > resources via devlink is fine but this seems to lean a bit into
>> > > > forwarding. How do other vendors do it? What's the mapping of VFs
>> > > > to ports?
>> > >
>> > > I don't understand the meaning of VFs here. If we are talking about PCI
>> > > VFs, other vendors follow PCI spec "9.3.3.3.1 VF Enable" section, which
>> > > talks about having one bit to enable all VFs at once. All these VFs will
>> > > have separate netdevs.
>> >
>> > Yes, that is the case here too (before and after).
>> >
>> > What we are talking about is the association of VFs to physical ports
>> > (in the case where a NIC has more than one physical port).
>>
>> We have devices with multiple ports too, but don't have such issues.
>> So it will help if you can provide more context here.
>>
>> I'm failing to see connection between physical ports and physical VFs.
>>
>> Are you saying that physical ports are actual PCI VFs, which spans L2 VFs,
>> which you want to assign to another port (PF)?
>
>No, a physical port is not a VF (nor a PF, FWIIW).
>
>The topic here is about two modes of behaviour.
>
>One, where VFs are associated with physical ports - conceptually one might
>think of this as some VFs and one phys port being plugged into one VEB,
>while other VFs and another phys port are plugged into another VEB.
>
>I believe this is the mode on most multi-port NICs.
>
>And another mode where all VFs are associated with one physical port,
>even if more than one is present. The NFP currently implements this model.
>
>This patch is about allowing NICs, in particular the NFP based NICs,
>to switch modes.

I don't understand the difference between the two modes, 
1) "where VFs are associated with physical ports"
2) "another mode where all VFs are associated with one physical port"

anyway here how it works for ConnectX devices, and i think the model should
be generalized to others as it simplifies the user life in my opinion.

Each physical port is represented by a PCI PF function.

devlink dev show
pci/0000:81:00.0 #port 1
pci/0000:81:00.1 #port 2

when you enable sriov, you enable it on a specific PF, eventually port,
meaning those vfs are only associated with that port.

# enable vfs on port 1
echo 4 > /sys/bus/pci/devices/0000:81:00.0/sriov_numvfs

# enable vfs on port 2
echo 4 > /sys/bus/pci/devices/0000:81:00.1/sriov_numvfs

$ devlink dev show
# port 1 PF
pci/0000:81:00.0

# port 2 PF
pci/0000:81:00.1

# port 1 VFs
pci/0000:81:00.2
pci/0000:81:00.3
pci/0000:81:00.4
pci/0000:81:00.5

# port 2 VFs
pci/0000:81:01.2
pci/0000:81:01.3
pci/0000:81:01.4
pci/0000:81:01.5

The pci address enumeration is device specific, but i don't think user
should care.