netdev - RE: [net-next v4 0/2] devlink: Add port function attribute for IO EQs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Mon, 8 Apr 2024 03:20:33 +0000
From: Parav Pandit <parav@...dia.com>
To: Zhu Yanjun <yanjun.zhu@...ux.dev>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>, "davem@...emloft.net" <davem@...emloft.net>,
	"edumazet@...gle.com" <edumazet@...gle.com>, "kuba@...nel.org"
	<kuba@...nel.org>, "pabeni@...hat.com" <pabeni@...hat.com>, "corbet@....net"
	<corbet@....net>, "dw@...idwei.uk" <dw@...idwei.uk>,
	"kalesh-anakkur.purayil@...adcom.com" <kalesh-anakkur.purayil@...adcom.com>
CC: Saeed Mahameed <saeedm@...dia.com>, "leon@...nel.org" <leon@...nel.org>,
	"jiri@...nulli.us" <jiri@...nulli.us>, Shay Drori <shayd@...dia.com>, Dan
 Jurgens <danielj@...dia.com>, Dima Chumak <dchumak@...dia.com>,
	"linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
	"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>
Subject: RE: [net-next v4 0/2] devlink: Add port function attribute for IO EQs


> From: Zhu Yanjun <yanjun.zhu@...ux.dev>
> Sent: Saturday, April 6, 2024 2:36 PM
> 
> 在 2024/4/6 3:05, Parav Pandit 写道:
> > Currently, PCI SFs and VFs use IO event queues to deliver netdev per
> > channel events. The number of netdev channels is a function of IO
> > event queues. In the second scenario of an RDMA device, the completion
> > vectors are also a function of IO event queues. Currently, an
> > administrator on the hypervisor has no means to provision the number
> > of IO event queues for the SF device or the VF device. Device/firmware
> > determines some arbitrary value for these IO event queues. Due to
> > this, the SF netdev channels are unpredictable, and consequently, the
> > performance is too.
> >
> > This short series introduces a new port function attribute: max_io_eqs.
> > The goal is to provide administrators at the hypervisor level with the
> > ability to provision the maximum number of IO event queues for a
> > function. This gives the control to the administrator to provision
> > right number of IO event queues and have predictable performance.
> >
> > Examples of when an administrator provisions (set) maximum number of
> > IO event queues when using switchdev mode:
> >
> >    $ devlink port show pci/0000:06:00.0/1
> >        pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum
> 0 vfnum 0
> >            function:
> >            hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 10
> >
> >    $ devlink port function set pci/0000:06:00.0/1 max_io_eqs 20
> >
> >    $ devlink port show pci/0000:06:00.0/1
> >        pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum
> 0 vfnum 0
> >            function:
> >            hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 20
> >
> > This sets the corresponding maximum IO event queues of the function
> > before it is enumerated. Thus, when the VF/SF driver reads the
> > capability from the device, it sees the value provisioned by the
> > hypervisor. The driver is then able to configure the number of
> > channels for the net device, as well as the number of completion
> > vectors for the RDMA device. The device/firmware also honors the
> > provisioned value, hence any VF/SF driver attempting to create IO EQs
> > beyond provisioned value results in an error.
> >
> > With above setting now, the administrator is able to achieve the 2x
> > performance on SFs with 20 channels. In second example when SF was
> > provisioned for a container with 2 cpus, the administrator provisioned
> > only
> > 2 IO event queues, thereby saving device resources.
> >
> 
> The following paragraph is the same with the above paragraph?
>
Ah, yes. I forgot to remove one of them while doing minor grammar changes.

 
> > With the above settings now in place, the administrator achieved 2x
> > performance with the SF device with 20 channels. In the second
> > example, when the SF was provisioned for a container with 2 CPUs, the
> > administrator provisioned only 2 IO event queues, thereby saving device
> resources.
> >
> > changelog:
> > v2->v3:
> > - limited to 80 chars per line in devlink
> > - fixed comments from Jakub in mlx5 driver to fix missing mutex unlock
> >    on error path
> > v1->v2:
> > - limited comment to 80 chars per line in header file
> > - fixed set function variables for reverse christmas tree
> > - fixed comments from Kalesh
> > - fixed missing kfree in get call
> > - returning error code for get cmd failure
> > - fixed error msg copy paste error in set on cmd failure
> >
> > Parav Pandit (2):
> >    devlink: Support setting max_io_eqs
> >    mlx5/core: Support max_io_eqs for a function
> >
> >   .../networking/devlink/devlink-port.rst       | 33 +++++++
> >   .../mellanox/mlx5/core/esw/devlink_port.c     |  4 +
> >   .../net/ethernet/mellanox/mlx5/core/eswitch.h |  7 ++
> >   .../mellanox/mlx5/core/eswitch_offloads.c     | 97 +++++++++++++++++++
> >   include/net/devlink.h                         | 14 +++
> >   include/uapi/linux/devlink.h                  |  1 +
> >   net/devlink/port.c                            | 53 ++++++++++
> >   7 files changed, 209 insertions(+)
> >