netdev - Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190304181107.1379e358@cakuba.netronome.com>
Date:   Mon, 4 Mar 2019 18:11:07 -0800
From:   Jakub Kicinski <jakub.kicinski@...ronome.com>
To:     Jason Gunthorpe <jgg@...lanox.com>
Cc:     Jiri Pirko <jiri@...nulli.us>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "oss-drivers@...ronome.com" <oss-drivers@...ronome.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Parav Pandit <parav@...lanox.com>
Subject: Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI
 ports

On Tue, 5 Mar 2019 01:30:19 +0000, Jason Gunthorpe wrote:
> On Mon, Mar 04, 2019 at 05:03:20PM -0800, Jakub Kicinski wrote:
> 
> > > Don't we already have devlink instances for every mlx5 physical port
> > > and VF as they are unique PCI functions?  
> > 
> > That's a very NIC-centric view of the world, though.  Equating devlink
> > instances to ports, and further to PCI devices.  Its fundamentally
> > different from what switches and some NICs do, where all ports are under
> > single devlink instance.  
> 
> I think, as a practical matter, it is a bit hard to recombine an asic
> that presents multiple PCI BDFs into a single SW object. It is tricky
> to give stable labels to things, to leave gaps to allow for uncertain
> discovey, to co-ordinate between multiple struct pci_device drivers
> probe functions, etc.

It is tricky indeed, hence my so far unsuccessful search for a stable
handle :/  One thing which would not make things easier tho, is if we
objects we use to model this scenario don't have clear meanings...

> And at least with devlink, if you have a object layer that is broader
> then PCI BDF, how do the devlink commands work? Are all BDFs just an
> alias for this hidden super object?

My thinking was that they'd alias.

> Do any drivers attempt to provide single instant made up of merged
> BDFs?

Not yet, but our NFP can do it.  NFP used to be single PF per host,
which made life easier, but the silicon team was persuaded to remove
that comfort :)

> In other words, is a PCI BDF really the largest granularity that
> devlink can address today?

Yes, DBDF is the largest today, _but_ most advanced devices (mlxsw, nfp)
have only one PF per host.  IOW we existed blissfully in a world where
devices either pipelined from port to PF or had only one PF.

> At least in RDMA we have drivers doing all combinations of this:
> multiple ports per BDF, one port per BDF, and one composite RDMA
> device formed by combining multiple BDFs worth of ports together.

Right, last but not least we have the case where there is one port but
multiple links (for NUMA, or just because 1 PCIe link can't really cope
with 200Gbps).  In that case which DBDF would the port go to? :(
Do all internal info of the ASIC (health, regions, sbs) get registered
twice?

> > > > You guys come from the RDMA side of the world, with which I'm less
> > > > familiar, and the soft bus + spawning devices seems to be a popular
> > > > design there.  Could you describe the advantages of that model for 
> > > > the sake of the netdev-only folks? :)    
> > > 
> > > I don't think we do this in RDMA at all yet, or maybe I'm not sure
> > > what you are thinking of?  
> > 
> > Mm.. I caught an Intel patch set recently which was talking about buses
> > and spawning devices.  It must have been a different kettle of fish.  
> 
> That sounds like scalable iov..
> 
> Jason