Message-ID: <e1a752df-d550-4e98-b275-371a8fe303a6@intel.com>
Date: Fri, 16 Feb 2024 13:44:46 -0800
From: Jacob Keller <jacob.e.keller@...el.com>
To: Jiri Pirko <jiri@...nulli.us>
CC: Jakub Kicinski <kuba@...nel.org>, William Tu <witu@...dia.com>,
	<bodong@...dia.com>, <jiri@...dia.com>, <netdev@...r.kernel.org>,
	<saeedm@...dia.com>, "aleksander.lobakin@...el.com"
	<aleksander.lobakin@...el.com>
Subject: Re: [RFC PATCH v3 net-next] Documentation: devlink: Add devlink-sd



On 2/16/2024 12:10 AM, Jiri Pirko wrote:
> Thu, Feb 15, 2024 at 06:41:31PM CET, jacob.e.keller@...el.com wrote:
>>
>>
>> On 2/15/2024 5:19 AM, Jiri Pirko wrote:
>>> Fri, Feb 09, 2024 at 02:26:33AM CET, kuba@...nel.org wrote:
>>>> On Fri, 2 Feb 2024 08:46:56 +0100 Jiri Pirko wrote:
>>>>> Fri, Feb 02, 2024 at 05:00:41AM CET, kuba@...nel.org wrote:
>>>>>> On Thu, 1 Feb 2024 11:13:57 +0100 Jiri Pirko wrote:  
>>>>>>> Wait a sec.  
>>>>>>
>>>>>> No, you wait a sec ;) Why do you think this belongs to devlink?
>>>>>> Two months ago you were complaining bitterly when people were
>>>>>> considering using devlink rate to control per-queue shapers.
>>>>>> And now it's fine to add queues as a concept to devlink?  
>>>>>
>>>>> Do you have a better suggestion how to model common pool object for
>>>>> multiple netdevices? This is the reason why devlink was introduced to
>>>>> provide a platform for common/shared things for a device that contains
>>>>> multiple netdevs/ports/whatever. But I may be missing something here,
>>>>> for sure.
>>>>
>>>> devlink just seems like the lowest common denominator, but the moment
>>>> we start talking about multi-PF devices it also gets wobbly :(
>>>
>>> You mean you see real to have a multi-PF device that allows to share the
>>> pools between the PFs? If, in theory, that exists, could this just be a
>>> limitation perhaps?
>>>
>>>
>>
>> I don't know offhand if we have a device which can share pools
>> specifically, but we do have multi-PF devices which have a lot of shared
>> resources. However, due to the multi-PF PCIe design. I looked into ways
>> to get a single devlink across the devices.. but ultimately got stymied
>> and gave up.
>>
>> This left us with accepting the limitation that each PF gets its own
>> devlink and can't really communicate with other PFs.
>>
>> The existing solution has just been to partition the shared resources
>> evenly across PFs, typically via firmware. No flexibility.
>>
>> I do think the best solution here would be to figure out a generic way
>> to tie multiple functions into a single devlink representing the device.
>> Then each function gets the set of devlink_port objects associated to
>> it. I'm not entirely sure how that would work. We could hack something
>> together with auxbus.. but thats pretty ugly. Some sort of orchestration
>> in the PCI layer that could identify when a device wants to have some
>> sort of "parent" driver which loads once and has ties to each of the
>> function drivers would be ideal.
> 
> Hmm, dpll does this. You basically share dpll device instance in between
> multiple pci functions. The same concept could be done for devlink. We
> have to figure out the backward compatibility. User now expects the
> instances per-pf.
> 
> 

Ya, ice started doing this over auxbus for PTP as well, because we had a
bunch of issues with E822 devices that couldn't be solved without some
communication across PF.

I'm not sure its actually worth trying to change how devlink works now,
its pretty ingrained at this point.

But I think it is a possibility that could have happened, and having a
centralized "this is the device owner" would have simplified a lot of
concepts for managing these sorts of shared resources which are not
owned by a single PF.

Powered by blists - more mailing lists