Message-ID: <c19e7dec-7aae-449d-b454-4078c8fbd926@nvidia.com>
Date: Tue, 6 May 2025 14:25:10 +0300
From: Mark Bloch <mbloch@...dia.com>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Moshe Shemesh <moshe@...dia.com>, netdev@...r.kernel.org,
 "David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
 Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>,
 Donald Hunter <donald.hunter@...il.com>, Jiri Pirko <jiri@...nulli.us>,
 Jonathan Corbet <corbet@....net>, Andrew Lunn <andrew+netdev@...n.ch>,
 Tariq Toukan <tariqt@...dia.com>
Subject: Re: [RFC net-next 0/5] devlink: Add unique identifier to devlink port
 function



On 05/05/2025 21:55, Jakub Kicinski wrote:
> On Sun, 4 May 2025 20:46:51 +0300 Mark Bloch wrote:
>> On the DPU (ARM), we see representors for each BDF. For simplicity,
>> assume each BDF corresponds to a single devlink port. So the ARM would
>> expose:
>>
>> PF0_HOST0_REP
>> UPLINK0_REP
>> PF0_HOST1_REP
>> UPLINK1_REP
>>
>> In devlink terms, we're referring to the c argument in phys_port_name,
>> which represents the controller, effectively indicating which host
>> the BDF belongs to.
>>
>> The problem we're addressing is matching the PF seen on a host to its
>> corresponding representor on the DPU. From the ARM side, we know that
>> this rep X belongs to pf0 on host y, but we don't which host is which.
>> From within each host, you can't tell which host you are, because all
>> see their PF as PF0.
>>
>> With the proposed feature (along with Jiri's changes), this becomes
>> trivial, you just match the function UID and you're done.
> 
> Thanks for explaining the setup. Could you please explain the user
> scenario now? Perhaps thinking of it as a sequence diagram would
> be helpful, but whatever is easiest, just make it concrete.
> 

It's a rough flow, but I believe it clearly illustrates the use case
we're targeting:
 
Some system configuration info:
 
- A static mapping file exists that defines the relationship between
  a host and the corresponding ARM/DPU host that manages it.
 
- OVN, OVS and Kubernetes are used to manage network connectivity and
  resource allocation.
 
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF
   configuration and sends the PF number and VF index to the central
   management software.
3. An agent on the DPU side detects the changes made on host X. Using
   the PF number and VF index, it identifies the corresponding
   representor, attaches it to an OVS bridge, and allows OVN to program
   the relevant steering rules.
 
This setup works well when the mapping file defines a one-to-one
relationship between a host and a single ARM/DPU host.
It's already supported in upstream today [1]
 
However, in a slightly more generic scenario like:
 
Control Host A: External host X
                External host Y
 
A single ARM/DPU host manages multiple external hosts. In this case, step
2—where only the PF number and VF index are sent is insufficient. During
step 3, the agent on the DPU reads the data but cannot determine which
external host created the VF. As a result, it cannot correctly associate
the representor with the appropriate OVS bridge.
 
To resolve this, we plan to modify step 2 to include the VUID along with
the PF number and VF index. The DPU-side agent will use the VUID to match
it with the FUID, identify the correct PF representor, and then use
standard devlink mechanisms to locate the corresponding VF representor.

1: https://github.com/ovn-kubernetes/ovn-kubernetes
You can look at: go-controller/pkg/util/dpu_annotations.go for more info.

Mark

>> As a side note, I believe this feature has merit even beyond this
>> specific use case. 
> 
> I also had that belief when I implemented something similar for the NFP
> long time ago. Jiri didn't like the solution / understand the problem 
> at the time. But it turned out not to matter in practice.

Powered by blists - more mailing lists