Message-ID:
 <PH0PR18MB4425CD5BD139261F035D19CDD95B2@PH0PR18MB4425.namprd18.prod.outlook.com>
Date: Thu, 14 Nov 2024 05:17:18 +0000
From: Shijith Thotton <sthotton@...vell.com>
To: Bjorn Helgaas <helgaas@...nel.org>
CC: "bhelgaas@...gle.com" <bhelgaas@...gle.com>,
        "ilpo.jarvinen@...ux.intel.com" <ilpo.jarvinen@...ux.intel.com>,
        "Jonathan.Cameron@...wei.com" <Jonathan.Cameron@...wei.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "rafael@...nel.org"
	<rafael@...nel.org>,
        "scott@...amperecomputing.com"
	<scott@...amperecomputing.com>,
        Jerin Jacob <jerinj@...vell.com>,
        Srujana
 Challa <schalla@...vell.com>,
        Vamsi Krishna Attunuru <vattunuru@...vell.com>
Subject: Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

>> >> >> This patch introduces a PCI hotplug controller driver for the OCTEON
>> >> >> PCIe device. The OCTEON PCIe device is a multi-function device where
>the
>> >> >> first function serves as the PCI hotplug controller.
>> >> >>
>> >> >>                +--------------------------------+
>> >> >>                |           Root Port            |
>> >> >>                +--------------------------------+
>> >> >>                                |
>> >> >>                               PCIe
>> >> >>                                |
>> >> >> +---------------------------------------------------------------+
>> >> >> |              OCTEON PCIe Multifunction Device                 |
>> >> >> +---------------------------------------------------------------+
>> >> >>              |                    |              |            |
>> >> >>              |                    |              |            |
>> >> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> >> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> >> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >> >>              |
>> >> >>              |
>> >> >> +-------------------------+
>> >> >> |   Controller Firmware   |
>> >> >> +-------------------------+
>> >> >>
>> >> >> The hotplug controller driver enables hotplugging of non-controller
>> >> >> functions within the same device. During probing, the driver removes
>> >> >> the non-controller functions and registers them as PCI hotplug slots.
>> >> >> These slots are added back by the driver, only upon request from the
>> >> >> device firmware.
>> >> >>
>> >> >> The controller uses MSI-X interrupts to notify the host of hotplug
>> >> >> events initiated by the OCTEON firmware. Additionally, the driver
>> >> >> allows users to enable or disable individual functions via sysfs slot
>> >> >> entries, as provided by the PCI hotplug framework.
>> >> >
>> >> >Can we say something here about what the benefit of this driver is?
>> >> >For example, does it save power?
>> >>
>> >> The driver enables hotplugging of non-controller functions within the
>device
>> >> without requiring a fully implemented switch, reducing both power
>> >consumption
>> >> and product cost.
>> >
>> >Reduced product cost is motivation for the hardware design, not for
>> >this hotplug driver.
>> >
>> >You didn't explicitly say that when function 0 hot-removes another
>> >function, it reduces overall power consumption.  But I assume that's
>> >the case?
>> >
>>
>> Yes, I will explain it in detail below
>>
>> >> >What causes the function 0 firmware to request a hot-add or
>> >> >hot-removal of another function?
>> >>
>> >> The firmware will enable the required number of non-controller
>> >> functions based on runtime demand, allowing control over these
>> >> functions. For example, in a vDPA scenario, each function could act
>> >> as a different type of device (such as net, crypto, or storage)
>> >> depending on the firmware configuration.
>> >
>> >What is the path for this runtime demand?  I assume function 0
>> >provides some interface to request a specific kind of functionality
>> >(net, crypo, storage, etc)?
>> >
>>
>> Right now, it done via firmware management console.
>>
>> >I don't know anything about vDPA, so if that's important here, it
>> >needs a little more context.
>> >
>> >> Hot removal is useful in cases of live firmware updates.
>> >
>> >So the idea is that function X is hot-removed, which forces the driver
>> >to let go of it, the firmware is updated, and X is hot-added again,
>> >and the driver binds to it again?
>> >
>>
>> I will explain the process in detail, which should also address the questions
>> below.
>>
>> >And somewhere in there is a reset of function X, and after the reset
>> >X is running the new firmware?
>> >
>> >Who/what initiates this whole path?  Some request to function 0,
>> >saying "please remove function X"?
>> >
>> >But I guess maybe it doesn't go through function 0, since octeon_hp
>> >claims function 0, and it doesn't provide that functionality.  Maybe
>> >the individual drivers for *other* functions know how to initiate
>> >these things, and those functions internally communicate with function
>> >0 to ask it to start a hot-remove/hot-add sequence?
>> >
>> >That wouldn't explain the power reduction plan, though.  A driver for
>> >function X could conceivably tell its device "I'm no longer needed"
>> >and function X could tell function 0 to remove it.  That might enable
>> >some power savings.  But that doesn't have a path to *re-enable*
>> >function X, since function X has been removed and there's no driver to
>> >ask for it to be hot-added again.
>> >
>> >Maybe there's some out-of-band management path that can tell function
>> >0 to do things, independent of PCIe?
>> >
>>
>> Our implementation aims to achieve two main objectives:
>>
>> 1. Enable changing a function's personality at runtime.
>> 2. Reduce power consumption.
>>
>> The OCTEON PCI device has multiple ARM cores running Linux, with its
>firmware
>> composed of multiple components. For example, the firmware includes
>components
>> like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any
>function
>> at runtime. The device firmware is accessible via a management console,
>allowing
>> components to be started or stopped. For each component, an associated
>function
>> is hot-added on the host to expose its functionality. Initially, after boot, only
>> Function 0 and the controller firmware are active.
>>
>> Here's a breakdown:
>>
>> At Time 0:
>> - Linux boots on the device, starting the controller firmware.
>>
>> At Time 1:
>> - The hotplug driver loads on the host, temporarily removing other functions.
>>
>> At Time 2:
>> - A network device firmware component starts on an ARM core (initiated
>through
>>   a console command).
>> - This component sets up the Function 1 configuration space, data, and other
>>   request handlers for network processing.
>> - The firmware issues a hot-add request to Function 0 (hotplug driver) on the
>>   host to enable Function 1.
>>
>> At Time 3:
>> - The Function 0 hotplug driver on the host receives the hot-add request and
>>   enables Function 1 on the host.
>> - A network driver binds to Function 1 based on device class and ID.
>>
>> At Time 4:
>> - The network device firmware component receives a stop signal.
>> - The firmware issues a hot-remove request for Function 1 on the host.
>> - The firmware component halts, reducing the device's power consumption.
>>
>> At Time 5:
>> - The Function 0 hotplug driver on the host receives the hot-remove request
>and
>>   disables Function 1 on the host.
>>
>> At Time 6:
>> - A crypto device firmware component starts on an ARM core.
>> - This component configures the Function 1 configuration space for crypto
>>   processing and sets up the required firmware handlers.
>> - The firmware issues a hot-add request to enable Function 1 on the host.
>>
>> At Time 7:
>> - The Function 0 hotplug driver on the host receives the hot-add request and
>enables Function 1 on the host.
>> - A crypto driver binds to Function 1 based on device class and ID.
>>
>> The firmware component for each function only runs and is hot-added when
>> needed. Only Function 0 and the controller firmware remain active
>> continuously. This dynamic control reduces power usage by keeping
>unnecessary
>> components off. Additionally, a single function can adapt its personality based
>> on the associated firmware component, enhancing flexibility.
>>
>> I hope this clarifies the implementation. Let me know if you have any
>> questions.
>
>Thanks very much!  I propose adding text like this to the commit log:
>
>  There is an out-of-band management console interface to firmware
>  running on function 0 whereby an administrator can disable functions
>  to save power or enable them with one of several personalities
>  (virtio-net, virtio-crypto, NVMe, etc) for the other functions.
>  Function 0 initiates hotplug events handled by this driver when the
>  other functions are enabled or disabled.
>
>I provisionally applied this to pci/hotplug-octeon, but will be happy
>to update the text if necessary.
>

Thank you. The commit log looks good. I hope no further action is required from
my side.

Shijith

Powered by blists - more mailing lists