Message-ID: <BY5PR02MB62604B87C66A1AD139A6F153BBF40@BY5PR02MB6260.namprd02.prod.outlook.com>
Date:   Tue, 1 Dec 2020 19:39:57 +0000
From:   Sonal Santan <sonals@...inx.com>
To:     Rob Herring <robh@...nel.org>
CC:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-fpga@...r.kernel.org" <linux-fpga@...r.kernel.org>,
        Max Zhen <maxz@...inx.com>, Lizhi Hou <lizhih@...inx.com>,
        Michal Simek <michals@...inx.com>,
        Stefano Stabellini <stefanos@...inx.com>,
        "devicetree@...r.kernel.org" <devicetree@...r.kernel.org>
Subject: RE: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview

Hi,

> -----Original Message-----
> From: Rob Herring <robh@...nel.org>
> Sent: Monday, November 30, 2020 10:09 AM
> To: Sonal Santan <sonals@...inx.com>
> Cc: linux-kernel@...r.kernel.org; Sonal Santan <sonals@...inx.com>; linux-
> fpga@...r.kernel.org; Max Zhen <maxz@...inx.com>; Lizhi Hou
> <lizhih@...inx.com>; Michal Simek <michals@...inx.com>; Stefano Stabellini
> <stefanos@...inx.com>; devicetree@...r.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
> 
> On Sat, Nov 28, 2020 at 5:02 PM Sonal Santan <sonal.santan@...inx.com>
> wrote:
> >
> > Hello,
> >
> > This patch series adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards,
> > https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
> >
> > The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was
> > posted before.
> >
> > ALVEO PLATFORM ARCHITECTURE
> >
> > Alveo PCIe FPGA based platforms have a static *shell* partition and a
> > partial re-configurable *user* partition. The shell partition is
> > automatically loaded from flash when host is booted and PCIe is
> > enumerated by BIOS. Shell cannot be changed till the next cold reboot. The
> shell exposes two PCIe physical functions:
> >
> > 1. management physical function
> > 2. user physical function
> >
> > The patch series includes Documentation/xrt.rst which describes Alveo
> > platform, xmgmt driver architecture and deployment model in more more
> detail.
> >
> > Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> > image using Vitis
> > https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> > tools. The image is packaged as xclbin and contains partial bitstream
> > for the user partition and necessary metadata. Users can dynamically
> > swap the image running on the user partition in order to switch between
> different workloads.
> >
> > ALVEO DRIVERS
> >
> > Alveo Linux kernel driver *xmgmt* binds to management physical
> > function of Alveo platform. The modular driver framework is organized
> > into several platform drivers which primarily handle the following
> functionality:
> >
> > 1.  Loading firmware container also called xsabin at driver attach
> > time 2.  Loading of user compiled xclbin with FPGA Manager integration
> > 3.  Clock scaling of image running on user partition 4.  In-band
> > sensors: temp, voltage, power, etc.
> > 5.  Device reset and rescan
> > 6.  Flashing static *shell* partition
> >
> > The platform drivers are packaged into *xrt-lib* helper module with a
> > well defined interfaces the details of which can be found in
> Documentation/xrt.rst.
> >
> > xmgmt driver is second generation Alveo management driver and
> > evolution of the first generation (out of tree) Alveo management
> > driver, xclmgmt. The sources of the first generation drivers were
> > posted on LKML last year--
> > https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx
> > .com/
> >
> > Changes since the first generation driver include the following: the
> > driver has been re-architected as data driven modular driver; the
> > driver has been split into xmgmt and xrt-lib; user physical function
> > driver has been removed from the patch series.
> >
> > Alveo/XRT security and platform architecture is documented on the
> > following GitHub pages:
> > https://xilinx.github.io/XRT/master/html/security.html
> > https://xilinx.github.io/XRT/master/html/platforms_partitions.html
> >
> > User physical function driver is not included in this patch series.
> >
> > TESTING AND VALIDATION
> >
> > xmgmt driver can be tested with full XRT open source stack which
> > includes user space libraries, board utilities and (out of tree) first
> > generation user physical function driver xocl. XRT open source runtime
> > stack is available at https://github.com/Xilinx/XRT. This patch series
> > has been validated on Alveo U50 platform.
> >
> > Complete documentation for XRT open source stack can be found here--
> > https://xilinx.github.io/XRT/master/html/index.html
> 
> I've not gotten into the patch details, but I'm not clear on what the lifecycle of
> the DT looks like here. What's the starting point and what manipulations to the
> DT are being done? I'm trying to understand if using libfdt is the right way
> versus operating on an unflattened tree.

The DT is created when *xmgmt* driver attaches to the device and reads the 
xsabin. The xsabin defines the shell and HW subsystems contained in the shell. 
Since the shell is live for the lifetime of the driver the DT is captured in the 
partition subdev. The partition then looks for "addressable_endpoints" node and 
walks the list of child end point nodes each of which is then copied into its own 
instance of subdev. The life cycle of the copied nodes is same as the owning subdev. 
All the DT nodes are released when the partition together with its child subdevs 
goes away which happens when the driver is unloaded.

xmgmt driver also collects all end points which advertise "pcie_physical_function 
= <0x01>" and then constructs a DT on the fly which is then sent to the user 
physical function driver via mailbox. This requires support for manipulating device 
tree nodes.

In the next revision of the driver we would like to add support for a variation of 
platform which has three partitions: base, shell and user-- 
https://xilinx.github.io/XRT/master/html/platforms_partitions.html#two-stage-platforms

In this model, *base* is initialized like described above. However *shell* can be 
changed dynamically by service provider. This means xmgmt would load DT 
corresponding to shell partition when shell is loaded and tear it down when shell 
is unloaded. The DT corresponding to *base* remains unaffected.

Thanks,
-Sonal
> 
> Rob

Powered by blists - more mailing lists