Message-ID: <20190325202810.GG2665@phenom.ffwll.local>
Date:   Mon, 25 Mar 2019 21:28:10 +0100
From:   Daniel Vetter <daniel@...ll.ch>
To:     sonal.santan@...inx.com
Cc:     dri-devel@...ts.freedesktop.org, gregkh@...uxfoundation.org,
        cyrilc@...inx.com, linux-kernel@...r.kernel.org, lizhih@...inx.com,
        michals@...inx.com, airlied@...hat.com
Subject: Re: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator driver

On Tue, Mar 19, 2019 at 02:53:55PM -0700, sonal.santan@...inx.com wrote:
> From: Sonal Santan <sonal.santan@...inx.com>
> 
> Hello,
> 
> This patch series adds drivers for Xilinx Alveo PCIe accelerator cards.
> These drivers are part of Xilinx Runtime (XRT) open source stack and
> have been deployed by leading FaaS vendors and many enterprise customers.

Cool, first fpga driver submitted to drm! And from a high level I think
this makes a lot of sense.

> PLATFORM ARCHITECTURE
> 
> Alveo PCIe platforms have a static shell and a reconfigurable (dynamic)
> region. The shell is automatically loaded from PROM when host is booted
> and PCIe is enumerated by BIOS. Shell cannot be changed till next cold
> reboot. The shell exposes two physical functions: management physical
> function and user physical function.
> 
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> image using SDx compiler. The FPGA image packaged as xclbin file can be
> loaded onto reconfigurable region. The image may contain one or more
> compute unit. Users can dynamically swap the full image running on the
> reconfigurable region in order to switch between different workloads.
> 
> XRT DRIVERS
> 
> XRT Linux kernel driver xmgmt binds to mgmt pf. The driver is modular and
> organized into several platform drivers which primarily handle the
> following functionality:
> 1.  ICAP programming (FPGA bitstream download with FPGA Mgr integration)
> 2.  Clock scaling
> 3.  Loading firmware container also called dsabin (embedded Microblaze
>     firmware for ERT and XMC, optional clearing bitstream)
> 4.  In-band sensors: temp, voltage, power, etc.
> 5.  AXI Firewall management
> 6.  Device reset and rescan
> 7.  Hardware mailbox for communication between two physical functions
> 
> XRT Linux kernel driver xocl binds to user pf. Like its peer, this driver
> is also modular and organized into several platform drivers which handle
> the following functionality:
> 1.  Device memory topology discovery and memory management
> 2.  Buffer object abstraction and management for client process
> 3.  XDMA MM PCIe DMA engine programming
> 4.  Multi-process aware context management
> 5.  Compute unit execution management (optionally with help of ERT) for
>     client processes
> 6.  Hardware mailbox for communication between two physical functions
> 
> The drivers export ioctls and sysfs nodes for various services. xocl
> driver makes heavy use of DRM GEM features for device memory management,
> reference counting, mmap support and export/import. xocl also includes a
> simple scheduler called KDS which schedules compute units and interacts
> with hardware scheduler running ERT firmware. The scheduler understands
> custom opcodes packaged into command objects and provides an asynchronous
> command done notification via POSIX poll.
> 
> More details on architecture, software APIs, ioctl definitions, execution
> model, etc. is available as Sphinx documentation--
> 
> https://xilinx.github.io/XRT/2018.3/html/index.html
> 
> The complete runtime software stack (XRT) which includes out of tree
> kernel drivers, user space libraries, board utilities and firmware for
> the hardware scheduler is open source and available at
> https://github.com/Xilinx/XRT

Before digging into the implementation side more I looked into the
userspace here. I admit I got lost a bit, since there's lots of
indirections and abstractions going on, but it seems like this is just a
fancy ioctl wrapper/driver backend abstractions. Not really something
applications would use.

>From the pretty picture on github it looks like there's some
opencl/ml/other fancy stuff sitting on top that applications would use. Is
that also available?

Thanks, Daniel

> 
> Thanks,
> -Sonal
> 
> Sonal Santan (6):
>   Add skeleton code: ioctl definitions and build hooks
>   Global data structures shared between xocl and xmgmt drivers
>   Add platform drivers for various IPs and frameworks
>   Add core of XDMA driver
>   Add management driver
>   Add user physical function driver
> 
>  drivers/gpu/drm/Kconfig                    |    2 +
>  drivers/gpu/drm/Makefile                   |    1 +
>  drivers/gpu/drm/xocl/Kconfig               |   22 +
>  drivers/gpu/drm/xocl/Makefile              |    3 +
>  drivers/gpu/drm/xocl/devices.h             |  954 +++++
>  drivers/gpu/drm/xocl/ert.h                 |  385 ++
>  drivers/gpu/drm/xocl/lib/Makefile.in       |   16 +
>  drivers/gpu/drm/xocl/lib/cdev_sgdma.h      |   63 +
>  drivers/gpu/drm/xocl/lib/libxdma.c         | 4368 ++++++++++++++++++++
>  drivers/gpu/drm/xocl/lib/libxdma.h         |  596 +++
>  drivers/gpu/drm/xocl/lib/libxdma_api.h     |  127 +
>  drivers/gpu/drm/xocl/mgmtpf/Makefile       |   29 +
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-core.c    |  960 +++++
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-core.h    |  147 +
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-cw.c      |   30 +
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-ioctl.c   |  148 +
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-reg.h     |  244 ++
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-sysfs.c   |  318 ++
>  drivers/gpu/drm/xocl/mgmtpf/mgmt-utils.c   |  399 ++
>  drivers/gpu/drm/xocl/subdev/dna.c          |  356 ++
>  drivers/gpu/drm/xocl/subdev/feature_rom.c  |  412 ++
>  drivers/gpu/drm/xocl/subdev/firewall.c     |  389 ++
>  drivers/gpu/drm/xocl/subdev/fmgr.c         |  198 +
>  drivers/gpu/drm/xocl/subdev/icap.c         | 2859 +++++++++++++
>  drivers/gpu/drm/xocl/subdev/mailbox.c      | 1868 +++++++++
>  drivers/gpu/drm/xocl/subdev/mb_scheduler.c | 3059 ++++++++++++++
>  drivers/gpu/drm/xocl/subdev/microblaze.c   |  722 ++++
>  drivers/gpu/drm/xocl/subdev/mig.c          |  256 ++
>  drivers/gpu/drm/xocl/subdev/sysmon.c       |  385 ++
>  drivers/gpu/drm/xocl/subdev/xdma.c         |  510 +++
>  drivers/gpu/drm/xocl/subdev/xmc.c          | 1480 +++++++
>  drivers/gpu/drm/xocl/subdev/xvc.c          |  461 +++
>  drivers/gpu/drm/xocl/userpf/Makefile       |   27 +
>  drivers/gpu/drm/xocl/userpf/common.h       |  157 +
>  drivers/gpu/drm/xocl/userpf/xocl_bo.c      | 1255 ++++++
>  drivers/gpu/drm/xocl/userpf/xocl_bo.h      |  119 +
>  drivers/gpu/drm/xocl/userpf/xocl_drm.c     |  640 +++
>  drivers/gpu/drm/xocl/userpf/xocl_drv.c     |  743 ++++
>  drivers/gpu/drm/xocl/userpf/xocl_ioctl.c   |  396 ++
>  drivers/gpu/drm/xocl/userpf/xocl_sysfs.c   |  344 ++
>  drivers/gpu/drm/xocl/version.h             |   22 +
>  drivers/gpu/drm/xocl/xclbin.h              |  314 ++
>  drivers/gpu/drm/xocl/xclfeatures.h         |  107 +
>  drivers/gpu/drm/xocl/xocl_ctx.c            |  196 +
>  drivers/gpu/drm/xocl/xocl_drm.h            |   91 +
>  drivers/gpu/drm/xocl/xocl_drv.h            |  783 ++++
>  drivers/gpu/drm/xocl/xocl_subdev.c         |  540 +++
>  drivers/gpu/drm/xocl/xocl_thread.c         |   64 +
>  include/uapi/drm/xmgmt_drm.h               |  204 +
>  include/uapi/drm/xocl_drm.h                |  483 +++
>  50 files changed, 28252 insertions(+)
>  create mode 100644 drivers/gpu/drm/xocl/Kconfig
>  create mode 100644 drivers/gpu/drm/xocl/Makefile
>  create mode 100644 drivers/gpu/drm/xocl/devices.h
>  create mode 100644 drivers/gpu/drm/xocl/ert.h
>  create mode 100644 drivers/gpu/drm/xocl/lib/Makefile.in
>  create mode 100644 drivers/gpu/drm/xocl/lib/cdev_sgdma.h
>  create mode 100644 drivers/gpu/drm/xocl/lib/libxdma.c
>  create mode 100644 drivers/gpu/drm/xocl/lib/libxdma.h
>  create mode 100644 drivers/gpu/drm/xocl/lib/libxdma_api.h
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/Makefile
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-core.c
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-core.h
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-cw.c
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-ioctl.c
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-reg.h
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-sysfs.c
>  create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-utils.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/dna.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/feature_rom.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/firewall.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/fmgr.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/icap.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/mailbox.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/mb_scheduler.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/microblaze.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/mig.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/sysmon.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/xdma.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/xmc.c
>  create mode 100644 drivers/gpu/drm/xocl/subdev/xvc.c
>  create mode 100644 drivers/gpu/drm/xocl/userpf/Makefile
>  create mode 100644 drivers/gpu/drm/xocl/userpf/common.h
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_bo.c
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_bo.h
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_drm.c
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_drv.c
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_ioctl.c
>  create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_sysfs.c
>  create mode 100644 drivers/gpu/drm/xocl/version.h
>  create mode 100644 drivers/gpu/drm/xocl/xclbin.h
>  create mode 100644 drivers/gpu/drm/xocl/xclfeatures.h
>  create mode 100644 drivers/gpu/drm/xocl/xocl_ctx.c
>  create mode 100644 drivers/gpu/drm/xocl/xocl_drm.h
>  create mode 100644 drivers/gpu/drm/xocl/xocl_drv.h
>  create mode 100644 drivers/gpu/drm/xocl/xocl_subdev.c
>  create mode 100644 drivers/gpu/drm/xocl/xocl_thread.c
>  create mode 100644 include/uapi/drm/xmgmt_drm.h
>  create mode 100644 include/uapi/drm/xocl_drm.h
> 
> --
> 2.17.0
> _______________________________________________
> dri-devel mailing list
> dri-devel@...ts.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Powered by blists - more mailing lists