Date:   Thu, 2 Apr 2020 17:21:34 +1100
From:   Andrew Donnellan <ajd@...ux.ibm.com>
To:     Dan Williams <dan.j.williams@...el.com>,
        "Alastair D'Silva" <alastair@...ilva.org>
Cc:     "Aneesh Kumar K . V" <aneesh.kumar@...ux.ibm.com>,
        "Oliver O'Halloran" <oohall@...il.com>,
        Benjamin Herrenschmidt <benh@...nel.crashing.org>,
        Paul Mackerras <paulus@...ba.org>,
        Michael Ellerman <mpe@...erman.id.au>,
        Frederic Barrat <fbarrat@...ux.ibm.com>,
        Arnd Bergmann <arnd@...db.de>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Vishal Verma <vishal.l.verma@...el.com>,
        Dave Jiang <dave.jiang@...el.com>,
        Ira Weiny <ira.weiny@...el.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Mauro Carvalho Chehab <mchehab+samsung@...nel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Rob Herring <robh@...nel.org>,
        Anton Blanchard <anton@...abs.org>,
        Krzysztof Kozlowski <krzk@...nel.org>,
        Mahesh Salgaonkar <mahesh@...ux.vnet.ibm.com>,
        Madhavan Srinivasan <maddy@...ux.vnet.ibm.com>,
        Cédric Le Goater <clg@...d.org>,
        Anju T Sudhakar <anju@...ux.vnet.ibm.com>,
        Hari Bathini <hbathini@...ux.ibm.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Greg Kurz <groug@...d.org>,
        Nicholas Piggin <npiggin@...il.com>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Alexey Kardashevskiy <aik@...abs.ru>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        linuxppc-dev <linuxppc-dev@...ts.ozlabs.org>,
        linux-nvdimm <linux-nvdimm@...ts.01.org>,
        Linux MM <linux-mm@...ck.org>
Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow
 it to be mapped

On 1/4/20 7:48 pm, Dan Williams wrote:
> On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva <alastair@...ilva.org> wrote:
>>
>> OpenCAPI LPC memory is allocated per link, but each link supports
>> multiple AFUs, and each AFU can have LPC memory assigned to it.
> 
> Is there an OpenCAPI primer to decode these objects and their
> associations that I can reference?

There isn't presently a primer that I think addresses these questions 
nicely (to my knowledge - Fred might have something he can link to?) - 
there are the specs published by the OpenCAPI Consortium at 
https://opencapi.org but they're really for hardware implementers.

We should probably expand what's currently documented in 
Documentation/userspace-api/accelerators/ocxl.rst generally, and this 
series should probably update that to include details on LPC.

To explain the specific objects here:

- A "link" is a point-to-point link between the host CPU, and a single 
OpenCAPI card. (We don't currently support cards making use of multiple 
links for increased bandwidth, though that is supported from a hardware 
point of view.)

- On POWER9, each link appears as a separate PCI domain, with a single 
bus, and the card appears as a single device.

- A device can have up to 8 functions, per PCI.

- An Attached Functional Unit (AFU) is the abstraction for a particular 
application function. Each PCI function defines the number of AFUs it 
has through a set of OpenCAPI-specific DVSECs, max 64 per function. The 
ocxl driver handles AFU discovery.

- On the host side, LPC memory is mapped by setting a single BAR for the 
whole link, but on the device side, LPC memory is requested on a per-AFU 
basis, through an AFU descriptor that is exposed through the 
aforementioned DVSECs. Hence the need to loop through the AFUs and get 
the total required LPC memory to work out the correct BAR value.

-- 
Andrew Donnellan              OzLabs, ADL Canberra
ajd@...ux.ibm.com             IBM Australia Limited

Powered by blists - more mailing lists