Message-ID: <720994f2-51b5-944e-bc35-716f571687e3@broadcom.com>
Date:   Fri, 25 Sep 2020 09:50:15 -0700
From:   Scott Branden <scott.branden@...adcom.com>
To:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Cc:     Arnd Bergmann <arnd@...db.de>, Kees Cook <keescook@...omium.org>,
        linux-kernel@...r.kernel.org,
        bcm-kernel-feedback-list@...adcom.com,
        Desmond Yan <desmond.yan@...adcom.com>,
        James Hu <james.hu@...adcom.com>
Subject: Re: [PATCH v3 2/3] misc: bcm-vk: add Broadcom VK driver

Hi Greg,

On 2020-09-24 10:10 p.m., Greg Kroah-Hartman wrote:
> On Thu, Sep 24, 2020 at 02:40:08PM -0700, Scott Branden wrote:
>>> Ugh, yes, it is uncommon because those are two different things.  Why do
>>> you need/want a misc driver to control a tty device?  Why do you need a
>>> tty device?  What really is this beast?
>> The beast consists of a PCI card.  The PCI card communicates to the host via shared memory in BAR space and MSIX interrupts.
>> The host communicates through shared memory in BAR space and generates mailbox interrupts.
> That describes any PCI card :)
>
>> In addition the PCI card has DMA access to host memory to access data for processing.
>> The misc driver handles these operations.  Multiple user space processes are accessing the misc device at the same time
>> to perform simultaenous offload operations.  Each process opens the device and sends multiple commands with write operations
>> and receives solicited and unsolicited responses read read operations.  In addition there are sysfs entries to collect statistics and errors.
>> Ioctl operations are present for loading new firmware images to the card and reset operations.
>>
>> In addition, the card has multiple physical UART connections to access consoles on multi processors on the card.
>> But, in a real system there is no serial cable connected from the card to the server.  And, there could be 16 PCIe cards
>> plugged into a server so that would make it even more unrealistic to access these consoles via a UART.
>>
>> Fortunately the data sent out to each physical UART exists in a circular buffer.  These circular buffer can be access via the host in BAR space.
>> So we added tty device nodes per UART (2 per PCIe).  These are not the misc device node, these are seperate tty device nodes that are opened
>> and behave like tty devices.
>>
>> We are not using the misc driver to control tty.
>>   1) The PCI driver is the foundaton of it which provides the physical interface to the card, 2) misc driver is a presentation to user space to do its regular message - this allows user to treat it as a char-device, so its like fopen/read/write/close etc. and 3) tty is an additional debug channel which could be on or off.
>>
>> Please suggest how we can access the misc device and tty device nodes other than how we have implemented it in a single driver?
> You haven't described what the card actually is for.  You describe how
> you have tried to hook it up to userspace, but what is the use-case
> here?
The driver is used for real-time high performance, high throughput, low latency offload compute engine operations.
These operations are used to offload tasks that take up far too much of a CPU to be run on a host which would only be able to run a few of these tasks.
It is used for such tasks as: video and image processing and crypto.
The card is capable of performing many of these task in parallel.  And, many of these cards can be plugged into the same system (say 16 cards).
So instead of a host taking all its time running a single one of these tasks, 100's of these tasks can be performed instead.
Of course, future cards will be able to perform even more operations, host memory bandwidth becoming a bottleneck.

The main operation of the card is to receive commands from various user space operations to offload processing of data.
This usually involves a sequence of offload operations.  The card will DMA the data from host memory and act upon it, signalling appropriate
events back to the host.  At some point the card will typically write the data back to host memory via DMA.  Various statistics and events can also
be queried via sysfs entries.  Crash and real time logs can also be retrieved.  User space processes need to be killed when the card resets or crashes so those tasks do not hang.  Console access to the card's processors is also required.

Most of this could be done differently with uio and daemons and ipc mechanisms.
The linux kernel already provides all the mechanisms that would need to be reinvented in user space.
User space data would also need to be passed from individual processes to a shared daemon, which is not desirable for security purposes vs. keeping it in the individual user space and passed to the kernel driver.
The scatter gather list creation would need to somehow be done.  I don't know if this could be added to a uio driver or if we would need another driver to do such operation.
I guess you would need a custom app to replace every standard app accessing the tty device.
It is just a better model to have kernel operations in the linux kernel and a user space library that opens the device and handles the higher level message through read/write operations.  And then control and debug handled via ioctl and sysfs.

A generic linux model should exist for the offload compute performance and features we are looking for, but doesn't right now.
It would need to have the same reliability and scalability we have and could be tuned as needed to maximize performance, security, and memory usage for the particular use case as needed.
>   Why even have a kernel driver at all and not do everything in a
> uio driver?
>
> Creating random misc devices and tty devices implies that the device
> does not fit into any of the different existing kernel apis, so you need
> to write your own "special" one, which is always worrying.
>
> Without actually knowing what this device is, it's hard to judge if you
> really should be using something existing or not.
Hopefully this helps explain why we have made the design decision to use a kernel module plus user space library to integrate with customer
applications and systems.
>
> thanks,
>
> greg k-h


Download attachment "smime.p7s" of type "application/pkcs7-signature" (4169 bytes)

Powered by blists - more mailing lists