Date:   Mon, 25 Jul 2022 15:52:59 +0900
From:   Jiho Chu <jiho.chu@...sung.com>
To:     gregkh@...uxfoundation.org, arnd@...db.de,
        linux-kernel@...r.kernel.org
Cc:     yelini.jeong@...sung.com, myungjoo.ham@...sung.com,
        Jiho Chu <jiho.chu@...sung.com>
Subject: [PATCH 0/9] Samsung Trinity NPU device driver

Hello,

My name is Jiho Chu, and working for device driver and system daemon for
several years at Samsung Electronics. 

Trinity Neural Processing Unit (NPU) series are hardware accelerators
for neural network processing in embedded systems, which are integrated
into application processors or SoCs. Trinity NPU is compatible with AMBA
bus architecture and first launched in 2018 with its first version for
vision processing, Trinity Version1 (TRIV1). Its second version, TRIV2,
is released in Dec, 2021. Another Trinity NPU for audio processing is
referred as TRIA.

TRIV2 is shipped for many models of 2022 Samsung TVs, providing
acceleration for various AI-based applications, which include image
recognition and picture quality improvements for streaming video, which
can be accessed via GStreamer and its neural network plugins,
NNStreamer.

In this patch set, it includes Trinity Vision 2 kernel device driver.
Trinity Vision 2 supports accelerating image inference process for
Convolution Neural Network (CNN). The CNN workload is executed by Deep
Learning Accelerator (DLA), and general Neural Network Layers are
executed by Digital Signal Processor (DSP). And there is a Control
Processor (CP) which can control DLA and DSP. These three IPs (DLA, DSP,
CP) are composing Trinity Vision 2 NPU, and the device driver mainly
supervise the CP to manage entire NPU.

Controlling DLA and DSP operations is performed with internal command
instructions. and the instructions for the Trinity is similar with
general processor's ISA, but it is specialized for Neural Processing
operations. The virtual ISA (vISA) is designed for calculating multiple
data with single operation, like modern SIMD processor. The device
driver loads a program to CP at start up, and the program can decode a
binary which is built with the vISA. We calls this decoding program as a
Instruction Decoding Unit (IDU) program. While running the NPU, the CP
executes IDU program to fetch and decode instructions which made up of
vISA, by the scheduling policy of the device driver.

These DLA, DSP and CP are loosely coupled using ARM's AMBA, so the
Trinity can easily communicate with most ARM processors. Each IPs
designed to have memory-mapped registers which can be used to control
the IP, and the CP provides Wait-For-Event (WFE) operation to subscribe
interrupt signals from the DLA and DSP. Also, embedded Direct Memory
Access Controller (DMAC) manages data communications between internal
SRAM and outer main memory, IOMMU module supports unified memory space.

A user can control the Trinity NPU with IOCTLs provided by driver. These
controls includes memory management operations to transfer model data
(HWMEM_ALLOC/HWMEM_DEALLOC), NPU workload control operations to submit
workload (RUN/STOP), and statistics operations to check current NPU
status. (STAT)

The device driver also implemented features for developers. It provides
sysfs control attributes like stop, suspend, sched_test, and profile.
Also, it provides status attributes like app status, a number of total
requests, a number of active requests and memory usages. For the tracing
operations, several ftrace events are defined and embedded for several
important points.

I would highly appreciate your feedback.
Review, question or anythings.


Thanks.
Jiho Chu

Jiho Chu (9):
  trinity: Add base driver
  tirnity: Add dma memory module
  trinity: Add load/unload IDU files
  trinity: Add schduler module
  trinity: Add sysfs debugfs module
  trinity: Add pm and ioctl feature
  trinity: Add profile module
  trinity: Add trace module
  MAINTAINERS: add TRINITY driver

 MAINTAINERS                                   |    7 +
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/trinity/Kconfig                  |   27 +
 drivers/misc/trinity/Makefile                 |   12 +
 drivers/misc/trinity/sched/core.c             |  170 ++
 drivers/misc/trinity/sched/priority.c         |  335 +++
 drivers/misc/trinity/sched/priority.h         |   18 +
 drivers/misc/trinity/sched/sched.h            |   52 +
 drivers/misc/trinity/trinity.c                | 1282 +++++++++++
 drivers/misc/trinity/trinity_common.h         |  434 ++++
 drivers/misc/trinity/trinity_debug.c          |  358 ++++
 drivers/misc/trinity/trinity_hwmem.c          |  438 ++++
 drivers/misc/trinity/trinity_hwmem.h          |   45 +
 drivers/misc/trinity/trinity_pm.c             |   76 +
 drivers/misc/trinity/trinity_resv_mem.c       |  264 +++
 drivers/misc/trinity/trinity_resv_mem.h       |   41 +
 drivers/misc/trinity/trinity_stat.c           |  893 ++++++++
 drivers/misc/trinity/trinity_stat.h           |   56 +
 drivers/misc/trinity/trinity_sysfs.c          |  864 ++++++++
 drivers/misc/trinity/trinity_trace.c          |   15 +
 drivers/misc/trinity/trinity_trace.h          |  406 ++++
 drivers/misc/trinity/trinity_vision2_drv.c    | 1893 +++++++++++++++++
 .../misc/trinity/trinity_vision2_profile.h    |  324 +++
 drivers/misc/trinity/trinity_vision2_regs.h   |  210 ++
 include/uapi/misc/trinity.h                   |  458 ++++
 26 files changed, 8680 insertions(+)
 create mode 100644 drivers/misc/trinity/Kconfig
 create mode 100644 drivers/misc/trinity/Makefile
 create mode 100644 drivers/misc/trinity/sched/core.c
 create mode 100644 drivers/misc/trinity/sched/priority.c
 create mode 100644 drivers/misc/trinity/sched/priority.h
 create mode 100644 drivers/misc/trinity/sched/sched.h
 create mode 100644 drivers/misc/trinity/trinity.c
 create mode 100644 drivers/misc/trinity/trinity_common.h
 create mode 100644 drivers/misc/trinity/trinity_debug.c
 create mode 100644 drivers/misc/trinity/trinity_hwmem.c
 create mode 100644 drivers/misc/trinity/trinity_hwmem.h
 create mode 100644 drivers/misc/trinity/trinity_pm.c
 create mode 100644 drivers/misc/trinity/trinity_resv_mem.c
 create mode 100644 drivers/misc/trinity/trinity_resv_mem.h
 create mode 100644 drivers/misc/trinity/trinity_stat.c
 create mode 100644 drivers/misc/trinity/trinity_stat.h
 create mode 100644 drivers/misc/trinity/trinity_sysfs.c
 create mode 100644 drivers/misc/trinity/trinity_trace.c
 create mode 100644 drivers/misc/trinity/trinity_trace.h
 create mode 100644 drivers/misc/trinity/trinity_vision2_drv.c
 create mode 100644 drivers/misc/trinity/trinity_vision2_profile.h
 create mode 100644 drivers/misc/trinity/trinity_vision2_regs.h
 create mode 100644 include/uapi/misc/trinity.h

-- 
2.25.1

Powered by blists - more mailing lists