Message-ID: <20260123080938.3367348-5-antoine.bouyer@nxp.com>
Date: Fri, 23 Jan 2026 09:09:31 +0100
From: Antoine Bouyer <antoine.bouyer@....com>
To: julien.vuillaumier@....com,
	alexi.birlinger@....com,
	daniel.baluta@....com,
	peng.fan@....com,
	frank.li@....com,
	jacopo.mondi@...asonboard.com,
	laurent.pinchart@...asonboard.com,
	mchehab@...nel.org,
	robh@...nel.org,
	krzk+dt@...nel.org,
	conor+dt@...nel.org,
	shawnguo@...nel.org,
	s.hauer@...gutronix.de,
	kernel@...gutronix.de,
	festevam@...il.com
Cc: linux-kernel@...r.kernel.org,
	linux-media@...r.kernel.org,
	devicetree@...r.kernel.org,
	linux-arm-kernel@...ts.infradead.org,
	Antoine Bouyer <antoine.bouyer@....com>
Subject: [RFC v1 04/11] media: Documentation: Add NXP neoisp driver documentation

Document the NXP neoisp driver both in the admin-guide for neoisp IP
description, and in the userpace-api for neoisp interface description.

Signed-off-by: Antoine Bouyer <antoine.bouyer@....com>
---
 .../admin-guide/media/nxp-neoisp-diagram.dot  |  22 ++
 .../admin-guide/media/nxp-neoisp.dot          |  16 ++
 .../admin-guide/media/nxp-neoisp.rst          | 189 ++++++++++++++++++
 .../admin-guide/media/v4l-drivers.rst         |   1 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-nxp-neoisp.rst          | 114 +++++++++++
 6 files changed, 343 insertions(+)
 create mode 100644 Documentation/admin-guide/media/nxp-neoisp-diagram.dot
 create mode 100644 Documentation/admin-guide/media/nxp-neoisp.dot
 create mode 100644 Documentation/admin-guide/media/nxp-neoisp.rst
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst

diff --git a/Documentation/admin-guide/media/nxp-neoisp-diagram.dot b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot
new file mode 100644
index 000000000000..ac0d950bc1dc
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot
@@ -0,0 +1,22 @@
+digraph G {
+  rankdir = "LR";
+  node [shape=rect];
+  splines = ortho;
+  label = "Neoisp pipeline diagram";
+  {In0, In1 } -> HC -> {HDR_decomp0, HDR_decomp1};
+  HDR_decomp0 -> OBWB0 -> HDR_merge;
+  HDR_decomp1 -> OBWB1 -> HDR_merge;
+  HDR_merge -> RGBIR -> IR_compression;
+  RGBIR -> Statistics;
+  RGBIR -> OBWB2 ->BNR -> Vignetting -> ColorTemp;
+  Vignetting -> Demosaic -> RGB2YUV -> DRC -> AF;
+  DRC-> NR -> EE -> DF -> Gamma -> Packetizer;
+  DRC -> DMAP -> DF[weight=2];
+  DRC -> CCONV -> CAS -> Gamma;
+  {rank = "same"; RGBIR, DRC}
+  {rank = "same"; AF, NR, DMAP}
+  {rank = "same"; IR_compression, Vignetting}
+  IR_compression -> AXIOutDMA;
+  Packetizer -> AXIOutDMA [weight=3];
+  AXIOutDMA -> {"Out0", "Out1"}
+}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.dot b/Documentation/admin-guide/media/nxp-neoisp.dot
new file mode 100644
index 000000000000..abcc2dc9bb55
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.dot
@@ -0,0 +1,16 @@
+digraph board {
+	rankdir=TB
+	n00000001 [label="{{<port0> 0 | <port1> 1 | <port2> 2} | neoisp\n | {<port3> 3 | <port4> 4 | <port5> 5}}", shape=Mrecord, style=filled, fillcolor=green]
+	n00000001:port3 -> n00000020 [style=dashed]
+	n00000001:port4 -> n00000022 [style=dashed]
+	n00000001:port5 -> n00000024 [style=dashed]
+	n0000000a [label="neoisp-input0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+	n0000000a -> n00000001:port0 [style=bold]
+	n00000010 [label="neoisp-input1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
+	n00000010 -> n00000001:port1 [style=dashed]
+	n00000016 [label="neoisp-params\n/dev/video2", shape=box, style=filled, fillcolor=yellow]
+	n00000016 -> n00000001:port2 [style=dashed]
+	n00000020 [label="neoisp-frame\n/dev/video3", shape=box, style=filled, fillcolor=yellow]
+	n00000022 [label="neoisp-ir\n/dev/video4", shape=box, style=filled, fillcolor=yellow]
+	n00000024 [label="neoisp-stats\n/dev/video5", shape=box, style=filled, fillcolor=yellow]
+}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.rst b/Documentation/admin-guide/media/nxp-neoisp.rst
new file mode 100644
index 000000000000..0632de6bc51d
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.rst
@@ -0,0 +1,189 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================================
+NXP Neo Image Signal Processor (neoisp)
+=======================================
+
+Introduction
+============
+
+Neoisp performs a set of image processing tasks on the RAW camera stream, where
+the input camera stream and Neoisp processed output image are stored in DDR or
+any system memory fast enough to keep up with Neoisp processing.
+The overall Neoisp operation is frame based, that is 1 complete image frame is
+read and output pixel by pixel, line by line wise.
+
+Revisions
+=========
+
+NXP Neoisp hw revisions are listed in UAPI in enum :c:type:`neoisp_version_e`.
+Version is stored in the Media device information, hw revision field,
+accessible from the ioctl MEDIA_IOC_DEVICE_INFO.
+
+There are 2 versions:
+
+- NEOISP_HW_V1: used at least in i.MX95 A0/A1 SoC, not targeted for production.
+- NEOISP_HW_V2: used at least in i.MX95 B0 SoC.
+
+Neoisp hardware
+===============
+
+The Neoisp registers and pipeline processing are documented in the NXP Image
+Signal Processor Specification document (under NDA).
+
+The NEO pipeline block diagram is shown below:
+
+.. kernel-figure:: nxp-neoisp-diagram.dot
+    :alt:   Diagram of neoisp pipeline
+    :align: center
+
+It is composed by the following HW blocks:
+
+- a Head Color (HC) selection block,
+- two HDR decompression blocks, one for each input,
+- three Optical Black correction and White Balance (OBWB) blocks at different
+  stages in the pipeline,
+- a HDR merge block for HDR image capture from the 2 input lines,
+- a RGB-IR to RGGB converter,
+- a Bayer Noise Reduction (BNR) block,
+- a Vignetting block, aka Lens Shading Correction (LSC),
+- a Demosaic block for RAW image conversion to RGB,
+- a RGB Color Correction Matrix (CCM) and Color Space Converter aka CSC or
+  RGB2YUV,
+- a Dynamic Range Compression (DRC) block,
+- the Denoising pipeline (composed by multiple blocks for Noise Reduction, Edge
+  Enhancement, Gamma Compensation, etc),
+- a Packetizer used for UV sub-sampling or RGB packing.
+
+All these blocks are controlled by SW through registers. Some of these
+registers are accessible by the uAPI, so that usespace application and Image
+Processing Algorithms (IPA) can configure them through the parameters buffers.
+
+Neoisp driver
+=============
+
+Neoisp driver is located under drivers/media/platform/nxp/neoisp.
+It uses the `V4L2 API` and the `V4L2 subdev API` to register capture and output
+video devices in addition to a subdevice for neoisp that connects the video
+devices in a single media graph realized using the `Media Controller (MC) API`.
+
+The media topology registered by Neoisp driver is represented below:
+
+.. kernel-figure:: nxp-neoisp.dot
+    :alt:   Diagram of neoisp media device topology
+    :align: center
+
+
+The media graph registers the following video device nodes:
+
+- neoisp-input0: output device for RAW frames to be submitted to the ISP for processing.
+- neoisp-input1: output device for RAW frames short capture in HDR merge mode.
+- neoisp-params: output meta device for parameters provided by user space 3A algorithms.
+- neoisp-frame: capture device for RGB/YUV pixels of the processed images.
+- neoisp-ir: capture device for the infra-red pixels of the processed images.
+- neoisp-stats: capture meta device for generated image statistics for user space 3A algorithms.
+
+neoisp-input0, neoisp-input1
+----------------------------
+
+Images to be processed by Neoisp are queued to the neoisp-input0 (and
+neoisp-input1 when in HDR mode) output device nodes. Supported image formats
+as input to the ISP are:
+
+- Raw bayer formats:
+
+  - 8 bits raw (V4L2_PIX_FMT_SRGGB8, V4L2_PIX_FMT_SBGGR8, V4L2_PIX_FMT_SGBRG8,
+    V4L2_PIX_FMT_SGRBG8)
+  - 10 bits raw (V4L2_PIX_FMT_SRGGB10, V4L2_PIX_FMT_SBGGR10,
+    V4L2_PIX_FMT_SGBRG10, V4L2_PIX_FMT_SGRBG10)
+  - 12 bits raw (V4L2_PIX_FMT_SRGGB12, V4L2_PIX_FMT_SBGGR12,
+    V4L2_PIX_FMT_SGBRG12, V4L2_PIX_FMT_SGRBG12)
+  - 14 bits raw (V4L2_PIX_FMT_SRGGB14, V4L2_PIX_FMT_SBGGR14,
+    V4L2_PIX_FMT_SGBRG14, V4L2_PIX_FMT_SGRBG14)
+  - 16 bits raw (V4L2_PIX_FMT_SRGGB16, V4L2_PIX_FMT_SBGGR16,
+    V4L2_PIX_FMT_SGBRG16, V4L2_PIX_FMT_SGRBG16)
+
+- Monochrome formats:
+
+  - 8 bits Monochrome (V4L2_PIX_FMT_GREY)
+  - 10 bits Monochrome (V4L2_PIX_FMT_Y10)
+  - 12 bits Monochrome (V4L2_PIX_FMT_Y12)
+  - 14 bits Monochrome (V4L2_PIX_FMT_Y14)
+  - 16 bits Monochrome (V4L2_PIX_FMT_Y16)
+
+.. note::
+   RGBIr camera sensors are supported as well, and can be used through user
+   space activation of the IR block.
+
+.. note::
+   neoisp-input1 link is mutable and should be enabled in case a short capture
+   image buffer is provided to the ISP for HDR merge.
+
+.. _neoisp_params:
+
+neoisp-params
+-------------
+
+The neoisp-params output meta device receives configuration data to be written
+to Neoisp registers and internal memory for desired input image processing.
+This v4l2 device accepts the `legacy parameters` format, or the `extensible
+parameters` format.
+
+When using the `legacy parameters` format, the parameters buffer is defined by
+structure :c:type:`neoisp_meta_params_s`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_PARAMS <v4l2-meta-fmt-neo-isp-params>` as dataformat.
+
+When using the `extensible parameters` format, the parameters buffer is defined
+by structure :c:type:`neoisp_ext_params_s`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_EXT_PARAMS <v4l2-meta-fmt-neo-isp-ext-params>` as
+dataformat.
+
+When the related media link is disabled, the image decoding will be done based
+on the default parameters of the ISP.
+
+neoisp-frame
+------------
+
+The capture device writes to memory the RGB or YUV pixels of the image processed
+by Neoisp when the media link is enabled. If the related media link is disabled,
+the processed image will be written to dummy buffer and not delivered to the
+neoisp-frame video device node.
+
+neoisp-ir
+---------
+
+The capture device writes to memory the RGBIr pixels of the image processed by
+Neoisp when the media link is enabled. If the related media link is disabled,
+the processed image will not be delivered to the neoisp-ir video device node.
+
+.. _neoisp_stats:
+
+neoisp-stats
+------------
+
+The neoisp-stats capture meta device provides statistics data generated by
+Neoisp hardware while processing the input image.
+This v4l2 device accepts the `legacy statistics` format, or the `extensible
+statistics` format.
+
+When using the `legacy statistics` format, the statistics buffer is defined by
+structure :c:type:`neoisp_meta_stats_s`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_STATS <v4l2-meta-fmt-neo-isp-stats>` as dataformat.
+
+When using the `extensible statistics` format, the statistics buffer is defined
+by structure :c:type:`neoisp_ext_stats_s`, and userspace should set
+:ref:`V4L2_META_FMT_NEO_ISP_EXT_STATS <v4l2-meta-fmt-neo-isp-ext-stats>` as
+dataformat.
+
+When the related media link is disabled, the decoding statistics will not be
+delivered to the neoisp-stats meta device node.
+
+Control
+=======
+
+To support additional neoisp hardware revisions, the read-only bitmask control
+`V4L2_CID_NEOISP_SUPPORTED_PARAMS_BLOCKS` can be used to query the list of
+supported blocks. Each bit represents the availability of the corresponding
+entry from the :c:type:`neoisp_param_block_type_e` enum. In current driver
+version, default and max values represent the blocks supported by the i.MX95
+SoC.
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index 393f83e8dc4d..5b6c71ae369d 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -21,6 +21,7 @@ Video4Linux (V4L) driver-specific documentation
 	ivtv
 	mali-c55
 	mgb4
+	nxp-neoisp
 	omap3isp
 	philips
 	qcom_camss
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 3e0cab153f0a..46268d955d3a 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -18,6 +18,7 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-d4xx
     metafmt-generic
     metafmt-intel-ipu3
+    metafmt-nxp-neoisp
     metafmt-pisp-be
     metafmt-pisp-fe
     metafmt-rkisp1
diff --git a/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
new file mode 100644
index 000000000000..bcde8edcb441
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
@@ -0,0 +1,114 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+********************************************************************************
+V4L2_META_FMT_NEO_ISP_PARAMS ('nnip'), V4L2_META_FMT_NEO_ISP_EXT_PARAMS ('nnep')
+********************************************************************************
+
+The Neoisp image signal processor is configured by userspace through a buffer
+of parameters to :ref:`neoisp-params <neoisp_params>` output device node using
+the :c:type:`v4l2_meta_format` interface.
+
+There are two methods that allow to configure the Neoisp: the `legacy
+parameters` configuration format and the `extensible parameters` configuration
+format.
+
+.. _v4l2-meta-fmt-neo-isp-params:
+
+Legacy parameters configuration format
+======================================
+
+When using the `legacy parameters` configuration format, parameters are passed
+to the :ref:`neoisp-params <neoisp_params>` metadata output video node using
+the `V4L2_META_FMT_NEO_ISP_PARAMS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`neoisp_meta_params_s` defined in ``nxp_neoisp.h``. So the structure
+can be obtained from the buffer by:
+
+.. code-block:: c
+
+	struct neoisp_meta_params_s *params = (struct neoisp_meta_params_s *) buffer;
+
+This method supports the Neoisp features currently available, it won't be
+maintained for future Neoisp versions. New applications should use the
+`extensible parameters` method then.
+
+.. _v4l2-meta-fmt-neo-isp-ext-params:
+
+Extensible parameters configuration format
+==========================================
+
+When using the `extensible parameters` configuration format, parameters are
+passed to the :ref:`neoisp-params <neoisp_params>` metadata output video node
+using the `V4L2_META_FMT_NEO_ISP_EXT_PARAMS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`v4l2_isp_params_buffer` defined in ``v4l2-isp.h``. The
+:c:type:`v4l2_isp_params_buffer` structure is designed to allow userspace to
+populate the data buffer with only the configuration data for the Neoisp blocks
+it intends to configure. The extensible parameters format design allows
+developers to define new block types to support new configuration parameters,
+and defines a versioning scheme so that it can be extended and versioned
+without breaking compatibility with existing applications.
+
+******************************************************************************
+V4L2_META_FMT_NEO_ISP_STATS ('nnis'), V4L2_META_FMT_NEO_ISP_EXT_STATS ('nnes')
+******************************************************************************
+
+The Neoisp image signal processor generates statistics data while processing an
+input image. These statistics are captured in a buffer and provided to
+userspace through the :ref:`neoisp-stats <neoisp_stats>` capture video node
+using the :c:type:`v4l2_meta_format` interface. The statistics data are
+processed by userspace application to produce the next Neoisp parameters.
+
+There are two methods to capture the statistics: the `legacy statistics` format
+and the `extensible statistics` format.
+
+.. _v4l2-meta-fmt-neo-isp-stats:
+
+Legacy statistics format
+========================
+
+When using the `legacy statistics` format, Neoisp statistics are passed from
+the :ref:`neoisp-stats <neoisp_stats>` metadata capture video node using the
+`V4L2_META_FMT_NEO_ISP_STATS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`neoisp_meta_stats_s` defined in ``nxp_neoisp.h``. So the structure can
+be obtained from the buffer by:
+
+.. code-block:: c
+
+	struct neoisp_meta_stats_s *params = (struct neoisp_meta_stats_s *) buffer;
+
+This method supports the Neoisp statistics currently available, it won't be
+maintained for future Neoisp versions. New applications should use the
+`extensible statistics` method then.
+
+.. _v4l2-meta-fmt-neo-isp-ext-stats:
+
+Extensible statistics format
+============================
+
+When using the `extensible statistics` format, the statistics buffer is passed
+from the :ref:`neoisp-stats <neoisp_stats>` metadata capture video node using
+the `V4L2_META_FMT_NEO_ISP_EXT_STATS` meta format.
+
+The buffer contains a single instance of the C structure
+:c:type:`v4l2_isp_stats_buffer` defined in ``v4l2-isp.h``. The
+:c:type:`v4l2_isp_stats_buffer` structure is designed to allow future Neoisp
+driver versions to populate the statistics buffer with future blocks
+statistics, and defines a versioning scheme so that it can be extended and
+versioned without breaking compatibility with existing applications.
+
+**********************
+Neoisp uAPI data types
+**********************
+
+This chapter describes the data types exposed to userspace by Neoisp driver.
+
+Some structure members are in a fixed-point format, in this case the related
+description will be ended by a fixed-point definition between parenthesis.
+
+.. kernel-doc:: include/uapi/linux/media/nxp/nxp_neoisp.h
+
-- 
2.52.0

Powered by blists - more mailing lists