Message-ID: <20251111000627.GA30837@pendragon.ideasonboard.com>
Date: Tue, 11 Nov 2025 02:06:27 +0200
From: Laurent Pinchart <laurent.pinchart@...asonboard.com>
To: Michael Riesch <michael.riesch@...labora.com>
Cc: Sakari Ailus <sakari.ailus@...ux.intel.com>,
	Mehdi Djait <mehdi.djait@...ux.intel.com>,
	Maxime Chevallier <maxime.chevallier@...tlin.com>,
	Théo Lebrun <theo.lebrun@...tlin.com>,
	Thomas Petazzoni <thomas.petazzoni@...tlin.com>,
	Gerald Loacker <gerald.loacker@...fvision.net>,
	Bryan O'Donoghue <bryan.odonoghue@...aro.org>,
	Markus Elfring <Markus.Elfring@....de>,
	Mauro Carvalho Chehab <mchehab@...nel.org>,
	Rob Herring <robh+dt@...nel.org>,
	Krzysztof Kozlowski <krzk+dt@...nel.org>,
	Conor Dooley <conor+dt@...nel.org>,
	Heiko Stuebner <heiko@...ech.de>,
	Kever Yang <kever.yang@...k-chips.com>,
	Nicolas Dufresne <nicolas.dufresne@...labora.com>,
	Sebastian Reichel <sebastian.reichel@...labora.com>,
	Collabora Kernel Team <kernel@...labora.com>,
	Paul Kocialkowski <paulk@...-base.io>,
	Alexander Shiyan <eagle.alexander923@...il.com>,
	Val Packett <val@...kett.cool>, Rob Herring <robh@...nel.org>,
	Philipp Zabel <p.zabel@...gutronix.de>, linux-media@...r.kernel.org,
	devicetree@...r.kernel.org, linux-kernel@...r.kernel.org,
	linux-arm-kernel@...ts.infradead.org,
	linux-rockchip@...ts.infradead.org,
	Mehdi Djait <mehdi.djait@...tlin.com>,
	Krzysztof Kozlowski <krzysztof.kozlowski@...aro.org>,
	Bryan O'Donoghue <bod@...nel.org>, Chen-Yu Tsai <wens@...e.org>
Subject: Re: [PATCH v14 00/18] media: rockchip: add a driver for the rockchip
 camera interface

On Mon, Nov 10, 2025 at 11:29:57AM +0100, Michael Riesch wrote:
> On 11/10/25 09:43, Sakari Ailus wrote:
> > On Fri, Nov 07, 2025 at 09:51:37PM +0100, Michael Riesch wrote:
> >> On 11/7/25 19:54, Laurent Pinchart wrote:
> >>> On Fri, Nov 07, 2025 at 07:41:59PM +0100, Michael Riesch wrote:
> >>>> On 11/7/25 18:32, Sakari Ailus wrote:
> >>>>> On Fri, Oct 24, 2025 at 02:51:29PM +0200, Michael Riesch via B4 Relay wrote:
> >>>>>> Habidere,
> >>>>>>
> >>>>>> This series introduces support for the Rockchip Camera Interface (CIF),
> >>>>>> which is featured in many Rockchip SoCs in different variations.
> >>>>>> For example, the PX30 Video Input Processor (VIP) is able to receive
> >>>>>> video data via the Digital Video Port (DVP, a parallel data interface)
> >>>>>> and transfer it into system memory using a double-buffering mechanism
> >>>>>> called ping-pong mode.
> >>>>>> The RK3568 Video Capture (VICAP) unit, on the other hand, features a
> >>>>>> DVP and a MIPI CSI-2 receiver that can receive video data independently
> >>>>>> (both using the ping-pong scheme).
> >>>>>> The different variants may have additional features, such as scaling
> >>>>>> and/or cropping.
> >>>>>> Finally, the RK3588 VICAP unit constitutes an essential piece of the
> >>>>>> camera interface with one DVP, six MIPI CSI-2 receivers, scale/crop
> >>>>>> units, and a data path multiplexer (to scaler units, to ISP, ...).
> >>>>>
> >>>>> I understand both RK3568 and RK3588 include an ISP. Do you have insight on
> >>>>> how would this work, should the support for the ISP be added later on?
> >>>>
> >>>> Short answer: Yes and yes.
> >>>>
> >>>> Long answer:
> >>>>
> >>>> The patch series at hand adds support for the PX30 VIP and the RK3568
> >>>> VICAP. I cannot really say something about the PX30, but on the RK3568
> >>>> VICAP and ISP are orthogonal (the ISP features its own MIPI CSI-2
> >>>> receiver, different from that introduced in this series). Thus, ISP
> >>>> support can be introduced anytime (whenever someone is motivated ;-)).
> >>>
> >>> Won't they both be connected to the same sensor though, and probably the
> >>> same D-PHY in the SoC ? They don't seem entirely separate to me.
> >>
> >> The MIPI CSI-2 DPHY is shared, indeed. Thus, they *maybe technically
> >> could be* connected to the same sensor, but I don't know whether that
> >> works and fail to see why anyone would to such a thing (if it is about
> >> raw capture, the MIPI CSI-2 receiver in the ISP can do that on its own).
> >>
> >> The DPHY can be operated in split mode, with two lanes for VICAP and two
> >> lanes for ISP. This is not implemented yet, but can be done at a later
> >> stage on PHY level (not media related). In this case, ISP and VICAP can
> >> receive data from different subdevices via CSI-2.
> > 
> > The two would be part of the same media graph in that case and as there are
> > two CSI-2 receivers and a single PHY, the PHY would probably need to have a
> > sub-device as well, to allow link configuration to be used to select where
> > the PHY is connected.
> > 
> > I don't think we have such a setup elsewhere, and supporting this would
> > require changes in the MC framework.
> 
> What follows is a response that also addresses issues raised during our
> off-list discussion.
> 
> First of all, I agree with you that the RK3568 HW is "a bit special" (to
> say the least) in that regard. Let's have an outlook to newer SoCs, such
> as the RK3588: Here, the MIPI CSI-2 DPHYs (there are two of them) with
> their split mode are present as well, but the assignment is fixed. For
> example, the RK3588 VICAP has six MIPI CSI-2 receiver units and six MIPI
> CSI-2 capture units. Units 1 and 2 handle a different MIPI PHY, units 3
> and 5 handle the DPHYs (without split mode), units 4 and/or 6 are active
> whenever DPHY 1 and/or 2 is in split mode.
> 
> I would model this by adding support for more than one (logical) PHYs
> (phy-cells = <1>;) and assigning the logical PHYs to the MIPI CSI-2
> receivers. There is not really a possibility to route anything at this
> point (routing is done in a MUX unit that takes the different MIPI CSI-2
> receivers as inputs).
> 
> Now back to the peculiar RK3568 situation: By default the split mode of
> the DPHY is off and both VICAP and ISP are able to receive the same data
> (from up to four lanes) with their MIPI CSI-2 receivers (not sure
> whether both can be active at the same time, though).

A common use case for capturing data from the same sensor through both
ISP and VICAP would be routing image data to the ISP and embedded data
to VICAP. Assuming the hardware is able to do this, there will be users
interested in this feature.

> There are two bits
> in the GRF that define the lanes that ISP and VICAP receive in split
> mode (lane 0/1 or lane 2/3). Not sure whether these bits are supposed to
> be changed during runtime.
> 
> I would suggest modelling this on PHY level in DT, e.g., by passing
> reasonable properties to the dphy node, such as
>     rockchip,dphy-split-mode;

Split mode should indeed be conveyed through DT, as it's a property of
the hardware.

>     rockchip,dphy-split-invert;
> where the former activates the split mode and assigns lanes 0/1 to the
> ISP and lanes 2/3 to the VICAP, and the latter inverts this assignment
> (lanes 2/3 to the ISP and lanes 0/1 to the VICAP). This would facilitate
> the reasonable use cases with reasonable effort.

This I'm less convinced about. The routing should be dynamic.

How do you envision the connections to the CSI-2 sources to be modelled
in DT ? We need ports and endpoints, and data-lanes properties. Where
would those reside on the RK3568 side ?

> Otherwise, to keep it perfectly general and most flexible and
> everything, we would have to introduce another subdevice indeed, which
> would be active on the RK3568 exclusively. Therefore, I don't see that
> the PHY driver introduces this subdevice, but a specialized (syscon?)
> MUX driver that deals with the RK3568 GRF bits. Something like this
> 
>                            |----------------------|     |-------------|
>  Sensor A --- /2 lanes --- | lane 0/1      to ISP | --- | ISP MIPI RX |
>                            |                      |     |-------------|
>                            |                      |
>                            |                      |     |-------------|
>  Sensor B --- /2 lanes --- | lane 2/3    to VICAP | --- |VICAP MIPI RX|
>                            |----------------------|     |-------------|
> 
> But IMHO this will be too much effort for corner use case that I doubt
> anyone will actually use.
> 
> What do you think:
>  - Let's keep the PHYs out of V4L2/MC, ok?
>  - Let's model the reasonable use cases with device tree properties in
>    the dphy DT node, ok?

I think the routing should be dynamic.

> > How does the media graph look like for the device at the moment?
> 
> Please take a look at the media graph in the documentation patch (PATCH
> v14 01/18). This is without the ISP, but gives an overview of what the
> RK3568 VICAP is capable of.
> 
> Best regards,
> Michael
> 
> >> BTW the ISP is able to process the data captured by VICAP, but
> >> apparently this includes a RAM round trip (VICAP captures to memory, ISP
> >> operates in mem2mem mode).
> >>
> >>> A block diagram that shows connections between the CSI-2 pins, D-PHY,
> >>> CSI-2 receivers, VICAP and ISP could help.
> >>>
> >>>> Once this patch series is merged, I'll push out changes that introduce
> >>>> support for the RK3588 VICAP. We can discuss the integration of any
> >>>> RK3588 ISP in this scope then -- and there may be some things to discuss
> >>>> as there the VICAP and the ISP(s) are directly connected by means of a
> >>>> MUX unit in the VICAP.
> >>>>
> >>>> Alright?

-- 
Regards,

Laurent Pinchart

Powered by blists - more mailing lists