Message-ID: <Z-6H6r0FIAbRtxwQ@kekkonen.localdomain>
Date: Thu, 3 Apr 2025 13:06:50 +0000
From: Sakari Ailus <sakari.ailus@...ux.intel.com>
To: Hans de Goede <hdegoede@...hat.com>
Cc: Bryan O'Donoghue <bryan.odonoghue@...aro.org>,
	Mauro Carvalho Chehab <mchehab@...nel.org>,
	Liam Girdwood <lgirdwood@...il.com>,
	Mark Brown <broonie@...nel.org>, Rob Herring <robh@...nel.org>,
	Krzysztof Kozlowski <krzk+dt@...nel.org>,
	Conor Dooley <conor+dt@...nel.org>,
	Bryan O'Donoghue <bod@...nel.org>, Hans de Goede <hansg@...nel.org>,
	Jingjing Xiong <jingjing.xiong@...el.com>,
	Hao Yao <hao.yao@...el.com>, Jim Lai <jim.lai@...el.com>,
	You-Sheng Yang <vicamo.yang@...onical.com>,
	Alan Stern <stern@...land.harvard.edu>,
	linux-kernel@...r.kernel.org, linux-media@...r.kernel.org,
	devicetree@...r.kernel.org
Subject: Re: [PATCH v2 2/2] media: i2c: ov02e10: add OV02E10 image sensor
 driver

Hi Hans,

On Tue, Apr 01, 2025 at 03:34:39PM +0200, Hans de Goede wrote:
> Hi Sakari,
> 
> On 1-Apr-25 11:42 AM, Sakari Ailus wrote:
> > Hi Hans, Bryan,
> > 
> > On Tue, Apr 01, 2025 at 10:12:35AM +0200, Hans de Goede wrote:
> >> Hi Bryan, Sakari,
> >>
> >> On 1-Apr-25 2:34 AM, Bryan O'Donoghue wrote:
> >>> On 27/03/2025 07:36, Sakari Ailus wrote:
> >>>>> +static u64 to_pixel_rate(u32 f_index)
> >>>>> +{
> >>>>> +    u64 pixel_rate = link_freq_menu_items[f_index] * 2 * OV02E10_DATA_LANES;
> >>>>> +
> >>>>> +    do_div(pixel_rate, OV02E10_RGB_DEPTH);
> >>>> The pixel rate control is for the pixel rate on the pixel array, not on the
> >>>> CSI-2 interface. Without binning or sub-sampling these may the same still,
> >>>> but this only works in special cases really.
> >>>
> >>> Hmm computer says no, I don't think I have understood this comment..
> >>>
> >>> Looking at other drivers, I'd say the above pattern is pretty common - taking ov8856 as an example that's pretty much equivalent logic to the above, ov08x40 does something similar.
> >>>
> >>> =>
> >>>
> >>> pixel_rate == link_freq * 2 * #oflanes / RGB_DEPTH
> >>>            => 360MHz * 2 * 2 / 10
> >>>            => 360000000 * 2 * 2 / 10
> >>>            => 144000000
> >>>
> >>> If I'm understanding you though you mean the pixel rate for the control V4L2_CID_PIXEL_RATE expressed here should be the resolution * the FPS / bits_per_pixel
> >>
> >> I have to agree with Bryan here that the pixelrate typically is const
> >> and directly derived from the link-frequency. Even the
> >> __v4l2_get_link_freq_ctrl() helper from drivers/media/v4l2-core/v4l2-common.c
> >> assumes this.
> > 
> > That is there to support old drivers that don't use the LINK_FREQ control.
> > All new ones do.
> > 
> >>
> >> binning / subsampling does not change anything wrt the pixelrate it
> >> just means that either the blanking becomes much bigger keeping
> >> vts / hts the same, or that the FPS becomes much higher.
> >>
> >> It is not like the sensor is sending an empty pixel on the CSI
> >> link every other pixel when binning, since there is no such
> >> thing as an empty pixel. So the sensor must go faster when doing
> >> horizontal binning to keep the CSI link filled effectively
> >> doubling the FPS, or requiring a much larger hblank after having
> >> taken only half the time sending pixels.
> > 
> > Please see
> > <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/drivers/camera-sensor.html#raw-camera-sensors>
> > and
> > <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/ext-ctrls-image-process.html#image-process-control-ids>.
> > 
> > As noted, this may be correct if the sensor doesn't use binning or
> > sub-sampling, but conceptually pixel rate on the pixel array and on the
> > CSI-2 bus are different. The PIXEL_RATE control is for the former albeit in
> > the past some drivers have presumably used it for the latter as well.
> 
> Ok, so here is what is written there:
> 
> "V4L2_CID_PIXEL_RATE (64-bit integer)
> 
>     Pixel sampling rate in the device’s pixel array. This control is read-only and its unit is pixels / second.
> 
>     Some devices use horizontal and vertical blanking to configure the frame rate. The frame rate can be calculated from the pixel rate, analogue crop rectangle as well as horizontal and vertical blanking. The pixel rate control may be present in a different sub-device than the blanking controls and the analogue crop rectangle configuration.
> 
>     The configuration of the frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz."
> 
> So when not bin-/skip-/averag-ing this matches with my understanding
> which is that:
> 
> FPS = pixelrate / ((mode.width + hblank) * (mode.height + vblank))
> 
> and also pixelrate = link_freq * 2 * #oflanes / RGB_DEPTH.
> 
> Since the ov02e10 driver does not do bin-/skip-/avera-ging,
> this definitely is correct for the ov02e10, so I don't really
> think there is an issue with the ov02e10 driver here.
> 
> I've been assuming in drivers which do do binning like the ov2680
> that this also holds true when binning. But what I'm hearing
> you say here is that the reported pixelrate should change when
> binning?

The pixel rate could be different in binned modes and non-binned modes. On
some devices (and configurations) it could be the same, but primarily it
depends on what the driver does.

> 
> I see how the calculation of the FPS should change, reading
> the V4L2 API using mode.width / mode.height is wrong and instead
> the analog crop should be used which is e.g. (2 * mode.width) x
> (2 * mode.height) when doing binning and I can see how this is
> sensible because this way when just disabling/enabling binning
> without changing the blanks the fps does not change.
> 
> Except that we don't have a proper API to select binning and
> instead is done transparently by drivers like the ov2680 driver ...
> 
> And I believe that e.g. libcamera atm simply implements:
> 
> FPS = pixelrate / ((mode.width + hblank) * (mode.height + vblank))
> 
> and thus assumes that the driver updates the hblank / vblank
> values it reports by adding width/height to the reported value
> to compensate for binning but I might be mistaken there.

The line length and frame values appear to be miscalculated but there are
driver bugs, too, and right now not all drivers expose the information
anyway. Register list based drivers will continue to be poor in this
respect though. But see below.

> 
> This also begs the question what a driver with arbitrary mode
> support like the ov2680 driver should do on a set_fmt() call
> where fmt.width / height don't match (either 1:1 or 1:2)
> with the analog crop. Should the driver then update the analog
> crop? ATM for modes smaller then the current analog crop,
> the ov2680 code simply adds extra in driver cropping on top of
> the selection API crop, without updating the selection API crop.
> Since the driver clamps the max accepted fmt width/height to
> the crop updating the selection API crop would be troublesome
> and break non selection API users when they want to switch back
> to a higher resolution mode ...
> 
> I don't think we really have properly specified (as in written down)
> how all this is supposed to work, especially as soon as binning comes
> into play. I think discussing all possible corner cases and trying
> to hammer out how this all is supposed to fit together would be
> a good summit for the media maintainers summit in Nice ?

I've been working on more elaborate camera sensor control recently, it's
related to streams support. The current set is here
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>. That
does not yet provide a solution for changing the mode for mode-based
drivers though.

> 
> Eitherway I believe that the current code in ov02e10 is fine as
> is for now since the ov02e10 driver only supports a single fixed
> mode with no binning.

Yes, I think so as well.

> 
> BTW the last sentence of the control description clearly needs work,
> framerate indeed is in Hz, but framerate is indirectly controlled through
> setting the blanking times, which are in pixels not Hz, so that last
> sentence is confusing.

"This" refers to the PIXEL_RATE control, not to the blanking controls. The
sentence could be moved to a separate paragraph to make it clearer it's
unrelated. Adding a reference to further driver API documentation in
camera-sensor.rst could be useful, too.

-- 
Regards,

Sakari Ailus

Powered by blists - more mailing lists