Message-ID: <6cf8d6db-c24e-461b-bd7d-a3dad25e4945@redhat.com>
Date: Tue, 1 Apr 2025 15:34:39 +0200
From: Hans de Goede <hdegoede@...hat.com>
To: Sakari Ailus <sakari.ailus@...ux.intel.com>
Cc: Bryan O'Donoghue <bryan.odonoghue@...aro.org>,
 Mauro Carvalho Chehab <mchehab@...nel.org>,
 Liam Girdwood <lgirdwood@...il.com>, Mark Brown <broonie@...nel.org>,
 Rob Herring <robh@...nel.org>, Krzysztof Kozlowski <krzk+dt@...nel.org>,
 Conor Dooley <conor+dt@...nel.org>, Bryan O'Donoghue <bod@...nel.org>,
 Hans de Goede <hansg@...nel.org>, Jingjing Xiong <jingjing.xiong@...el.com>,
 Hao Yao <hao.yao@...el.com>, Jim Lai <jim.lai@...el.com>,
 You-Sheng Yang <vicamo.yang@...onical.com>,
 Alan Stern <stern@...land.harvard.edu>, linux-kernel@...r.kernel.org,
 linux-media@...r.kernel.org, devicetree@...r.kernel.org
Subject: Re: [PATCH v2 2/2] media: i2c: ov02e10: add OV02E10 image sensor
 driver

Hi Sakari,

On 1-Apr-25 11:42 AM, Sakari Ailus wrote:
> Hi Hans, Bryan,
> 
> On Tue, Apr 01, 2025 at 10:12:35AM +0200, Hans de Goede wrote:
>> Hi Bryan, Sakari,
>>
>> On 1-Apr-25 2:34 AM, Bryan O'Donoghue wrote:
>>> On 27/03/2025 07:36, Sakari Ailus wrote:
>>>>> +static u64 to_pixel_rate(u32 f_index)
>>>>> +{
>>>>> +    u64 pixel_rate = link_freq_menu_items[f_index] * 2 * OV02E10_DATA_LANES;
>>>>> +
>>>>> +    do_div(pixel_rate, OV02E10_RGB_DEPTH);
>>>> The pixel rate control is for the pixel rate on the pixel array, not on the
>>>> CSI-2 interface. Without binning or sub-sampling these may the same still,
>>>> but this only works in special cases really.
>>>
>>> Hmm computer says no, I don't think I have understood this comment..
>>>
>>> Looking at other drivers, I'd say the above pattern is pretty common - taking ov8856 as an example that's pretty much equivalent logic to the above, ov08x40 does something similar.
>>>
>>> =>
>>>
>>> pixel_rate == link_freq * 2 * #oflanes / RGB_DEPTH
>>>            => 360MHz * 2 * 2 / 10
>>>            => 360000000 * 2 * 2 / 10
>>>            => 144000000
>>>
>>> If I'm understanding you though you mean the pixel rate for the control V4L2_CID_PIXEL_RATE expressed here should be the resolution * the FPS / bits_per_pixel
>>
>> I have to agree with Bryan here that the pixelrate typically is const
>> and directly derived from the link-frequency. Even the
>> __v4l2_get_link_freq_ctrl() helper from drivers/media/v4l2-core/v4l2-common.c
>> assumes this.
> 
> That is there to support old drivers that don't use the LINK_FREQ control.
> All new ones do.
> 
>>
>> binning / subsampling does not change anything wrt the pixelrate it
>> just means that either the blanking becomes much bigger keeping
>> vts / hts the same, or that the FPS becomes much higher.
>>
>> It is not like the sensor is sending an empty pixel on the CSI
>> link every other pixel when binning, since there is no such
>> thing as an empty pixel. So the sensor must go faster when doing
>> horizontal binning to keep the CSI link filled effectively
>> doubling the FPS, or requiring a much larger hblank after having
>> taken only half the time sending pixels.
> 
> Please see
> <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/drivers/camera-sensor.html#raw-camera-sensors>
> and
> <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/ext-ctrls-image-process.html#image-process-control-ids>.
> 
> As noted, this may be correct if the sensor doesn't use binning or
> sub-sampling, but conceptually pixel rate on the pixel array and on the
> CSI-2 bus are different. The PIXEL_RATE control is for the former albeit in
> the past some drivers have presumably used it for the latter as well.

Ok, so here is what is written there:

"V4L2_CID_PIXEL_RATE (64-bit integer)

    Pixel sampling rate in the device’s pixel array. This control is read-only and its unit is pixels / second.

    Some devices use horizontal and vertical blanking to configure the frame rate. The frame rate can be calculated from the pixel rate, analogue crop rectangle as well as horizontal and vertical blanking. The pixel rate control may be present in a different sub-device than the blanking controls and the analogue crop rectangle configuration.

    The configuration of the frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz."

So when not bin-/skip-/averag-ing this matches with my understanding
which is that:

FPS = pixelrate / ((mode.width + hblank) * (mode.height + vblank))

and also pixelrate = link_freq * 2 * #oflanes / RGB_DEPTH.

Since the ov02e10 driver does not do bin-/skip-/avera-ging,
this definitely is correct for the ov02e10, so I don't really
think there is an issue with the ov02e10 driver here.

I've been assuming in drivers which do do binning like the ov2680
that this also holds true when binning. But what I'm hearing
you say here is that the reported pixelrate should change when
binning?

I see how the calculation of the FPS should change, reading
the V4L2 API using mode.width / mode.height is wrong and instead
the analog crop should be used which is e.g. (2 * mode.width) x
(2 * mode.height) when doing binning and I can see how this is
sensible because this way when just disabling/enabling binning
without changing the blanks the fps does not change.

Except that we don't have a proper API to select binning and
instead is done transparently by drivers like the ov2680 driver ...

And I believe that e.g. libcamera atm simply implements:

FPS = pixelrate / ((mode.width + hblank) * (mode.height + vblank))

and thus assumes that the driver updates the hblank / vblank
values it reports by adding width/height to the reported value
to compensate for binning but I might be mistaken there.

This also begs the question what a driver with arbitrary mode
support like the ov2680 driver should do on a set_fmt() call
where fmt.width / height don't match (either 1:1 or 1:2)
with the analog crop. Should the driver then update the analog
crop? ATM for modes smaller then the current analog crop,
the ov2680 code simply adds extra in driver cropping on top of
the selection API crop, without updating the selection API crop.
Since the driver clamps the max accepted fmt width/height to
the crop updating the selection API crop would be troublesome
and break non selection API users when they want to switch back
to a higher resolution mode ...

I don't think we really have properly specified (as in written down)
how all this is supposed to work, especially as soon as binning comes
into play. I think discussing all possible corner cases and trying
to hammer out how this all is supposed to fit together would be
a good summit for the media maintainers summit in Nice ?

Eitherway I believe that the current code in ov02e10 is fine as
is for now since the ov02e10 driver only supports a single fixed
mode with no binning.

BTW the last sentence of the control description clearly needs work,
framerate indeed is in Hz, but framerate is indirectly controlled through
setting the blanking times, which are in pixels not Hz, so that last
sentence is confusing.

Regards,

Hans




> 
>>
>> (and the same applies to vts when vertical binning)
>>
>>> pixel_rate = wdith x height x fps / bpp
>>>            => 1928 * 1088 * 30 / 10
>>>            => 6292992
>>>
>>> i.e. the pixel rate not related to the CSI2 link frequency ?
>>
>> No the pixel-rate control includes vblank + hblank "pixels"
>> and is in pixels/sec so no dividing by bpp, iow it is:
>>
>> vts * hts * fps
>>
>> and this must match
>>
>> link_freq * 2 * #oflanes / RGB_DEPTH
> 

Powered by blists - more mailing lists