Message-ID: <a25fd4e7aca4f49e0793698915dbb5fe99bbe8f2.camel@ndufresne.ca>
Date: Fri, 19 Dec 2025 10:18:34 -0500
From: Nicolas Dufresne <nicolas@...fresne.ca>
To: Hans Verkuil <hverkuil+cisco@...nel.org>, Hirokazu Honda
	 <hiroh@...omium.org>
Cc: Dmitry Osipenko <dmitry.osipenko@...labora.com>, Mauro Carvalho Chehab	
 <mchehab@...nel.org>, Tomasz Figa <tfiga@...omium.org>, Benjamin Gaignard	
 <benjamin.gaignard@...labora.com>, Daniel Almeida
 <daniel.almeida@...labora.com>, 	linux-media@...r.kernel.org,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH v1] media: videobuf2: Allow applications customize data
 offsets of capture buffers

Hi Hans,

Le mercredi 17 décembre 2025 à 11:02 +0100, Hans Verkuil a écrit :
> > For me, the most central issue in V4L2 is that the memory allocation/importation
> > is bound to the operation queues. That brings all sort of issues such 
> > 
> > - We can't queue twice the same frame
> > - We can't mix external buffer with device allocated buffer
> > - All buffers must have the exact same stride
> 
> The three limitations above are all technically possible to implement with the
> current vb2 framework/streaming uAPI, it's just that nobody was ever motivated
> enough to add support for it.

I don't see how technically it is possible without new uAPI to support
heterogeneous strides, nor queuing twice the same frame while running in MMAP,
or mix device memory and externally imported memory. Please feel free to
enlighten me if you have some spare time.

Perhaps worth to mention that this is about doing this without creating glitches
or jumps cause by expensive drain and streamoff/on sequences.

> 
> > - Application is responsible for caching which memory goes to which v4l2_buffer
> 
> True, but is this really a big deal?

Maybe a lesser deal, but its extra complexity for both sides. The current bug
being that if you use import mode only to workaround "queuing twice" issue, you
will endup with two mapping (which in some stateful codec firmware is not
allowed by firmwares). So on top of the lookup userspace is doing to match
buffer ids to their memory (have to cache pointers and fd and all), the driver
(or vb2) should also implement caching. My proposal imply solving that dual
mapping issue for both current and future stream mode.

In userspace, there is also cases, where the video buffers comes from other
process, and you don't really know if two FD values points to the same dmabuf.
This is the kind of scenarios the DRM subsystem had to deal in compositors, in
our case that would be something such as pipewire. This caching is either micro
optimization or simply to support firmware limitation, but a guarantee to have 1
memory object for one chunk is in my opinion achievable and allow reducing
complexity.

> 
> > - Attempting to import a buffer requires a free spot in the queue
> 
> True.
> 
> > 
> > This adds on top of the v4l2_buffer structure limitation we have been targeting
> > so far. With the growth of modern standard API (think Vulkan Video notably), it
> > becomes apparent that the model is too inflexible. This inability to separate
> > memory allocation and importation from operations creates a lot of complexity in
> > user-space, leading to complicated bugs.
> > 
> > I've been quite about it, since until now I didn't have a solution in mind, but
> > I recently come with some ideas. I'll will try develop these ideas, at least in
> > prose for now and come up with an RFC, hopefully somewhere beginning of January
> > 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it
> > will be a starter to go go beyond just fixing what we see. In fact, this next
> > step is for me doing to be quite decisive if I continue doing codecs in V4L2 or
> > not in the long run. But I'm sure this is not just about video codecs.
> 
> I'm looking forward to your RFC!
> 
> What is important for me is that whatever we come up with, it is something that
> existing drivers can easily support. A new streaming/buffer allocation uAPI that
> can only be supported by new drivers will need very, very good reasons for it to
> be accepted.
> 
> The nice thing about the v4l2_buffer_ext proposal was that it can easily be
> supported by all drivers.
> 
> From a technical perspective both struct v4l2_buffer and struct v4l2_format have
> reached end-of-life: they are full of historical cruft, they are inefficient, the
> 32-bit/time32 compatibility code is awful and hard to maintain.

I want to get a lot further in the proposal writing before going into extended
debate, but there is absolutely nothing set in stone. I already mention that my
proposal it to create traction, and I will have no problem moving direction base
on feedback.

I can reassure you that part of the plan is to be able to implement that for all
drivers. In fact, I think its a good requirement for any proposal here. My
proposal will not replace VB2 or the format. Perhaps few bits of the format will
be replicated somewhere else, but not all of it. Perhaps we'll get into breaking
down the gigantic "format" structure into manageable part in the future, but
this aspect of the _ext proposal, with DRM formats and modifiers is orthogonal
to what I have in mind.

Also, the vast majority of what I want to proposal can be implemented inside
vb2, without changing any drivers or userspace. But the full gain, I believe,
will requires a new v4l2_memory type.

Feel free to think of your own ideas toward these goals, our best bet for the
future is if we can combine best ideas together.

regards,
Nicolas

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)

Powered by blists - more mailing lists