[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <4afabd718f632965314794946d4b31cec739a3d6.camel@collabora.com>
Date: Wed, 26 Jul 2023 16:02:49 -0400
From: Nicolas Dufresne <nicolas.dufresne@...labora.com>
To: Paul Kocialkowski <paul.kocialkowski@...tlin.com>
Cc: Michael Grzeschik <mgr@...gutronix.de>,
linux-kernel@...r.kernel.org, linux-media@...r.kernel.org,
Hans Verkuil <hverkuil@...all.nl>,
Sakari Ailus <sakari.ailus@....fi>,
Andrzej Pietrasiewicz <andrzej.p@...labora.com>,
Michael Tretter <m.tretter@...gutronix.de>,
Jernej Škrabec <jernej.skrabec@...il.com>,
Chen-Yu Tsai <wens@...e.org>,
Samuel Holland <samuel@...lland.org>,
Thomas Petazzoni <thomas.petazzoni@...tlin.com>
Subject: Re: Stateless Encoding uAPI Discussion and Proposal
Le mardi 25 juillet 2023 à 11:09 +0200, Paul Kocialkowski a écrit :
> Hi Nicolas,
>
> On Mon 24 Jul 23, 10:03, Nicolas Dufresne wrote:
> > Le vendredi 21 juillet 2023 à 20:19 +0200, Michael Grzeschik a écrit :
> > > > As a result, we cannot expect that any given encoder is able to produce frames
> > > > for any set of headers. Reporting related constraints and limitations (beyond
> > > > profile/level) seems quite difficult and error-prone.
> > > >
> > > > So it seems that keeping header generation in-kernel only (close to where the
> > > > hardware is actually configured) is the safest approach.
> > >
> > > For the case with the rkvenc, the headers are also not created by the
> > > kernel driver. Instead we use the gst_h264_bit_writer_sps/pps functions
> > > that are part of the codecparsers module.
> >
> > One level of granularity we can add is split headers (like SPS/PPS) and
> > slice/frame headers.
>
> Do you mean asking the driver to return a buffer with only SPS/PPS and then
> return another buffer with the slice/frame header?
>
> Looks like there's already a control for it: V4L2_CID_MPEG_VIDEO_HEADER_MODE
> which takes either
> - V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE: looks like what you're suggesting
> - V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME: usual case
>
> So that could certainly be supported to easily allow userspace to stuff extra
> NALUs in-between.
Good point, indeed.
>
> > It remains that in some cases, like HEVC, when the slice
> > header is byte aligned, it can be nice to be able to handle it at application
> > side in order to avoid limiting SVC support (and other creative features) by our
> > API/abstraction limitations.
>
> Do you see something in the headers that we expect the kernel to generate that
> would need specific changes to support features like SVC?
Getting the kernel to set the layer IDs, unless we have a full SVC configuration
would just be extra indirections. That being said, if we mention HEVC, these IDs
can be modified in-place as they use a fixed number of bytes. If you can split
the headers appart, generating per layer headers in application makes a lot of
sense.
Traditionally, slice headers are made by stateless accelerators, but not the
SPS/PPS and friend.
>
> From what I can see there's a svc_extension_flag that's only set for specific
> NALUs (prefix_nal_unit/lice_layer_extension) so these could be inserted by
> userspace.
>
> Also I'm not very knowledgeable about SVC so it's not very clear to me if it's
> possible to take an encoder that doesn't support SVC and turn the resulting
> stream into something SVC-ready by adding extra NAL units or if the encoder
> should be a lot more involved.
You can use any encoders to create a temporal SVC. Its only about the
referencing pattern, made so you can reduce the framerate (dividing by 2
usually).
For spatial layer, the encoders need scaling capabilities. I'm not totally sure
how multi-view work, but this is most likely just using left eye as reference
(not having an I frame ever for the second eye).
>
> Also do you know if we have stateful codecs supporting SVC?
We don't at the moment, they all produce headers with layer id hardcoded to 0 as
far as I'm aware. The general plan (if it had continued) might have been to
offer a memu based control, and drivers could offer from a list of preset SVC
pattern. Mimicking what browsers needs:
https://www.w3.org/TR/webrtc-svc/
>
> > I think a certain level of "per CODEC" reasoning is
> > also needed. Just like, I would not want to have to ask the kernel to generate
> > user data SEI and other in-band data.
>
> Yeah it looks like there is definitely a need for adding extra NALUs from
> userspace without passing that data to the kernel.
>
> Cheers,
>
> Paul
>
Powered by blists - more mailing lists