Message-ID: <445aa6ea-2630-4fce-955a-e9df422fdf7d@oss.nxp.com>
Date: Wed, 3 Sep 2025 10:12:44 +0800
From: "Ming Qian(OSS)" <ming.qian@....nxp.com>
To: Nicolas Dufresne <nicolas@...fresne.ca>, mchehab@...nel.org,
 hverkuil-cisco@...all.nl
Cc: sebastian.fricke@...labora.com, shawnguo@...nel.org,
 s.hauer@...gutronix.de, kernel@...gutronix.de, festevam@...il.com,
 linux-imx@....com, xiahong.bao@....com, eagle.zhou@....com,
 imx@...ts.linux.dev, linux-media@...r.kernel.org,
 linux-kernel@...r.kernel.org, linux-arm-kernel@...ts.infradead.org
Subject: Re: [PATCH] media: amphion: Drop the sequence header after seek for
 VC1L

Hi Nicolas,

On 9/3/2025 1:36 AM, Nicolas Dufresne wrote:
> Hi,
> 
> just adding to my anwer,
> 
> Le mardi 02 septembre 2025 à 13:01 -0400, Nicolas Dufresne a écrit :
>> Le lundi 01 septembre 2025 à 17:41 +0800, Ming Qian(OSS) a écrit :
>>>
>>>
> 
> [...}
> 
>>>>
>>>> Nicolas
>>>
>>> I tested this with gstreaer, not FFMPEG,
>>> And I checked the gstreamer code in our repository, Then I found the
>>> following related code:
>>>
>>>     } else if (g_str_equal (mimetype, "video/x-wmv")) {
>>>       const gchar *format = gst_structure_get_string (structure, "format");
>>>       if (format) {
>>>         if (!g_ascii_strcasecmp (format, "WVC1"))
>>>           fourcc = V4L2_PIX_FMT_VC1_ANNEX_G;
>>>         else if (!g_ascii_strcasecmp (format, "WMV3"))
>>>           fourcc = V4L2_PIX_FMT_VC1_ANNEX_L;
>>>       }
>>>
>>> Basically it processes WMV3 into VC1_ANNEX_L, and WVC1 to VC1_ANNEX_G.
>>> I didn't found them in the upstream gstreamer repository.
>>> Now I'm not sure if it is correct userspace behavior.
>>
>> Its a little concerning, since we are in the largely untested territory.
>> Without
>> proper documentation and with all the downstream changes done to userspace, we
>> can easily endup with NXP considering this is the right mapping and let's say
>> Qualcomm or Samsung thinking differently. Since this is for upstream, we need
>> to
>> ensure this is concistant. Have you reached to other driver maintainers
>> already
>> to discuss and resolve the subject in a way it works for everyone ?
> 
> So I checked Samsung implementation and your interpretation seems to be the
> same. They MAP VC1_ANNEX_G to VC1 Advanced Profile Decoding in their
> driver. Venus drivers does not care and just map both to VC1.
> 
> If I quote here Wikipedia's Window Media Video page:
>     The Simple and Main profile levels in WMV 9 are compliant with the same
>     profile levels in the VC-1 specification.[13] The Advanced Profile in VC-1 is
>     implemented in a new WMV format called Windows Media Video 9 Advanced
>     Profile. It improves compression efficiency for interlaced content and is
>     made transport-independent, making it able to be encapsulated in an MPEG
>     transport stream or RTP packet format. The format is not compatible with
>     previous WMV 9 formats, however.[14]
> 
> 
> It matches well with the fact Annex G introduce start codes and inline sequence
> headers, since you absolutely need that to stream over MPEG TS. GStreamer uses
> video/x-wmv as a family, and format=WVC1 for the advanced profiles, and WMV3 for
> everything else it supports.
> 
> I think you should go ahead and upstream this mapping fix into GStreamer. V4L2
> documentation should perhaps mention "Advanced Profile" to help devs.
> 
> Though, this gives me the impression that codec_data can be inline for ANNEX G.
> 
> Nicolas
> 

I also checked the ffmpeg, it also treats VC1 as the advanced profiles,
and WMV3 for Simple and Main Profiles.

And below is from the VC-1's wiki page:

	WMV3
	The Simple and Main Profiles of VC-1 remained completely
	faithful to the existing WMV3 implementation, making WMV3
	bitstreams fully VC-1 compliant. The WMV3 codec was designed to
	primarily support progressive encoding for computer displays. An
	interlaced encoding mode was implemented, but quickly became
	deprecated when Microsoft started implementing WMV Advanced
	Profile. Whereas WMV3 progressive encoding was implemented using
	the YUV 4:2:0 color sampling scheme, the deprecated interlaced
	mode was implemented using the less common YUV 4:1:1 sampling
	scheme.

	The Windows Media Video 9 (WMV3) codec implements the Simple and
	Main modes of the VC-1 codec standard, providing high-quality
	video for streaming and downloading. "It provides support for a
	wide range of bit rates, from high-definition content at
	one-half to one-third the bit rate of MPEG-2, to low-bit-rate
	Internet video delivered over a dial-up modem. This codec also
	supports professional-quality downloadable video with two-pass
	and variable bit rate (VBR) encoding."[7]


	WVC1
	WVC1, also known as Windows Media Video 9 Advanced Profile,
	implements a more recent and fully compliant Advanced Profile of
	the VC-1 codec standard. It offers support for interlaced
	content and is transport independent. With the previous version
	of the Windows Media Video 9 Series codec, users could deliver
	progressive content at data rates as low as one-third that of
	the MPEG-2 codec and still get equivalent or comparable quality
	to MPEG-2[citation needed]. The Windows Media Video 9 Advanced
	Profile codec also offers this same improvement in encoding
	efficiency with interlaced contents[citation needed]. A decoder
	for WVC1 is included in Windows Media Player 11, which is
	bundled with Windows Vista and is available as a download for
	Windows XP. This implementation is supported in Microsoft
	Silverlight.

And we will prepare a gstreamer patch first, and also try to add a
description of the advanced profile to V4L2 Documentation.

Thank you again for your patience.

Regards,
Ming

>>
>>>
>>> And the reason of this issue is the below code in gstreamer, that the
>>> v4l2decoder may only send codec data after seek.
>>>
>>>       codec_data = self->input_state->codec_data;
>>>
>>>       /* We are running in byte-stream mode, so we don't know the
>>> headers, but
>>>        * we need to send something, otherwise the decoder will refuse to
>>>        * initialize.
>>>        */
>>>       if (codec_data) {
>>>         gst_buffer_ref (codec_data);
>>>       } else {
>>>         codec_data = gst_buffer_ref (frame->input_buffer);
>>>         processed = TRUE;
>>>       }
>>
>> That is truncating a bit too much of the context. The "processed" boolean is
>> set
>> when the codec data and frame is combined. In the case the codec data is only
>> to
>> be found in caps, it will queue the codec data and immediately queue the frame
>> data. This is perfectly valid with the way the stateful decoder specification
>> is
>> written.
>>
>> In practice, GStreamer stateful decoder is multi-threaded, so it will fill the
>> OUTPUT queue with following frames too. What you can do to make your driver
>> more
>> flexible is whenever you didn't find a frame in a buffer, merge this buffer
>> with
>> the next one, and do that until there is no more space in one buffer. This way
>> you don't copy all the time like ring buffers do, but you can survive abitrary
>> splits of byte-stream.
>>
>> Nicolas
>>
>>>
>>> Regards,
>>> Ming
>>>
>>>
>>>>
>>>>>    
>>>>>    	ret = vpu_malone_insert_scode_seq(scode,
>>>>> MALONE_CODEC_ID_VC1_SIMPLE, sizeof(rcv_seqhdr));
>>>>>    	if (ret < 0)

Powered by blists - more mailing lists