Message-ID: <CAJ+vNU1-gq2nB5oOcgo9zj5xR+qxmqyKAS-N4De1A_8SNRO_Aw@mail.gmail.com>
Date:   Fri, 3 Dec 2021 08:46:58 -0800
From:   Tim Harvey <tharvey@...eworks.com>
To:     Nicolas Dufresne <nicolas@...fresne.ca>
Cc:     Ezequiel Garcia <ezequiel@...guardiasur.com.ar>,
        Adam Ford <aford173@...il.com>,
        linux-media <linux-media@...r.kernel.org>,
        Schrempf Frieder <frieder.schrempf@...tron.de>,
        Marek Vasut <marek.vasut@...il.com>,
        Jagan Teki <jagan@...rulasolutions.com>,
        Adam Ford-BE <aford@...conembedded.com>,
        cstevens@...conembedded.com,
        Philipp Zabel <p.zabel@...gutronix.de>,
        Mauro Carvalho Chehab <mchehab@...nel.org>,
        Rob Herring <robh+dt@...nel.org>,
        Shawn Guo <shawnguo@...nel.org>,
        Sascha Hauer <s.hauer@...gutronix.de>,
        Pengutronix Kernel Team <kernel@...gutronix.de>,
        Fabio Estevam <festevam@...il.com>,
        NXP Linux Team <linux-imx@....com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Heiko Stuebner <heiko@...ech.de>,
        Lucas Stach <l.stach@...gutronix.de>,
        Joakim Zhang <qiangqing.zhang@....com>,
        Alice Guo <alice.guo@....com>, Peng Fan <peng.fan@....com>,
        "open list:HANTRO VPU CODEC DRIVER" 
        <linux-rockchip@...ts.infradead.org>,
        "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS" 
        <devicetree@...r.kernel.org>,
        "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" 
        <linux-arm-kernel@...ts.infradead.org>,
        open list <linux-kernel@...r.kernel.org>,
        "open list:STAGING SUBSYSTEM" <linux-staging@...ts.linux.dev>
Subject: Re: [RFC 0/5] arm64: imx8mm: Enable Hantro VPUs

On Thu, Dec 2, 2021 at 8:34 PM Nicolas Dufresne <nicolas@...fresne.ca> wrote:
>
> Le mardi 30 novembre 2021 à 11:28 -0800, Tim Harvey a écrit :
> > On Tue, Nov 30, 2021 at 6:00 AM Ezequiel Garcia
> > <ezequiel@...guardiasur.com.ar> wrote:
> > >
> > > Hi Tim,
> > >
> > > On Mon, 29 Nov 2021 at 16:36, Tim Harvey <tharvey@...eworks.com> wrote:
> > > >
> > > > On Mon, Nov 29, 2021 at 10:59 AM Adam Ford <aford173@...il.com> wrote:
> > > ..
> > > > >
> > > >
> > > > Adam,
> > > >
> > > > What deps did you install in order to get v4l2codecs building? I
> > > > installed libgudev-1.0-dev based on Nicolas' suggestion and rebuilt
> > > > (not sure if I needed to re-configure somehow) but there is still
> > > > nothing in build/subprojects/gst-plugins-bad/sys/v4l2codecs/. A 'meson
> > > > configure' tells me that v4l2codecs is set to 'auto' but I'm not sure
> > > > how to find out what dependencies are needed or what may be missing.
> > > >
> > >
> > > At least in my case (Centps-derivative), this is what I've done:
> > >
> > > ...
> > > gst-plugins-bad| Run-time dependency gudev-1.0 found: NO (tried
> > > pkgconfig and cmake)
> > >
> > > Installed gudev ... and then:
> > >
> > > ...
> > > gst-plugins-bad| Dependency gudev-1.0 found: YES 232 (cached)
> > > ...
> > > gst-plugins-bad 1.19.3.1
> > >
> > >     Plugins               : accurip, adpcmdec, adpcmenc, aiff, asfmux,
> > > audiobuffersplit, audiofxbad, audiomixmatrix, audiolatency,
> > > audiovisualizers, autoconvert, bayer,
> > >                             camerabin, codecalpha, coloreffects,
> > > debugutilsbad, dvbsubenc, dvbsuboverlay, dvdspu, faceoverlay,
> > > festival, fieldanalysis, freeverb, frei0r,
> > >                             gaudieffects, gdp, geometrictransform,
> > > id3tag, inter, interlace, ivfparse, ivtc, jp2kdecimator, jpegformat,
> > > rfbsrc, midi, mpegpsdemux,
> > >                             mpegpsmux, mpegtsdemux, mpegtsmux, mxf,
> > > netsim, rtponvif, pcapparse, pnm, proxy, legacyrawparse,
> > > removesilence, rist, rtmp2, rtpmanagerbad,
> > >                             sdpelem, segmentclip, siren, smooth,
> > > speed, subenc, switchbin, timecode, transcode, videofiltersbad,
> > > videoframe_audiolevel, videoparsersbad,
> > >                             videosignal, vmnc, y4mdec, decklink, dvb,
> > > fbdevsink, ipcpipeline, nvcodec, shm, v4l2codecs, hls, sctp
> > >
> > > GStreamer current master build fails. It's a known issue which will be
> > > fixed today:
> > >
> > > [...]
> > > [8/9] Compiling C object
> > > subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > > FAILED: subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > > cc -Isubprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p
> > > -Isubprojects/gst-plugins-bad/sys/v4l2codecs
> > > -I../subprojects/gst-plugins-bad/sys/v4l2codecs
> > > -Isubprojects/gst-plugins-bad -I../subprojects/gst-plugins-bad
> > > -Isubprojects/gstreamer/libs -I../subprojects/gstreamer/libs
> > > -Isubprojects/gstreamer -I../subprojects/gstreamer
> > > -Isubprojects/gst-plugins-bad/gst-libs
> > > -I../subprojects/gst-plugins-bad/gst-libs
> > > -Isubprojects/gst-plugins-base/gst-libs
> > > -I../subprojects/gst-plugins-base/gst-libs -Isubprojects/orc
> > > -I../subprojects/orc -Isubprojects/gstreamer/gst
> > > -Isubprojects/gst-plugins-base/gst-libs/gst/video
> > > -Isubprojects/gst-plugins-base/gst-libs/gst/pbutils
> > > -Isubprojects/gst-plugins-base/gst-libs/gst/audio
> > > -Isubprojects/gst-plugins-base/gst-libs/gst/tag
> > > -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include
> > > -I/usr/include/gudev-1.0 -fdiagnostics-color=always
> > > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O2 -g -fvisibility=hidden
> > > -fno-strict-aliasing -DG_DISABLE_DEPRECATED -Wmissing-prototypes
> > > -Wdeclaration-after-statement -Wold-style-definition
> > > -Wmissing-declarations -Wredundant-decls -Wwrite-strings -Wformat
> > > -Wformat-security -Winit-self -Wmissing-include-dirs -Waddress
> > > -Wno-multichar -Wvla -Wpointer-arith -fPIC -pthread -DHAVE_CONFIG_H
> > > -MD -MQ subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > > -MF subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o.d
> > > -o subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > > -c ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:92:3:
> > > error: unknown type name ‘grefcount’
> > >    grefcount ref_count;
> > >    ^~~~~~~~~
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > > function ‘gst_v4l2_codec_vp9_dec_picture_data_new’:
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:106:3:
> > > warning: implicit declaration of function ‘g_ref_count_init’; did you
> > > mean ‘g_cond_init’? [-Wimplicit-function-declaration]
> > >    g_ref_count_init (&pic_data->ref_count);
> > >    ^~~~~~~~~~~~~~~~
> > >    g_cond_init
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > > function ‘gst_v4l2_codec_vp9_dec_picture_data_ref’:
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:118:3:
> > > warning: implicit declaration of function ‘g_ref_count_inc’; did you
> > > mean ‘g_strv_contains’? [-Wimplicit-function-declaration]
> > >    g_ref_count_inc (&data->ref_count);
> > >    ^~~~~~~~~~~~~~~
> > >    g_strv_contains
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > > function ‘gst_v4l2_codec_vp9_dec_picture_data_unref’:
> > > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:125:7:
> > > warning: implicit declaration of function ‘g_ref_count_dec’
> > > [-Wimplicit-function-declaration]
> > >    if (g_ref_count_dec (&data->ref_count)) {
> > >        ^~~~~~~~~~~~~~~
> > > ninja: build stopped: subcommand failed.
> > >
> > > Hope this helps get you started!
> > > Ezequiel
> >
> > Ezequiel and Nicolas,
> >
> > Thanks - I did manage to get gstreamer 1.19.3 built successfully with
> > v4l2codecs finally by getting the correct dependencies. I've attempted
> > to software encode from another system and decode/display on the IMX8M
> > Mini but thus far have not been successful.
> >
> > I see that v4l2codecs plugin v4l2slh264dec/v4l2slmpeg2dec/v4l2slvp8dec
> > and these all can output video/x-raw NV12/YUY2 which kmssink should
> > accept so I'm attempting the following :
> >
> > # vp8 encode from x86
> > gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 !
> > vp8enc ! rtpvp8pay ! udpsink host=172.24.33.15 port=9001
> > # vp8 decode on imx8mm@....24.33.15 which has a 800x480 display
> > [gst-main] root@...al-venice:~/gstreamer/build# gst-launch-1.0 -v
> > udpsrc port=9001 caps = "application/x-rtp, media=(string)video,
> > clock-rate=(int)90000, encoding-name=(string)VP8, payload=(int)96,
> > ssrc=(uint)2745262155, timestamp-offset=(uint)2515032683,
> > seqnum-offset=(uint)19579, a-framerate=(string)30" ! rtpvp8depay !
> > v4l2slvp8dec ! kmssink
> > Setting pipeline to PAUSED ...
> > Pipeline is live and does not need PREROLL ...
> > /GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 800
> > /GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 480
> > Pipeline is PREROLLED ...
> > Setting pipeline to PLAYING ...
> > /GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps =
> > application/x-rtp, media=(string)video, clock-rate=(int)90000,
> > encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)2745262155,
> > timestamp-offset=(uint)2515032683, seqnum-offset=(uint)19579,
> > a-framerate=(string)30
> > New clock: GstSystemClock
> > /GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:sink: caps =
> > application/x-rtp, media=(string)video, clock-rate=(int)90000,
> > encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)2745262155,
> > timestamp-offset=(uint)2515032683, seqnum-offset=(uint)19579,
> > a-framerate=(string)30
> > /GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:src: caps =
> > video/x-vp8, framerate=(fraction)0/1, height=(int)480, width=(int)800,
> > profile=(string)0
> > ERROR: from element /GstPipeline:pipeline0/GstUDPSrc:udpsrc0: Internal
> > data stream error.
> > Additional debug info:
> > ../subprojects/gstreamer/libs/gst/base/gstbasesrc.c(3127):
> > gst_base_src_loop (): /GstPipeline:pipeline0/GstUDPSrc:udpsrc0:
> > streaming stopped, reason not-negotiated (-4)
> > Execution ended after 0:00:02.076839644
> > Setting pipeline to NULL ...
> > Freeing pipeline ...
> >
> > I'm getting the same thing when trying to use h264.
> >
> > I've never quite been able to grasp how to debug GStreamer's
> > negotiation issues. If I end with fakesink it appears to decode so it
> > must be the v4l2slvp8dec to kmssink. I tried forcing the pixel format
> > using 'v4l2slvp8dec ! "video/x-raw,format=(string)NV12" ! kmssink' but
> > I still get the negotiation error.
> >
> > What interrupts should I be seeing in /proc/interrupts? I don't see
> > anything vpu/hantro related there.
> >
> > I also want to make sure I have a basic understanding of the vpu
> > drivers and usersapce on the IMX8M Mini. The IMX6Q/DL that I'm more
> > familiar with has a vpu that is supported by the GStreamer video4linux
> > plugin which shows the following (on GStreamer 1.16.2):
> >   v4l2jpegenc: V4L2 JPEG Encoder
> >   v4l2jpegdec: V4L2 JPEG Decoder
> >   v4l2h264enc: V4L2 H.264 Encoder
> >   v4l2mpeg4enc: V4L2 MPEG4 Encoder
> >   v4l2mpeg4dec: V4L2 MPEG4 Decoder
> >   v4l2mpeg2dec: V4L2 MPEG2 Decoder
> >   v4l2h264dec: V4L2 H264 Decoder
> > The IMX6Q/DL also has an IPU that has an M2M driver that provides the
> > following for scaling/colorspace conversion:
> >   v4l2convert: V4L2 Video Converter
> >
> > I believe what I'm reading is that the IMX8M Mini Hantro codecs are
> > 'stateful' where more software is required to drive them and is
>
> 'stateless'. the rest is right.
>
> > supported by the newer v4l2codecs plugin. I haven't been able to
> > understand what kernel version/requirements the v4l2codecs plugin
> > users/requires.
>
> After H264 debacle with 5.9, 5.10 and 5.11 API break and GStreamer not getting
> enough release to support all of these we started merging support for CODECs
> only when the stable uAPI land. I made an exception for VP9 as it is already
> applied in the media tree and didn't want to miss 1.20 release.
>
> So to answer you question, it depends on when the CODEC uAPI landed.
>

Ok, thanks for the explanation.

> >
> > I'm also trying to understand how we can get scaling/colorspace
> > conversion on the IMX8M Mini. The IMX8M lacks an IPU... is there some
> > way to utilize scaling/colorspace conversion from the 2D GPU bound to
> > the etnaviv driver?
>
> The concept of the mini, is that you would be using th encoder for anything that
> isn't going to the display. So they only integrated the Hantro PP on the
> encoder. Unfortunately, you'll have to be patient for mainline stateless encoder
> support, we barely scratch the surface of this subject, but its being worked on.

After some searching for Hantro PP I see that the IMXMQ (IMX8M
Dual/QuadLite/Quad) mentions Hantro PP. From the IMX8MDQLQRM section
14.1.2.1 Decoder Features:
<quote>
Video post-processing features
 - Frame rotation 90 degrees left/right
 - Frame mirroring horizontally/vertically
 - Frame cropping
 - Frame conversion from YCbCr formats to 16-bit or 32-bit RGB formats
 - Frame scaling with maximum up-scaling factor of 3
 - Two rectangular or alpha blending masks for output frame

 The post-processing features can be used in pipeline with the
decoder. The postprocessing features can also be used as stand-alone,
without performing any decoding
</quote>

The above is under the VPU_G1 section and the same is not mentioned
for VPU_G2 and the IMX8MQ doesn't have encode support. Where do you
see that the IMX8MM has the Hantro PP on the H1? I know the TRM's lack
a lot of info so perhaps you know more about the internals than what
the TRM states.

I also found on a forum
(https://community.nxp.com/t5/i-MX-Processors/imx8mq-Hantro-G1-scaling/m-p/1285343)
that NXP's BSP doesn't use the Hantro for scaling (and likely not csc
either) and they use the GPU instead. I'm still unclear if/how you
could tap into the 2D GPU to use its scaling/conversion if it's bound
to the etnaviv driver.

> Unlike the IMX8MQ, you don't have the option to output YUYV (packed yuv 4:2:2)
> which would satisfy the 2D GPU support hoold to the DMABuf import path.
>
> When the display driver gets ready and upstream (it's been only 2-3 years now),
> you'll get linear NV12 support along with G2 compression support (this one is
> not supported by the GPU, so it will be tricky to expose in userland). I don't
> think the display support 4L4 tiles, but you GPU most likely can do with the
> right shared and texelFetch() or vulkan equivalent if that exist on that target.

Do you mean the Samsung Exynos DRM driver (which doesn't yet have
support for IMX8MM) or drivers/gpu/drm/mxsfb?

I'm currently using a pretty old patchset that adds IMX8MM support to
the drm/exynos driver. I'm way out of my realm when talking about
GPU/VPU and display drivers.

Best regards,

Tim

Powered by blists - more mailing lists