linux-kernel - Re: [RFC V3 0/2] arm64: imx8mm: Enable Hantro VPUs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 2 Dec 2021 14:54:36 -0800
From:   Tim Harvey <tharvey@...eworks.com>
To:     Adam Ford <aford173@...il.com>,
        Ezequiel Garcia <ezequiel@...guardiasur.com.ar>,
        Nicolas Dufresne <nicolas@...fresne.ca>
Cc:     Lucas Stach <l.stach@...gutronix.de>,
        linux-media <linux-media@...r.kernel.org>,
        Adam Ford-BE <aford@...conembedded.com>,
        Hans Verkuil <hverkuil-cisco@...all.nl>,
        Rob Herring <robh+dt@...nel.org>,
        Shawn Guo <shawnguo@...nel.org>,
        Sascha Hauer <s.hauer@...gutronix.de>,
        Pengutronix Kernel Team <kernel@...gutronix.de>,
        Fabio Estevam <festevam@...il.com>,
        NXP Linux Team <linux-imx@....com>,
        Philipp Zabel <p.zabel@...gutronix.de>,
        Mauro Carvalho Chehab <mchehab@...nel.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        devicetree <devicetree@...r.kernel.org>,
        arm-soc <linux-arm-kernel@...ts.infradead.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        "open list:HANTRO VPU CODEC DRIVER" 
        <linux-rockchip@...ts.infradead.org>,
        "open list:STAGING SUBSYSTEM" <linux-staging@...ts.linux.dev>
Subject: Re: [RFC V3 0/2] arm64: imx8mm: Enable Hantro VPUs

On Thu, Dec 2, 2021 at 1:00 PM Adam Ford <aford173@...il.com> wrote:
>
> On Thu, Dec 2, 2021 at 12:54 PM Tim Harvey <tharvey@...eworks.com> wrote:
> >
> > On Thu, Dec 2, 2021 at 4:29 AM Adam Ford <aford173@...il.com> wrote:
> > >
> > > On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@...il.com> wrote:
> > > >
> > > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear
> > > > to be related to the video decoders used on the i.MX8MQ, but because of
> > > > how the Mini handles the power domains, the VPU driver does not need to
> > > > handle all the functions, so a new compatible flag is required.
> > > >
> > > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c
> > > > This branch has support for VP9.
> > > >
> > > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory
> > > >
> > > > V3 of this series has several changes:
> > > >
> > > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9
> > > > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting
> > > > Fix missing reg-names from vpu_g2 node.
> > > > Apply patch [1] to manage the power domains powering down.
> > > > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/
> > > >
> > > > With the above, the following Fluster scores are produced:
> > > >
> > > > G1:
> > > > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0
> > > > Ran 90/135 tests successfully               in 74.406 secs
> > > >
> > > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0
> > > > Ran 55/61 tests successfully               in 8.080 secs
> > > >
> > > > G2:
> > > > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0
> > > > Ran 127/303 tests successfully               in 203.873 secs
> > > >
> > > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches.
> > > >
> > >
> > > I should note, that both interrupts appear to be triggering.
> > >
> > > # cat /proc/interrupts |grep codec
> > >  57:      13442          0          0          0     GICv3  39 Level
> > >   38300000.video-codec
> > >  58:       7815          0          0          0     GICv3  40 Level
> > >   38310000.video-codec
> > >
> >
> > Adam,
> >
> > On another thread you had let me know that you also removed the reset
> > from the pgc_vpumix power domain which does appear to resolve the
> > hang:
> >
> > diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> > b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> > index eb9dcd9d1a31..31710af544dc 100644
> > --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> > +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> > @@ -681,7 +681,6 @@
> >                                                 clocks = <&clk
> > IMX8MM_CLK_VPU_DEC_ROOT>;
> >                                                 assigned-clocks =
> > <&clk IMX8MM_CLK_VPU_BUS>;
> >                                                 assigned-clock-parents
> > = <&clk IMX8MM_SYS_PLL1_800M>;
> > -                                               resets = <&src
> > IMX8MQ_RESET_VPU_RESET>;
> >                                         };
> >
> >                                         pgc_vpu_g1: power-domain@7 {
> >
> > That would make such a patch have a 'Fixes commit d39d4bb15310
> > ("arm64: dts: imx8mm: add GPC node")' but of course that vpu domain
> > isn't active until your series so I'm not sure if we should send this
> > separate or squash it with "arm64: dts: imx8mm: Enable VPU-G1 and
> > VPU-G2". I'm also not clear if removing the reset requires some
> > further discussion with Lucas.
>
> Unless there is objection from Lucas, I'll likely make it the first
> patch in the series marking it with a fixes tag so it gets backported,
> then the rest of the series would be adding the bindings, update the
> driver and adding the G1 and G2 nodes.
>

Adam,

I've also gotten decode+display working for vp8/h264 using this series
and gstreamer-1.19.3 (although I have to use software colorspace
conversion)

# source: vp8 software encode on x86
gst-launch-1.0 -v videotestsrc ! vp8enc ! rtpvp8pay ! udpsink
host=172.24.33.15 port=9001
# sink: vp8 hardware decode on imx8mm
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp,
media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP8,
payload=(int)96, ssrc=(uint)3363940374,
timestamp-offset=(uint)3739685909, seqnum-offset=(uint)28161,
a-framerate=(string)30' ! rtpvp8depay ! v4l2slvp8dec ! videoconvert !
kmssink

# source: h264 software encode on x86
gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 !
x264enc ! rtph264pay ! udpsink host=172.24.33.15 port=9001
# sink: h264 hardware decode on imx8mm
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp,
media=(string)video, clock-rate=(int)90000,
encoding-name=(string)H264, packetization-mode=(string)1,
profile-level-id=(string)64001f,
sprop-parameter-sets=(string)"Z2QAH6zZQMg9sBagwCC0oAAAAwAgAAAHkeMGMsA\=\,aOvssiw\=",
payload=(int)96, ssrc=(uint)2753453329,
timestamp-offset=(uint)3593065282, seqnum-offset=(uint)12297,
a-framerate=(string)30' ! rtph264depay ! v4l2slh264dec ! videoconvert
! kmssink

# source: vp9 software encode on x86
gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 !
vp9enc ! rtpvp9pay ! udpsink host=172.24.33.15 port=9001
# sink: vp9 hardware decode on imx8mm
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp,
media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP9,
payload=(int)96, ssrc=(uint)2246741422,
timestamp-offset=(uint)3441735424, seqnum-offset=(uint)30250,
a-framerate=(string)30' ! rtpvp9depay ! v4l2slvp9dec ! fakesink
^^^ this fails with no-negotiated

Note I have to use videoconvert because v4l2slvp8dev src is
NV12/YUY2/NV12_32L32 and from testing only BGRx appears compatible
with kmssink (even though gst-inspect kmssink says it can sink
NV12/YUY2). With the 800x480 resolution of my display the CPU overhead
of software colorspcae conversion with videoconvert only about 9%

I haven't yet gotten vp9 decode+display working yet as 'rtpvp9depay !
v4l2slvp9dec ! fakesink' does not negotiate and it might be because my
vp9enc source is on an old gstreamer 1.16.

When you post the next series please add:
Tested-By: Tim Harvey <tharvey@...eworks.com>