netdev - Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20240813105822.GD24634@pendragon.ideasonboard.com>
Date: Tue, 13 Aug 2024 13:58:22 +0300
From: Laurent Pinchart <laurent.pinchart@...asonboard.com>
To: Tomasz Figa <tomasz.figa@...il.com>
Cc: Daniel Vetter <daniel.vetter@...ll.ch>,
	Sakari Ailus <sakari.ailus@....fi>,
	Ricardo Ribalda Delgado <ricardo.ribalda@...il.com>,
	Dan Williams <dan.j.williams@...el.com>,
	James Bottomley <James.Bottomley@...senpartnership.com>,
	ksummit@...ts.linux.dev, linux-cxl@...r.kernel.org,
	linux-rdma@...r.kernel.org, netdev@...r.kernel.org, jgg@...dia.com
Subject: Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

On Tue, Aug 13, 2024 at 07:33:59PM +0900, Tomasz Figa wrote:
> 2024年8月13日(火) 19:27 Laurent Pinchart <laurent.pinchart@...asonboard.com>:
> > On Tue, Aug 13, 2024 at 07:17:07PM +0900, Tomasz Figa wrote:
> > > 2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@...ll.ch>:
> > > >
> > > > On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@....fi> wrote:
> > > > > This is also very different from GPUs or accel devices that are built to be
> > > > > user-programmable. If I'd compare ISPs to different devices, then the
> > > > > closest match would probably be video codecs -- which also use V4L2.
> > > >
> > > > Really just aside, but I figured I should correct this. DRM supports
> > > > plenty of video codecs. They're all tied to gpus, but the real reason
> > > > really is that the hw has decent command submission support so that
> > > > running the entire codec in userspace except the basic memory and
> > > > batch execution and synchronization handling in the kernel is a
> > > > feasible design.
> > >
> > > FWIW, V4L2 also has an interface for video decoders that require
> > > bitstream processing in software, it's called the V4L2 Stateless
> > > Decoder interface [1]. It defines low level data structures that map
> > > directly to the particular codec specification, so the kernel
> > > interface is generic and the userspace doesn't need to have
> > > hardware-specific components. Hardware that consumes command buffers
> > > can be supported simply by having the kernel driver fill the command
> > > buffers as needed (as opposed to writing the registers directly).
> > > On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
> > > KMS interface for display controllers, rather than a command buffer
> > > passthrough, even though some display controllers actually are driven
> > > by command buffers.
> > > So arguably it's possible and practical to do both command
> > > buffer-based and fixed interfaces for both display controllers and
> > > video codecs. Do you happen to know some background behind why one or
> > > the other was chosen for each of them in DRM?
> > >
> > > For how it applies to ISPs, there are both types of ISPs out in the
> > > wild, some support command buffers, while some are programmed directly
> > > via registers.
> >
> > Could you provide examples of ISPs that use command buffers ? The
> > discussion has remained fairly vague so far, which I think hinders
> > progress.
> >
> > > For the former, I can see some loss of flexibility if
> > > the command buffers are hidden behind a fixed function API, because
> > > the userspace would only be able to do what the kernel driver supports
> > > internally, which could make some use case-specific optimizations very
> > > challenging if not impossible.
> >
> > Let's try to discuss this with specific examples.
> 
> AFAIK Intel IPU6 and newer, Qualcomm and MediaTek ISPs use command
> buffers natively.

At the hardware level, firmware level, or both ? Is there a way we can
get more information about the structure of the command buffer and how
it is handled by the ISP for any of those three platforms ?

> > > [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
> > >
> > > > And actually good, because your kernel wont ever blow
> > > > up trying to parse complex media formats because it just doesn't.

-- 
Regards,

Laurent Pinchart