linux-kernel - Re: [PATCH] Export shmem_file_setup and shmem

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1217785943.23437.330.camel@koto.keithp.com>
Date:	Sun, 03 Aug 2008 10:52:23 -0700
From:	Keith Packard <keithp@...thp.com>
To:	John Stoffel <john@...ffel.org>
Cc:	keithp@...thp.com, Christoph Hellwig <hch@...radead.org>,
	Eric Anholt <eric@...olt.net>, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] Export shmem_file_setup and shmem_getpage for DRM-GEM

On Sun, 2008-08-03 at 08:49 -0400, John Stoffel wrote:

> Why?  If you need thousands of files, won't the overhead of managing
> them dynamically start to be a big load as well?  

We're not having any trouble with upwards of 40000 shmem file objects at
this point. Most of these are around 4 pages in size, although they
range up to around 16MB. A typical 3D game will use around 100MB or so
of this kind of data at one time.

> I assume (sorry for not looking at the code in depth) that you're
> trying to setup specific regions of memory with various attributes for
> the DRI/DRM/GEM/TTF access to Video cards?  

Not really; these are objects used in the rendering process, vertices,
command buffers, constant buffers, textures, programs, rendering
surfaces and various other state buffers.  Lots of these are write-once
(from the CPU) and read-many (from the GPU), like textures, programs and
vertices. Some of these are effectively streamed from the CPU to the
GPU.

Each sequence of rendering commands requires that a set of these objects
be pinned to physical memory and bound to the graphics translation table
within the GPU.

Oh, and I allocate enough that I need to make them pageable as well;
under memory pressure, I can pull objects back out of the GTT binding
table and unpin the pages, letting shmem write them to disk.

> Just seeing your statement that you wanted to add ioctls made me
> shudder and try to visuallize a better way to do this.  

I could use fds if the kernel supported applications needing many
thousand of them, and also some way to keep these FDs from polluting the
fd space used by select(2) (yeah, I know, don't use select).

> I realize you're trying to make the drivers generic so that the *BSD
> port is simple too, but... thousands of files (per X server?  per
> xclient) just seems like the wrong level.

I'm not worried about BSD; I want to build the right API for Linux at
this point.

And, yes, thousands of objects per 3D application. Each X pixmap will
end up in one of these objects as well, and so something like Firefox
will likely allocate several hundred. To keep the rendering engine busy,
we'll have a couple hundred command buffers allocated as well, of 16KB
apiece.

> Maybe you just need to open the *entire* card with one FD and then
> have your own VFS like abstraction in there, which doesn't require
> lots of filehandles?  

That's what I'm doing at present; the card is opened with a single fd
and then these shmem objects are allocated within an ioctl from there.
What I don't want to do is use a single linear address space for these
objects; it's just useless overhead.

> Dunno... just trying to figure out what you're try to mediate here
> with ioctl() and how it could be improved/simplified.

Suggestions welcome; I just need a few thousand multi-page allocations,
which the kernel deals with quite easily these days.

-- 
keith.packard@...el.com

Download attachment "signature.asc" of type "application/pgp-signature" (190 bytes)