linux-kernel - RE: [RFC 00/10] implement alternative and much simpler id allocator

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BY2PR21MB0036CE87953D422A5AA06498CB9C0@BY2PR21MB0036.namprd21.prod.outlook.com>
Date:   Fri, 16 Dec 2016 21:09:07 +0000
From:   Matthew Wilcox <mawilcox@...rosoft.com>
To:     Rasmus Villemoes <linux@...musvillemoes.dk>
CC:     Tejun Heo <tj@...nel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Lai Jiangshan <jiangshanlai@...il.com>,
        "Jens Axboe" <axboe@...nel.dk>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
        "Andrew Morton" <akpm@...ux-foundation.org>
Subject: RE: [RFC 00/10] implement alternative and much simpler id allocator

From: Rasmus Villemoes [mailto:linux@...musvillemoes.dk]
> On Fri, Dec 16 2016, Matthew Wilcox <mawilcox@...rosoft.com> wrote:
> > Thanks for your work on this; you've really put some effort into
> > proving your work has value.  My motivation was purely aesthetic, but
> > you've got some genuine savings here (admittedly it's about a quarter
> > of a cent's worth of memory with DRAM selling for $10/GB).
> > Nevertheless, that adds up over a billion devices, and there are still
> > people trying to fit Linux into 4MB embedded devices.
> >
> 
> Yeah, my main motivation was embedded devices which don't have the
> luxury of measuring their RAM in GB. E.g., it's crazy that the
> watchdog_ida effectively use more memory than the .text of the watchdog
> subsystem, and similarly for the kthread workers, etc., etc.. I didn't
> mean for my patches to go in as is, more to provoke some discussion. I
> wasn't aware of your reimplementation, but it seems that may make the
> problem go away.

It certainly shrinks the problem down to a size where it may not be worth introducing another implementation.

> > On a 64-bit machine, your tIDA root is 24 bytes; my new IDA root is 16
> > bytes.  If you allocate only one entry, you'll allocate 8 bytes.
> > Thanks to the slab allocator, that gets rounded up to 32 bytes.  I
> > allocate the full 128 byte leaf, but I store the pointer to it in the
> > root (unlike the IDR, the radix tree doesn't need to allocate a layer
> > for a single entry).  So tIDA wins on memory consumption between 1 and
> > 511 IDs, and newIDA is slightly ahead between 512 and 1023 IDs.
> 
> This sounds good. I think there may still be a lot of users that never
> allocate more than a handful of IDAs, making a 128 byte allocation still
> somewhat excessive. One thing I considered was (exactly as it's done for
> file descriptor tables) to embed a single word in the struct ida and
> use that initially; I haven't looked closely at newIDA, so I don't know
> how easy that would be or if its worth the complexity.

Heh, I was thinking about that too.  The radix tree supports "exceptional entries" which have the bottom bit set.  On a 64-bit machine, we could use 62 of the bits in the radix tree root to store the ID bitmap.  I'm a little wary of the potential complexity, but we should try it out.

Did you come up with any fun tests that could be added to the test-suite?  It feels a little slender right now.