linux-kernel - Re: kernel decompressor interface

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <4BB238D3.7050808@zytor.com>
Date:	Tue, 30 Mar 2010 10:45:55 -0700
From:	"H. Peter Anvin" <hpa@...or.com>
To:	Ferenc Wagner <wferi@...f.hu>
CC:	Alain Knaff <alain@...ff.lu>, linux-kernel@...r.kernel.org,
	Phillip Lougher <phillip@...gher.demon.co.uk>
Subject: Re: kernel decompressor interface

On 03/30/2010 09:10 AM, Ferenc Wagner wrote:
> Hi,
> 
> While working with SquashFS code recently, I got the impression that the
> current decompress_fn interface isn't best suited for general use: it
> rules out real scatter/gather operation, which -- one hopes -- is a
> general feature of stream decompressors.  For example, if one has to
> decompress data from a series of buffer_heads into a bunch of (cache)
> pages (typical operation in compressed file systems), the inflate
> interface in zlib.h provides the possibility of changing input and
> output buffer addresses, but decompress_fn does not, necessitating extra
> memory copying.  On the other hand, the latter is admittedly simpler.
> 
> However, I feel like their advantages could be combined by making the
> fill() and flush() callbacks able to change the buffer addresses, like
> int fill(void **, unsigned int) for example.  Even better, some context
> could also be passed around, so that they can easily carry private state.
> 
> A cursory look at the LZMA decompressor implementation suggests that
> this would be straightforward change (although I haven't done it).  Do
> you think that
> 1. it could be done for other decompressors (lzo, bunzip) as well, and
> 2. it should be done in any form (interface replacement or extension)?

I think the event-based interfaces in zlib.h are pretty good, myself,
and since they are so widely deployed they are also widely mimiced,
which makes it a good interface to rely on.  A simpler "one shot"
interface can then be implemented as a wrapper (since it just means:
create the context, point it at the two buffers, and then tell it to to
run to the end.)

LZO has some super-compact variants that don't support start and stop
operation, but even the "full featured" versions are small enough, that
I don't think it matters for something as large as the Linux kernel.  It
mattered to me when I was dealing with self-decompressing objects on the
order of a few tens of kilobytes.

	-hpa
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/