lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Mon, 28 Jul 2014 06:10:18 -0400
From:	Austin S Hemmelgarn <>
To:	Nick Krause <>
CC:	"" <>,
Subject: Re: Multi Core Support for compression in compression.c

On 07/27/2014 11:21 PM, Nick Krause wrote:
> On Sun, Jul 27, 2014 at 10:56 PM, Austin S Hemmelgarn
> <> wrote:
>> On 07/27/2014 04:47 PM, Nick Krause wrote:
>>> This may be a bad idea , but compression in brtfs seems to be only
>>> using one core to compress.
>>> Depending on the CPU used and the amount of cores in the CPU we can
>>> make this much faster
>>> with multiple cores. This seems bad by my reading at least I would
>>> recommend for writing compression
>>> we write a function to use a certain amount of cores based on the load
>>> of the system's CPU not using
>>> more then 75% of the system's CPU resources as my system when idle has
>>> never needed more
>>> then one core of my i5 2500k to run when with interrupts for opening
>>> eclipse are running. For reading
>>> compression on good core seems fine to me as testing other compression
>>> software for reads , it's
>>> way less CPU intensive.
>>> Cheers Nick
>> We would probably get a bigger benefit from taking an approach like
>> SquashFS has recently added, that is, allowing multi-threaded
>> decompression fro reads, and decompressing directly into the pagecache.
>>  Such an approach would likely make zlib compression much more scalable
>> on large systems.
> Austin,
> That seems better then my idea as you seem to be more up to date on
> brtfs devolopment.
> If you and the other developers of brtfs are interested in adding this
> as a feature please let
> me known as I would like to help improve brtfs as the file system as
> an idea is great just
> seems like it needs a lot of work :).
> Nick
I wouldn't say that I am a BTRFS developer (power user maybe?), but I
would definitely say that parallelizing compression on writes would be a
good idea too (especially for things like lz4, which IIRC is either in
3.16 or in the queue for 3.17).  Both options would be a lot of work,
but almost any performance optimization would.  I would almost say that
it would provide a bigger performance improvement to get BTRFS to
intelligently stripe reads and writes (at the moment, any given worker
thread only dispatches one write or read to a single device at a time,
and any given write() or read() syscall gets handled by only one worker).

Download attachment "smime.p7s" of type "application/pkcs7-signature" (2967 bytes)

Powered by blists - more mailing lists