lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20250129134226.GC5777@twin.jikos.cz>
Date: Wed, 29 Jan 2025 14:42:26 +0100
From: David Sterba <dsterba@...e.cz>
To: Daniel Vacek <neelx@...e.com>
Cc: Chris Mason <clm@...com>, Josef Bacik <josef@...icpanda.com>,
	David Sterba <dsterba@...e.com>, Nick Terrell <terrelln@...com>,
	linux-btrfs@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v3] btrfs/zstd: enable negative compression levels mount
 option

In next patch iteration if there's one please fix the subject so there's
"btrfs: zstd: ...".

On Tue, Jan 28, 2025 at 03:51:04PM +0100, Daniel Vacek wrote:
> This patch allows using the fast modes (negative compression levels) of zstd
> as a mount option.
> 
> As per the results, the compression ratio is lower:
> 
> for level in {-10..-1} 1 2 3; \
> do printf "level %3d\n" $level; \
>   mount -o compress=zstd:$level /dev/sdb /mnt/test/; \
>   grep sdb /proc/mounts; \
>   cp -r /usr/bin       /mnt/test/; sync; compsize /mnt/test/bin; \
>   cp -r /usr/share/doc /mnt/test/; sync; compsize /mnt/test/doc; \
>   cp    enwik9         /mnt/test/; sync; compsize /mnt/test/enwik9; \
>   cp    linux-6.13.tar /mnt/test/; sync; compsize /mnt/test/linux-6.13.tar; \
>   rm -r /mnt/test/{bin,doc,enwik9,linux-6.13.tar}; \
>   umount /mnt/test/; \
> done |& tee results | \
> awk '/^level/{print}/^TOTAL/{print$3"\t"$2"  |"}' | paste - - - - -
> 
> 		266M	bin  |	45M	doc  |	953M	wiki |	1.4G	source
> =============================+===============+===============+===============+
> level -10	171M	64%  |	28M	62%  |	631M	66%  |	512M	34%  |
> level  -9	165M	62%  |	27M	61%  |	615M	64%  |	493M	33%  |
> level  -8	161M	60%  |	27M	59%  |	598M	62%  |	475M	32%  |
> level  -7	155M	58%  |	26M	58%  |	582M	61%  |	457M	30%  |
> level  -6	151M	56%  |	25M	56%  |	565M	59%  |	437M	29%  |
> level  -5	145M	54%  |	24M	55%  |	545M	57%  |	417M	28%  |
> level  -4	139M	52%  |	23M	52%  |	520M	54%  |	391M	26%  |
> level  -3	135M	50%  |	22M	50%  |	495M	51%  |	369M	24%  |
> level  -2	127M	47%  |	22M	48%  |	470M	49%  |	349M	23%  |
> level  -1	120M	45%  |	21M	47%  |	452M	47%  |	332M	22%  |
> level   1	110M	41%  |	17M	39%  |	362M	38%  |	290M	19%  |
> level   2	106M	40%  |	17M	38%  |	349M	36%  |	288M	19%  |
> level   3	104M	39%  |	16M	37%  |	340M	35%  |	276M	18%  |

Thanks, this is a good overview. The ratio comparing the now default
level 3 against the fastest level -10 is between 1.6-1.9x, with roughly
linear progression with each level.

We still need the compression times. This may be harder to get from
kernel, so one possibility is to add tracepoints or gather the
statistics inside kernel before/after compression calls and then print
to the log.

Or maybe the 'zstd' utility can be configured to run its benchmarks
close to what kernel does, ie. the block size, memory limit and maybe
more. I've skimmed the manual page and there's enough to choose from.

The last thing is to implement a userspace microbenchmark. Recently
mkfs.btrfs learned compression support so you could reuse this code for
that.

> 
> Signed-off-by: Daniel Vacek <neelx@...e.com>
> ---
> Changes in v3:
>  * Correct the accidentally replaced string in `clip_level()` function.

The code looks good now. Thanks.

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ