| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20250129224253.GF5777@twin.jikos.cz> Date: Wed, 29 Jan 2025 23:42:54 +0100 From: David Sterba <dsterba@...e.cz> To: Daniel Vacek <neelx@...e.com> Cc: dsterba@...e.cz, Chris Mason <clm@...com>, Josef Bacik <josef@...icpanda.com>, David Sterba <dsterba@...e.com>, Nick Terrell <terrelln@...com>, linux-btrfs@...r.kernel.org, linux-kernel@...r.kernel.org Subject: Re: [PATCH] btrfs/zstd: enable negative compression levels mount option On Tue, Jan 28, 2025 at 09:46:02AM +0100, Daniel Vacek wrote: > > > ==============================================================+=============================================== > > > level -15 0m0,261s 0m0,329s 141M 100% | 0m2,511s 0m2,794s 598M 40% | > > > level -14 0m0,145s 0m0,291s 141M 100% | 0m1,829s 0m2,443s 581M 39% | > > > level -13 0m0,141s 0m0,289s 141M 100% | 0m1,832s 0m2,347s 566M 38% | > > > level -12 0m0,140s 0m0,291s 141M 100% | 0m1,879s 0m2,246s 548M 37% | > > > level -11 0m0,133s 0m0,271s 141M 100% | 0m1,789s 0m2,257s 530M 35% | > > > > I found an old mail asking ZSTD people which realtime levels are > > meaningful, the -10 was mentioned as a good cut-off. The numbers above > > confirm that although this is on a small sample. > > The limit is really arbitrary. We may as well not even set one and > leave it to the user. It's not like we allocate additional memory or > any other resources. Yes, it is arbitrary, the criteria is what are the practical benefits of keeping the levels, what is the trade off compression/speed. It does not matter much if the realtime level is ultra fast when it is not able to reduce the size, given the constraints. I agree that we should leave it up to user but I'd say it may be unclear which level select. The level number is translated to some internal compression logic so even very high numbers are allowed. The testing file I've used is output of 'find -ls', so mostly textual data, high level of redundancy. Generated by a series of "zstd -b$i --fast=$i zstd-test" -1#zstd-test : 6126812 -> 1017901 (x6.019), 1055.7 MB/s, 2826.0 MB/s -2#zstd-test : 6126812 -> 1065645 (x5.749), 1143.0 MB/s, 3005.8 MB/s -3#zstd-test : 6126812 -> 1128314 (x5.430), 1204.2 MB/s, 3153.6 MB/s -4#zstd-test : 6126812 -> 1210669 (x5.061), 1220.5 MB/s, 3172.8 MB/s -5#zstd-test : 6126812 -> 1280004 (x4.787), 1242.7 MB/s, 3221.3 MB/s -6#zstd-test : 6126812 -> 1371974 (x4.466), 1259.8 MB/s 3277.1 MB/s -7#zstd-test : 6126812 -> 1440333 (x4.254), 1304.4 MB/s, 3293.0 MB/s -8#zstd-test : 6126812 -> 1496071 (x4.095), 1356.9 MB/s, 3391.2 MB/s -9#zstd-test : 6126812 -> 1566788 (x3.910), 1452.9 MB/s, 3613.6 MB/s -10#std-test : 6126812 -> 1613304 (x3.798), 1497.1 MB/s, 3738.7 MB/s -11#std-test : 6126812 -> 1685217 (x3.636), 1541.4 MB/s, 3829.2 MB/s -12#std-test : 6126812 -> 1766056 (x3.469), 1556.0 MB/s, 3875.6 MB/s -13#std-test : 6126812 -> 1826338 (x3.355), 1563.3 MB/s, 3886.6 MB/s -14#std-test : 6126812 -> 1899464 (x3.226), 1623.8 MB/s, 3963.5 MB/s -15#std-test : 6126812 -> 1989219 (x3.080), 1641.9 MB/s, 3901.3 MB/s -16#std-test : 6126812 -> 2057101 (x2.978), 1646.0 MB/s, 3781.2 MB/s -17#std-test : 6126812 -> 2101038 (x2.916), 1639.9 MB/s, 3809.8 MB/s -18#std-test : 6126812 -> 2151170 (x2.848), 1667.4 MB/s, 3887.3 MB/s -19#std-test : 6126812 -> 2185255 (x2.804), 1675.8 MB/s, 3932.5 MB/s -20#std-test : 6126812 -> 2228180 (x2.750), 1655.2 MB/s, 3983.3 MB/s -21#std-test : 6126812 -> 2275855 (x2.692), 1748.4 MB/s, 4188.4 MB/s -22#std-test : 6126812 -> 2324370 (x2.636), 1758.7 MB/s, 4245.0 MB/s -23#std-test : 6126812 -> 2373789 (x2.581), 1810.8 MB/s, 4345.4 MB/s -24#std-test : 6126812 -> 2423045 (x2.529), 1856.4 MB/s, 4326.7 MB/s -25#std-test : 6126812 -> 2470722 (x2.480), 1860.7 MB/s, 4425.6 MB/s -26#std-test : 6126812 -> 2519621 (x2.432), 1872.8 MB/s, 4447.8 MB/s -27#std-test : 6126812 -> 2571890 (x2.382), 1865.0 MB/s, 4432.7 MB/s -28#std-test : 6126812 -> 2630088 (x2.330), 1892.7 MB/s, 4480.3 MB/s -29#std-test : 6126812 -> 2678598 (x2.287), 1892.3 MB/s, 4458.1 MB/s -30#std-test : 6126812 -> 2730233 (x2.244), 1886.9 MB/s, 4415.0 MB/s -100#td-test : 6126812 -> 3878531 (x1.580), 2546.7 MB/s, 7076.8 MB/s -101#td-test : 6126812 -> 3873153 (x1.582), 2549.6 MB/s, 7053.4 MB/s -102#td-test : 6126812 -> 3878200 (x1.580), 2572.9 MB/s 7173.3 MB/s -103#td-test : 6126812 -> 3900441 (x1.571), 2581.2 MB/s 7164.9 MB/s -104#td-test : 6126812 -> 3902470 (x1.570), 2559.0 MB/s, 7133.3 MB/s -105#td-test : 6126812 -> 3912874 (x1.566), 2596.4 MB/s, 7150.9 MB/s -106#td-test : 6126812 -> 3917867 (x1.564), 2553.8 MB/s, 7260.1 MB/s -107#td-test : 6126812 -> 3930050 (x1.559), 2599.6 MB/s, 7284.5 MB/s -108#td-test : 6126812 -> 3946701 (x1.552), 2622.5 MB/s, 7288.1 MB/s -109#td-test : 6126812 -> 3954437 (x1.549), 2605.8 MB/s, 7334.4 MB/s -110#td-test : 6126812 -> 3955273 (x1.549), 2601.0 MB/s, 7389.1 MB/s -1000#d-test : 6126812 -> 5982024 (x1.024), 9253.0 MB/s, 22210.0 MB/s -1001#d-test : 6126812 -> 5986021 (x1.024), 9120.1 MB/s, 23034.3 MB/s -1002#d-test : 6126812 -> 5984005 (x1.024), 9086.7 MB/s, 22827.0 MB/s -1003#d-test : 6126812 -> 5978004 (x1.025), 9199.4 MB/s, 21718.3 MB/s -1004#d-test : 6126812 -> 5977133 (x1.025), 9120.3 MB/s, 21995.3 MB/s -1005#d-test : 6126812 -> 5979294 (x1.025), 9231.6 MB/s, 22204.5 MB/s -1006#d-test : 6126812 -> 5981595 (x1.024), 9175.6 MB/s, 21794.4 MB/s -1007#d-test : 6126812 -> 5989283 (x1.023), 9298.9 MB/s, 24239.1 MB/s -1008#d-test : 6126812 -> 5986954 (x1.023), 9242.1 MB/s, 23809.4 MB/s -1009#d-test : 6126812 -> 5988383 (x1.023), 9199.3 MB/s, 24322.1 MB/s -1010#d-test : 6126812 -> 5985304 (x1.024), 9221.2 MB/s, 23711.9 MB/s Up to -15 it's 3x improvement which translates to about 33% of the original size. And this is only for rough estimate, kernel compression could be slightly worse due to slightly different parameters. We can let it to -15, so it's same number as the upper limit.
Powered by blists - more mailing lists