linux-ext4 - Re: Transparent compression with ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+rD4x-hh1bzN3YWiGY-EdSpjDxiA-6HzRO14d0pfdb789mS6Q@mail.gmail.com>
Date: Tue, 21 Jan 2025 09:42:14 +0000
From: Artem Blagodarenko <artem.blagodarenko@...il.com>
To: "Theodore Ts'o" <tytso@....edu>
Cc: Gerhard Wiesinger <lists@...singer.com>, linux-ext4@...r.kernel.org
Subject: Re: Transparent compression with ext4 - especially with zstd

Hi Gerhard, Theodore,

>even for a hyperscaler cloud company
>(and even there, it's unclear that transparent compression is really
>needed).

Regarding exascale storages. Lustre FS (which uses EXT4 (LDISKFS) as a
backend) has a “Client-side data compression” project (LU-10026) which
adds transparent compression with an extendable set of algorithms. The
initial release includes gzip, lz4, lz4hc, lzo, zstd, zstdfast
algorithms with levels.
More details are in the LUG and LAD 2023-2024 years presentations.

Best regards,
Artem Blagodarenko

From: Theodore Ts'o <tytso@....edu>
Date: Tuesday, 21 January 2025 at 04:01
To: Gerhard Wiesinger <lists@...singer.com>
Cc: linux-ext4@...r.kernel.org <linux-ext4@...r.kernel.org>
Subject: Re: Transparent compression with ext4 - especially with zstd

On Sun, Jan 19, 2025 at 03:37:27PM +0100, Gerhard Wiesinger wrote:
>
> Are there any plans to include transparent compression with ext4 (especially
> with zstd)?

I'm not aware of anyone in the ext4 deveopment commuity working on
something like this.  Fully transparent compression is challenging,
since supporting random writes into a compressed file is tricky.
There are solutions (for example, the Stac patent which resulted in
Microsoft to pay $120 million dollars), but even ignoring the
intellectual property issues, they tend to compromise the efficiency
of the compression.

More to the point, given how cheap byte storage tends to be (dollars
per IOPS tend to be far more of a constraint than dollars per GB),
it's unclear what the business case would be for any company to fund
development work in this area, when the cost of a slightly large HDD
or SSD is going to be far cheaper than the necessary software
engineering investrment needed, even for a hyperscaler cloud company
(and even there, it's unclear that transparent compression is really
needed).

What is the business and/or technical problem which you are trying to
solve?

Cheers,

                                        - Ted