linux-kernel - [PATCH v6 00/23] zram: introduce custom comp backends API

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240712051850.484318-1-senozhatsky@chromium.org>
Date: Fri, 12 Jul 2024 14:18:11 +0900
From: Sergey Senozhatsky <senozhatsky@...omium.org>
To: Andrew Morton <akpm@...ux-foundation.org>,
	Minchan Kim <minchan@...nel.org>
Cc: linux-kernel@...r.kernel.org,
	Sergey Senozhatsky <senozhatsky@...omium.org>
Subject: [PATCH v6 00/23] zram: introduce custom comp backends API

This series introduces support for run-time compression algorithms
tuning, so users, for instance, can adjust compression/acceleration
levels and provide pre-trained compression/decompression dictionaries
which certain algorithms support.

At this point we stop supporting (old/deprecated) comp API.  We may
add new acomp API support in the future, but before that zram needs
to undergo some major rework (we are not ready for async compression).

Some benchmarks for reference (look at column #2)

*** zstd
/sys/block/zram0/mm_stat
1750650880 504575194 514392064        0 514392064        1        0    34204    34204

*** zstd level=-1
/sys/block/zram0/mm_stat
1750638592 591816152 603758592        0 603758592        1        0    34288    34288

*** zstd level=8
/sys/block/zram0/mm_stat
1750659072 486924248 496377856        0 496377856        1        0    34204    34204

*** zstd dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750634496 465853994 475230208        0 475230208        1        0    34185    34185

*** zstd level=8 dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750650880 430760956 439967744        0 439967744        1        0    34185    34185

*** lz4
/sys/block/zram0/mm_stat
1750663168 664194239 676970496        0 676970496        1        0    34288    34288

*** lz4 dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750650880 619901052 632061952        0 632061952        1        0    34278    34278

*** lz4 level=5 dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750650880 727180082 740884480        0 740884480        1        0    34438    34438

v5..v6:
- const-ify zcomp ops (Thomas Weissschuh)

Sergey Senozhatsky (23):
  lib: zstd: export API needed for dictionary support
  lib: lz4hc: export LZ4_resetStreamHC symbol
  lib: zstd: fix null-deref in ZSTD_createCDict_advanced2()
  zram: introduce custom comp backends API
  zram: add lzo and lzorle compression backends support
  zram: add lz4 compression backend support
  zram: add lz4hc compression backend support
  zram: add zstd compression backend support
  zram: pass estimated src size hint to zstd
  zram: add zlib compression backend support
  zram: add 842 compression backend support
  zram: check that backends array has at least one backend
  zram: introduce zcomp_params structure
  zram: recalculate zstd compression params once
  zram: extend comp_algorithm attr write handling
  zram: add support for dict comp config
  zram: introduce zcomp_req structure
  zram: introduce zcomp_ctx structure
  zram: move immutable comp params away from per-CPU context
  zram: add dictionary support to lz4
  zram: add dictionary support to lz4hc
  zram: add dictionary support to zstd backend
  Documentation/zram: add documentation for algorithm parameters

 Documentation/admin-guide/blockdev/zram.rst |  45 +++-
 drivers/block/zram/Kconfig                  |  76 +++++--
 drivers/block/zram/Makefile                 |   8 +
 drivers/block/zram/backend_842.c            |  61 ++++++
 drivers/block/zram/backend_842.h            |  10 +
 drivers/block/zram/backend_deflate.c        | 146 +++++++++++++
 drivers/block/zram/backend_deflate.h        |  10 +
 drivers/block/zram/backend_lz4.c            | 127 +++++++++++
 drivers/block/zram/backend_lz4.h            |  10 +
 drivers/block/zram/backend_lz4hc.c          | 128 +++++++++++
 drivers/block/zram/backend_lz4hc.h          |  10 +
 drivers/block/zram/backend_lzo.c            |  59 +++++
 drivers/block/zram/backend_lzo.h            |  10 +
 drivers/block/zram/backend_lzorle.c         |  59 +++++
 drivers/block/zram/backend_lzorle.h         |  10 +
 drivers/block/zram/backend_zstd.c           | 226 ++++++++++++++++++++
 drivers/block/zram/backend_zstd.h           |  10 +
 drivers/block/zram/zcomp.c                  | 195 ++++++++---------
 drivers/block/zram/zcomp.h                  |  71 ++++--
 drivers/block/zram/zram_drv.c               | 117 +++++++++-
 drivers/block/zram/zram_drv.h               |   1 +
 include/linux/zstd.h                        | 167 +++++++++++++++
 lib/lz4/lz4hc_compress.c                    |   1 +
 lib/zstd/compress/zstd_compress.c           |   2 +
 lib/zstd/zstd_compress_module.c             |  49 +++++
 lib/zstd/zstd_decompress_module.c           |  36 ++++
 26 files changed, 1505 insertions(+), 139 deletions(-)
 create mode 100644 drivers/block/zram/backend_842.c
 create mode 100644 drivers/block/zram/backend_842.h
 create mode 100644 drivers/block/zram/backend_deflate.c
 create mode 100644 drivers/block/zram/backend_deflate.h
 create mode 100644 drivers/block/zram/backend_lz4.c
 create mode 100644 drivers/block/zram/backend_lz4.h
 create mode 100644 drivers/block/zram/backend_lz4hc.c
 create mode 100644 drivers/block/zram/backend_lz4hc.h
 create mode 100644 drivers/block/zram/backend_lzo.c
 create mode 100644 drivers/block/zram/backend_lzo.h
 create mode 100644 drivers/block/zram/backend_lzorle.c
 create mode 100644 drivers/block/zram/backend_lzorle.h
 create mode 100644 drivers/block/zram/backend_zstd.c
 create mode 100644 drivers/block/zram/backend_zstd.h

-- 
2.45.2.993.g49e7a77208-goog