| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20231121010318.524570-1-nickrterrell@gmail.com>
Date: Mon, 20 Nov 2023 17:03:16 -0800
From: Nick Terrell <nickrterrell@...il.com>
To: linux-kernel@...r.kernel.org
Cc: Yann Collet <cyan@...a.com>, Nick Terrell <terrelln@...a.com>,
Nick Terrell <nickrterrell@...il.com>,
Kernel Team <Kernel-team@...com>,
Giovanni Cabiddu <giovanni.cabiddu@...el.com>,
Linus Torvalds <torvalds@...ux-foundation.org>
Subject: [PATCH 0/2] zstd: import upstream v1.5.5
From: Nick Terrell <terrelln@...a.com>
Import upstream zstd v1.5.5 to expose upstream's QAT integration.
The following changes since commit 77618db346455129424fadbbaec596a09feaf3bb:
zstd: Fix array-index-out-of-bounds UBSAN warning (2023-11-14 17:12:52 -0800)
are also available in the Git repository at:
https://github.com/terrelln/linux.git tags/2023-11-20-zstd-v1.5.5-v1
In addition to keeping the kernel's copy of zstd up to date, this update
was requested by Intel to expose upstream zstd's external match provider
API to the kernel, which allows QAT to accelerate the LZ match finding
stage.
The result of the series is that zstd is cleanly imported from the
tag v1.5.5-kernel [0], which is signed with upstream's signing key
EF8FE99528B52FFD [1]. Meaning that running this script creates no diff:
export ZSTD=/path/to/repo/zstd/
export LINUX=/path/to/repo/linux/
cd "$ZSTD/contrib/linux-kernel"
git checkout v1.5.5-kernel
make import LINUX="$LINUX"
The first patch updates to v1.5.5 with three cherry picked backports
from the Linux kernel and the upstream dev branch. The second patch
negates a regression in decompression speed caused by the update.
The series has been tested on x86-64, and has been boot tested with a
zstd compressed kernel on i386 and aarch64. I benchmarked the series
on x86-64 with gcc-13.2.1 on an Intel i9-9900K by measuring the
performance of compressed filesystems read & writes.
Component, Level, C. time delta, size delta, D. time delta
Btrfs , 1, -1.5%, +0.0%, +1.8%
Btrfs , 3, -5.8%, +0.0%, +3.2%
Btrfs , 5, -5.7%, +0.0%, +0.0%
Btrfs , 7, -6.5%, +0.0%, +0.4%
Btrfs , 9, -6.9%, +0.0%, +1.5%
Squashfs , 1, N/A, 0.0%, +1.0%
For levels 3 and above, there is a ~6% reduction in write+compression time.
There is also a small increase in read+decompression time.
If everything looks good, I will submit this patch set with the rest of my
tree in the v6.8 merge window.
[0] https://github.com/facebook/zstd/tree/v1.5.5-kernel
[1] https://keyserver.ubuntu.com/pks/lookup?search=EF8FE99528B52FFD&fingerprint=on&op=index
Nick Terrell (2):
zstd: import upstream v1.5.5
zstd: Backport Huffman speed improvement from upstream
include/linux/zstd.h | 2 +-
include/linux/zstd_errors.h | 23 +-
include/linux/zstd_lib.h | 697 +++++--
lib/zstd/Makefile | 2 +-
lib/zstd/common/allocations.h | 56 +
lib/zstd/common/bits.h | 149 ++
lib/zstd/common/bitstream.h | 53 +-
lib/zstd/common/compiler.h | 14 +-
lib/zstd/common/cpu.h | 3 +-
lib/zstd/common/debug.c | 3 +-
lib/zstd/common/debug.h | 3 +-
lib/zstd/common/entropy_common.c | 42 +-
lib/zstd/common/error_private.c | 12 +-
lib/zstd/common/error_private.h | 3 +-
lib/zstd/common/fse.h | 89 +-
lib/zstd/common/fse_decompress.c | 94 +-
lib/zstd/common/huf.h | 222 +--
lib/zstd/common/mem.h | 2 +-
lib/zstd/common/portability_macros.h | 26 +-
lib/zstd/common/zstd_common.c | 38 +-
lib/zstd/common/zstd_deps.h | 16 +-
lib/zstd/common/zstd_internal.h | 99 +-
lib/zstd/compress/clevels.h | 3 +-
lib/zstd/compress/fse_compress.c | 59 +-
lib/zstd/compress/hist.c | 3 +-
lib/zstd/compress/hist.h | 3 +-
lib/zstd/compress/huf_compress.c | 372 ++--
lib/zstd/compress/zstd_compress.c | 1762 ++++++++++++-----
lib/zstd/compress/zstd_compress_internal.h | 333 +++-
lib/zstd/compress/zstd_compress_literals.c | 155 +-
lib/zstd/compress/zstd_compress_literals.h | 25 +-
lib/zstd/compress/zstd_compress_sequences.c | 7 +-
lib/zstd/compress/zstd_compress_sequences.h | 3 +-
lib/zstd/compress/zstd_compress_superblock.c | 47 +-
lib/zstd/compress/zstd_compress_superblock.h | 3 +-
lib/zstd/compress/zstd_cwksp.h | 149 +-
lib/zstd/compress/zstd_double_fast.c | 129 +-
lib/zstd/compress/zstd_double_fast.h | 6 +-
lib/zstd/compress/zstd_fast.c | 582 ++++--
lib/zstd/compress/zstd_fast.h | 6 +-
lib/zstd/compress/zstd_lazy.c | 518 ++---
lib/zstd/compress/zstd_lazy.h | 7 +-
lib/zstd/compress/zstd_ldm.c | 11 +-
lib/zstd/compress/zstd_ldm.h | 3 +-
lib/zstd/compress/zstd_ldm_geartab.h | 3 +-
lib/zstd/compress/zstd_opt.c | 187 +-
lib/zstd/compress/zstd_opt.h | 3 +-
lib/zstd/decompress/huf_decompress.c | 770 ++++---
lib/zstd/decompress/zstd_ddict.c | 9 +-
lib/zstd/decompress/zstd_ddict.h | 3 +-
lib/zstd/decompress/zstd_decompress.c | 261 ++-
lib/zstd/decompress/zstd_decompress_block.c | 283 ++-
lib/zstd/decompress/zstd_decompress_block.h | 8 +-
.../decompress/zstd_decompress_internal.h | 7 +-
lib/zstd/decompress_sources.h | 2 +-
lib/zstd/zstd_common_module.c | 5 +-
lib/zstd/zstd_compress_module.c | 2 +-
lib/zstd/zstd_decompress_module.c | 4 +-
58 files changed, 4787 insertions(+), 2594 deletions(-)
create mode 100644 lib/zstd/common/allocations.h
create mode 100644 lib/zstd/common/bits.h
--
2.42.1
Powered by blists - more mailing lists