linux-kernel - Re: [PATCH v4 0/10] perf: enable compression of record mode trace to save storage space

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190313143726.GG6676@krava>
Date:   Wed, 13 Mar 2019 15:37:26 +0100
From:   Jiri Olsa <jolsa@...hat.com>
To:     Alexey Budankov <alexey.budankov@...ux.intel.com>
Cc:     Arnaldo Carvalho de Melo <acme@...nel.org>,
        Namhyung Kim <namhyung@...nel.org>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Andi Kleen <ak@...ux.intel.com>,
        linux-kernel <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v4 0/10] perf: enable compression of record mode trace to
 save storage space

On Thu, Feb 28, 2019 at 11:35:24AM +0300, Alexey Budankov wrote:
> 
> The patch set implements runtime trace compression (-z option) in 
> record mode and trace auto decompression in report and inject modes. 
> Streaming Zstandard (Zstd) API (zstd) is used for compression and 
> decompression of data that come from kernel mmaped data buffers.
> 
> Usage of implemented -z,--compression_level=n option provides ~3-5x 
> avg. trace file size reduction on variety of tested workloads what 
> saves storage space on larger server systems where trace file size 
> can easily reach several tens or even hundreds of GiBs, especially 
> when profiling with dwarf-based stacks and tracing of context switches.
> Implemented -f,--mmap-flush option can be used to avoid compressing 
> every single byte of data and increase compression ratio at the same 
> time lowering tool runtime overhead. Default option value is 1 what 
> is equal to the current perf record implementation. The option is 
> independent from -z setting and doesn't vary with compression level:
> 
>   $ tools/perf/perf record -z 1 -e cycles -- matrix.gcc
>   $ tools/perf/perf record --aio=1 -z 1 -e cycles -- matrix.gcc
>   $ tools/perf/perf record -z 1 -f 1024 -e cycles -- matrix.gcc
>   $ tools/perf/perf record --aio=1 -z 1 -f 1024 -e cycles -- matrix.gcc
> 
> Runtime compression overhead has been measured for serial and AIO 
> trace writing modes when profiling matrix multiplication workload 
> with the following results:
> 
>     -------------------------------------------------------------
>     | SERIAL			  | AIO-1                       |
> -----------------------------------------------------------------
> |-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
> |----------------------------------------------------------------
> | 0 | 1,00   | 1,000    179,424   | 1,00   | 1,000    187,527   |
> | 1 | 1,04   | 8,427    181,148   | 1,01   | 8,474    188,562   |
> | 2 | 1,07   | 8,055    186,953   | 1,03   | 7,912    191,773   |
> | 3 | 1,04   | 8,283    181,908   | 1,03   | 8,220    191,078   |
> | 5 | 1,09   | 8,101    187,705   | 1,05   | 7,780    190,065   |
> | 8 | 1,05   | 9,217    179,191   | 1,12   | 6,111    193,024   |
> -----------------------------------------------------------------
> 
> OVH = (Execution time with -z N) / (Execution time with -z 0)
> 
> ratio - compression ratio
> size  - number of bytes that was compressed
> 
> size ~= trace file x ratio
> 
> See complete description of measurement conditions with details below.
> 
> Introduced compression functionality can be disabled or configured from 
> the command line using NO_LIBZSTD and LIBZSTD_DIR defines:
> 
>   $ make -C tools/perf NO_LIBZSTD=1 clean all
>   $ make -C tools/perf LIBZSTD_DIR=/path/to/zstd/sources/ clean all
> 
> If your system has some version of the zstd package preinstalled then 
> the build system finds and uses it during the build. Auto detection 
> feature status is reported just before compilation starts, as usual.
> If you still prefer to compile with some version of zstd that is not 
> preinstalled you have capability to refer the compilation to that 
> version using LIBZSTD_DIR define.
> 
> See 'perf test' results below for enabled and disabled (NO_LIBZSTD=1)
> feature configurations.
> 
> ---
> Alexey Budankov (10):
>   feature: implement libzstd check, LIBZSTD_DIR and NO_LIBZSTD defines
>   perf record: implement -f,--mmap-flush=<threshold> option
>   perf session: define bytes_transferred and bytes_compressed metrics
>   perf record: implement COMPRESSED event record and its attributes
>   perf mmap: implement dedicated memory buffer for data compression
>   perf util: introduce Zstd based streaming compression API
>   perf record: implement -z,--compression_level=n option and compression
>   perf report: implement record trace decompression
>   perf inject: enable COMPRESSED records decompression
>   perf tests: implement Zstd comp/decomp integration test

the record messages disappeared totaly:

[jolsa@...va perf]$ ./perf record -z 1 ls
[jolsa@...va perf]$ ./perf record -z 22 ls
[jolsa@...va perf]$ ./perf record ls
[jolsa@...va perf]$


seems like we bail out for some reason with no message:

[jolsa@...va perf]$ ./perf record ls
[jolsa@...va perf]$ ./perf report --stat
WARNING: The perf.data file's data size field is 0 which is unexpected.
Was the 'perf record' command properly terminated?
Error:
failed to process sample

jirka