netdev - Re: [PATCH btf 0/3] Add BTF types deduplication algorithm

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <007ee25b-9ad4-aab9-657f-0200bf6a3a98@fb.com>
Date:   Thu, 31 Jan 2019 07:18:52 +0000
From:   Yonghong Song <yhs@...com>
To:     Andrii Nakryiko <andriin@...com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Alexei Starovoitov <ast@...com>, Martin Lau <kafai@...com>,
        "acme@...nel.org" <acme@...nel.org>,
        "daniel@...earbox.net" <daniel@...earbox.net>,
        Kernel Team <Kernel-team@...com>,
        Edward Cree <ecree@...arflare.com>
Subject: Re: [PATCH btf 0/3] Add BTF types deduplication algorithm


add Edward.

On 1/30/19 10:58 PM, Andrii Nakryiko wrote:
> This patch series adds BTF deduplication algorithm to libbpf. This algorithm
> allows to take BTF type information containing duplicate per-compilation unit
> information and reduce it to equivalent set of BTF types with no duplication without
> loss of information. It also deduplicates strings and removes those strings that
> are not referenced from any BTF type (and line information in .BTF.ext section,
> if any).
> 
> Algorithm also resolves struct/union forward declarations into concrete BTF types
> across multiple compilation units to facilitate better deduplication ratio. If
> undesired, this resolution can be disabled through specifying corresponding options.
> 
> When applied to BTF data emitted by pahole's DWARF->BTF converter, it reduces
> the overall size of .BTF section by about 65x, from about 112MB to 1.75MB, leaving
> only 29247 out of initial 3073497 BTF type descriptors.
> 
> Algorithm with minor differences and preliminary results before FUNC/FUNC_PROTO
> support is also described more verbosely at:
> https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf-enhancement.html
> 
> Andrii Nakryiko (3):
>    btf: extract BTF type size calculation
>    btf: add BTF types deduplication algorithm
>    selftests/btf: add initial BTF dedup tests
> 
>   tools/lib/bpf/btf.c                    | 1851 +++++++++++++++++++++++-
>   tools/lib/bpf/btf.h                    |   11 +
>   tools/lib/bpf/libbpf.map               |    3 +
>   tools/testing/selftests/bpf/test_btf.c |  535 ++++++-
>   4 files changed, 2333 insertions(+), 67 deletions(-)
>