netdev - Re: [PATCH bpf v2 0/3] bpf: invalidate unused part of bpf_prog

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <110CB061-8DA5-400B-AAE3-13FAFE0ADE90@fb.com>
Date:   Sat, 7 May 2022 19:36:39 +0000
From:   Song Liu <songliubraving@...com>
To:     Linus Torvalds <torvalds@...ux-foundation.org>
CC:     bpf <bpf@...r.kernel.org>, Networking <netdev@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Andrii Nakryiko <andrii@...nel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Kernel Team <Kernel-team@...com>,
        Christoph Hellwig <hch@...radead.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Song Liu <song@...nel.org>
Subject: Re: [PATCH bpf v2 0/3] bpf: invalidate unused part of bpf_prog_pack



> On May 6, 2022, at 11:50 PM, Song Liu <songliubraving@...com> wrote:
> 
> 
> 
>> On Apr 27, 2022, at 11:48 PM, Song Liu <songliubraving@...com> wrote:
>> 
>> Hi Linus, 
>> 
>> Thanks for your thorough analysis of the situation, which make a lot of
>> sense. 
>> 
>>> On Apr 27, 2022, at 6:45 PM, Linus Torvalds <torvalds@...ux-foundation.org> wrote:
>>> 
>>> On Wed, Apr 27, 2022 at 3:24 PM Song Liu <songliubraving@...com> wrote:
>>>> 
>>>> Could you please share your suggestions on this set? Shall we ship it
>>>> with 5.18?
>>> 
>>> I'd personally prefer to just not do the prog_pack thing at all, since
>>> I don't think it was actually in a "ready to ship" state for this
>>> merge window, and the hugepage mapping protection games I'm still
>>> leery of.
>>> 
>>> Yes, the hugepage protection things probably do work from what I saw
>>> when I looked through them, but that x86 vmalloc hugepage code was
>>> really designed for another use (non-refcounted device pages), so the
>>> fact that it all actually seems surprisingly ok certainly wasn't
>>> because the code was designed to do that new case.
>>> 
>>> Does the prog_pack thing work with small pages?
>>> 
>>> Yes. But that wasn't what it was designed for or its selling point, so
>>> it all is a bit suspect to me.
>> 
>> prog_pack on small pages can also reduce the direct map fragmentation.
>> This is because libbpf uses tiny BPF programs to probe kernel features. 
>> Before prog_pack, all these BPF programs can fragment the direct map.
>> For example, runqslower (tools/bpf/runqslower/) loads total 7 BPF programs 
>> (3 actual programs and 4 tiny probe programs). All these programs may 
>> cause direct map fragmentation. With prog_pack, OTOH, these BPF programs 
>> would fit in a single page (or even share pages with other tools). 
> 
> Here are some performance data from our web service production benchmark, 
> which is the biggest service in our fleet. We compare 3 kernels:    
> 
>  nopack: no bpf_prog_pack; IOW, the same behavior as 5.17
>  4kpack: use bpf_prog_pack on 4kB pages (same as 5.18-rc5)
>  2mpack: use bpf_prog_pack on 2MB pages
> 
> The benchmark measures system throughput under latency constraints. 
> 4kpack provides 0.5% to 0.7% more throughput than nopack. 
> 2mpack provides 0.6% to 0.9% more throughput than nopack. 
> 
> So the data has confirmed:
> 1. Direct map fragmentation has non-trivial impact on system performance;
> 2. While 2MB pages are preferred, bpf_prog_pack on 4kB pages also gives 
>   Significant performance improvements.  

Please note that 0.5% is a huge improvement for our fleet. I believe this
is also significant for other companies with many thousand servers. 

Thanks,
Song