netdev - Re: [Linuxarm] Re: [PATCH RFC 0/7] add socket to netdev page frag recycling support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5fdc5223-7d67-fed7-f691-185dcb2e3d80@gmail.com>
Date:   Wed, 25 Aug 2021 09:29:06 -0700
From:   David Ahern <dsahern@...il.com>
To:     Eric Dumazet <edumazet@...gle.com>,
        Yunsheng Lin <linyunsheng@...wei.com>
Cc:     David Miller <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Alexander Duyck <alexander.duyck@...il.com>,
        Russell King <linux@...linux.org.uk>,
        Marcin Wojtas <mw@...ihalf.com>, linuxarm@...neuler.org,
        Yisen Zhuang <yisen.zhuang@...wei.com>,
        Salil Mehta <salil.mehta@...wei.com>,
        Thomas Petazzoni <thomas.petazzoni@...tlin.com>,
        Jesper Dangaard Brouer <hawk@...nel.org>,
        Ilias Apalodimas <ilias.apalodimas@...aro.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        John Fastabend <john.fastabend@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Will Deacon <will@...nel.org>,
        Matthew Wilcox <willy@...radead.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Fenghua Yu <fenghua.yu@...el.com>,
        Roman Gushchin <guro@...com>, Peter Xu <peterx@...hat.com>,
        "Tang, Feng" <feng.tang@...el.com>, Jason Gunthorpe <jgg@...pe.ca>,
        mcroce@...rosoft.com, Hugh Dickins <hughd@...gle.com>,
        Jonathan Lemon <jonathan.lemon@...il.com>,
        Alexander Lobakin <alobakin@...me>,
        Willem de Bruijn <willemb@...gle.com>,
        wenxu <wenxu@...oud.cn>, Cong Wang <cong.wang@...edance.com>,
        Kevin Hao <haokexin@...il.com>,
        Aleksandr Nogikh <nogikh@...gle.com>,
        Marco Elver <elver@...gle.com>, Yonghong Song <yhs@...com>,
        kpsingh@...nel.org, Andrii Nakryiko <andrii@...nel.org>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>,
        netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        chenhao288@...ilicon.com,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        David Ahern <dsahern@...nel.org>, memxor@...il.com,
        linux@...pel-privat.de, Antoine Tenart <atenart@...nel.org>,
        Wei Wang <weiwan@...gle.com>, Taehee Yoo <ap420073@...il.com>,
        Arnd Bergmann <arnd@...db.de>,
        Mat Martineau <mathew.j.martineau@...ux.intel.com>,
        aahringo@...hat.com, ceggers@...i.de, yangbo.lu@....com,
        Florian Westphal <fw@...len.de>, xiangxia.m.yue@...il.com,
        linmiaohe <linmiaohe@...wei.com>, Christoph Hellwig <hch@....de>
Subject: Re: [Linuxarm] Re: [PATCH RFC 0/7] add socket to netdev page frag
 recycling support

On 8/23/21 8:04 AM, Eric Dumazet wrote:
>>
>>
>> It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory
>> compact and memory isolation, as the test system has a lot of memory installed
>> (about 500G, only 3-4G is used), so I used the below patch to test the max
>> possible performance improvement when making TCP frags twice bigger, and
>> the performance improvement went from about 30Gbit to 32Gbit for one thread
>> iperf tcp flow in IOMMU strict mode,
> 
> This is encouraging, and means we can do much better.
> 
> Even with SKB_FRAG_PAGE_ORDER  set to 4, typical skbs will need 3 mappings
> 
> 1) One for the headers (in skb->head)
> 2) Two page frags, because one TSO packet payload is not a nice power-of-two.

interesting observation. I have noticed 17 with the ZC API. That might
explain the less than expected performance bump with iommu strict mode.

> 
> The first issue can be addressed using a piece of coherent memory (128
> or 256 bytes per entry in TX ring).
> Copying the headers can avoid one IOMMU mapping, and improve IOTLB
> hits, because all
> slots of the TX ring buffer will use one single IOTLB slot.
> 
> The second issue can be solved by tweaking a bit
> skb_page_frag_refill() to accept an additional parameter
> so that the whole skb payload fits in a single order-4 page.
> 
>