linux-kernel - Re: [PATCH v4 2/2] arm64: support batched/deferred tlb shootdown during page reclamation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <bc44cf85-aee9-03ca-9911-dbd904a43cc8@huawei.com>
Date:   Fri, 28 Oct 2022 09:20:08 +0800
From:   Yicong Yang <yangyicong@...wei.com>
To:     Punit Agrawal <punit.agrawal@...edance.com>,
        Barry Song <21cnbao@...il.com>
CC:     <yangyicong@...ilicon.com>, <corbet@....net>,
        <peterz@...radead.org>, <arnd@...db.de>,
        <linux-kernel@...r.kernel.org>, <darren@...amperecomputing.com>,
        <huzhanyuan@...o.com>, <lipeifeng@...o.com>,
        <zhangshiming@...o.com>, <guojian@...o.com>, <realmz6@...il.com>,
        <linux-mips@...r.kernel.org>, <openrisc@...ts.librecores.org>,
        <linux-mm@...ck.org>, <x86@...nel.org>,
        <linux-arm-kernel@...ts.infradead.org>,
        <linuxppc-dev@...ts.ozlabs.org>, <akpm@...ux-foundation.org>,
        <linux-riscv@...ts.infradead.org>, <linux-s390@...r.kernel.org>,
        <wangkefeng.wang@...wei.com>, <xhao@...ux.alibaba.com>,
        <prime.zeng@...ilicon.com>, Barry Song <v-songbaohua@...o.com>,
        Nadav Amit <namit@...are.com>, Mel Gorman <mgorman@...e.de>,
        <catalin.marinas@....com>, <will@...nel.org>,
        <linux-doc@...r.kernel.org>,
        Anshuman Khandual <anshuman.khandual@....com>
Subject: Re: [PATCH v4 2/2] arm64: support batched/deferred tlb shootdown
 during page reclamation

On 2022/10/27 22:19, Punit Agrawal wrote:
> 
> [ Apologies for chiming in late in the conversation ]
> 
> Anshuman Khandual <anshuman.khandual@....com> writes:
> 
>> On 9/28/22 05:53, Barry Song wrote:
>>> On Tue, Sep 27, 2022 at 10:15 PM Yicong Yang <yangyicong@...wei.com> wrote:
>>>>
>>>> On 2022/9/27 14:16, Anshuman Khandual wrote:
>>>>> [...]
>>>>>
>>>>> On 9/21/22 14:13, Yicong Yang wrote:
>>>>>> +static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
>>>>>> +{
>>>>>> +    /* for small systems with small number of CPUs, TLB shootdown is cheap */
>>>>>> +    if (num_online_cpus() <= 4)
>>>>>
>>>>> It would be great to have some more inputs from others, whether 4 (which should
>>>>> to be codified into a macro e.g ARM64_NR_CPU_DEFERRED_TLB, or something similar)
>>>>> is optimal for an wide range of arm64 platforms.
>>>>>
>>>
>>> I have tested it on a 4-cpus and 8-cpus machine. but i have no machine
>>> with 5,6,7
>>> cores.
>>> I saw improvement on 8-cpus machines and I found 4-cpus machines don't need
>>> this patch.
>>>
>>> so it seems safe to have
>>> if (num_online_cpus()  < 8)
>>>
>>>>
>>>> Do you prefer this macro to be static or make it configurable through kconfig then
>>>> different platforms can make choice based on their own situations? It maybe hard to
>>>> test on all the arm64 platforms.
>>>
>>> Maybe we can have this default enabled on machines with 8 and more cpus and
>>> provide a tlbflush_batched = on or off to allow users enable or
>>> disable it according
>>> to their hardware and products. Similar example: rodata=on or off.
>>
>> No, sounds bit excessive. Kernel command line options should not be added
>> for every possible run time switch options.
>>
>>>
>>> Hi Anshuman, Will,  Catalin, Andrew,
>>> what do you think about this approach?
>>>
>>> BTW, haoxin mentioned another important user scenarios for tlb bach on arm64:
>>> https://lore.kernel.org/lkml/393d6318-aa38-01ed-6ad8-f9eac89bf0fc@linux.alibaba.com/
>>>
>>> I do believe we need it based on the expensive cost of tlb shootdown in arm64
>>> even by hardware broadcast.
>>
>> Alright, for now could we enable ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH selectively
>> with CONFIG_EXPERT and for num_online_cpus()  > 8 ?
> 
> When running the test program in the commit in a VM, I saw benefits from
> the patches at all sizes from 2, 4, 8, 32 vcpus. On the test machine,
> ptep_clear_flush() went from ~1% in the unpatched version to not showing
> up.
> 

Maybe you're booting VM on a server with more than 32 cores and Barry tested
on his 4 CPUs embedded platform. I guess a 4 CPU VM is not fully equivalent to
a 4 CPU real machine as the tbli and dsb in the VM may influence the host
as well.

> Yicong mentioned that he didn't see any benefit for <= 4 CPUs but is
> there any overhead? I am wondering what are the downsides of enabling
> the config by default.
> 
> Thanks,
> Punit
> .
>