linux-kernel - Re: [PATCH] mm: Make count list_lru_one::nr

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <fbb67bef-c13f-7fcb-fa6a-e3a7f6e5c82b@virtuozzo.com>
Date:   Thu, 28 Sep 2017 10:48:55 +0300
From:   Kirill Tkhai <ktkhai@...tuozzo.com>
To:     Andrew Morton <akpm@...ux-foundation.org>
Cc:     vdavydov.dev@...il.com, apolyakov@...et.ru,
        linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        aryabinin@...tuozzo.com
Subject: Re: [PATCH] mm: Make count list_lru_one::nr_items lockless

On 28.09.2017 00:15, Andrew Morton wrote:
> On Tue, 19 Sep 2017 18:06:33 +0300 Kirill Tkhai <ktkhai@...tuozzo.com> wrote:
> 
>> During the reclaiming slab of a memcg, shrink_slab iterates
>> over all registered shrinkers in the system, and tries to count
>> and consume objects related to the cgroup. In case of memory
>> pressure, this behaves bad: I observe high system time and
>> time spent in list_lru_count_one() for many processes on RHEL7
>> kernel (collected via $perf record --call-graph fp -j k -a):
>>
>> 0,50%  nixstatsagent  [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
>> 0,26%  nixstatsagent  [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
>> 0,23%  nixstatsagent  [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
>> 0,15%  nixstatsagent  [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
>> 0,15%  nixstatsagent  [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
>>
>> 0,94%  mysqld         [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
>> 0,57%  mysqld         [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
>> 0,51%  mysqld         [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
>> 0,32%  mysqld         [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
>> 0,32%  mysqld         [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
>>
>> 0,73%  sshd           [kernel.vmlinux]  [k] _raw_spin_lock                [k] _raw_spin_lock
>> 0,35%  sshd           [kernel.vmlinux]  [k] shrink_slab                   [k] shrink_slab
>> 0,32%  sshd           [kernel.vmlinux]  [k] super_cache_count             [k] super_cache_count
>> 0,21%  sshd           [kernel.vmlinux]  [k] __list_lru_count_one.isra.2   [k] _raw_spin_lock
>> 0,21%  sshd           [kernel.vmlinux]  [k] list_lru_count_one            [k] __list_lru_count_one.isra.2
>>
>> This patch aims to make super_cache_count() (and other functions,
>> which count LRU nr_items) more effective.
>> It allows list_lru_node::memcg_lrus to be RCU-accessed, and makes
>> __list_lru_count_one() count nr_items lockless to minimize
>> overhead introduced by locking operation, and to make parallel
>> reclaims more scalable.
> 
> And...  what were the effects of the patch?  Did you not run the same
> performance tests after applying it?

I've just detected the such high usage of shrink slab on production node. It's rather
difficult to make it use another kernel, than it uses, only kpatches are possible.
So, I haven't estimated how it acts on node's performance.
On test node I see, that the patch obviously removes raw_spin_lock from perf profile.
So, it's a little bit untested in this way.