linux-kernel - Re: [PATCH] mm: fix hanging shrinker management on long do_shrink

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <4e2d959a-0b0e-30aa-59b4-8e37728e9793@virtuozzo.com>
Date:   Mon, 2 Dec 2019 19:36:03 +0300
From:   Andrey Ryabinin <aryabinin@...tuozzo.com>
To:     Pavel Tikhomirov <ptikhomirov@...tuozzo.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        linux-kernel@...r.kernel.org, cgroups@...r.kernel.org,
        linux-mm@...ck.org
Cc:     Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Vladimir Davydov <vdavydov.dev@...il.com>,
        Roman Gushchin <guro@...com>,
        Shakeel Butt <shakeelb@...gle.com>,
        Chris Down <chris@...isdown.name>,
        Yang Shi <yang.shi@...ux.alibaba.com>,
        Tejun Heo <tj@...nel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        "Kirill A . Shutemov" <kirill.shutemov@...ux.intel.com>,
        Konstantin Khorenko <khorenko@...tuozzo.com>,
        Kirill Tkhai <ktkhai@...tuozzo.com>,
        Trond Myklebust <trond.myklebust@...merspace.com>,
        Anna Schumaker <anna.schumaker@...app.com>,
        "J. Bruce Fields" <bfields@...ldses.org>,
        Chuck Lever <chuck.lever@...cle.com>,
        linux-nfs@...r.kernel.org,
        Alexander Viro <viro@...iv.linux.org.uk>,
        linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH] mm: fix hanging shrinker management on long
 do_shrink_slab


On 11/30/19 12:45 AM, Pavel Tikhomirov wrote:
> We have a problem that shrinker_rwsem can be held for a long time for
> read in shrink_slab, at the same time any process which is trying to
> manage shrinkers hangs.
> 
> The shrinker_rwsem is taken in shrink_slab while traversing shrinker_list.
> It tries to shrink something on nfs (hard) but nfs server is dead at
> these moment already and rpc will never succeed. Generally any shrinker
> can take significant time to do_shrink_slab, so it's a bad idea to hold
> the list lock here.
> 
> We have a similar problem in shrink_slab_memcg, except that we are
> traversing shrinker_map+shrinker_idr there.
> 
> The idea of the patch is to inc a refcount to the chosen shrinker so it
> won't disappear and release shrinker_rwsem while we are in
> do_shrink_slab, after that we will reacquire shrinker_rwsem, dec
> the refcount and continue the traversal.
> 
> We also need a wait_queue so that unregister_shrinker can wait for the
> refcnt to become zero. Only after these we can safely remove the
> shrinker from list and idr, and free the shrinker.
> 
> I've reproduced the nfs hang in do_shrink_slab with the patch applied on
> ms kernel, all other mount/unmount pass fine without any hang.
> 
> Here is a reproduction on kernel without patch:
> 
> 1) Setup nfs on server node with some files in it (e.g. 200)
> 
> [server]# cat /etc/exports
> /vz/nfs2 *(ro,no_root_squash,no_subtree_check,async)
> 
> 2) Hard mount it on client node
> 
> [client]# mount -ohard 10.94.3.40:/vz/nfs2 /mnt
> 
> 3) Open some (e.g. 200) files on the mount
> 
> [client]# for i in $(find /mnt/ -type f | head -n 200); \
>   do setsid sleep 1000 &>/dev/null <$i & done
> 
> 4) Kill all openers
> 
> [client]# killall sleep -9
> 
> 5) Put your network cable out on client node
> 
> 6) Drop caches on the client, it will hang on nfs while holding
>   shrinker_rwsem lock for read
> 
> [client]# echo 3 > /proc/sys/vm/drop_caches
> 
>   crash> bt ...
>   PID: 18739  TASK: ...  CPU: 3   COMMAND: "bash"
>    #0 [...] __schedule at ...
>    #1 [...] schedule at ...
>    #2 [...] rpc_wait_bit_killable at ... [sunrpc]
>    #3 [...] __wait_on_bit at ...
>    #4 [...] out_of_line_wait_on_bit at ...
>    #5 [...] _nfs4_proc_delegreturn at ... [nfsv4]
>    #6 [...] nfs4_proc_delegreturn at ... [nfsv4]
>    #7 [...] nfs_do_return_delegation at ... [nfsv4]
>    #8 [...] nfs4_evict_inode at ... [nfsv4]
>    #9 [...] evict at ...
>   #10 [...] dispose_list at ...
>   #11 [...] prune_icache_sb at ...
>   #12 [...] super_cache_scan at ...
>   #13 [...] do_shrink_slab at ...
>   #14 [...] shrink_slab at ...
>   #15 [...] drop_slab_node at ...
>   #16 [...] drop_slab at ...
>   #17 [...] drop_caches_sysctl_handler at ...
>   #18 [...] proc_sys_call_handler at ...
>   #19 [...] vfs_write at ...
>   #20 [...] ksys_write at ...
>   #21 [...] do_syscall_64 at ...
>   #22 [...] entry_SYSCALL_64_after_hwframe at ...
> 
> 7) All other mount/umount activity now hangs with no luck to take
>   shrinker_rwsem for write.
> 
> [client]# mount -t tmpfs tmpfs /tmp
> 
>   crash> bt ...
>   PID: 5464   TASK: ...  CPU: 3   COMMAND: "mount"
>    #0 [...] __schedule at ...
>    #1 [...] schedule at ...
>    #2 [...] rwsem_down_write_slowpath at ...
>    #3 [...] prealloc_shrinker at ...
>    #4 [...] alloc_super at ...
>    #5 [...] sget at ...
>    #6 [...] mount_nodev at ...
>    #7 [...] legacy_get_tree at ...
>    #8 [...] vfs_get_tree at ...
>    #9 [...] do_mount at ...
>   #10 [...] ksys_mount at ...
>   #11 [...] __x64_sys_mount at ...
>   #12 [...] do_syscall_64 at ...
>   #13 [...] entry_SYSCALL_64_after_hwframe at ...
> 
 

I don't think this patch solves the problem, it only fixes one minor symptom of it.
The actual problem here the reclaim hang in the nfs.
It means that any process, including kswapd, may go into nfs inode reclaim and stuck there.

Even mount() itself has GFP_KERNEL allocations in its path, so it still might stuck there even with your patch.

I think this should be handled on nfs/vfs level by making  inode eviction during reclaim more asynchronous.