linux-kernel - Re: [PATCH v4 0/6] fs/dcache: Limit # of negative dentries

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <8c7f3be0-0d15-5919-2eb2-f4c9c41ec68f@redhat.com>
Date:   Thu, 5 Oct 2017 11:41:26 -0400
From:   Waiman Long <longman@...hat.com>
To:     Alexander Viro <viro@...iv.linux.org.uk>
Cc:     linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Jan Kara <jack@...e.cz>,
        "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Ingo Molnar <mingo@...nel.org>,
        Miklos Szeredi <mszeredi@...hat.com>,
        Matthew Wilcox <willy@...radead.org>,
        Larry Woodman <lwoodman@...hat.com>,
        James Bottomley <James.Bottomley@...senPartnership.com>,
        "Wangkai (Kevin C)" <wangkai86@...wei.com>
Subject: Re: [PATCH v4 0/6] fs/dcache: Limit # of negative dentries

On 09/18/2017 02:20 PM, Waiman Long wrote:
>  
> A rogue application can potentially create a large number of negative
> dentries in the system consuming most of the memory available even if
> memory controller is enabled to limit memory usage. This can impact
> performance of other applications running on the system.
>
> We have customers seeing soft lockup and unresponsive system when
> tearing down a container because of the large number of negative
> dentries accumulated during its up time that had to be cleaned up at
> exit time when the container's filesystem was unmounted. So we need
> to do something about it.
>
> This patchset introduces changes to the dcache subsystem to limit
> the number of negative dentries allowed to be created thus limiting
> the amount of memory that can be consumed by negative dentries.
>
> Patch 1 just relocates the postion of the dentry_kill() function.
>
> Patch 2 tracks the number of negative dentries present in the LRU
> lists and reports it in /proc/sys/fs/dentry-state.
>
> Patch 3 sets a limit on the number of negative dentries allowable as a
> small percentage (2%) of total system memory. So the larger the system,
> the more negative dentries can be allowed. Once the limit is reached,
> new negative dentries will be killed after use.
>
> Patch 4 enables automatic pruning of least recently used negative
> dentries when it is close to the limit so that we won't end up killing
> recently used negative dentries.
>
> Patch 5 shows the number of forced negative dentry killings in
> /proc/sys/fs/dentry-state.
>
> Patch 6 enables auto-tuneup of free pool negative dentry count to
> no more than the maximum number of positive dentries ever used.
>
> With a 4.13 based kernel, the positive & negative dentries lookup rates
> (lookups per second) after initial boot on a 36-core 50GB memory VM
> with and without the patch were as follows:
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Positive dentry lookup      840269           845762
>   Negative dentry lookup     1903405          1962514
>   Negative dentry creation   6817957          6928768
>
> The last row refers to the creation rate of 1 millions negative
> dentries. With 50GB of memory, 1 millions negative dentries can be
> created with the patched kernel without any pruning or dentry killing.
>
> Ignoring some inherent noise in the test results, there wasn't any
> noticeable difference in term of lookup and negative dentry creation
> performance with or without this patch.
>
> By creating 10 millions negative dentries, however, the performance
> differed.
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Negative dentry creation   651663            190105
>
>
> For the patched kernel, the corresponding dentry-state was:
>
>   1608833	1590416	45	0	1579878	8286952
>
> This was expected as negative dentry creation throttling with forced
> dentry deletion happened in this case.
>
> Running the AIM7 high-systime workload on the same VM, the baseline
> performance was 186770 jobs/min. By running a single-thread rogue
> negative dentry creation program in the background until the patched
> kernel with 2% limit started throttling, the performance was 183746
> jobs/min. On an unpatched kernel with memory almost exhausted and
> memory shrinker was kicked in, the performance was 148997 jobs/min.
>
> So the patch does protect the system from suffering significant
> performance degradation in case a negative dentry creation rogue
> program is runninig in the background.
>
> Waiman Long (6):
>   fs/dcache: Relocate dentry_kill() after lock_parent()
>   fs/dcache: Track & report number of negative dentries
>   fs/dcache: Limit numbers of negative dentries
>   fs/dcache: Enable automatic pruning of negative dentries
>   fs/dcache: Track count of negative dentries forcibly killed
>   fs/dcache: Autotuning of negative dentry limit
>
>  fs/dcache.c              | 462 +++++++++++++++++++++++++++++++++++++++++++----
>  include/linux/dcache.h   |   8 +-
>  include/linux/list_lru.h |   1 +
>  mm/list_lru.c            |   4 +-
>  4 files changed, 439 insertions(+), 36 deletions(-)
>
Ping!

Any comments on this v4 patchset. We have customers seeing problems
because of the unlimited growth of negative dentries. I am hoping that
at least part of this patchset, in some form, can be merged eventually.

Thanks,
Longman