Date:   Mon, 28 Aug 2017 14:59:15 -0400
From:   Waiman Long <longman@...hat.com>
To:     Alexander Viro <viro@...iv.linux.org.uk>,
        Jonathan Corbet <corbet@....net>
Cc:     linux-kernel@...r.kernel.org, linux-doc@...r.kernel.org,
        linux-fsdevel@...r.kernel.org,
        "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Ingo Molnar <mingo@...nel.org>,
        Miklos Szeredi <mszeredi@...hat.com>,
        Matthew Wilcox <willy@...radead.org>,
        Larry Woodman <lwoodman@...hat.com>,
        James Bottomley <James.Bottomley@...senPartnership.com>,
        "Wangkai (Kevin,C)" <wangkai86@...wei.com>
Subject: Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries

On 08/28/2017 01:58 PM, Waiman Long wrote:
> On 07/28/2017 02:34 PM, Waiman Long wrote:
>>  v2->v3:
>>   - Add a faster pruning rate when the free pool is closed to depletion.
>>   - As suggested by James Bottomley, add an artificial delay waiting
>>     loop before killing a negative dentry and properly clear the
>>     DCACHE_KILL_NEGATIVE flag if killing doesn't happen.
>>   - Add a new patch to track number of negative dentries that are
>>     forcifully killed.
>>
>>  v1->v2:
>>   - Move the new nr_negative field to the end of dentry_stat_t structure
>>     as suggested by Matthew Wilcox.
>>   - With the help of Miklos Szeredi, fix incorrect locking order in
>>     dentry_kill() by using lock_parent() instead of locking the parent's
>>     d_lock directly.
>>   - Correctly account for positive to negative dentry transitions.
>>   - Automatic pruning of negative dentries will now ignore the reference
>>     bit in negative dentries but not the regular shrinking.
>>
>> A rogue application can potentially create a large number of negative
>> dentries in the system consuming most of the memory available. This
>> can impact performance of other applications running on the system.
>>
>> This patchset introduces changes to the dcache subsystem to limit
>> the number of negative dentries allowed to be created thus limiting
>> the amount of memory that can be consumed by negative dentries.
>>
>> Patch 1 tracks the number of negative dentries used and disallow
>> the creation of more when the limit is reached.
>>
>> Patch 2 enables /proc/sys/fs/dentry-state to report the number of
>> negative dentries in the system.
>>
>> Patch 3 enables automatic pruning of negative dentries when it is
>> close to the limit so that we won't end up killing recently used
>> negative dentries.
>>
>> Patch 4 prevents racing between negative dentry pruning and umount
>> operation.
>>
>> Patch 5 shows the number of forced negative dentry killings in
>> /proc/sys/fs/dentry-state. End users can then tune the neg_dentry_pc=
>> kernel boot parameter if they want to reduce forced negative dentry
>> killings.
>>
>> Waiman Long (5):
>>   fs/dcache: Limit numbers of negative dentries
>>   fs/dcache: Report negative dentry number in dentry-state
>>   fs/dcache: Enable automatic pruning of negative dentries
>>   fs/dcache: Protect negative dentry pruning from racing with umount
>>   fs/dcache: Track count of negative dentries forcibly killed
>>
>>  Documentation/admin-guide/kernel-parameters.txt |   7 +
>>  fs/dcache.c                                     | 451 ++++++++++++++++++++++--
>>  include/linux/dcache.h                          |   8 +-
>>  include/linux/list_lru.h                        |   1 +
>>  mm/list_lru.c                                   |   4 +-
>>  5 files changed, 435 insertions(+), 36 deletions(-)
>>
> With a 4.13 based kernel, the positive & negative dentries lookup rates
> (lookups per second) after initial boot on a 32GB memory VM with and
> without the patch were as follows:
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Positive dentry lookup      844881           842618
>   Negative dentry lookup     1865158          1901875
>   Negative dentry creation    609887           617215
>
> The last row refers to the creation rate of 10 millions negative
> dentries with the negative dentry limit set to 50% (> 80M dentries).
> Ignoring some inherent noise in the test results, there wasn't any
> noticeable difference in term of lookup and negative dentry creation
> performance with or without this patch.
>
> If the limit was set to 5% (the default), the 10M negative dentry
> creation rate dropped to 199565 and the dentry-state was:
>
> 2344754 2326486 45      0       2316533 7494261
>
> This was expected as negative dentry creation throttling with forced
> dentry deletion happened in this case.
>
> IOW, this patch does not cause any regression in term of lookup and
> negative dentry creation performance as long as the limit hasn't been
> reached.

Another performance data point about running the AIM7 highsystime
workload on a 36-core 32G VM is as follows:

Running the AIM7 high-systime workload on the VM, the baseline
performance was 186770 jobs/min. By running a single-thread rogue
negative dentry creation program in the background until the patched
kernel with 5% limit started throttling, the performance was 183746
jobs/min. On an unpatched kernel with memory almost exhausted and
memory shrinker was kicked in, the performance was 148997 jobs/min.

So the patch does protect the system from suffering significant
performance degradation in case a negative dentry creation rogue
program is runninig in the background.

Cheers,
Longman

Powered by blists - more mailing lists