netdev - Re: [PATCH RFC 00/30] Overhaul NFSD filecache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 23 Jun 2022 01:01:25 +0000
From:   Chuck Lever III <chuck.lever@...cle.com>
To:     Dave Chinner <david@...morbit.com>
CC:     Wang Yugui <wangyugui@...-tech.com>,
        Linux NFS Mailing List <linux-nfs@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "tgraf@...g.ch" <tgraf@...g.ch>, Jeff Layton <jlayton@...hat.com>
Subject: Re: [PATCH RFC 00/30] Overhaul NFSD filecache



> On Jun 22, 2022, at 8:21 PM, Dave Chinner <david@...morbit.com> wrote:
> 
> On Wed, Jun 22, 2022 at 07:04:39PM +0000, Chuck Lever III wrote:
>>> more detail in attachment file(531.dmesg)
>>> 
>>> local.config of fstests:
>>> 	export NFS_MOUNT_OPTIONS="-o rw,relatime,vers=4.2,nconnect=8"
>>> changes of generic/531
>>> 	max_allowable_files=$(( 1 * 1024 * 1024 / $nr_cpus / 2 ))
>> 
>> Changed from:
>> 
>> 	max_allowable_files=$(( $(cat /proc/sys/fs/file-max) / $nr_cpus / 2 ))
>> 
>> For my own information, what's $nr_cpus in your test?
>> 
>> Aside from the max_allowable_files setting, can you tell how the
>> test determines when it should stop creating files? Is it looking
>> for a particular error code from open(2), for instance?
>> 
>> On my client:
>> 
>> [cel@...isot generic]$ cat /proc/sys/fs/file-max
>> 9223372036854775807
>> [cel@...isot generic]$
> 
> $ echo $((2**63 - 1))
> 9223372036854775807
> 
> i.e. LLONG_MAX, or "no limit is set".
> 
>> I wonder if it's realistic to expect an NFSv4 server to support
>> that many open files. Is 9 quintillion files really something
>> I'm going to have to engineer for, or is this just a crazy
>> test?
> 
> The test does not use the value directly - it's a max value for
> clamping:
> 
> max_files=$((50000 * LOAD_FACTOR))
> max_allowable_files=$(( $(cat /proc/sys/fs/file-max) / $nr_cpus / 2 ))
> test $max_allowable_files -gt 0 && test $max_files -gt $max_allowable_files && \
>        max_files=$max_allowable_files
> ulimit -n $max_files
> 
> i.e. the result should be
> 
> max_files = max(50000, max_allowable_files)
> 
> So the test should only be allowing 50,000 open unlinked files to be
> created before unmounting.

Looking at my testing, it's ~50,000 per worker thread, and there are
2 workers per physical core on the client. But thankfully, this is
much smaller than 9 quintillion.


> Which means there's lots of silly
> renaming going on at the client and so the server is probably seeing
> 100,000 unique file handles across the test....
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@...morbit.com

--
Chuck Lever