linux-kernel - Re: [RFC PATCH] getvalues(2) prototype

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d0e2573a-7736-bb3e-9f6a-5fa25e6d31a2@ddn.com>
Date:   Wed, 23 Mar 2022 11:26:11 +0100
From:   Bernd Schubert <bschubert@....com>
To:     Greg KH <gregkh@...uxfoundation.org>,
        Miklos Szeredi <mszeredi@...hat.com>
Cc:     linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-api@...r.kernel.org, linux-man@...r.kernel.org,
        linux-security-module@...r.kernel.org, Karel Zak <kzak@...hat.com>,
        Ian Kent <raven@...maw.net>,
        David Howells <dhowells@...hat.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Al Viro <viro@...iv.linux.org.uk>,
        Christian Brauner <christian@...uner.io>,
        Amir Goldstein <amir73il@...il.com>,
        James Bottomley <James.Bottomley@...senpartnership.com>,
        Dharmendra Singh <dsingh@....com>
Subject: Re: [RFC PATCH] getvalues(2) prototype

On 3/23/22 08:16, Greg KH wrote:
> On Tue, Mar 22, 2022 at 08:27:12PM +0100, Miklos Szeredi wrote:
>> Add a new userspace API that allows getting multiple short values in a
>> single syscall.
>>
>> This would be useful for the following reasons:
>>
>> - Calling open/read/close for many small files is inefficient.  E.g. on my
>>    desktop invoking lsof(1) results in ~60k open + read + close calls under
>>    /proc and 90% of those are 128 bytes or less.
> 
> As I found out in testing readfile():
> 	https://lore.kernel.org/r/20200704140250.423345-1-gregkh@linuxfoundation.org
> 
> microbenchmarks do show a tiny improvement in doing something like this,
> but that's not a real-world application.
> 
> Do you have anything real that can use this that shows a speedup?

Add in network file systems. Demonstrating that this is useful locally 
and with micro benchmarks - yeah, helps a bit to make it locally faster. 
But the real case is when thousands of clients are handled by a few 
network servers. Even reducing wire latency for a single client would 
make a difference here.

There is a bit of chicken-egg problem - it is a bit of work to add to 
file systems like NFS (or others that are not the kernel), but the work 
won't be made there before there is no syscall for it. To demonstrate it 
on NFS one also needs a an official protocol change first. And then 
applications also need to support that new syscall first.
I had a hard time explaining weather physicist back in 2009 that it is 
not a good idea to have millions of 512B files on  Lustre. With recent 
AI workload this gets even worse.

This is the same issue in fact with the fuse patches we are creating 
(https://lwn.net/Articles/888877/). Miklos asked for benchmark numbers - 
we can only demonstrate slight effects locally, but out goal is in fact 
to reduce network latencies and server load.

- Bernd