| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <161161025063.2537118.2009249444682241405.stgit@warthog.procyon.org.uk>
Date: Mon, 25 Jan 2021 21:30:50 +0000
From: David Howells <dhowells@...hat.com>
To: Trond Myklebust <trondmy@...merspace.com>,
Anna Schumaker <anna.schumaker@...app.com>,
Steve French <sfrench@...ba.org>,
Dominique Martinet <asmadeus@...ewreck.org>
Cc: linux-afs@...ts.infradead.org,
Dave Wysochanski <dwysocha@...hat.com>,
Matthew Wilcox <willy@...radead.org>,
"Matthew Wilcox (Oracle)" <willy@...radead.org>,
dhowells@...hat.com, Jeff Layton <jlayton@...hat.com>,
David Wysochanski <dwysocha@...hat.com>,
Matthew Wilcox <willy@...radead.org>,
Alexander Viro <viro@...iv.linux.org.uk>,
linux-cachefs@...hat.com, linux-afs@...ts.infradead.org,
linux-nfs@...r.kernel.org, linux-cifs@...r.kernel.org,
ceph-devel@...r.kernel.org, v9fs-developer@...ts.sourceforge.net,
linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: [PATCH 00/32] Network fs helper library & fscache kiocb API [ver #2]
Here's a set of patches to do two things:
(1) Add a helper library to handle the new VM readahead interface. This
is intended to be used unconditionally by the filesystem (whether or
not caching is enabled) and provides a common framework for doing
caching, transparent huge pages and, in the future, possibly fscrypt
and read bandwidth maximisation. It also allows the netfs and the
cache to align, expand and slice up a read request from the VM in
various ways; the netfs need only provide a function to read a stretch
of data to the pagecache and the helper takes care of the rest.
(2) Add an alternative fscache/cachfiles I/O API that uses the kiocb
facility to do async DIO to transfer data to/from the netfs's pages,
rather than using readpage with wait queue snooping on one side and
vfs_write() on the other. It also uses less memory, since it doesn't
do buffered I/O on the backing file.
Note that this uses SEEK_HOLE/SEEK_DATA to locate the data available
to be read from the cache. Whilst this is an improvement from the
bmap interface, it still has a problem with regard to a modern
extent-based filesystem inserting or removing bridging blocks of
zeros. Fixing that requires a much greater overhaul.
This is a step towards overhauling the fscache API. The change is opt-in
on the part of the network filesystem. A netfs should not try to mix the
old and the new API because of conflicting ways of handling pages and the
PG_fscache page flag and because it would be mixing DIO with buffered I/O.
Further, the helper library can't be used with the old API.
This does not change any of the fscache cookie handling APIs or the way
invalidation is done.
In the near term, I intend to deprecate and remove the old I/O API
(fscache_allocate_page{,s}(), fscache_read_or_alloc_page{,s}(),
fscache_write_page() and fscache_uncache_page()) and eventually replace
most of fscache/cachefiles with something simpler and easier to follow.
The patchset contains five parts:
(1) Some helper patches, including provision of an ITER_XARRAY iov
iterator and a function to do readahead expansion.
(2) Patches to add the netfs helper library.
(3) A patch to add the fscache/cachefiles kiocb API
(4) Patches to add support in AFS for this.
(5) Patches from David Wysochanski to add support in NFS for this.
Jeff Layton also has patches for Ceph for this, though they're not included
on this branch.
With this, AFS without a cache passes all expected xfstests; with a cache,
there's an extra failure, but that's also there before these patches.
Fixing that probably requires a greater overhaul. Ceph and NFS also pass
the expected tests.
These patches can be found also on:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=fscache-netfs-lib
Changes
=======
(v2) Fix some bugs and add NFS support.
David
---
Dave Wysochanski (8):
NFS: Clean up nfs_readpage() and nfs_readpages()
NFS: In nfs_readpage() only increment NFSIOS_READPAGES when read succeeds
NFS: Refactor nfs_readpage() and nfs_readpage_async() to use nfs_readdesc
NFS: Call readpage_async_filler() from nfs_readpage_async()
NFS: Add nfs_pageio_complete_read() and remove nfs_readpage_async()
NFS: Allow internal use of read structs and functions
NFS: Convert to the netfs API and nfs_readpage to use netfs_readpage
NFS: Convert readpage to readahead and use netfs_readahead for fscache
David Howells (24):
iov_iter: Add ITER_XARRAY
vm: Add wait/unlock functions for PG_fscache
mm: Implement readahead_control pageset expansion
vfs: Export rw_verify_area() for use by cachefiles
netfs: Make a netfs helper module
netfs: Provide readahead and readpage netfs helpers
netfs: Add tracepoints
netfs: Gather stats
netfs: Add write_begin helper
netfs: Define an interface to talk to a cache
fscache, cachefiles: Add alternate API to use kiocb for read/write to cache
afs: Disable use of the fscache I/O routines
afs: Pass page into dirty region helpers to provide THP size
afs: Print the operation debug_id when logging an unexpected data version
afs: Move key to afs_read struct
afs: Don't truncate iter during data fetch
afs: Log remote unmarshalling errors
afs: Set up the iov_iter before calling afs_extract_data()
afs: Use ITER_XARRAY for writing
afs: Wait on PG_fscache before modifying/releasing a page
afs: Extract writeback extension into its own function
afs: Prepare for use of THPs
afs: Use the fs operation ops to handle FetchData completion
afs: Use new fscache read helper API
fs/Kconfig | 1 +
fs/Makefile | 1 +
fs/afs/Kconfig | 1 +
fs/afs/dir.c | 225 ++++---
fs/afs/file.c | 472 ++++----------
fs/afs/fs_operation.c | 4 +-
fs/afs/fsclient.c | 108 +--
fs/afs/inode.c | 7 +-
fs/afs/internal.h | 57 +-
fs/afs/rxrpc.c | 150 ++---
fs/afs/write.c | 610 +++++++++--------
fs/afs/yfsclient.c | 82 +--
fs/cachefiles/Makefile | 1 +
fs/cachefiles/interface.c | 5 +-
fs/cachefiles/internal.h | 9 +
fs/cachefiles/rdwr2.c | 412 ++++++++++++
fs/fscache/Kconfig | 1 +
fs/fscache/Makefile | 3 +-
fs/fscache/internal.h | 3 +
fs/fscache/page.c | 2 +-
fs/fscache/page2.c | 116 ++++
fs/fscache/stats.c | 1 +
fs/internal.h | 5 -
fs/netfs/Kconfig | 23 +
fs/netfs/Makefile | 5 +
fs/netfs/internal.h | 97 +++
fs/netfs/read_helper.c | 1155 +++++++++++++++++++++++++++++++++
fs/netfs/stats.c | 59 ++
fs/nfs/file.c | 2 +-
fs/nfs/fscache.c | 206 +++---
fs/nfs/fscache.h | 66 +-
fs/nfs/internal.h | 8 +
fs/nfs/pagelist.c | 2 +
fs/nfs/read.c | 244 ++++---
fs/read_write.c | 1 +
include/linux/fs.h | 1 +
include/linux/fscache-cache.h | 4 +
include/linux/fscache.h | 28 +-
include/linux/netfs.h | 167 +++++
include/linux/nfs_fs.h | 5 +-
include/linux/nfs_iostat.h | 2 +-
include/linux/nfs_page.h | 1 +
include/linux/nfs_xdr.h | 1 +
include/linux/pagemap.h | 16 +
include/net/af_rxrpc.h | 2 +-
include/trace/events/afs.h | 74 +--
include/trace/events/netfs.h | 201 ++++++
mm/filemap.c | 18 +
mm/readahead.c | 70 ++
net/rxrpc/recvmsg.c | 9 +-
50 files changed, 3457 insertions(+), 1286 deletions(-)
create mode 100644 fs/cachefiles/rdwr2.c
create mode 100644 fs/fscache/page2.c
create mode 100644 fs/netfs/Kconfig
create mode 100644 fs/netfs/Makefile
create mode 100644 fs/netfs/internal.h
create mode 100644 fs/netfs/read_helper.c
create mode 100644 fs/netfs/stats.c
create mode 100644 include/linux/netfs.h
create mode 100644 include/trace/events/netfs.h
Powered by blists - more mailing lists