Message-ID: <158861259989.340223.11154339813692528296.stgit@warthog.procyon.org.uk>
Date:   Mon, 04 May 2020 18:16:39 +0100
From:   David Howells <dhowells@...hat.com>
To:     Trond Myklebust <trondmy@...merspace.com>,
        Anna Schumaker <anna.schumaker@...app.com>,
        Steve French <sfrench@...ba.org>,
        Jeff Layton <jlayton@...hat.com>
Cc:     dhowells@...hat.com, Matthew Wilcox <willy@...radead.org>,
        Alexander Viro <viro@...iv.linux.org.uk>,
        linux-afs@...ts.infradead.org, linux-nfs@...r.kernel.org,
        linux-cifs@...r.kernel.org, ceph-devel@...r.kernel.org,
        v9fs-developer@...ts.sourceforge.net,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: [RFC PATCH 60/61] fscache: Document the new netfs API

Document the netfs parts of the rewritten caching API.

Signed-off-by: David Howells <dhowells@...hat.com>
---

 Documentation/filesystems/caching/netfs-api.txt |  488 ++++++++++++++++++-----
 1 file changed, 372 insertions(+), 116 deletions(-)

diff --git a/Documentation/filesystems/caching/netfs-api.txt b/Documentation/filesystems/caching/netfs-api.txt
index a39de0a4f336..5f8f031b423f 100644
--- a/Documentation/filesystems/caching/netfs-api.txt
+++ b/Documentation/filesystems/caching/netfs-api.txt
@@ -28,11 +28,13 @@ This document contains the following sections:
 	- Index registration
 	- Data file registration
 	- Miscellaneous object registration
-	- Index and data file consistency
 	- Miscellaneous cookie operations
 	- Cookie unregistration
-	- Index invalidation
-	- Data file invalidation
+	- Coherency management
+	- Preparing to access data
+	- I/O request shaping
+	- Submitting I/O requests
+	- Read helpers
 	- FS-Cache specific page flag
 
 
@@ -88,9 +90,9 @@ Indices are used for two purposes:
 
 However, since it's unlikely that any two netfs's are going to want to define
 their index hierarchies in quite the same way, FS-Cache tries to impose as few
-restraints as possible on how an index is structured and where it is placed in
-the tree.  The netfs can even mix indices and data files at the same level, but
-it's not recommended.
+restrictions as possible on how an index is structured and where it is placed
+in the tree.  The netfs can even mix indices and data files at the same level,
+but it's not recommended.
 
 Each index entry consists of a key of indeterminate length plus some auxiliary
 data, also of indeterminate length.
@@ -151,7 +153,7 @@ This takes a text string as the name and returns a representation of a tag.  It
 will never return an error.  It may return a dummy tag, however, if it runs out
 of memory; this will inhibit caching with this tag.
 
-Any representation so obtained must be released by passing it to this function:
+The tag must be released by passing it to this function:
 
 	void fscache_release_cache_tag(struct fscache_cache_tag *tag);
 
@@ -169,114 +171,93 @@ the path to the file:
 
 	struct fscache_cookie *
 	fscache_acquire_cookie(struct fscache_cookie *parent,
-			       const struct fscache_object_def *def,
+			       enum fscache_cookie_type type,
+			       const char *type_name,
+			       u8 advice,
+			       struct fscache_cache_tag *preferred_cache,
 			       const void *index_key,
 			       size_t index_key_len,
 			       const void *aux_data,
 			       size_t aux_data_len,
-			       void *netfs_data,
-			       loff_t object_size,
-			       bool enable);
+			       loff_t object_size);
 
-This function creates an index entry in the index represented by parent,
-filling in the index entry by calling the operations pointed to by def.
+This function creates an index entry in the index represented by parent.  The
+entry is of the given type (FSCACHE_COOKIE_TYPE_*) and should be given a small
+type name for display purposes in /proc (trimmed to 7 chars).
+
+The advice flags inform the cache as to how best to handle the object:
+
+ (*) FSCACHE_ADV_SINGLE_CHUNK - The object is all one piece and will only be
+     written or read as a whole.  Without this, it is assumed that the object
+     can be treated as granular.
+
+ (*) FSCACHE_ADV_WRITE_CACHE - Write local changes to the cache.
+
+ (*) FSCACHE_ADV_WRITE_NOCACHE - Don't write local changes to the cache, but
+     rather invalidate the object and cease caching it.
+
+A tag indicating the preferred cache for this object can be given in
+preferred_cache.  If not given, this will be derived from the parent cookie.
 
 A unique key that represents the object within the parent must be pointed to by
-index_key and is of length index_key_len.
+index_key and is of length index_key_len.  A warning will be logged if a cookie
+with this key already exists and NULL will be returned.
 
 An optional blob of auxiliary data that is to be stored within the cache can be
 pointed to with aux_data and should be of length aux_data_len.  This would
-typically be used for storing coherency data.
+typically be used for storing coherency data and will be checked against
+whatever is in the cache when the cache object is located.
 
-The netfs may pass an arbitrary value in netfs_data and this will be presented
-to it in the event of any calling back.  This may also be used in tracing or
-logging of messages.
+object_size indicates the expected size of the object.  This is used for
+trimming the cache object on release rather than coherency checks.
 
-Note that this function never returns an error - all errors are handled
-internally.  It may, however, return NULL to indicate no cookie.  It is quite
-acceptable to pass this token back to this function as the parent to another
-acquisition (or even to the relinquish cookie, read page and write page
-functions - see below).
+That this function never returns an error - all errors are handled internally.
+It may, however, return NULL to indicate no cookie.  It is quite acceptable to
+pass this token back to this function as the parent to another acquisition (or
+even to the relinquish cookie, read and write functions - see below).
 
-Note also that no indices are actually created in a cache until a non-index
-object needs to be created somewhere down the hierarchy.  Furthermore, an index
-may be created in several different caches independently at different times.
-This is all handled transparently, and the netfs doesn't see any of it.
+Note also that no cache activity will take place until the cache is 'used' to
+access a data object.  This is handled transparently, and the netfs doesn't see
+any of it.
 
 For example, with AFS, a cell would be added to the primary index.  This index
 entry would have a dependent inode containing volume mappings within this cell:
 
 	cell->cache =
 		fscache_acquire_cookie(afs_cache_netfs.primary_index,
-				       &afs_cell_cache_index_def,
+				       FSCACHE_COOKIE_TYPE_INDEX,
+				       "AFS.cell",
+				       0, NULL,
 				       cell->name, strlen(cell->name),
-				       NULL, 0,
-				       cell, 0, true);
+				       NULL, 0, 0);
 
 And then a particular volume could be added to that index by ID, creating
 another index for vnodes (AFS inode equivalents):
 
 	volume->cache =
 		fscache_acquire_cookie(volume->cell->cache,
-				       &afs_volume_cache_index_def,
+				       FSCACHE_COOKIE_TYPE_INDEX,
+				       "AFS.vol",
 				       &volume->vid, sizeof(volume->vid),
-				       NULL, 0,
-				       volume, 0, true);
+				       NULL, 0, 0);
 
 
 ======================
 DATA FILE REGISTRATION
 ======================
 
-The fourth step is to request a data file be created in the cache.  This is
-identical to index cookie acquisition.  The only difference is that the type in
-the object definition should be something other than index type.
+The fourth step is to register a data file with the in the cache.  This is more
+or less identical to index cookie acquisition.
 
 	vnode->cache =
 		fscache_acquire_cookie(volume->cache,
-				       &afs_vnode_cache_object_def,
+				       FSCACHE_COOKIE_TYPE_DATAFILE,
+				       "AFS.vnode",
+				       FSCACHE_ADV_SINGLE_CHUNK,
+				       NULL,
 				       &key, sizeof(key),
 				       &aux, sizeof(aux),
-				       vnode, vnode->status.size, true);
-
-
-=================================
-MISCELLANEOUS OBJECT REGISTRATION
-=================================
-
-An optional step is to request an object of miscellaneous type be created in
-the cache.  This is almost identical to index cookie acquisition.  The only
-difference is that the type in the object definition should be something other
-than index type.  While the parent object could be an index, it's more likely
-it would be some other type of object such as a data file.
-
-	xattr->cache =
-		fscache_acquire_cookie(vnode->cache,
-				       &afs_xattr_cache_object_def,
-				       &xattr->name, strlen(xattr->name),
-				       NULL, 0,
-				       xattr, strlen(xattr->val), true);
-
-Miscellaneous objects might be used to store extended attributes or directory
-entries for example.
-
-
-===============================
-INDEX AND DATA FILE CONSISTENCY
-===============================
-
-To request an update of the index data for an index or other object, the
-following function should be called:
-
-	void fscache_update_cookie(struct fscache_cookie *cookie,
-				   const void *aux_data);
-
-This function will update the cookie's auxiliary data buffer from aux_data if
-that is non-NULL and then schedule this to be stored on disk.  The update
-method in the parent index definition will be called to transfer the data.
-
-Note that partial updates may happen automatically at other times, such as when
-data blocks are added to a data file object.
+				       vnode->status.size);
 
 
 ===============================
@@ -306,65 +287,342 @@ COOKIE UNREGISTRATION
 To get rid of a cookie, this function should be called.
 
 	void fscache_relinquish_cookie(struct fscache_cookie *cookie,
-				       const void *aux_data,
 				       bool retire);
 
-If retire is non-zero, then the object will be marked for recycling, and all
-copies of it will be removed from all active caches in which it is present.
-Not only that but all child objects will also be retired.
+If retire is non-zero, then the object and, if an index, all its children will
+be invalidated, and all copies of will be removed from all active caches in
+which it is present.
 
-If retire is zero, then the object may be available again when next the
+If retire is false, then the object may be available again when next the
 acquisition function is called.  Retirement here will overrule the pinning on a
 cookie.
 
-The cookie's auxiliary data will be updated from aux_data if that is non-NULL
-so that the cache can lazily update it on disk.
-
 One very important note - relinquish must NOT be called for a cookie unless all
-the cookies for "child" indices, objects and pages have been relinquished
-first.
+the cookies for "child" cookies have been relinquished first and the cookie has
+been un-used.
 
 
-==================
-INDEX INVALIDATION
-==================
+====================
+COHERENCY MANAGEMENT
+====================
 
-There is no direct way to invalidate an index subtree.  To do this, the caller
-should relinquish and retire the cookie they have, and then acquire a new one.
+Coherency is managed at a number of points in the lifecycle of a cache object.
+Firstly, as previously mentioned, coherency data is stored in the auxiliary
+data buffer in the cookie.  This is initially filled in at cookie acquisition,
+but can be updated arbitrarily by calling:
 
+	void fscache_update_cookie(struct fscache_cookie *cookie,
+				   const void *aux_data,
+				   const loff_t *object_size);
 
-======================
-DATA FILE INVALIDATION
-======================
+This will update the cookie's auxiliary data buffer from aux_data if that is
+non-NULL and the cached size of the object if object_size is non-NULL.
+
+An opportunity to update the coherency information is also provided in the
+parameters passed to fscache_unuse_cookie().
+
+Beyond that, a data object cookie can also be arbitrarily invalidated at any
+time by calling:
+
+	void fscache_invalidate(struct fscache_cookie *cookie,
+				unsigned int flags);
+
+or by passing true to the 'retire' argument of fscache_relinquish_cookie() when
+the cookie is discarded.  Index cookies may only be retired and not arbitrarily
+invalidated as they refer to an entire subtree.
+
+Invalidation is typically done when a foreign change is detected on the server
+with which a network filesystem is communicating.
+
+With fscache_invalidate(), in-progress I/O operations to the cache object will
+be cancelled as best they can and upcoming I/O operations will be made to wait
+until the invalidation is completed.
+
+The flags parameter can be used to qualify the level of invalidation:
+
+ (*) FSCACHE_INVAL_LIGHT - Invalidation need not do anything if the object has
+     already been invalidated and a temporary object is in use.
+
+
+========================
+PREPARING TO ACCESS DATA
+========================
+
+Before a data object can be accessed, the netfs must tell the cache that it
+wants to actively use an object.  This can be done with:
+
+	void fscache_use_cookie(struct fscache_cookie *cookie,
+				bool will_modify);
+
+This allows the cache to prepare any resources, open files, etc. so that it can
+deal with upcoming requests.  At this time, the objects on disk will be looked
+up and/or created as necessary.
+
+The cookie parameter indicates the object to be accessed and the will_modify
+parameter indicates whether this is for the purpose of caching a local write
+to the netfs (note that *reading* in the netfs may result in writes to the
+cache - this is not what this flag is used for).
+
+When the netfs has finished with an object, it should call the counterpart
+function:
+
+	void fscache_unuse_cookie(struct fscache_cookie *cookie,
+				  const void *aux_data,
+				  const loff_t *object_size);
+
+This drops the activity count on the cookie and provides an opportunity to
+update the cached coherency data and object size.
+
+It is permitted to 'use' a cookie multiple times concurrently - say for every
+open of a netfs file.  The number of users is counted and the object state is
+kept in memory until the users drops off to zero.  The state may be kept around
+for a while after that in case further operations are performed as it may cache
+the result of slow, synchronous filesystem operations.
+
+
+===================
+I/O REQUEST SHAPING
+===================
+
+When making a read or a write on the cache, the request needs to be 'shaped'.
+This requires making the request align to the granularity of the cache (for
+content-tracking purposes) and the I/O block size of the backing filesystem
+(for direct-I/O purposes).
+
+The way to do this is to fill in an instance of the following structure:
+
+	struct fscache_extent {
+		pgoff_t		start;
+		pgoff_t		block_end;
+		pgoff_t		limit;
+		unsigned int	dio_block_size;
+	};
+
+where 'start' and 'block_end' define the pages in a contiguous read that is
+intended to be made.  'limit' indicates the maximum page that can be read and
+should be ULONG_MAX if no limit is set.  Note that both 'block_end' and 'limit'
+should be set to the page index after the last page included in the proposed
+set.  dio_block_size can be ignored at this point.
+
+For instance, to read the first two pages of a file:
+
+	.start		= 0,
+	.block_end	= 2,
+	.limit		= ULONG_MAX,
+
+Then the following function can be called:
+
+	unsigned int fscache_shape_extent(struct fscache_cookie *cookie,
+					  struct fscache_extent *extent,
+					  loff_t i_size, bool for_write);
 
-Sometimes it will be necessary to invalidate an object that contains data.
-Typically this will be necessary when the server tells the netfs of a foreign
-change - at which point the netfs has to throw away all the state it had for an
-inode and reload from the server.
+where 'i_size' indicates the size of the netfs file (proposed size in the case
+of a write that increases it) and 'for_write' indicates if this is for a local
+write to the netfs.
 
-To indicate that a cache object should be invalidated, the following function
-can be called:
+For a granular object (ie. not single-chunk) and if for_write is false, this
+will:
 
-	void fscache_invalidate(struct fscache_cookie *cookie);
+ (1) Round start down and round block_end up or down to the nearest granule
+     boundaries, such that the extent does not cross a data-hole transition in
+     the cache (ie. it's all-read or all-write) and doesn't transgress the
+     stated limit.
 
-All extant storage, retrieval and attribute change ops at this point are
-cancelled and discarded.  Some future operations will be rejected until the
-cache has had a chance to insert a barrier in the operations queue.  After
-that, operations will be queued again behind the invalidation operation.
+ (2) Round limit down to indicate the maximum size of a single read the cache
+     can entertain.
 
-The invalidation operation will perform an attribute change operation and an
-auxiliary data update operation as it is very likely these will have changed.
+In the granular case, if for_write is true, then the writes only need to be
+DIO-aligned as it is assumed that the caller has pre-fetched the block from the
+cache.
+
+Note that the cache granule size may vary between cache objects and may vary
+within a cache object.
+
+For a single-chunk object, start is reduced to 0 and block_end and limit are
+set to the EOF, indicating that the cache wants the whole chunk.
+
+In all cases, on return dio_block_size will be set to the cache's I/O block
+size and the function will return 0, FSCACHE_READ_FROM_CACHE or
+FSCACHE_WRITE_TO_CACHE to indicate whether any I/O can/should be done to/from
+the cache.
+
+
+=======================
+SUBMITTING I/O REQUESTS
+=======================
+
+Data I/O on a cache object is done using a reference-counted request descriptor
+and four functions.  To begin an I/O operation, the descriptor must be
+allocated by the netfs and then initialised by calling:
+
+	void fscache_init_io_request(struct fscache_io_request *req,
+				     struct fscache_cookie *cookie,
+				     const struct fscache_io_request_ops *ops);
+
+This sets fields in the request, allocates any resources it needs and takes a
+reference on the cookie.  Once the references have run out, the cache's
+resources in the I/O request descriptor are cleaned up with:
+
+	void fscache_free_io_request(struct fscache_io_request *req)
+
+I/O is initiated with one of two functions, one to read:
+
+	int fscache_read(struct fscache_io_request *req,
+			 struct iov_iter *iter);
+
+and one to write:
+
+	int fscache_write(struct fscache_io_request *req,
+			  struct iov_iter *iter);
+
+Both of them use an I/O iterator, 'iter', to indicate the data/buffer to be
+written/read into.  Other parameters are defined by the request descriptor,
+including the position and length of the transfer and an optional function to
+be called on completion of the I/O operation:
+
+	struct fscache_io_request {
+		loff_t		pos;
+		loff_t		len;
+		void (*io_done)(struct fscache_io_request *);
+
+A number of fields are then filled in upon completion of the I/O operation:
+
+		loff_t		transferred;
+		short		error;
+		bool		data_from_server;
+		bool		data_from_cache;
+		...
+	};
+
+including the amount of data transferred, any error that occurred and the
+source of the data.
+
+The request descriptor must also be supplied with a table of operations:
+
+	struct fscache_io_request_ops {
+		bool (*is_still_valid)(struct fscache_io_request *);
+		void (*get)(struct fscache_io_request *);
+		void (*put)(struct fscache_io_request *);
+		...
+	};
+
+including methods to find out if a cache operation is still valid after a wait
+to begin an operation and methods to get or put a reference on the operation.
+
+[!] Note that the 'put' method may be called in softirq context and be unable
+    to sleep.  A 'work' member is available in the request struct to allow
+    cleanup to be offloaded to a workqueue.
+
+Note that there are other fields and operations in these structures that
+pertain to the read helpers rather than the above I/O operations and are
+discussed there.
+
+The read and write ops will return 0 on immediate success, -EIOCBQUEUED if the
+operation has been queued for asynchronous I/O, -ESTALE if the cache object
+became invalid, -ENODATA if there's no data to read or -ENOBUFS if caching is
+unavailable.  Apart from -EIOCBQUEUED, the error will also be placed in the
+request descriptor.
+
+
+============
+READ HELPERS
+============
+
+Read helpers are available that do all the work of shaping I/O, reading from
+the cache, falling back to issue a read to the server and writing the result to
+the cache.
+
+The first helper is for reading granulated I/O that is read asynchronously in
+units of pages:
+
+	int fscache_read_helper(struct fscache_io_request *req,
+				struct fscache_extent *extent,
+				struct page **requested_page,
+				struct list_head *pages,
+				enum fscache_read_type type,
+				unsigned int aop_flags);
+
+This must be called with a partially prepared request.  It will fill in the
+pos, len and io_done fields for itself.
+
+This function can be called in one of three modes:
+
+ (1) Multipage read (ie. readpages).
+
+     In this case, 'type' should be set to FSCACHE_READ_PAGE_LIST and 'pages'
+     should be pointed at a list of pages that is in reverse order of index.
+     'extent' should indicate the proposed contiguous run.  requested_page
+     should be NULL.  'aop_flags' is ignored.
+
+ (2) Read to locked page (ie. readpage).
+
+     In this case, 'type' should be set to FSCACHE_READ_LOCKED_PAGE and
+     '*requested_page' should point to the page to be read.  'pages' should be
+     NULL.  'aop_flags' is ignored.
+
+ (3) Prefetch for write (ie. write_begin).
+
+     In this case, 'type' should be set to FSCACHE_READ_FOR_WRITE and
+     '*requested_page' should point to either a pointer to the page to be read
+     or to a NULL pointer that will be filled in.  'pages' should be NULL.
+     'aop_flags' indicates the AOP_FLAG_* to be used when getting the primary
+     page.  On success, '*requested_page' will be filled in with the primary
+     page and the caller will be left needing to unlock it and release the ref
+     on it.
+
+Two additional request operations must also be provided:
+
+	struct fscache_io_request_ops {
+		...
+		void (*issue_op)(struct fscache_io_request *);
+		void (*done)(struct fscache_io_request *);
+	};
+
+'issue_op' will be called to issue a request to the server and 'done' will be
+called upon completion of the read phase of the request (though an asynchronous
+write to the cache may still be outstanding).
+
+fscache_read_helper() will attempt to expand/shape the request to fit the
+cache, filling in holes and padding both sides with extra pages attached to the
+cache, though it will abandon that if it finds an uptodate page in the way.
+
+It will attempt to read from the cache first, and if this is unavailable or
+unsuccessful, it will use 'issue_op' to talk to the server.  In the latter
+case, if successful, it will set the PG_fscache bits on the pages involved and
+begin a write to the cache.
+
+
+The second helper is for single-chunk I/O that is read or written synchronously
+as a complete unit:
+
+	int fscache_read_helper_single(
+		struct fscache_io_request *req,
+		int (*check)(struct fscache_io_request *req));
+
+This also requires a prepared request descriptor, but also requires the caller
+to have done the work in creating and locking those pages.  It waits for
+previous writes to the cache to get out of the way and then issues reads
+against the cache and/or the server and writes the data to the cache much in
+the same way as fscache_read_helper().  Reads will, however, be synchronous.
+Upon a successful return, the data is guaranteed to have finished being read,
+though writing to the cache may still be in progress.
 
 
 ===========================
 FS-CACHE SPECIFIC PAGE FLAG
 ===========================
 
-FS-Cache makes use of a page flag, PG_private_2, for its own purpose.  This is
-given the alternative name PG_fscache.
+FS-Cache makes use of a page flag, PG_private_2, to indicate that a page is
+undergoing write to the cache.  This is given the alternative name PG_fscache.
+
+The netfs should wait on PG_fscache in:
 
-The netfs can use this information in methods such as releasepage() to
-determine whether it needs to uncache a page or update it.
+	releasepage()
+	invalidatepage()
+	write_begin() - after calling fscache_read_helper()
+	writepage()
+	writepages()
+	page_mkwrite()
 
 Furthermore, if this bit is set, releasepage() and invalidatepage() operations
 will be called on a page to get rid of it, even if PG_private is not set.
@@ -372,13 +630,11 @@ will be called on a page to get rid of it, even if PG_private is not set.
 This bit does not overlap with such as PG_private.  This means that FS-Cache
 can be used with a filesystem that uses the block buffering code.
 
-There are a number of operations defined on this flag:
+There are a number of operations defined for this flag:
 
 	int PageFsCache(struct page *page);
 	void SetPageFsCache(struct page *page)
 	void ClearPageFsCache(struct page *page)
 	int TestSetPageFsCache(struct page *page)
 	int TestClearPageFsCache(struct page *page)
-
-These functions are bit test, bit set, bit clear, bit test and set and bit
-test and clear operations on PG_fscache.
+	wait_on_page_fscache(struct page *page);

Powered by blists - more mailing lists