Message-Id: <20221011131552.23833-1-zhujia.zj@bytedance.com>
Date:   Tue, 11 Oct 2022 21:15:47 +0800
From:   Jia Zhu <zhujia.zj@...edance.com>
To:     dhowells@...hat.com, xiang@...nel.org, jefflexu@...ux.alibaba.com
Cc:     linux-cachefs@...hat.com, linux-erofs@...ts.ozlabs.org,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        yinxin.x@...edance.com, Jia Zhu <zhujia.zj@...edance.com>
Subject: [PATCH 0/5] Introduce daemon failover mechanism to recover from crashing

Changes since RFC:
1. Solve the conflict with patch "cachefiles: make on-demand request distribution fairer" 
2. Add some code comments.
3. Optimize some structures to make the code more readable.
4. Extract cachefiles_ondemand_skip_req() from cachefiles_ondemand_daemon_read()
   to make codes more intuitional.

[Background]
============
In ondemand read mode, if user daemon closes anonymous fd(e.g. daemon
crashes), subsequent read and inflight requests based on these fd will
return -EIO.
Even if above mentioned case is tolerable for some individual users, but
when it happenens in real cloud service production environment, such IO
errors will be passed to cloud service users and impact its working jobs.
It's terrible for cloud service stability.

[Design]
========
This patchset introduce three states for ondemand object:
CLOSE: Object which just be allocated or closed by user daemon.
OPEN: Object which related OPEN request has been processed correctly.
REOPENING: Object which has been closed, and is drived to open by a read
request.

[Flow Path]
===========
[Daemon Crash] 
0. Daemon use UDS send/receive fd to keep and pass the fd reference of
   "/dev/cachefiles".
1. User daemon crashes -> restart and recover dev fd's reference.
2. User daemon write "restore" to device.
   2.1 Reset the object's state from CLOSE to OPENING.
   2.2 Init a work which reinit the object and add it to wq. (daemon can
       get rid of kernel space and handle that open request).
3. The user of upper filesystem won't notice that the daemon ever crashed
   since the inflight IO is restored and handled correctly.

[Daemon Close fd]
1. User daemon closes an anonymous fd.
2. User daemon reads a READ request which the associated anonymous fd was
   closed and init a work which re-open the object.
3. User daemon handles above open request normally.
4. The user of upper filesystem won't notice that the daemon ever closed
   any fd since the closed object is re-opened and related request was
   handled correctly.

[Test]
======
There is a testcase for above mentioned scenario.
A user process read the file by fscache ondemand reading.
At the same time, we kill the daemon constantly.
The expected result is that the file read by user is consistent with
original, and the user doesn't notice that daemon has ever been killed.

https://github.com/userzj/demand-read-cachefilesd/commits/failover-test

[GitWeb]
========
https://github.com/userzj/linux/tree/fscache-failover-v2

Jia Zhu (5):
  cachefiles: introduce object ondemand state
  cachefiles: extract ondemand info field from cachefiles_object
  cachefiles: resend an open request if the read request's object is
    closed
  cachefiles: narrow the scope of triggering EPOLLIN events in ondemand
    mode
  cachefiles: add restore command to recover inflight ondemand read
    requests

 fs/cachefiles/daemon.c    |  14 +++-
 fs/cachefiles/interface.c |   6 ++
 fs/cachefiles/internal.h  |  71 ++++++++++++++++-
 fs/cachefiles/ondemand.c  | 155 +++++++++++++++++++++++++++++---------
 4 files changed, 205 insertions(+), 41 deletions(-)

-- 
2.20.1

Powered by blists - more mailing lists