Message-ID: <4F82F16C.7030303@parallels.com>
Date:	Mon, 09 Apr 2012 18:25:48 +0400
From:	Stanislav Kinsbursky <skinsbursky@...allels.com>
To:	Jeff Layton <jlayton@...hat.com>
CC:	"bfields@...ldses.org" <bfields@...ldses.org>,
	"Trond.Myklebust@...app.com" <Trond.Myklebust@...app.com>,
	"linux-nfs@...r.kernel.org" <linux-nfs@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: Grace period

09.04.2012 17:47, Jeff Layton пишет:
> On Mon, 09 Apr 2012 15:24:19 +0400
> Stanislav Kinsbursky<skinsbursky@...allels.com>  wrote:
>
>> 07.04.2012 03:40, bfields@...ldses.org пишет:
>>> On Fri, Apr 06, 2012 at 09:08:26PM +0400, Stanislav Kinsbursky wrote:
>>>> Hello, Bruce.
>>>> Could you, please, clarify this reason why grace list is used?
>>>> I.e. why list is used instead of some atomic variable, for example?
>>>
>>> Like just a reference count?  Yeah, that would be OK.
>>>
>>> In theory it could provide some sort of debugging help.  (E.g. we could
>>> print out the list of "lock managers" currently keeping us in grace.)  I
>>> had some idea we'd make those lock manager objects more complicated, and
>>> might have more for individual containerized services.
>>
>> Could you share this idea, please?
>>
>> Anyway, I have nothing against lists. Just was curious, why it was used.
>> I added Trond and lists to this reply.
>>
>> Let me explain, what is the problem with grace period I'm facing right know, and
>> what I'm thinking about it.
>> So, one of the things to be containerized during "NFSd per net ns" work is the
>> grace period, and these are the basic components of it:
>> 1) Grace period start.
>> 2) Grace period end.
>> 3) Grace period check.
>> 3) Grace period restart.
>>
>> So, the simplest straight-forward way is to make all internal stuff:
>> "grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and
>> "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be
>> per-net as well.
>> In this case:
>> 1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has
>> to be moves there from "lockd()") and "nfs4_state_start()".
>> 2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to
>> be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
>> 3) Check - looks easy. There is either svc_rqst or net context can be passed to
>> function.
>> 4) Restart - this is a tricky place. It would be great to restart grace period
>> only for the networks namespace of the sender of the kill signal. So, the idea
>> is to check siginfo_t for the pid of sender, then try to locate the task, and if
>> found, then get sender's networks namespace, and restart grace period only for
>> this namespace (of course, if lockd was started for this namespace - see below).
>>
>> If task not found, of it's lockd wasn't started for it's namespace, then grace
>> period can be either restarted for all namespaces, of just silently dropped.
>> This is the place where I'm not sure, how to do. Because calling grace period
>> for all namespaces will be overkill...
>>
>> There also another problem with the "task by pid" search, that found task can be
>> actually not sender (which died already), but some other new task with the same
>> pid number. In this case, I think, we can just neglect this probability and
>> always assume, that we located sender (if, of course, lockd was started for
>> sender's network namespace).
>>
>> Trond, Bruce, could you, please, comment this ideas?
>>
>
> I can comment and I'm not sure that will be sufficient.
>

Hi, Jeff. Thanks for the comment.

> The grace period has a particular purpose. It keeps nfsd or lockd from
> handing out stateful objects (e.g. locks) before clients have an
> opportunity to reclaim them. Once the grace period expires, there is no
> more reclaim allowed and "normal" lock and open requests can proceed.
>
> Traditionally, there has been one nfsd or lockd "instance" per host.
> With that, we were able to get away with a relatively simple-minded
> approach of a global grace period that's gated on nfsd or lockd's
> startup and shutdown.
>
> Now, you're looking at making multiple nfsd or lockd "instances". Does
> it make sense to make this a per-net thing? Here's a particularly
> problematic case to illustrate what I mean:
>
> Suppose I have a filesystem that's mounted and exported in two
> different containers. You start up one container and then 60s later,
> start up the other. The grace period expires in the first container and
> that nfsd hands out locks that conflict with some that have not been
> reclaimed yet in the other.
>
> Now, we can just try to say "don't export the same fs from more than
> one container". But we all know that people will do it anyway, since
> there's nothing that really stops you from doing so.
>

Yes, I see. But situation you described is existent already.
I.e. you can replace containers with the same file system by two nodes, sharing 
the same distributed file system (like Lustre and GPFS), and you'll experience 
the same problem in such case.

> What probably makes more sense is making the grace period a per-sb
> property, and coming up with a set of rules for the fs going into and
> out of "grace" status.
>
> Perhaps a way for different net namespaces to "subscribe" to a
> particular fs, and don't take the fs out of grace until all of the
> grace period timers pop? If a fs attempts to subscribe after the fs
> comes out of grace, then its subscription would be denied and reclaim
> attempts would get NFS4ERR_NOGRACE or the NLM equivalent.
>

This raises another problem. Imagine, that grace period has elapsed for some 
container and then you start nfsd in another one. New grace period will affect 
all both of them. And that's even worse from my pow.

-- 
Best regards,
Stanislav Kinsbursky
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists