linux-kernel - Re: 3.0+ NFS issues (bisected)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <502E7F84.3060003@msgid.tls.msk.ru>
Date:	Fri, 17 Aug 2012 21:29:40 +0400
From:	Michael Tokarev <mjt@....msk.ru>
To:	"J. Bruce Fields" <bfields@...ldses.org>
CC:	"Myklebust, Trond" <Trond.Myklebust@...app.com>,
	"linux-nfs@...r.kernel.org" <linux-nfs@...r.kernel.org>,
	Linux-kernel <linux-kernel@...r.kernel.org>,
	Eric Dumazet <eric.dumazet@...il.com>
Subject: Re: 3.0+ NFS issues (bisected)

On 17.08.2012 21:26, Michael Tokarev wrote:
> On 17.08.2012 21:18, J. Bruce Fields wrote:
>> On Fri, Aug 17, 2012 at 09:12:38PM +0400, Michael Tokarev wrote:
> []
>>> So we're calling svc_recv in a tight loop, eating
>>> all available CPU.  (The above is with just 2 nfsd
>>> threads).
>>>
>>> Something is definitely wrong here.  And it happens mure more
>>> often after the mentioned commit (f03d78db65085).
>>
>> Oh, neat.  Hm.  That commit doesn't really sound like the cause, then.
>> Is that busy-looping reproduceable on kernels before that commit?
> 
> Note I bisected this issue to this commit.  I haven't seen it
> happening before this commit, and reverting it from 3.0 or 3.2
> kernel makes the problem to go away.
> 
> I guess it is looping there:
> 
> 
> net/sunrpc/svc_xprt.c:svc_recv()
> ...
>         len = 0;
> ...
>         if (test_bit(XPT_LISTENER, &xprt->xpt_flags)) {
> ...
>         } else if (xprt->xpt_ops->xpo_has_wspace(xprt)) {  <=== here -- has no wspace due to memory...
> ...  len = <something>
>         }
> 
>         /* No data, incomplete (TCP) read, or accept() */
>         if (len == 0 || len == -EAGAIN)
>                 goto out;
> ...
> out:
>         rqstp->rq_res.len = 0;
>         svc_xprt_release(rqstp);
>         return -EAGAIN;
> }
> 
> I'm trying to verify this theory...

Yes.  I inserted a printk there, and all these million times while
we're waiting in this EAGAIN loop, this printk is triggering:

....
[21052.533053]  svc_recv: !has_wspace
[21052.533070]  svc_recv: !has_wspace
[21052.533087]  svc_recv: !has_wspace
[21052.533105]  svc_recv: !has_wspace
[21052.533122]  svc_recv: !has_wspace
[21052.533139]  svc_recv: !has_wspace
[21052.533156]  svc_recv: !has_wspace
[21052.533174]  svc_recv: !has_wspace
[21052.533191]  svc_recv: !has_wspace
[21052.533208]  svc_recv: !has_wspace
[21052.533226]  svc_recv: !has_wspace
[21052.533244]  svc_recv: !has_wspace
[21052.533265] calling svc_recv: 1228163 times (err=-4)
[21052.533403] calling svc_recv: 1226616 times (err=-4)
[21052.534520] nfsd: last server has exited, flushing export cache

(I stopped nfsd since it was flooding the log).

I can only guess that before that commit, we always had space,
now we don't anymore, and are looping like crazy.

Thanks,

/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/