linux-kernel - Re: NFS regression? Odd delays and lockups accessing an NFS export.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <48B567F5.2090605@opengridcomputing.com>
Date:	Wed, 27 Aug 2008 09:43:01 -0500
From:	Tom Tucker <tom@...ngridcomputing.com>
To:	"J. Bruce Fields" <bfields@...ldses.org>
CC:	Trond Myklebust <trond.myklebust@....uio.no>,
	Ian Campbell <ijc@...lion.org.uk>,
	John Ronciak <john.ronciak@...il.com>,
	Grant Coady <gcoady.lk@...il.com>,
	linux-kernel@...r.kernel.org, neilb@...e.de,
	linux-nfs@...r.kernel.org,
	Jeff Kirsher <jeffrey.t.kirsher@...el.com>,
	Jesse Brandeburg <jesse.brandeburg@...el.com>,
	Bruce Allan <bruce.w.allan@...el.com>,
	PJ Waskiewicz <peter.p.waskiewicz.jr@...el.com>,
	John Ronciak <john.ronciak@...el.com>,
	e1000-devel@...ts.sourceforge.net
Subject: Re: NFS regression? Odd delays and lockups accessing an NFS export.

J. Bruce Fields wrote:
> On Mon, Aug 25, 2008 at 11:04:08AM -0500, Tom Tucker wrote:
>> Trond Myklebust wrote:
>>> On Sun, 2008-08-24 at 23:09 +0100, Ian Campbell wrote:
>>>> (added some quoting from previous mail to save replying twice)
>>>>
>>>> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
>>>>> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
>>>>>> >From the tcpdump, it looks as if the NFS server is failing to close the
>>>>>> socket, when the client closes its side. You therefore end up getting
>>>>>> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
>>>>>>
>>>>>> Is the server keeping the client in this state for a very long
>>>>>> period?
>>>> Well, it had been around an hour and a half on this occasion. Next time
>>>> it happens I can wait longer but I'm pretty sure I've come back from
>>>> time away and it's been wedged for at least a day. How long would you
>>>> expect it to remain in this state for?
>>> The server should ideally start to close the socket as soon as it
>>> receives the FIN from the client. I'll have a look at the code.
>>>
>> I don't think it should matter how long the connection stays in FIN WAIT,
>> the client should reconnect anyway.
>>
>> Since the client seems to be the variable, I would think it might be an
>> issue with the client reconnect logic?
>>
>> That said, 2.6.25 is when the server side transport switch logic went in.
> 
> Any chance you could help Trond figure out why the server might be doing
> this?
> 
> If not, I'll get to it, but not as soon as I should.
> 

Sure. I've actually tried to reproduce it here unsuccessfully.

As a starter, I would suggest turning on transport debugging:

# echo 256 > /proc/sys/sunrpc/rpc_debug

Here are my thoughts on how it is supposed to work. Trond if this doesn't match
your understanding, please let me know.

For the case where the client closes the connection first
because it's timeout is shorter (5min vs. 6), the sequence of events should be:


client					server

close
sends FIN goes to FIN-WAIT-1
					receives FIN replies with ACK
					and goes to CLOSE-WAIT
receives ACK goes to FIN-WAIT-2
					calls tcp_state_change callback on socket
					svc_tcp_state_change sets XPT_CLOSE on
					the transport and enqueues transport for
					servicing by svc thread.

					thread gets dequeued, calls svc_recv
					svc_recv sees XPT_CLOSE and calls
					svc_xprt_destroy that closes the socket

					TCP sends FIN/ACK after close
receives FIN/ACK goes to TIME-WAIT

since state is TIME-WAIT and
reuse port is set, we can re-connect

There's a couple places we could be getting stuck:

- The state-change callback never gets called.
- The state-change callback gets called but svsk is null and it ignores the event
- svc_tcp_state_change enqueues the transport, but due to a race bug, the transport
   doesn't actually get queued and since there is no activity it never closes
- something else

The message log with RPCDBG_SVCXPRT (256) enabled as shown above would help figure
out where it's happening.

If Ian is willing to create the log (or already has one), I'm certainly willing to
look at it.

Tom


> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/