netdev - [PATCH 1/1] SUNRPC: Handle major timeout in xprt_adjust

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:   Sun,  4 Apr 2021 21:29:26 -0400
From:   Chris Dion <Christopher.Dion@...l.com>
To:     linux-nfs@...r.kernel.org
Cc:     trond.myklebust@...merspace.com,
        Chris Dion <Christopher.Dion@...l.com>,
        "J. Bruce Fields" <bfields@...ldses.org>,
        Chuck Lever <chuck.lever@...cle.com>,
        Anna Schumaker <anna.schumaker@...app.com>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Olga Kornievskaia <olga.kornievskaia@...il.com>,
        netdev@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: [PATCH 1/1] SUNRPC: Handle major timeout in xprt_adjust_timeout()

Currently if a major timeout value is reached, but the minor value has
not been reached, an ETIMEOUT will not be sent back to the caller.
This can occur if the v4 server is not responding to requests and
retrans is configured larger than the default of two.

For example, A TCP mount with a configured timeout value of 50 and a
retransmission count of 3 to a v4 server which is not responding:

1. Initial value and increment set to 5s, maxval set to 20s, retries at 3
2. Major timeout is set to 20s, minor timeout set to 5s initially
3. xport_adjust_timeout() is called after 5s, retry with 10s timeout,
   minor timeout is bumped to 10s
4. And again after another 10s, 15s total time with minor timeout set
   to 15s
5. After 20s total time xport_adjust_timeout is called as major timeout is
   reached, but skipped because the minor timeout is not reached
       - After this time the cpu spins continually calling
       	 xport_adjust_timeout() and returning 0 for 10 seconds.
	 As seen on perf sched:
   	 39243.913182 [0005]  mount.nfs[3794] 4607.938      0.017   9746.863
6. This continues until the 15s minor timeout condition is reached (in
   this case for 10 seconds). After which the ETIMEOUT is processed
   back to the caller, the cpu spinning stops, and normal operations
   continue

Fixes: 7de62bc09fe6 ("SUNRPC dont update timeout value on connection reset")
Signed-off-by: Chris Dion <Christopher.Dion@...l.com>
---
Hello,

We recently have noticed an issue where we see a cpu spinning for a few
seconds at a time bouncing between xprt_adjust_timeout() and call_connect().
This is ocurring with an v4 client mounting a non-responsive server with a
low timeout and retransmissions set. In this case if the major timeout is
not serviced there appears to be a spin/bouncing between these functions
until such timeout is returned. It made sense to service the major timeout
in this scenario.  I am not clear on the implications on the original
implementation here but I will submit for your review.

Regards,
Chris

 net/sunrpc/xprt.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 691ccf8049a4..7fe975c1000a 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -698,9 +698,9 @@ int xprt_adjust_timeout(struct rpc_rqst *req)
 	const struct rpc_timeout *to = req->rq_task->tk_client->cl_timeout;
 	int status = 0;
 
-	if (time_before(jiffies, req->rq_minortimeo))
-		return status;
 	if (time_before(jiffies, req->rq_majortimeo)) {
+		if (time_before(jiffies, req->rq_minortimeo))
+			return status;
 		if (to->to_exponential)
 			req->rq_timeout <<= 1;
 		else
-- 
2.18.2