linux-kernel - Re: Question: How to customize retransmission timeout of unacknowledged NFS v3 TCP packet?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAPKjjnrez+hip+VBVrLT_g6Uzxd5DGSCoCSXQnRLx-qXT09yQA@mail.gmail.com>
Date: Wed, 29 May 2024 18:18:54 +0800
From: Zhitao Li <zhitao.li@...rtx.com>
To: Trond Myklebust <trond.myklebust@...merspace.com>, Anna Schumaker <anna@...nel.org>, 
	Chuck Lever <chuck.lever@...cle.com>, Jeff Layton <jlayton@...nel.org>, Neil Brown <neilb@...e.de>, 
	Olga Kornievskaia <kolga@...app.com>, Dai Ngo <Dai.Ngo@...cle.com>, Tom Talpey <tom@...pey.com>
Cc: linux-nfs@...r.kernel.org, linux-kernel@...r.kernel.org, 
	Ping Huang <huangping@...rtx.com>
Subject: Re: Question: How to customize retransmission timeout of
 unacknowledged NFS v3 TCP packet?

Essentially, we need a mechanism to quickly reconnect with new
nfs-server nodes for failover.
I also tried to adjust mount options like "timeo" to 10s and "retrans"
to 1,  and found that they don't work, either.  It seems that the NFS
v3 client always tries to reconnect after some request hangs for 3
minutes no matter what "timeo" and "retrans" is.

On Wed, May 29, 2024 at 6:10 PM Zhitao Li <zhitao.li@...rtx.com> wrote:
>
> Hi, dear community,
>
> In our NFS environment, NFS client mounts remote NFS export with its
> VIP. The VIP can be assigned to another server node for failover.
> However, the NFS client sends the unacknowledged packet 50s+ after the
> VIP is ready on the new node, which is because of the exponential
> backoff retransmission algorithm.  I tried to set this parameter
> "tcp_retries2" smaller so that the NFS client can reconnect with the
> new node more quickly, but this parameter didn't take effect. From
> tcpdump entries as follows,
>   1. At "2024-05-29 11:47:00",  ARP is updated.
>   2. At "2024-05-29 11:47:52" ,  the NFS client retried to send the packet.
>   3. Then the connection is reset and a new connection starts.
>
> I guess the parameter just takes effect for applications and doesn't
> take effect for kernel modules like the NFS client. Could anyone give
> some advice to customize  retransmission timeout of unacknowledged NFS
> v3 TCP packet?
>
>
> OS: Linux kernel v6.7.0
> NFS mount options:
> vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport
>
> tcp_retries2:
> [root@...play zhitaoli]# sysctl -w net.ipv4.tcp_retries2=5
> net.ipv4.tcp_retries2 = 5
> [root@...play zhitaoli]# cat /proc/sys/net/ipv4/tcp_retries2
> 5
>
> tcpdump entries:
>
> 2024-05-29 11:46:02.331891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973659245 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:02.542836 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973659456 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:02.751013 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973659664 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:03.166958 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973660080 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:04.046882 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973660960 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:05.710910 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973662624 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:09.039310 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973665952 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:16.017889 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973672930 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:29.326891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973686240 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:46:55.950915 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973712864 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:47:00.379844 52:54:00:13:1f:34 > Broadcast, ethertype
> ARP (0x0806), length 60: Reply 10.125.1.85 is-at 52:54:00:13:1f:34,
> length 46
>
> 2024-05-29 11:47:52.271192 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> options [nop,nop,TS val 1973769184 ecr 28456
> 58566], length 124: NFS request xid 1954624602 120 access fh
> Unknown/43000001180100000000000000DE40020000000000F439000000000000000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE
>
> 2024-05-29 11:47:52.272041 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> ethertype IPv4 (0x0800), length 54: 10.125.1.85.nfs >
> 10.125.1.214.58428: Flags [R], seq 1148562527, win 0, length 0
>
> 2024-05-29 11:47:52.272909 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> ethertype IPv4 (0x0800), length 74: 10.125.1.214.58428 >
> 10.125.1.85.nfs: Flags [S], seq 1734997801, win 32120, options [mss
> 1460,sackOK,TS val 1973769186 ecr 0,nop,wscale 7], length 0
>
> 2024-05-29 11:47:52.273503 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> ethertype IPv4 (0x0800), length 74: 10.125.1.85.nfs >
> 10.125.1.214.58428: Flags [S.], seq 1078843840, ack 1734997802, win
> 28960, options [mss 1460,sackOK,TS val 2235915769 ecr
> 1973769186,nop,wscale 7], length 0
>
>
> Best regards,
> Zhitao Li