linux-stable/net/sunrpc
Siddharth Kawar 943d045a6d SUNRPC: fix shutdown of NFS TCP client socket
NFS server Duplicate Request Cache (DRC) algorithms rely on NFS clients
reconnecting using the same local TCP port. Unique NFS operations are
identified by the per-TCP connection set of XIDs. This prevents file
corruption when non-idempotent NFS operations are retried.

Currently, NFS client TCP connections are using different local TCP ports
when reconnecting to NFS servers.

After an NFS server initiates shutdown of the TCP connection, the NFS
client's TCP socket is set to NULL after the socket state has reached
TCP_LAST_ACK(9). When reconnecting, the new socket attempts to reuse
the same local port but fails with EADDRNOTAVAIL (99). This forces the
socket to use a different local TCP port to reconnect to the remote NFS
server.

State Transition and Events:
TCP_CLOSE_WAIT(8)
TCP_LAST_ACK(9)
connect(fail EADDRNOTAVAIL(99))
TCP_CLOSE(7)
bind on new port
connect success

dmesg excerpts showing reconnect switching from TCP local port of 926 to
763 after commit 7c81e6a9d7:
[13354.947854] NFS call  mkdir testW
...
[13405.654781] RPC:       xs_tcp_state_change client 00000000037d0f03...
[13405.654813] RPC:       state 8 conn 1 dead 0 zapped 1 sk_shutdown 1
[13405.654826] RPC:       xs_data_ready...
[13405.654892] RPC:       xs_tcp_state_change client 00000000037d0f03...
[13405.654895] RPC:       state 9 conn 0 dead 0 zapped 1 sk_shutdown 3
[13405.654899] RPC:       xs_tcp_state_change client 00000000037d0f03...
[13405.654900] RPC:       state 9 conn 0 dead 0 zapped 1 sk_shutdown 3
[13405.654950] RPC:       xs_connect scheduled xprt 00000000037d0f03
[13405.654975] RPC:       xs_bind 0.0.0.0:926: ok (0)
[13405.654980] RPC:       worker connecting xprt 00000000037d0f03 via tcp
			  to 10.101.6.228 (port 2049)
[13405.654991] RPC:       00000000037d0f03 connect status 99 connected 0
			  sock state 7
[13405.655001] RPC:       xs_tcp_state_change client 00000000037d0f03...
[13405.655002] RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[13405.655024] RPC:       xs_connect scheduled xprt 00000000037d0f03
[13405.655038] RPC:       xs_bind 0.0.0.0:763: ok (0)
[13405.655041] RPC:       worker connecting xprt 00000000037d0f03 via tcp
			  to 10.101.6.228 (port 2049)
[13405.655065] RPC:       00000000037d0f03 connect status 115 connected 0
			  sock state 2

State Transition and Events with patch applied:
TCP_CLOSE_WAIT(8)
TCP_LAST_ACK(9)
TCP_CLOSE(7)
connect(reuse of port succeeds)

dmesg excerpts showing reconnect on same TCP local port of 936 with patch
applied:
[  257.139935] NFS: mkdir(0:59/560857152), testQ
[  257.139937] NFS call  mkdir testQ
...
[  307.822702] RPC:       state 8 conn 1 dead 0 zapped 1 sk_shutdown 1
[  307.822714] RPC:       xs_data_ready...
[  307.822817] RPC:       xs_tcp_state_change client 00000000ce702f14...
[  307.822821] RPC:       state 9 conn 0 dead 0 zapped 1 sk_shutdown 3
[  307.822825] RPC:       xs_tcp_state_change client 00000000ce702f14...
[  307.822826] RPC:       state 9 conn 0 dead 0 zapped 1 sk_shutdown 3
[  307.823606] RPC:       xs_tcp_state_change client 00000000ce702f14...
[  307.823609] RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[  307.823629] RPC:       xs_tcp_state_change client 00000000ce702f14...
[  307.823632] RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[  307.823676] RPC:       xs_connect scheduled xprt 00000000ce702f14
[  307.823704] RPC:       xs_bind 0.0.0.0:936: ok (0)
[  307.823709] RPC:       worker connecting xprt 00000000ce702f14 via tcp
			  to 10.101.1.30 (port 2049)
[  307.823748] RPC:       00000000ce702f14 connect status 115 connected 0
			  sock state 2
...
[  314.916193] RPC:       state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
[  314.916251] RPC:       xs_connect scheduled xprt 00000000ce702f14
[  314.916282] RPC:       xs_bind 0.0.0.0:936: ok (0)
[  314.916292] RPC:       worker connecting xprt 00000000ce702f14 via tcp
			  to 10.101.1.30 (port 2049)
[  314.916342] RPC:       00000000ce702f14 connect status 115 connected 0
			  sock state 2

Fixes: 7c81e6a9d7 ("SUNRPC: Tweak TCP socket shutdown in the RPC client")
Signed-off-by: Siddharth Rajendra Kawar <sikawar@microsoft.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2023-03-23 15:50:16 -04:00
..
auth_gss SUNRPC: Properly terminate test case arrays 2023-02-27 16:27:51 -05:00
xprtrdma SUNRPC: Remove ->xpo_secure_port() 2023-02-20 09:20:55 -05:00
.kunitconfig SUNRPC: Add KDF-HMAC-SHA2 Kunit tests 2023-02-20 09:20:51 -05:00
addr.c nfsd: don't alloc under spinlock in rpc_parse_scope_id 2021-09-21 17:51:47 -04:00
auth_null.c SUNRPC: Add rpc_auth::au_ralign field 2019-02-14 11:48:36 -05:00
auth_unix.c SUNRPC: Fix unx_lookup_cred() allocation 2022-03-22 15:52:55 -04:00
auth.c NFS client updates for Linux 5.20 2022-08-10 14:04:32 -07:00
backchannel_rqst.c NFS client updates for Linux 5.20 2022-08-10 14:04:32 -07:00
cache.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
clnt.c NFS: fix disabling of swap 2023-02-15 10:33:00 -05:00
debugfs.c SUNRPC: Cache deferral injection 2022-05-19 12:25:38 -04:00
fail.h SUNRPC: Cache deferral injection 2022-05-19 12:25:38 -04:00
Kconfig SUNRPC: Add KUnit tests for rpcsec_krb5.ko 2023-02-20 09:20:46 -05:00
Makefile sunrpc: Create a sunrpc directory under /sys/kernel/ 2021-07-08 14:03:23 -04:00
netns.h SUNRPC: Fix occasional warning when destroying gss_krb5_enctypes 2023-02-20 09:20:57 -05:00
rpc_pipe.c fs: allocate inode by using alloc_inode_sb() 2022-03-22 15:57:03 -07:00
rpcb_clnt.c SUNRPC: Make the rpciod and xprtiod slab allocation modes consistent 2022-03-22 15:52:55 -04:00
sched.c NFS Client Updates for Linux 6.1 2022-10-13 09:58:42 -07:00
socklib.c use less confusing names for iov_iter direction initializers 2022-11-25 13:01:55 -05:00
socklib.h SUNRPC: Refactor xs_sendpages() 2020-03-16 12:04:33 -04:00
stats.c SUNRPC: Use per-CPU counters to tally server RPC counts 2023-02-20 09:20:32 -05:00
sunrpc_syms.c sunrpc: add IDs to multipath 2021-07-08 14:03:23 -04:00
sunrpc.h treewide: Replace GPLv2 boilerplate/reference with SPDX - gpl-2.0_149.RULE 2022-06-10 14:51:35 +02:00
svc_xprt.c SUNRPC: Remove ->xpo_secure_port() 2023-02-20 09:20:55 -05:00
svc.c SUNRPC: Fix a server shutdown leak 2023-03-08 08:46:41 -05:00
svcauth_unix.c SUNRPC: Fix whitespace damage in svcauth_unix.c 2023-02-20 09:20:52 -05:00
svcauth.c SUNRPC: Hoist init_decode out of svc_authenticate() 2023-02-20 09:20:19 -05:00
svcsock.c NFSD 6.3 Release Notes 2023-02-22 14:21:40 -08:00
sysctl.c net/sunrpc: fix useless comparison in proc_do_xprt() 2020-11-08 16:28:25 -05:00
sysfs.c SUNRPC: make kobj_type structures constant 2023-02-14 16:21:52 -05:00
sysfs.h SUNRPC: take a xprt offline using sysfs 2021-07-08 14:03:24 -04:00
timer.c treewide: Add SPDX license identifier for missed files 2019-05-21 10:50:45 +02:00
xdr.c NFSD 6.3 Release Notes 2023-02-22 14:21:40 -08:00
xprt.c timers: Get rid of del_singleshot_timer_sync() 2022-11-24 15:09:10 +01:00
xprtmultipath.c SUNRPC: Directly use ida_alloc()/free() 2022-10-03 11:26:36 -04:00
xprtsock.c SUNRPC: fix shutdown of NFS TCP client socket 2023-03-23 15:50:16 -04:00