netdev - [PATCH v2] SUNRPC: Use zero-copy to perform socket send operations

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <160641091184.350621.18059939865682566764.stgit@klimt.1015granger.net>
Date:   Thu, 26 Nov 2020 12:17:58 -0500
From:   Chuck Lever <chuck.lever@...cle.com>
To:     netdev@...r.kernel.org, linux-nfs@...r.kernel.org
Subject: [PATCH v2] SUNRPC: Use zero-copy to perform socket send operations

Daire Byrne reports a ~50% aggregrate throughput regression on his
Linux NFS server after commit da1661b93bf4 ("SUNRPC: Teach server to
use xprt_sock_sendmsg for socket sends"), which replaced
kernel_send_page() calls in NFSD's socket send path with calls to
sock_sendmsg() using iov_iter.

Investigation showed that tcp_sendmsg() was not using zero-copy to
send the xdr_buf's bvec pages, but instead was relying on memcpy.

Set up the socket and each msghdr that bears bvec pages to use the
zero-copy mechanism in tcp_sendmsg. The server side enables full
zero-copy, but the client side does not, because pages passed to the
the RPC layer are not guaranteed to be stable. They include unlocked
page cache pages as well as O_DIRECT pages.

Reported-by: Daire Byrne <daire@...g.com>
BugLink: https://bugzilla.kernel.org/show_bug.cgi?id=209439
Fixes: da1661b93bf4 ("SUNRPC: Teach server to use xprt_sock_sendmsg for socket sends")
Signed-off-by: Chuck Lever <chuck.lever@...cle.com>
---
 net/sunrpc/socklib.c |    5 ++++-
 net/sunrpc/svcsock.c |    1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

Hi-

Because I got a 0-day regression report this morning based on the
original version of this patch, I thought I should repost with a
revised version that is corrected to disable zero-copy on the
client-side, as we previously discussed.

Changes since v1:
- Disable zero-copy optimization on the client-side

diff --git a/net/sunrpc/socklib.c b/net/sunrpc/socklib.c
index d52313af82bc..af47596a7bdd 100644
--- a/net/sunrpc/socklib.c
+++ b/net/sunrpc/socklib.c
@@ -226,9 +226,12 @@ static int xprt_send_pagedata(struct socket *sock, struct msghdr *msg,
 	if (err < 0)
 		return err;

+	msg->msg_flags |= MSG_ZEROCOPY;
 	iov_iter_bvec(&msg->msg_iter, WRITE, xdr->bvec, xdr_buf_pagecount(xdr),
 		      xdr->page_len + xdr->page_base);
-	return xprt_sendmsg(sock, msg, base + xdr->page_base);
+	err = xprt_sendmsg(sock, msg, base + xdr->page_base);
+	msg->msg_flags &= ~MSG_ZEROCOPY;
+	return err;
 }

 /* Common case:
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index b248f2349437..6f199cbb055d 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1176,6 +1176,7 @@ static void svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
 		svsk->sk_datalen = 0;
 		memset(&svsk->sk_pages[0], 0, sizeof(svsk->sk_pages));

+		sock_set_flag(sk, SOCK_ZEROCOPY);
 		tcp_sk(sk)->nonagle |= TCP_NAGLE_OFF;

 		set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags);