netdev - [RFC PATCH bpf-next 2/2] i40e: avoid xdp_do_redirect() call when "redirect_tail

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Tue,  9 Jun 2020 19:26:22 +0200
From:   Björn Töpel <bjorn.topel@...il.com>
To:     ast@...nel.org, daniel@...earbox.net, bpf@...r.kernel.org,
        john.fastabend@...il.com, toke@...hat.com
Cc:     Björn Töpel <bjorn.topel@...el.com>,
        magnus.karlsson@...el.com, netdev@...r.kernel.org,
        brouer@...hat.com, maciej.fijalkowski@...el.com
Subject: [RFC PATCH bpf-next 2/2] i40e: avoid xdp_do_redirect() call when "redirect_tail_call" is set

From: Björn Töpel <bjorn.topel@...el.com>

If an XDP program, where all the bpf_redirect_map() calls are tail
calls (as defined by the previous commit), the driver does not need to
explicitly call xdp_do_redirect().

The driver checks the active XDP program, and notifies the BPF helper
indirectly via xdp_set_redirect_tailcall().

This is just a naive, as-simple-as-possible implementation, calling
xdp_set_redirect_tailcall() for each packet.

For the AF_XDP rxdrop benchmark the pps went from 21.5 to 23.2 Mpps.

Signed-off-by: Björn Töpel <bjorn.topel@...el.com>
---
 drivers/net/ethernet/intel/i40e/i40e_txrx.c | 14 ++++++++++++--
 drivers/net/ethernet/intel/i40e/i40e_xsk.c  | 14 ++++++++++++--
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
index f9555c847f73..0d8dbdc4f47e 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
@@ -2186,6 +2186,7 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring,
 	int err, result = I40E_XDP_PASS;
 	struct i40e_ring *xdp_ring;
 	struct bpf_prog *xdp_prog;
+	bool impl_redir = false;
 	u32 act;
 
 	rcu_read_lock();
@@ -2194,6 +2195,11 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring,
 	if (!xdp_prog)
 		goto xdp_out;
 
+	if (xdp_prog->redirect_tail_call) {
+		impl_redir = true;
+		xdp_set_redirect_tailcall(rx_ring->xdp_prog, xdp,
+					  rx_ring->netdev);
+	}
 	prefetchw(xdp->data_hard_start); /* xdp_frame write */
 
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
@@ -2205,8 +2211,12 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring,
 		result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring);
 		break;
 	case XDP_REDIRECT:
-		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
-		result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+		if (impl_redir) {
+			result = I40E_XDP_REDIR;
+		} else {
+			err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+			result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+		}
 		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c
index 7276580cbe64..0826f551649f 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c
@@ -146,6 +146,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp)
 	int err, result = I40E_XDP_PASS;
 	struct i40e_ring *xdp_ring;
 	struct bpf_prog *xdp_prog;
+	bool impl_redir = false;
 	u32 act;
 
 	rcu_read_lock();
@@ -153,6 +154,11 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp)
 	 * this path is enabled by setting an XDP program.
 	 */
 	xdp_prog = READ_ONCE(rx_ring->xdp_prog);
+	if (xdp_prog->redirect_tail_call) {
+		impl_redir = true;
+		xdp_set_redirect_tailcall(rx_ring->xdp_prog, xdp,
+					  rx_ring->netdev);
+	}
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
 
 	switch (act) {
@@ -163,8 +169,12 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp)
 		result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring);
 		break;
 	case XDP_REDIRECT:
-		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
-		result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+		if (impl_redir) {
+			result = I40E_XDP_REDIR;
+		} else {
+			err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+			result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+		}
 		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
-- 
2.25.1