netdev - Re: [PATCH net] net: avoid indirect memory pressure calls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 28 Feb 2023 18:42:03 +0100
From:   Eric Dumazet <edumazet@...gle.com>
To:     Florian Westphal <fw@...len.de>, Brian Vazquez <brianvv@...gle.com>
Cc:     netdev@...r.kernel.org, davem@...emloft.net, kuba@...nel.org,
        pabeni@...hat.com, shakeelb@...gle.com, soheil@...gle.com
Subject: Re: [PATCH net] net: avoid indirect memory pressure calls

On Tue, Feb 28, 2023 at 5:44 PM Eric Dumazet <edumazet@...gle.com> wrote:
>
> On Fri, Feb 24, 2023 at 7:49 PM Florian Westphal <fw@...len.de> wrote:
> >
> > There is a noticeable tcp performance regression (loopback or cross-netns),
> > seen with iperf3 -Z (sendfile mode) when generic retpolines are needed.
> >
> > With SK_RECLAIM_THRESHOLD checks gone number of calls to enter/leave
> > memory pressure happen much more often. For TCP indirect calls are
> > used.
> >
> > We can't remove the if-set-return short-circuit check in
> > tcp_enter_memory_pressure because there are callers other than
> > sk_enter_memory_pressure.  Doing a check in the sk wrapper too
> > reduces the indirect calls enough to recover some performance.
> >
> > Before,
> > 0.00-60.00  sec   322 GBytes  46.1 Gbits/sec                  receiver
> >
> > After:
> > 0.00-60.04  sec   359 GBytes  51.4 Gbits/sec                  receiver
> >

BTW I was curious why Google was not seeing this, and it appears Brian Vasquez
forgot to upstream this change...

commit 5ea2f21d6c1078d2c563cb455ad5877b4ada94e1
Author: Brian Vazquez <brianvv@...gle.com>
Date:   Thu Mar 3 19:09:49 2022 -0800

    PRODKERNEL: net-directcall: annotate tcp_leave_memory_pressure and
    tcp_getsockopt

    Switch to upstream infra to make rebase easier

    Tested: built and booted on lab machine

    Upstream-Plan: 150254871
    Effort: net-directcall

diff --git a/net/core/sock.c b/net/core/sock.c
index 05032b399c873984e5297898d647905ca9f21f2e..54cb989dc162f3982380ac12cf5a150214e209a2
100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2647,10 +2647,13 @@ static void sk_enter_memory_pressure(struct sock *sk)
        sk->sk_prot->enter_memory_pressure(sk);
 }

+INDIRECT_CALLABLE_DECLARE(void tcp_leave_memory_pressure(struct sock *sk));
+
 static void sk_leave_memory_pressure(struct sock *sk)
 {
        if (sk->sk_prot->leave_memory_pressure) {
-               sk->sk_prot->leave_memory_pressure(sk);
+               INDIRECT_CALL_1(sk->sk_prot->leave_memory_pressure,
+                               tcp_leave_memory_pressure, sk);
        } else {
                unsigned long *memory_pressure = sk->sk_prot->memory_pressure;

@@ -3439,6 +3442,10 @@ int sock_recv_errqueue(struct sock *sk, struct
msghdr *msg, int len,
 }
 EXPORT_SYMBOL(sock_recv_errqueue);

+INDIRECT_CALLABLE_DECLARE(int tcp_getsockopt(struct sock *sk, int level,
+                                            int optname, char __user *optval,
+                                            int __user *optlen));
+
 /*
  *     Get a socket option on an socket.
  *
@@ -3451,7 +3458,8 @@ int sock_common_getsockopt(struct socket *sock,
int level, int optname,
 {
        struct sock *sk = sock->sk;

-       return sk->sk_prot->getsockopt(sk, level, optname, optval, optlen);
+       return INDIRECT_CALL_1(sk->sk_prot->getsockopt, tcp_getsockopt,
+                               sk, level, optname, optval, optlen);
 }
 EXPORT_SYMBOL(sock_common_getsockopt);