netdev - Re: [PATCH net] net: fix sk_memory_allocated

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACSApvbc__=J-_hNReeDTNAj2_KJjQaseP+QnY2aQgQGbvkQwA@mail.gmail.com>
Date: Sun, 21 Apr 2024 17:06:45 -0400
From: Soheil Hassas Yeganeh <soheil@...gle.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: "David S . Miller" <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>, 
	Paolo Abeni <pabeni@...hat.com>, netdev@...r.kernel.org, 
	Willem de Bruijn <willemb@...gle.com>, Neal Cardwell <ncardwell@...gle.com>, eric.dumazet@...il.com, 
	Jonathan Heathcote <jonathan.heathcote@....co.uk>
Subject: Re: [PATCH net] net: fix sk_memory_allocated_{add|sub} vs softirqs

On Sun, Apr 21, 2024 at 1:52 PM Eric Dumazet <edumazet@...gle.com> wrote:
>
> Jonathan Heathcote reported a regression caused by blamed commit
> on aarch64 architecture.
>
> x86 happens to have irq-safe __this_cpu_add_return()
> and __this_cpu_sub(), but this is not generic.
>
> I think my confusion came from "struct sock" argument,
> because these helpers are called with a locked socket.
> But the memory accounting is per-proto (and per-cpu after
> the blamed commit). We might cleanup these helpers later
> to directly accept a "struct proto *proto" argument.
>
> Switch to this_cpu_add_return() and this_cpu_xchg()
> operations, and get rid of preempt_disable()/preempt_enable() pairs.
>
> Fast path becomes a bit faster as a result :)
>
> Many thanks to Jonathan Heathcote for his awesome report and
> investigations.
>
> Fixes: 3cd3399dd7a8 ("net: implement per-cpu reserves for memory_allocated")
> Reported-by: Jonathan Heathcote <jonathan.heathcote@....co.uk>
> Closes: https://lore.kernel.org/netdev/VI1PR01MB42407D7947B2EA448F1E04EFD10D2@VI1PR01MB4240.eurprd01.prod.exchangelabs.com/
> Signed-off-by: Eric Dumazet <edumazet@...gle.com>
> Cc: Soheil Hassas Yeganeh <soheil@...gle.com>

Acked-by: Soheil Hassas Yeganeh <soheil@...gle.com>

Very nice catch! and thank you for the fix!

> ---
>  include/net/sock.h | 38 ++++++++++++++++++++------------------
>  1 file changed, 20 insertions(+), 18 deletions(-)
>
> diff --git a/include/net/sock.h b/include/net/sock.h
> index f57bfd8a2ad2deaedf3f351325ab9336ae040504..b4b553df7870c0290ae632c51828ad7161ba332d 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -1410,32 +1410,34 @@ sk_memory_allocated(const struct sock *sk)
>  #define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT))
>  extern int sysctl_mem_pcpu_rsv;
>
> +static inline void proto_memory_pcpu_drain(struct proto *proto)
> +{
> +       int val = this_cpu_xchg(*proto->per_cpu_fw_alloc, 0);
> +
> +       if (val)
> +               atomic_long_add(val, proto->memory_allocated);
> +}
> +
>  static inline void
> -sk_memory_allocated_add(struct sock *sk, int amt)
> +sk_memory_allocated_add(const struct sock *sk, int val)
>  {
> -       int local_reserve;
> +       struct proto *proto = sk->sk_prot;
>
> -       preempt_disable();
> -       local_reserve = __this_cpu_add_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
> -       if (local_reserve >= READ_ONCE(sysctl_mem_pcpu_rsv)) {
> -               __this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
> -               atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
> -       }
> -       preempt_enable();
> +       val = this_cpu_add_return(*proto->per_cpu_fw_alloc, val);
> +
> +       if (unlikely(val >= READ_ONCE(sysctl_mem_pcpu_rsv)))
> +               proto_memory_pcpu_drain(proto);
>  }
>
>  static inline void
> -sk_memory_allocated_sub(struct sock *sk, int amt)
> +sk_memory_allocated_sub(const struct sock *sk, int val)
>  {
> -       int local_reserve;
> +       struct proto *proto = sk->sk_prot;
>
> -       preempt_disable();
> -       local_reserve = __this_cpu_sub_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
> -       if (local_reserve <= -READ_ONCE(sysctl_mem_pcpu_rsv)) {
> -               __this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
> -               atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
> -       }
> -       preempt_enable();
> +       val = this_cpu_sub_return(*proto->per_cpu_fw_alloc, val);
> +
> +       if (unlikely(val <= -READ_ONCE(sysctl_mem_pcpu_rsv)))
> +               proto_memory_pcpu_drain(proto);
>  }
>
>  #define SK_ALLOC_PERCPU_COUNTER_BATCH 16
> --
> 2.44.0.769.g3c40516874-goog
>