netdev - Re: panic in udp_init() when using FORCE_NR

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89i+5DoHFh-2MvLy740ikLdV-sE8pEEM+R=i0i77Pyc1ADQ@mail.gmail.com>
Date: Tue, 13 Jun 2023 22:19:33 +0200
From: Eric Dumazet <edumazet@...gle.com>
To: Ricardo Nabinger Sanchez <rnsanchez@...il.com>
Cc: "David S. Miller" <davem@...emloft.net>, Willem de Bruijn <willemdebruijn.kernel@...il.com>, 
	netdev@...r.kernel.org
Subject: Re: panic in udp_init() when using FORCE_NR_CPUS

On Tue, Jun 13, 2023 at 9:56 PM Ricardo Nabinger Sanchez
<rnsanchez@...il.com> wrote:
>
> Hello,
>
> I have hit again an old panic that, in the past, I could not check in
> more depth.  But today I was able to pinpoint to a single config knob:
>
> $ diff -u /mnt/tmp/Kernel/linux-6.4-rc6/.config{.old,}
> --- /mnt/tmp/Kernel/linux-6.4-rc6/.config.old   2023-06-13
> 10:34:11.881004307 -0300 +++
> /mnt/tmp/Kernel/linux-6.4-rc6/.config   2023-06-13
> 13:42:46.396967635 -0300 @@ -4996,7 +4996,7 @@ CONFIG_SGL_ALLOC=y
>  CONFIG_CHECK_SIGNATURE=y
>  CONFIG_CPUMASK_OFFSTACK=y
> -CONFIG_FORCE_NR_CPUS=y
> +# CONFIG_FORCE_NR_CPUS is not set
>  CONFIG_CPU_RMAP=y
>  CONFIG_DQL=y
>  CONFIG_GLOB=y
>

Sure, but you did not give NR_CPUS value ?

Also posting the stack trace might be useful.

> Today's build is 6.4-rc6 tarball from kernel.org and, if I enable
> FORCE_NR_CPUS, it panic()s after doing kmalloc(), with
>
>         "UDP: failed to alloc udp_busylocks"
>
> Backtrace leads to:
>
> void __init udp_init(void)
> {
>         unsigned long limit;
>         unsigned int i;
>
>         udp_table_init(&udp_table, "UDP");
>         limit = nr_free_buffer_pages() / 8;
>         limit = max(limit, 128UL);
>         sysctl_udp_mem[0] = limit / 4 * 3;
>         sysctl_udp_mem[1] = limit;
>         sysctl_udp_mem[2] = sysctl_udp_mem[0] * 2;
>
>         /* 16 spinlocks per cpu */
>         udp_busylocks_log = ilog2(nr_cpu_ids) + 4;
>         udp_busylocks = kmalloc(sizeof(spinlock_t) << udp_busylocks_log,
>                                 GFP_KERNEL);
>         if (!udp_busylocks)
>                 panic("UDP: failed to alloc udp_busylocks\n");
>         for (i = 0; i < (1U << udp_busylocks_log); i++)
>                 spin_lock_init(udp_busylocks + i);
>
>         if (register_pernet_subsys(&udp_sysctl_ops))
>                 panic("UDP: failed to init sysctl parameters.\n");
>
> #if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_PROC_FS)
>         bpf_iter_register();
> #endif
> }
>
> For your convenience, this panic() is from the following commit:
>
> commit 4b272750dbe6f92a8d39a0ee1c7bd50d6cc1a2c8
> Author: Eric Dumazet <edumazet@...gle.com>
> Date:   Thu Dec 8 11:41:54 2016 -0800
>
>     udp: add busylocks in RX path
>
>     Idea of busylocks is to let producers grab an extra spinlock
>     to relieve pressure on the receive_queue spinlock shared by
>     consumer.
>     This behavior is requested only once socket receive queue is above
>     half occupancy.
>
>     Under flood, this means that only one producer can be in line
>     trying to acquire the receive_queue spinlock.
>
>     These busylock can be allocated on a per cpu manner, instead of a
>     per socket one (that would consume a cache line per socket)
>
>     This patch considerably improves UDP behavior under stress,
>     depending on number of NIC RX queues and/or RPS spread.
>
> This is very early in the boot process so I don't have textual output
> to paste, and the screen is pretty much on 80x25.  Let me know if you
> really need the backtrace/dump.
>
> It should reproduce on as early as 6.1-rc5 (which was the first time I
> tried to enable FORCE_NR_CPUS, as far as I can tell), but most likely
> any version since the mentioned commit went upstream.
>
> For reference, I'll leave the full .config.old in this Github gist:
>
>         https://gist.github.com/rnsanchez/fd60d25625a0459b4ee10b653fc11f93
>
> If you need to know which CPU I'm using:
>
> Architecture:            x86_64
>   CPU op-mode(s):        32-bit, 64-bit
>   Address sizes:         39 bits physical, 48 bits virtual
>   Byte Order:            Little Endian
> CPU(s):                  12
>   On-line CPU(s) list:   0-11
> Vendor ID:               GenuineIntel
>   Model name:            Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
>     CPU family:          6
>     Model:               158
>     Thread(s) per core:  2
>     Core(s) per socket:  6
>     Socket(s):           1
>     Stepping:            10
>     CPU(s) scaling MHz:  93%
>     CPU max MHz:         4600.0000
>     CPU min MHz:         800.0000
>     BogoMIPS:            6399.96
>
> Let me know if you need any more information.
>
> Please keep me in Cc:.
>
> Best regards,
>
> --
> Ricardo Nabinger Sanchez
>
>     Dedique-se a melhorar seus esforços.
>     Todas as suas conquistas evolutivas não foram resultado dos
>     deuses, das outras consciências, ou do acaso, mas unicamente
>     da sua transpiração. ---Waldo Vieira, Léxico de Ortopensatas