Message-ID: <20230613165654.3e75eda8@espresso.lan.box>
Date: Tue, 13 Jun 2023 16:56:54 -0300
From: Ricardo Nabinger Sanchez <rnsanchez@...il.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: "David S. Miller" <davem@...emloft.net>, Willem de Bruijn
 <willemdebruijn.kernel@...il.com>, netdev@...r.kernel.org
Subject: panic in udp_init() when using FORCE_NR_CPUS

Hello,

I have hit again an old panic that, in the past, I could not check in
more depth.  But today I was able to pinpoint to a single config knob:

$ diff -u /mnt/tmp/Kernel/linux-6.4-rc6/.config{.old,}
--- /mnt/tmp/Kernel/linux-6.4-rc6/.config.old	2023-06-13
10:34:11.881004307 -0300 +++
/mnt/tmp/Kernel/linux-6.4-rc6/.config	2023-06-13
13:42:46.396967635 -0300 @@ -4996,7 +4996,7 @@ CONFIG_SGL_ALLOC=y
 CONFIG_CHECK_SIGNATURE=y
 CONFIG_CPUMASK_OFFSTACK=y
-CONFIG_FORCE_NR_CPUS=y
+# CONFIG_FORCE_NR_CPUS is not set
 CONFIG_CPU_RMAP=y
 CONFIG_DQL=y
 CONFIG_GLOB=y

Today's build is 6.4-rc6 tarball from kernel.org and, if I enable
FORCE_NR_CPUS, it panic()s after doing kmalloc(), with

	"UDP: failed to alloc udp_busylocks"

Backtrace leads to:

void __init udp_init(void)
{
        unsigned long limit;
        unsigned int i;

        udp_table_init(&udp_table, "UDP");
        limit = nr_free_buffer_pages() / 8; 
        limit = max(limit, 128UL);
        sysctl_udp_mem[0] = limit / 4 * 3; 
        sysctl_udp_mem[1] = limit;
        sysctl_udp_mem[2] = sysctl_udp_mem[0] * 2; 

        /* 16 spinlocks per cpu */
        udp_busylocks_log = ilog2(nr_cpu_ids) + 4; 
        udp_busylocks = kmalloc(sizeof(spinlock_t) << udp_busylocks_log,
                                GFP_KERNEL);
        if (!udp_busylocks)
                panic("UDP: failed to alloc udp_busylocks\n");
        for (i = 0; i < (1U << udp_busylocks_log); i++) 
                spin_lock_init(udp_busylocks + i);

        if (register_pernet_subsys(&udp_sysctl_ops))
                panic("UDP: failed to init sysctl parameters.\n");

#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_PROC_FS)
        bpf_iter_register();
#endif
}

For your convenience, this panic() is from the following commit:

commit 4b272750dbe6f92a8d39a0ee1c7bd50d6cc1a2c8
Author: Eric Dumazet <edumazet@...gle.com>
Date:   Thu Dec 8 11:41:54 2016 -0800

    udp: add busylocks in RX path
    
    Idea of busylocks is to let producers grab an extra spinlock
    to relieve pressure on the receive_queue spinlock shared by
    consumer. 
    This behavior is requested only once socket receive queue is above
    half occupancy.
    
    Under flood, this means that only one producer can be in line
    trying to acquire the receive_queue spinlock.
    
    These busylock can be allocated on a per cpu manner, instead of a
    per socket one (that would consume a cache line per socket)
    
    This patch considerably improves UDP behavior under stress,
    depending on number of NIC RX queues and/or RPS spread.

This is very early in the boot process so I don't have textual output
to paste, and the screen is pretty much on 80x25.  Let me know if you
really need the backtrace/dump.

It should reproduce on as early as 6.1-rc5 (which was the first time I
tried to enable FORCE_NR_CPUS, as far as I can tell), but most likely
any version since the mentioned commit went upstream.

For reference, I'll leave the full .config.old in this Github gist:

	https://gist.github.com/rnsanchez/fd60d25625a0459b4ee10b653fc11f93

If you need to know which CPU I'm using:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    CPU family:          6
    Model:               158
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            10
    CPU(s) scaling MHz:  93%
    CPU max MHz:         4600.0000
    CPU min MHz:         800.0000
    BogoMIPS:            6399.96

Let me know if you need any more information.

Please keep me in Cc:.

Best regards,

-- 
Ricardo Nabinger Sanchez

    Dedique-se a melhorar seus esforços.
    Todas as suas conquistas evolutivas não foram resultado dos
    deuses, das outras consciências, ou do acaso, mas unicamente
    da sua transpiração. ---Waldo Vieira, Léxico de Ortopensatas

Powered by blists - more mailing lists