Date:   Sat, 8 May 2021 13:48:37 +0300
From:   Martin Zaharinov <micron10@...il.com>
To:     Paolo Abeni <pabeni@...hat.com>
Cc:     netdev <netdev@...r.kernel.org>, Wei Wang <weiwan@...gle.com>,
        "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Alexander Duyck <alexanderduyck@...com>,
        Hannes Frederic Sowa <hannes@...essinduktion.org>,
        Eric Dumazet <edumazet@...gle.com>, alobakin@...me
Subject: Bug Report Napi kthread rcd 

Hi all 
One more bug report .
Kernel is 5.12.1 

If you need more info I will write.

Server run with 200 users with nat 

[81402.540906] rcu: INFO: rcu_sched self-detected stall on CPU
[81402.540909] rcu: 5-....: (3314 ticks this GP) idle=74e/1/0x4000000000000000 softirq=4979878/4979878 fqs=2554 last_accelerate: a926/c0a0 dyntick_enabled: 1
[81402.540911] (t=6001 jiffies g=7517749 q=44479)
[81402.540913] NMI backtrace for cpu 5
[81402.540914] CPU: 5 PID: 36 Comm: ksoftirqd/5 Tainted: G O 5.12.1 #1
[81402.540916] Hardware name: Supermicro Super Server/X10SRD-F, BIOS 3.3 10/28/2020
[81402.540917] Call Trace:
[81402.540919]
[81402.540920] dump_stack+0x65/0x7d
[81402.540924] ? lapic_can_unplug_cpu+0x70/0x70
[81402.540927] nmi_trigger_cpumask_backtrace.cold+0x40/0x4d
[81402.540929] rcu_dump_cpu_stacks+0xbe/0xec
[81402.540932] rcu_sched_clock_irq.cold+0x195/0x3f1
[81402.540934] ? enqueue_task_fair+0x796/0xbd0
[81402.540938] update_process_times+0x88/0xc0
[81402.540942] tick_sched_timer+0x7f/0x110
[81402.540944] ? tick_nohz_dep_set_task+0x80/0x80
[81402.540945] __hrtimer_run_queues+0x10b/0x1b0
[81402.540947] hrtimer_interrupt+0x10a/0x420
[81402.540949] __sysvec_apic_timer_interrupt+0x47/0x60
[81402.540952] sysvec_apic_timer_interrupt+0x65/0x90
[81402.540955]
[81402.540955] asm_sysvec_apic_timer_interrupt+0xf/0x20
[81402.540959] RIP: 0010:console_unlock+0x366/0x5e0
[81402.540961] Code: ff ff 8b 05 44 5f b2 01 85 c0 75 66 c7 05 3a 5f b2 01 01 00 00 00 e9 0f fd ff ff e8 f4 1c 00 00 48 85 db 74 01 fb 8b 54 24 0c <85> d2 0f 84 4a fd ff ff e8 1d 2b 7c 00 e9 40 fd ff ff 4d 85 ff 74
[81402.540963] RSP: 0018:ffff9dc980203a80 EFLAGS: 00000206
[81402.540964] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
[81402.540965] RDX: 0000000000000000 RSI: 0000000000000087 RDI: ffffffff82b59898
[81402.540966] RBP: 0000000000000000 R08: ffff9786814db080 R09: 0000000000000000
[81402.540966] R10: ffff9786a85bf260 R11: ffff9786f7bd7cf0 R12: 0000000000000048
[81402.540967] R13: 0000000000000000 R14: 20c49ba5e353f7cf R15: 0000000000000000
[81402.540968] ? common_interrupt+0x14/0xa0
[81402.540969] ? asm_common_interrupt+0x1b/0x40
[81402.540971] vprintk_default+0x5a/0x150
[81402.540972] printk+0x43/0x45
[81402.540975] create_nat_session+0x1c5e/0x1cfd [xt_NAT]
[81402.540978] ipt_do_table+0x2e5/0x670 [ip_tables]
[81402.540980] ? ip_route_input_noref+0xa8/0x1e0
[81402.540983] nf_hook_slow+0x36/0xa0
[81402.540986] ip_forward+0x40d/0x450
[81402.540987] ? ip4_obj_hashfn+0xc0/0xc0
[81402.540989] process_backlog+0x11a/0x230
[81402.540992] __napi_poll+0x1f/0x130
[81402.540994] net_rx_action+0x239/0x2f0
[81402.540996] ? run_timer_softirq+0x730/0x880
[81402.540998] __do_softirq+0xaf/0x1da
[81402.541000] run_ksoftirqd+0x15/0x20
[81402.541004] smpboot_thread_fn+0xb3/0x140
[81402.541006] ? sort_range+0x20/0x20
[81402.541008] kthread+0xea/0x120
[81402.541010] ? kthread_park+0x80/0x80
[81402.541012] ret_from_fork+0x1f/0x30
[81416.300055] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: {
[81476.311498] rcu: INFO: rcu_sched self-detected stall on CPU
[81476.311500] rcu: 3-....: (1 GPs behind) idle=86a/1/0x4000000000000000 softirq=4703397/4703398 fqs=2596 last_accelerate: c5ff/dd71 dyntick_enabled: 1
[81476.311503] (t=6001 jiffies g=7517753 q=82419)
[81476.311505] NMI backtrace for cpu 3
[81476.311506] CPU: 3 PID: 527214 Comm: kworker/3:2 Tainted: G O 5.12.1 #1
[81476.311507] Hardware name: Supermicro Super Server/X10SRD-F, BIOS 3.3 10/28/2020
[81476.311509] Workqueue: rcu_gp wait_rcu_exp_gp
[81476.311512] Call Trace:
[81476.311514]
[81476.311515] dump_stack+0x65/0x7d
[81476.311519] ? lapic_can_unplug_cpu+0x70/0x70
[81476.311521] nmi_trigger_cpumask_backtrace.cold+0x40/0x4d
[81476.311523] rcu_dump_cpu_stacks+0xbe/0xec
[81476.311527] rcu_sched_clock_irq.cold+0x195/0x3f1
[81476.311529] ? timekeeping_advance+0x34e/0x540
[81476.311531] update_process_times+0x88/0xc0
[81476.311534] tick_sched_timer+0x7f/0x110
[81476.311536] ? tick_nohz_dep_set_task+0x80/0x80
[81476.311537] __hrtimer_run_queues+0x10b/0x1b0
[81476.311539] hrtimer_interrupt+0x10a/0x420
[81476.311541] __sysvec_apic_timer_interrupt+0x47/0x60
[81476.311544] sysvec_apic_timer_interrupt+0x65/0x90
[81476.311547]
[81476.311547] asm_sysvec_apic_timer_interrupt+0xf/0x20
[81476.311551] RIP: 0010:console_unlock+0x366/0x5e0
[81476.311554] Code: ff ff 8b 05 44 5f b2 01 85 c0 75 66 c7 05 3a 5f b2 01 01 00 00 00 e9 0f fd ff ff e8 f4 1c 00 00 48 85 db 74 01 fb 8b 54 24 0c <85> d2 0f 84 4a fd ff ff e8 1d 2b 7c 00 e9 40 fd ff ff 4d 85 ff 74
[81476.311555] RSP: 0018:ffff9dc980313cc0 EFLAGS: 00000206
[81476.311556] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
[81476.311557] RDX: 0000000000000000 RSI: 0000000000000087 RDI: ffffffff82b59898
[81476.311557] RBP: 0000000000000000 R08: ffff9786814db080 R09: 0000000000000000
[81476.311558] R10: ffff9786a85bac10 R11: ffff97872e90acf0 R12: 0000000000000048
[81476.311559] R13: 0000000000000000 R14: 20c49ba5e353f7cf R15: 0000000000000000
[81476.311560] vprintk_default+0x5a/0x150
[81476.311562] printk+0x43/0x45
[81476.311563] synchronize_rcu_expedited_wait.cold+0x20/0x2db
[81476.311565] rcu_exp_wait_wake+0xc/0x110
[81476.311567] process_one_work+0x1ec/0x350
[81476.311569] worker_thread+0x4f/0x4d0
[81476.311570] ? process_one_work+0x350/0x350
[81476.311571] kthread+0xea/0x120
[81476.311573] ? kthread_park+0x80/0x80
[81476.311574] ret_from_fork+0x1f/0x30
[81551.199572] } 19586 jiffies s: 14473 root: 0x0/.

Powered by blists - more mailing lists