Message-ID: <35827e6d-58df-469f-138b-06f8bcd21c60@itcare.pl>
Date:   Fri, 30 Nov 2018 16:00:04 +0100
From:   Paweł Staszewski <pstaszewski@...are.pl>
To:     Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: Weird traces 4.20.0-rc3+ / RIP: 0010:fib6_walk_continue+0x37/0xe6


Traces attached below:



[310658.536190] rcu: INFO: rcu_sched self-detected stall on CPU
[310658.536195] rcu:    15-....: (322 ticks this GP) 
idle=fca/1/0x4000000000000002 softirq=50617185/50617185 fqs=64
[310658.536195] rcu:     (t=15049 jiffies g=84272013 q=4728)
[310658.536200] NMI backtrace for cpu 15
[310658.536203] CPU: 15 PID: 87 Comm: ksoftirqd/15 Tainted: G        
W         4.20.0-rc3+ #1
[310658.536204] Call Trace:
[310658.536208]  <IRQ>
[310658.536214]  dump_stack+0x46/0x5c
[310658.536218]  nmi_cpu_backtrace+0x72/0x81
[310658.536222]  ? irq_force_complete_move+0x65/0x65
[310658.536224]  nmi_trigger_cpumask_backtrace+0x4c/0xbf
[310658.536228]  rcu_dump_cpu_stacks+0x80/0xaa
[310658.536231]  rcu_check_callbacks+0x213/0x500
[310658.536234]  ? tick_init_highres+0xe/0xe
[310658.536237]  update_process_times+0x23/0x47
[310658.536239]  tick_sched_timer+0x102/0x13a
[310658.536242]  __hrtimer_run_queues+0x105/0x205
[310658.536244]  ? ktime_get_update_offsets_now+0x31/0x8f
[310658.536247]  hrtimer_interrupt+0x85/0x177
[310658.536251]  smp_apic_timer_interrupt+0x8c/0xff
[310658.536253]  apic_timer_interrupt+0xf/0x20
[310658.536254]  </IRQ>
[310658.536258] RIP: 0010:fib6_walk_continue+0x37/0xe6
[310658.536260] Code: 02 0f 0b 48 8b 43 18 48 85 c0 0f 84 c5 00 00 00 8b 
53 28 83 fa 01 74 1e 72 0c 83 fa 02 74 3c 83 fa 03 74 6b eb e1 48 8b 50 
08 <48> 85 d2 75 10 c7 43 28 01 00 00 00 48 8b 50 10 48 85 d2 74 0d 48
[310658.536261] RSP: 0018:ffffc90003583a20 EFLAGS: 00000297 ORIG_RAX: 
ffffffffffffff13
[310658.536262] RAX: ffff88885103b800 RBX: ffffc90003583a58 RCX: 
ffff88885103b800
[310658.536263] RDX: 0000000000000000 RSI: ffff88882868c1c0 RDI: 
ffff8888257dc500
[310658.536264] RBP: ffffffff820d8f00 R08: ffffc90003583b18 R09: 
0000000000000000
[310658.536265] R10: 07387eb000000000 R11: 07387eb000000000 R12: 
ffffffff820d9980
[310658.536266] R13: ffffffff817567ed R14: 0000000000000000 R15: 
ffffc90003583b18
[310658.536268]  ? call_fib6_entry_notifiers+0x59/0x59
[310658.536272]  fib6_walk+0x59/0x76
[310658.536274]  fib6_clean_tree+0x52/0x6c
[310658.536276]  ? fib6_del+0x1da/0x1da
[310658.536278]  ? call_fib6_entry_notifiers+0x59/0x59
[310658.536280]  __fib6_clean_all+0x55/0x71
[310658.536282]  fib6_run_gc+0x85/0xe6
[310658.536285]  ip6_dst_gc+0x74/0xbf
[310658.536288]  dst_alloc+0x70/0x84
[310658.536290]  ip6_dst_alloc+0x1c/0x59
[310658.536293]  icmp6_dst_alloc+0x39/0xd9
[310658.536295]  ndisc_send_skb+0x8e/0x274
[310658.536298]  ? __kmalloc_reserve.isra.43+0x28/0x6a
[310658.536300]  ndisc_send_ns+0x135/0x15e
[310658.536302]  ? ndisc_solicit+0xdd/0x106
[310658.536304]  ndisc_solicit+0xdd/0x106
[310658.536306]  ? lock_timer_base+0x3d/0x61
[310658.536308]  ? neigh_table_init+0x1f9/0x1f9
[310658.536310]  ? neigh_probe+0x44/0x55
[310658.536312]  neigh_probe+0x44/0x55
[310658.536314]  neigh_timer_handler+0x192/0x1ca
[310658.536316]  call_timer_fn+0x51/0x125
[310658.536319]  run_timer_softirq+0x13c/0x172
[310658.536322]  ? __switch_to+0x16c/0x3be
[310658.536324]  __do_softirq+0xec/0x273
[310658.536329]  ? sort_range+0x17/0x17
[310658.536331]  run_ksoftirqd+0x13/0x1b
[310658.536334]  smpboot_thread_fn+0x123/0x138
[310658.536336]  kthread+0xe5/0xea
[310658.536338]  ? kthread_destroy_worker+0x39/0x39
[310658.536340]  ret_from_fork+0x1f/0x30
[310658.938348] ixgbe 0000:06:00.1 enp6s0f1: initiating reset due to tx 
timeout
[310660.685477] ixgbe 0000:84:00.0 enp132s0f0: initiating reset due to 
tx timeout
[310661.484424] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310662.652232] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310663.620879] ixgbe 0000:84:00.1 enp132s0f1: initiating reset due to 
tx timeout
[310664.605672] ixgbe 0000:06:00.1 enp6s0f1: initiating reset due to tx 
timeout
[310666.775352] ixgbe 0000:84:00.0 enp132s0f0: initiating reset due to 
tx timeout
[310667.565003] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310668.349902] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310669.534595] ixgbe 0000:84:00.1 enp132s0f1: initiating reset due to 
tx timeout
[310670.311528] ixgbe 0000:06:00.1 enp6s0f1: initiating reset due to tx 
timeout
[310673.682156] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310673.876020] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310674.264810] ixgbe 0000:06:00.0 enp6s0f0: initiating reset due to tx 
timeout
[310675.436519] ixgbe 0000:84:00.1 enp132s0f1: initiating reset due to 
tx timeout
[310679.393312] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310680.777320] ixgbe 0000:84:00.1 enp132s0f1: initiating reset due to 
tx timeout
[310684.782413] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310685.176739] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310686.561306] ixgbe 0000:84:00.1 enp132s0f1: initiating reset due to 
tx timeout
[310690.345435] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310690.539338] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310695.462443] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310696.248873] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310701.390266] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310701.583983] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310707.500178] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310707.899755] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310713.323668] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310713.529235] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310718.348882] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310719.338749] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310723.472844] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310724.856151] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310729.412495] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310730.384927] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310735.321275] ixgbe 0000:82:00.1 enp130s0f1: initiating reset due to 
tx timeout
[310735.521540] ixgbe 0000:04:00.1 enp4s0f1: initiating reset due to tx 
timeout
[310736.112308] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[310736.112312] rcu:    0-...!: (1 GPs behind) 
idle=a32/1/0x4000000000000000 softirq=50052670/50052671 fqs=24
[310736.112314] rcu:    1-...!: (1 GPs behind) 
idle=92a/1/0x4000000000000000 softirq=49346489/49346491 fqs=24
[310736.112316] rcu:    3-...!: (1 GPs behind) 
idle=86e/1/0x4000000000000000 softirq=49448793/49448794 fqs=24
[310736.112317] rcu:    4-...!: (1 GPs behind) 
idle=a46/1/0x4000000000000000 softirq=71183328/71183329 fqs=24
[310736.112319] rcu:    5-...!: (1 GPs behind) 
idle=22e/1/0x4000000000000000 softirq=68490023/68490109 fqs=24
[310736.112320] rcu:    6-...!: (1 GPs behind) 
idle=5d2/1/0x4000000000000000 softirq=48704478/48704479 fqs=24
[310736.112322] rcu:    7-...!: (1 GPs behind) 
idle=ffe/1/0x4000000000000000 softirq=49004815/49004817 fqs=24
[310736.112323] rcu:    9-...!: (1 GPs behind) 
idle=9da/1/0x4000000000000000 softirq=48909452/48909453 fqs=24
[310736.112325] rcu:    10-...!: (1 GPs behind) 
idle=486/1/0x4000000000000000 softirq=48988888/48988889 fqs=24
[310736.112327] rcu:    13-...!: (1 GPs behind) 
idle=e6a/1/0x4000000000000000 softirq=50819185/50819186 fqs=24
[310736.112328] rcu:    14-...!: (1 GPs behind) 
idle=6c6/1/0x4000000000000000 softirq=50971707/50971708 fqs=24
[310736.112330] rcu:    15-...!: (299 ticks this GP) 
idle=fca/1/0x4000000000000000 softirq=50617187/50617187 fqs=24
[310736.112332] rcu:    19-...!: (1 GPs behind) 
idle=2ba/1/0x4000000000000000 softirq=50386737/50386738 fqs=24
[310736.112333] rcu:    20-...!: (1 GPs behind) 
idle=d3a/1/0x4000000000000000 softirq=50550053/50550054 fqs=24
[310736.112334] rcu:    (detected by 12, t=15021 jiffies, g=84272017, 
q=7242)
[310736.112338] Sending NMI from CPU 12 to CPUs 0:
[310736.112438] NMI backtrace for cpu 0
[310736.112439] CPU: 0 PID: 10 Comm: ksoftirqd/0 Tainted: G W         
4.20.0-rc3+ #1
[310736.112439] RIP: 0010:fib6_age+0x4/0x29
[310736.112440] Code: 89 4c 24 10 ff 40 74 e8 7b 77 01 00 48 8b 4c 24 20 
65 48 33 0c 25 28 00 00 00 74 05 e8 24 42 8f ff 48 83 c4 28 c3 f6 47 56 
40 <48> 8b 15 08 e8 8a 00 74 14 48 8b 4f 30 48 85 c9 74 0b 83 c8 ff 48
[310736.112441] RSP: 0018:ffffc900032639b0 EFLAGS: 00000246
[310736.112442] RAX: 0000000000000000 RBX: ffffc90003263a28 RCX: 
0000000000000000
[310736.112443] RDX: 0000000000000000 RSI: ffffc90003263ae8 RDI: 
ffff888857784400
[310736.112443] RBP: ffff888857784400 R08: 0000000000190120 R09: 
0000000002000500
[310736.112444] R10: 00000000c9390000 R11: 000000009f95f700 R12: 
ffffffff820d9980
[310736.112445] R13: ffffffff817567ed R14: 0000000000000000 R15: 
ffffc90003263ae8
[310736.112445] FS:  0000000000000000(0000) GS:ffff88846f800000(0000) 
knlGS:0000000000000000
[310736.112446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[310736.112446] CR2: 00007f724a69c000 CR3: 00000004388b0003 CR4: 
00000000001606f0
[310736.112447] Call Trace:
[310736.112447]  fib6_clean_node+0x72/0xcf
[310736.112448]  fib6_walk_continue+0x89/0xe6
[310736.112448]  fib6_walk+0x59/0x76
[310736.112449]  fib6_clean_tree+0x52/0x6c
[310736.112449]  ? fib6_del+0x1da/0x1da
[310736.112450]  ? call_fib6_entry_notifiers+0x59/0x59
[310736.112450]  __fib6_clean_all+0x55/0x71
[310736.112450]  fib6_run_gc+0x85/0xe6
[310736.112451]  ip6_dst_gc+0x74/0xbf
[310736.112451]  dst_alloc+0x70/0x84
[310736.112452]  ip6_dst_alloc+0x1c/0x59
[310736.112452]  ip6_pol_route+0x1b0/0x23b
[310736.112453]  fib6_rule_lookup+0x19/0x4e
[310736.112453]  ip6_dst_lookup_tail+0x100/0x18c
[310736.112453]  ip6_dst_lookup_flow+0x40/0x7f
[310736.112454]  inet6_sk_rebuild_header+0xd3/0x172
[310736.112454]  ? tcp_write_timer_handler+0x1c5/0x1c5
[310736.112455]  ? __tcp_retransmit_skb+0x94/0x4cd
[310736.112455]  __tcp_retransmit_skb+0x94/0x4cd
[310736.112456]  ? tcp_write_timer_handler+0x1c5/0x1c5
[310736.112456]  tcp_retransmit_skb+0xd/0x60
[310736.112457]  tcp_retransmit_timer+0x358/0x57c
[310736.112457]  tcp_write_timer_handler+0x6f/0x1c5
[310736.112458]  tcp_write_timer+0x29/0x55
[310736.112458]  call_timer_fn+0x51/0x125
[310736.112459]  run_timer_softirq+0x13c/0x172
[310736.112459]  ? run_rebalance_domains+0x12f/0x207
[310736.112459]  __do_softirq+0xec/0x273
[310736.112460]  ? sort_range+0x17/0x17
[310736.112460]  run_ksoftirqd+0x13/0x1b
[310736.112461]  smpboot_thread_fn+0x123/0x138
[310736.112461]  kthread+0xe5/0xea
[310736.112462]  ? kthread_destroy_worker+0x39/0x39
[310736.112462]  ret_from_fork+0x1f/0x30

Powered by blists - more mailing lists