Message-ID: <09A22A95-6BBD-48EB-A2FE-42BF6244F751@fb.com>
Date:   Mon, 28 May 2018 18:34:57 +0000
From:   Song Liu <songliubraving@...com>
To:     Thomas Gleixner <tglx@...utronix.de>
CC:     Song Liu <liu.song.a23@...il.com>,
        Tariq Toukan <tariqt@...lanox.com>,
        Dmitry Safonov <0x7f454c46@...il.com>,
        open list <linux-kernel@...r.kernel.org>,
        Maor Gottlieb <maorg@...lanox.com>,
        "Kernel Team" <Kernel-team@...com>
Subject: Re: WARNING and PANIC in irq_matrix_free



> On May 28, 2018, at 3:53 AM, Thomas Gleixner <tglx@...utronix.de> wrote:
> 
> On Fri, 25 May 2018, Song Liu wrote:
>> On Wed, May 23, 2018 at 1:49 AM, Thomas Gleixner <tglx@...utronix.de> wrote:
>>> On Wed, 23 May 2018, Tariq Toukan wrote:
>>>> I have your patch merged into my internal branch, it prints the following:
>>>> 
>>>> [ 4898.226258] Trying to clear prev_vector: 0
>>>> [ 4898.226439] Trying to clear prev_vector: 0
>>>> 
>>>> i.e. vector(0) is lower than FIRST_EXTERNAL_VECTOR.
>>> 
>>> Could you please enable the vector and irq matrix trace points and capture
>>> the trace when this happens?
> 
> Does the patch below fix it?
> 
> Thanks,
> 
> 	tglx
> 
> 8<-------------------
> diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
> index bb6f7a2148d7..54af3d4884b1 100644
> --- a/arch/x86/kernel/apic/vector.c
> +++ b/arch/x86/kernel/apic/vector.c
> @@ -148,6 +148,7 @@ static void apic_update_vector(struct irq_data *irqd, unsigned int newvec,
> 	 * prev_vector for this and the offlined target case.
> 	 */
> 	apicd->prev_vector = 0;
> +	apicd->move_in_progress = false;
> 	if (!apicd->vector || apicd->vector == MANAGED_IRQ_SHUTDOWN_VECTOR)
> 		goto setnew;
> 	/*

This doesn't fix the issue with bnxt. Here is a trace with this patch:

[  569.222495] WARNING: CPU: 20 PID: 0 at kernel/irq/matrix.c:373 irq_matrix_free+0x32/0xd0
[  569.238811] BUG: unable to handle kernel
[  569.238811] NULL pointer dereference
[  569.238812]  at 0000000000000000
[  569.238812] IP: bnxt_poll+0x163/0x830
[  569.238812] PGD 0
[  569.238812] P4D 0
[  569.238813] Oops: 0002 [#1] SMP PTI
[  569.238813] Modules linked in:
[  569.238813]  nfsv3
[  569.238814]  nfs
[  569.238814]  fscache
[  569.238814]  ip6table_raw
[  569.238814]  ip6table_mangle
[  569.238815]  iptable_raw
[  569.238815]  iptable_mangle
[  569.238815]  ip6table_filter
[  569.238816]  xt_NFLOG
[  569.238816]  xt_comment
[  569.238816]  xt_statistic
[  569.238816]  iptable_filter
[  569.238817]  nfnetlink_log
[  569.238817]  tcp_diag
[  569.238817]  inet_diag
[  569.238817]  sb_edac
[  569.238818]  x86_pkg_temp_thermal
[  569.238818]  intel_powerclamp
[  569.238818]  coretemp
[  569.238818]  kvm_intel
[  569.238818]  kvm
[  569.238819]  irqbypass
[  569.238819]  iTCO_wdt
[  569.238819]  iTCO_vendor_support
[  569.238819]  lpc_ich
[  569.238819]  efivars
[  569.238820]  mfd_core
[  569.238820]  i2c_i801
[  569.238820]  ipmi_si
[  569.238820]  ipmi_devintf
[  569.238820]  ipmi_msghandler
[  569.238821]  button
[  569.238821]  acpi_cpufreq
[  569.238821]  sch_fq_codel
[  569.238821]  nfsd
[  569.238821]  nfs_acl
[  569.238822]  lockd
[  569.238822]  auth_rpcgss
[  569.238822]  oid_registry
[  569.238822]  grace
[  569.238822]  sunrpc
[  569.238823]  fuse
[  569.238823]  loop
[  569.238823]  efivarfs
[  569.238823]  autofs4
[  569.238824] CPU: 20 PID: 0 Comm: swapper/20 Not tainted 4.16.0-00391-g3742c6a #813
[  569.238824] Hardware name: Quanta Leopard ORv2-DDR4/Leopard ORv2-DDR4, BIOS F06_3B12 08/17/2017
[  569.238824] RIP: 0010:bnxt_poll+0x163/0x830
[  569.238825] RSP: 0018:ffff883ffef83b18 EFLAGS: 00010006
[  569.238825] RAX: 000000002c000000 RBX: 0000000000000000 RCX: 0000000000000000
[  569.238825] RDX: 0000000000000000 RSI: ffff881fdc53c000 RDI: ffff881f55a1da80
[  569.238826] RBP: ffff883ffef83b60 R08: 0000000000000011 R09: 000000004fbf4c8d
[  569.238826] R10: ffff883ffef83ab0 R11: ffff883fec6eee02 R12: ffff881ff6be2780
[  569.238826] R13: 0000000000000000 R14: ffff881f55a1da80 R15: 0000000000000000
[  569.238827] FS:  0000000000000000(0000) GS:ffff883ffef80000(0000) knlGS:0000000000000000
[  569.238827] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  569.238827] CR2: 0000000000000000 CR3: 000000000220a003 CR4: 00000000003606e0
[  569.238827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  569.238828] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  569.238828] Call Trace:
[  569.238828]  <IRQ>
[  569.238828]  ? __skb_tx_hash+0x94/0xb0
[  569.238829]  netpoll_poll_dev+0xc5/0x1a0
[  569.238829]  netpoll_send_skb_on_dev+0x12c/0x200
[  569.238829]  netpoll_send_udp+0x2d5/0x410
[  569.238829]  write_ext_msg+0x1e7/0x200
[  569.238830]  ? scnprintf+0x3a/0x70
[  569.238830]  console_unlock+0x35c/0x530
[  569.238830]  vprintk_emit+0x225/0x340
[  569.238830]  ? irq_matrix_free+0x32/0xd0
[  569.238831]  vprintk_default+0x1f/0x30
[  569.238831]  vprintk_func+0x35/0x70
[  569.238831]  printk+0x43/0x4b
[  569.238832]  ? irq_matrix_free+0x32/0xd0
[  569.238832]  __warn+0x6f/0x150
[  569.238832]  ? irq_matrix_free+0x32/0xd0
[  569.238832]  report_bug+0x83/0xe0
[  569.238833]  do_invalid_op+0x2c/0x70
[  569.238833]  invalid_op+0x1b/0x40
[  569.238833] RIP: 0010:irq_matrix_free+0x32/0xd0
[  569.238834] RSP: 0018:ffff883ffef83f58 EFLAGS: 00010002
[  569.238834] RAX: 0000000000000014 RBX: 0000000000014140 RCX: 0000000000000000
[  569.238835] RDX: 0000000000000000 RSI: 0000000000000014 RDI: ffff881fff028c00
[  569.238835] RBP: ffff883ffef83fc0 R08: ffff883ffefa1c40 R09: 0000000000000000
[  569.238836] R10: 0000000000000000 R11: 0000000000000000 R12: ffff883ffefa4e00
[  569.238836] R13: 0000000000000000 R14: 0000000000000014 R15: ffff881fff028c00
[  569.238836]  ? free_moved_vector+0x60/0x110
[  569.238836]  smp_irq_move_cleanup_interrupt+0x91/0xa9
[  569.238837]  irq_move_cleanup_interrupt+0xc/0x20
[  569.238837]  </IRQ>
[  569.238837] RIP: 0010:cpuidle_enter_state+0xb6/0x2c0
[  569.238837] RSP: 0018:ffffc9000c5d7e80 EFLAGS: 00000246
[  569.238838]  ORIG_RAX: ffffffffffffffdf
[  569.238838] RAX: ffff883ffefa1080 RBX: ffffe8ffff780d00 RCX: 000000000000001f
[  569.238838] RDX: 20c49ba5e353f7cf RSI: 0000000000000004 RDI: 0000000000000000
[  569.238838] RBP: ffffc9000c5d7eb8 R08: fffa7b7805587a32 R09: ffffffff822ec140
[  569.238839] R10: ffffc9000c5d7e60 R11: 0000000000004817 R12: 0000000000000004
[  569.238839] R13: 0000000000000004 R14: 0000000000000014 R15: 0000008487066592
[  569.238839]  cpuidle_enter+0x17/0x20
[  569.238839]  call_cpuidle+0x2d/0x40
[  569.238840]  do_idle+0x109/0x1a0
[  569.238840]  cpu_startup_entry+0x1d/0x20
[  569.238840]  start_secondary+0x10e/0x120
[  569.238840]  secondary_startup_64+0xa5/0xb0
[  569.238841] Code:
[  569.238841] 89
[  569.238841] 02
[  569.238841] 41
[  569.238841] f6
[  569.238842] 44
[  569.238842] 24
[  569.238842] 36
[  569.238842] 40
[  569.238842] 74
[  569.238842] 02
[  569.238843] 89
[  569.238843] 02
[  569.238843] 8b
[  569.238843] 44
[  569.238843] 24
[  569.238843] 1c
[  569.238844] 49
[  569.238844] 8b
[  569.238844] 96
[  569.238844] c8
[  569.238844] 00
[  569.238844] 00
[  569.238845] 00
[  569.238845] 41
[  569.238845] 89
[  569.238845] 86
[  569.238845] c0
[  569.238845] 00
[  569.238846] 00
[  569.238846] 00
[  569.238846] 41
[  569.238846] 23
[  569.238846] 84
[  569.238846] 24
[  569.238847] d4
[  569.238847] 00
[  569.238847] 00
[  569.238847] 00
[  569.238847] 0d
[  569.238847] 00
[  569.238848] 00
[  569.238848] 00
[  569.238848] 2c
[  569.238848] <89>
[  569.238848] 02
[  569.238848] 85
[  569.238849] db
[  569.238849] 0f
[  569.238849] 85
[  569.238849] e7
[  569.238849] 00
[  569.238849] 00
[  569.238850] 00
[  569.238850] f6
[  569.238850] 44
[  569.238850] 24
[  569.238850] 1b
[  569.238850] 01
[  569.238851] 74
[  569.238851] 40
[  569.238851] 49
[  569.238851] 8b
[  569.238851] 96
[  569.238851] a8
[  569.238852] RIP: bnxt_poll+0x163/0x830 RSP: ffff883ffef83b18
[  569.238852] CR2: 0000000000000000

Powered by blists - more mailing lists