linux-kernel - ARM/gic-v4: deadlock occurred

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Sun, 5 May 2019 10:26:18 +0800
From:   Heyi Guo <guoheyi@...wei.com>
To:     <linux-kernel@...r.kernel.org>
CC:     Marc Zyngier <marc.zyngier@....com>,
        wanghaibin 00208455 <wanghaibin.wang@...wei.com>
Subject: ARM/gic-v4: deadlock occurred

Hi folks,

We observed deadlocks after enabling GICv4 and PCI passthrough on ARM64 virtual
machines, when not pinning VCPU to physical CPU.

We observed below warnings after enabling lockdep debug in kernel:

[  362.847021] =====================================================
[  362.855643] WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
[  362.864840] 4.19.34+ #7 Tainted: G        W
[  362.872314] -----------------------------------------------------
[  362.881034] CPU 0/KVM/51468 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
[  362.890504] 00000000659c1dc9 (fs_reclaim){+.+.}, at: fs_reclaim_acquire.part.22+0x0/0x48
[  362.901413]
[  362.901413] and this task is already holding:
[  362.912976] 000000007318873f (&dev->event_map.vlpi_lock){....}, at: its_irq_set_vcpu_affinity+0x134/0x638
[  362.928626] which would create a new lock dependency:
[  362.936837]  (&dev->event_map.vlpi_lock){....} -> (fs_reclaim){+.+.}
[  362.946449]
[  362.946449] but this new dependency connects a HARDIRQ-irq-safe lock:
[  362.960877]  (&irq_desc_lock_class){-.-.}
[  362.960880]
[  362.960880] ... which became HARDIRQ-irq-safe at:
[  362.981234]   lock_acquire+0xf0/0x258
[  362.988337]   _raw_spin_lock+0x54/0x90
[  362.995543]   handle_fasteoi_irq+0x2c/0x198
[  363.003205]   generic_handle_irq+0x34/0x50
[  363.010787]   __handle_domain_irq+0x68/0xc0
[  363.018500]   gic_handle_irq+0xf4/0x1e0
[  363.025913]   el1_irq+0xc8/0x180
[  363.032683]   _raw_spin_unlock_irq+0x40/0x60
[  363.040512]   finish_task_switch+0x98/0x258
[  363.048254]   __schedule+0x350/0xca8
[  363.055359]   schedule+0x40/0xa8
[  363.062098]   worker_thread+0xd8/0x410
[  363.069340]   kthread+0x134/0x138
[  363.076070]   ret_from_fork+0x10/0x18
[  363.083111]
[  363.083111] to a HARDIRQ-irq-unsafe lock:
[  363.095213]  (fs_reclaim){+.+.}
[  363.095216]
[  363.095216] ... which became HARDIRQ-irq-unsafe at:
[  363.114527] ...
[  363.114530]   lock_acquire+0xf0/0x258
[  363.126269]   fs_reclaim_acquire.part.22+0x3c/0x48
[  363.134206]   fs_reclaim_acquire+0x2c/0x38
[  363.141363]   kmem_cache_alloc_trace+0x44/0x368
[  363.148892]   acpi_os_map_iomem+0x9c/0x208
[  363.155934]   acpi_os_map_memory+0x28/0x38
[  363.162831]   acpi_tb_acquire_table+0x58/0x8c
[  363.170021]   acpi_tb_validate_table+0x34/0x58
[  363.177162]   acpi_tb_get_table+0x4c/0x90
[  363.183741]   acpi_get_table+0x94/0xc4
[  363.190020]   find_acpi_cpu_topology_tag+0x54/0x240
[  363.197404]   find_acpi_cpu_topology_package+0x28/0x38
[  363.204985]   init_cpu_topology+0xdc/0x1e4
[  363.211498]   smp_prepare_cpus+0x2c/0x108
[  363.217882]   kernel_init_freeable+0x130/0x508
[  363.224699]   kernel_init+0x18/0x118
[  363.230624]   ret_from_fork+0x10/0x18
[  363.236611]
[  363.236611] other info that might help us debug this:
[  363.236611]
[  363.251604] Chain exists of:
[  363.251604]   &irq_desc_lock_class --> &dev->event_map.vlpi_lock --> fs_reclaim
[  363.251604]
[  363.270508]  Possible interrupt unsafe locking scenario:
[  363.270508]
[  363.282238]        CPU0                    CPU1
[  363.289228]        ----                    ----
[  363.296189]   lock(fs_reclaim);
[  363.301726]                                local_irq_disable();
[  363.310122] lock(&irq_desc_lock_class);
[  363.319143] lock(&dev->event_map.vlpi_lock);
[  363.328617]   <Interrupt>
[  363.333713]     lock(&irq_desc_lock_class);
[  363.340414]
[  363.340414]  *** DEADLOCK ***
[  363.340414]
[  363.353682] 5 locks held by CPU 0/KVM/51468:
[  363.360412]  #0: 00000000eeb852a5 (&vdev->igate){+.+.}, at: vfio_pci_ioctl+0x2f8/0xed0
[  363.370915]  #1: 000000002ab491f7 (lock#9){+.+.}, at: irq_bypass_register_producer+0x6c/0x1d0
[  363.382139]  #2: 000000000d9fd5c6 (&its->its_lock){+.+.}, at: kvm_vgic_v4_set_forwarding+0xd0/0x188
[  363.396625]  #3: 00000000232bdc47 (&irq_desc_lock_class){-.-.}, at: __irq_get_desc_lock+0x60/0xa0
[  363.408486]  #4: 000000007318873f (&dev->event_map.vlpi_lock){....}, at: its_irq_set_vcpu_affinity+0x134/0x638


Then we found that irq_set_vcpu_affinity() in kernel/irq/manage.c aquires an
antomic context by irq_get_desc_lock() at the beginning, but in
its_irq_set_vcpu_affinity() (drivers/irqchip/irq-gic-v3-its.c) we are still
using mutext_lock, kcalloc, kfree, etc, which we think should be forbidden in
atomic context.

Though the issue is observed in 4.19.34, we don't find any related fixes in the mainline yet.

Please advise.

Thanks,

Heyi