| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sat, 05 Mar 2022 15:40:25 +0000
From: Marc Zyngier <maz@...nel.org>
To: John Garry <john.garry@...wei.com>
Cc: Thomas Gleixner <tglx@...utronix.de>,
chenxiang <chenxiang66@...ilicon.com>,
Shameer Kolothum <shameerali.kolothum.thodi@...wei.com>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"liuqi (BA)" <liuqi115@...wei.com>, <wangxiongfeng2@...wei.com>,
David Decotigny <decot@...gle.com>
Subject: Re: PCI MSI issue for maxcpus=1
[+ David, who was chasing something similar]
Hi John,
On Fri, 04 Mar 2022 12:53:31 +0000,
John Garry <john.garry@...wei.com> wrote:
>
> > ...
>
> >
> > [ 7.961007] valid_col+0x14/0x24
> > [ 7.964223] its_send_single_command+0x4c/0x150
> > [ 7.968741] its_irq_domain_activate+0xc8/0x104
> > [ 7.973259] __irq_domain_activate_irq+0x5c/0xac
> > [ 7.977865] __irq_domain_activate_irq+0x38/0xac
> > [ 7.982471] irq_domain_activate_irq+0x3c/0x64
> > [ 7.986902] __msi_domain_alloc_irqs+0x1a8/0x2f4
> > [ 7.991507] msi_domain_alloc_irqs+0x20/0x2c
> > [ 7.995764] __pci_enable_msi_range+0x2ec/0x590
> > [ 8.000284] pci_alloc_irq_vectors_affinity+0xe0/0x140
> > [ 8.005410] hisi_sas_v3_probe+0x300/0xbe0
> > [ 8.009494] local_pci_probe+0x44/0xb0
> > [ 8.013232] work_for_cpu_fn+0x20/0x34
> > [ 8.016969] process_one_work+0x1d0/0x354
> > [ 8.020966] worker_thread+0x2c0/0x470
> > [ 8.024703] kthread+0x17c/0x190
> > [ 8.027920] ret_from_fork+0x10/0x20
> > [ 8.031485] ---[ end trace bb67cfc7eded7361 ]---
> >
>
> ...
>
> > Ah, of course. the CPU hasn't booted yet, so its collection isn't
> > mapped. I was hoping that the core code would keep the interrupt in
> > shutdown state, but it doesn't seem to be the case...
> >
> > > Apart from this, I assume that if another cpu comes online later in
> > > the affinity mask I would figure that we want to target the irq to
> > > that cpu (which I think we would not do here).
> >
> > That's probably also something that should come from core code, as
> > we're not really in a position to decide this in the ITS driver.
> > .
>
>
> Hi Marc,
>
> Have you had a chance to consider this issue further?
>
> So I think that x86 avoids this issue as it uses matrix.c, which
> handles CPUs being offline when selecting target CPUs for managed
> interrupts.
>
> So is your idea still that core code should keep the interrupt in
> shutdown state (for no CPUs online in affinity mask)?
Yup. I came up with this:
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 2bdfce5edafd..97e9eb9aecc6 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -823,6 +823,19 @@ static int msi_init_virq(struct irq_domain *domain, int virq, unsigned int vflag
if (!(vflags & VIRQ_ACTIVATE))
return 0;
+ if (!(vflags & VIRQ_CAN_RESERVE)) {
+ /*
+ * If the interrupt is managed but no CPU is available
+ * to service it, shut it down until better times.
+ */
+ if (irqd_affinity_is_managed(irqd) &&
+ !cpumask_intersects(irq_data_get_affinity_mask(irqd),
+ cpu_online_mask)) {
+ irqd_set_managed_shutdown(irqd);
+ return 0;
+ }
+ }
+
ret = irq_domain_activate_irq(irqd, vflags & VIRQ_CAN_RESERVE);
if (ret)
return ret;
With this in place, I get the following results (VM booted with 4
vcpus and maxcpus=1, the virtio device is using managed interrupts):
root@...ian:~# cat /proc/interrupts
CPU0
10: 2298 GICv3 27 Level arch_timer
12: 84 GICv3 33 Level uart-pl011
49: 0 GICv3 41 Edge ACPI:Ged
50: 0 ITS-MSI 16384 Edge virtio0-config
51: 2088 ITS-MSI 16385 Edge virtio0-req.0
52: 0 ITS-MSI 16386 Edge virtio0-req.1
53: 0 ITS-MSI 16387 Edge virtio0-req.2
54: 0 ITS-MSI 16388 Edge virtio0-req.3
55: 11641 ITS-MSI 32768 Edge xhci_hcd
56: 0 ITS-MSI 32769 Edge xhci_hcd
IPI0: 0 Rescheduling interrupts
IPI1: 0 Function call interrupts
IPI2: 0 CPU stop interrupts
IPI3: 0 CPU stop (for crash dump) interrupts
IPI4: 0 Timer broadcast interrupts
IPI5: 0 IRQ work interrupts
IPI6: 0 CPU wake-up interrupts
Err: 0
root@...ian:~# echo 1 >/sys/devices/system/cpu/cpu2/online
root@...ian:~# cat /proc/interrupts
CPU0 CPU2
10: 2530 90 GICv3 27 Level arch_timer
12: 103 0 GICv3 33 Level uart-pl011
49: 0 0 GICv3 41 Edge ACPI:Ged
50: 0 0 ITS-MSI 16384 Edge virtio0-config
51: 2097 0 ITS-MSI 16385 Edge virtio0-req.0
52: 0 0 ITS-MSI 16386 Edge virtio0-req.1
53: 0 12 ITS-MSI 16387 Edge virtio0-req.2
54: 0 0 ITS-MSI 16388 Edge virtio0-req.3
55: 13487 0 ITS-MSI 32768 Edge xhci_hcd
56: 0 0 ITS-MSI 32769 Edge xhci_hcd
IPI0: 38 45 Rescheduling interrupts
IPI1: 3 3 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
Would this solve your problem?
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
Powered by blists - more mailing lists