linux-kernel - RE: Virtualizing MSI-X on IMS via VFIO

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <87o8buuyfy.ffs@nanos.tec.linutronix.de>
Date:   Fri, 25 Jun 2021 10:43:13 +0200
From:   Thomas Gleixner <tglx@...utronix.de>
To:     "Tian\, Kevin" <kevin.tian@...el.com>,
        Alex Williamson <alex.williamson@...hat.com>
Cc:     Jason Gunthorpe <jgg@...dia.com>,
        "Dey\, Megha" <megha.dey@...el.com>,
        "Raj\, Ashok" <ashok.raj@...el.com>,
        "Pan\, Jacob jun" <jacob.jun.pan@...el.com>,
        "Jiang\, Dave" <dave.jiang@...el.com>,
        "Liu\, Yi L" <yi.l.liu@...el.com>,
        "Lu\, Baolu" <baolu.lu@...el.com>,
        "Williams\, Dan J" <dan.j.williams@...el.com>,
        "Luck\, Tony" <tony.luck@...el.com>,
        "Kumar\, Sanjay K" <sanjay.k.kumar@...el.com>,
        LKML <linux-kernel@...r.kernel.org>, KVM <kvm@...r.kernel.org>,
        Kirti Wankhede <kwankhede@...dia.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Marc Zyngier <maz@...nel.org>,
        Bjorn Helgaas <helgaas@...nel.org>
Subject: RE: Virtualizing MSI-X on IMS via VFIO

On Fri, Jun 25 2021 at 05:21, Kevin Tian wrote:
>> From: Alex Williamson <alex.williamson@...hat.com>
>> So caching/latching occurs on unmask for MSI-X, but I can't find
>> similar statements for MSI.  If you have, please note them.  It's
>> possible MSI is per interrupt.
>
> I checked PCI Local Bus Specification rev3.0. At that time MSI and
> MSI-X were described/compared together in almost every paragraph 
> in 6.8.3.4 (Per-vector Masking and Function Masking). The paragraph
> that you cited is the last one in that section. It's a pity that MSI is
> not clarified in this paragraph but it gives me the impression that 
> MSI function is not permitted to cache address and data values. 
> Later after MSI and MSI-X descriptions were split into separate 
> sections in PCIe spec, this impression is definitely weakened a lot.
>
> If true, this even implies that software is free to change data/addr
> when MSI is unmasked, which is sort of counter-intuitive to most
> people.

Yes, software is free to do that and it has to deal with the
consequences. See arch/x86/kernel/apic/msi.c::msi_set_affinity().

> Then I further found below thread:
>
> https://lore.kernel.org/lkml/1468426713-31431-1-git-send-email-marc.zyngier@arm.com/
>
> It identified a device which does latch the message content in a
> MSI-capable device, forcing the kernel to startup irq early before
> enabling MSI capability.
>
> So, no answer and let's see whether Thomas can help identify
> a better proof.

As I said to Alex: The MSI specification is and always was blury and the
behaviour in detail is implementation defined. IOW, what might work on
device A is not guaranteed to work on device B.

> p.s. one question to Thomas. As Alex cited above, software must 
> not modify the Address, Data, or Steering Tag fields of an MSI-X
> entry while it is unmasked. However this rule might be violated
> today in below flow:
>
> request_irq()
>     __setup_irq()
>         irq_startup()
>             __irq_startup()
>                 irq_enable()
>                     unmask_irq() <<<<<<<<<<<<<
>         irq_setup_affinity()
>             irq_do_set_affinity()
>                 msi_set_affinity() // when IR is disabled
>                     irq_msi_update_msg()
>                         pci_msi_domain_write_msg() <<<<<<<<<<<<<<
>
> Isn't above have msi-x entry updated after it's unmasked? 

Dammit, I could swear that we had masking at the core or PCI level at
some point. Let me dig into this.

Thanks,

        tglx