linux-kernel - Re: [PATCH] pci/iov: VFs are never multifunction

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAErSpo7xEKzWF8oetskQAcLcwJUw4VPByXgbMUkurVcK2caRgA@mail.gmail.com>
Date:	Thu, 9 Jan 2014 16:20:50 -0700
From:	Bjorn Helgaas <bhelgaas@...gle.com>
To:	Alex Williamson <alex.williamson@...hat.com>,
	Nishank Trivedi <nistrive@...co.com>
Cc:	"linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
	sathya.perla@...lex.com,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	ajit.khaparde@...lex.com, Don Dutile <ddutile@...hat.com>,
	subbu.seetharaman@...lex.com, Myron Stowe <myron.stowe@...hat.com>
Subject: Re: [PATCH] pci/iov: VFs are never multifunction

[+to Nishank]

On Thu, Jan 9, 2014 at 2:58 PM, Alex Williamson
<alex.williamson@...hat.com> wrote:
> On Thu, 2014-01-09 at 14:39 -0700, Bjorn Helgaas wrote:
>> On Thu, Jan 9, 2014 at 11:25 AM, Alex Williamson
>> <alex.williamson@...hat.com> wrote:
>> > On Thu, 2014-01-09 at 11:08 -0700, Bjorn Helgaas wrote:
>> >> On Thu, Jan 9, 2014 at 8:36 AM, Alex Williamson
>> >> <alex.williamson@...hat.com> wrote:
>> >> > Per the SR-IOV spec rev 1.1:
>> >> >
>> >> > 3.4.1.9 Header Type (Offset 0Eh)
>> >> >
>> >> > "... For VFs, this register must be RO Zero."
>> >> >
>> >> > Unfortunately some devices get this wrong, ex. Emulex OneConnect 10Gb
>> >> > NIC.  When they do it makes us handle ACS testing and therefore IOMMU
>> >> > groups as if they were actual multifunction devices and require ACS
>> >> > capabilities to make sure there's no peer-to-peer between functions.
>> >> > VFs are never traditional multifunction devices, so simply clear this
>> >> > bit before we get any further into setup.
>> >>
>> >> This seems reasonable.  Do you have "lspci -vvxxxx" output for this
>> >> device?  I'd like to save it for future reference.
>> >
>> > Sure, here's a VF:
>> >
>> > 09:04.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
>> >         Subsystem: Emulex Corporation Device e722
>>
>> Thanks!  I put this in
>> https://bugzilla.kernel.org/show_bug.cgi?id=68431, and I'll add a
>> reference to the changelog.
>>
>> But I wonder if we can make this slightly more generic by doing
>> something like this in pci_setup_device():
>>
>>   dev->multifunction = (PCI_FUNC(dev->devfn) == 0) && (hdr_type & 0x80);
>>
>> That's basically what lspci does in pci_generic_scan_bus(), and
>> section 3.2.2.3.4 of the PCI 3.0 spec sort of implies that we should
>> only look at the bit 7 of the header type for function 0:
>>
>>   If a single function device is detected (i.e., bit 7 in the Header
>>   Type register of function 0 is 0), no more functions for that
>>   Device Number will be checked. If a multi-function device is
>>   detected (i.e., bit 7 in the Header Type register of function 0
>>   is 1), then all remaining Function Numbers will be checked.
>
> We could do that and rely only on pci_scan_slot() to set multifunction=1
> for the other functions, but that doesn't completely solve this problem.
> VFs can occupy function zero and the example device would still set
> multifunction with that test.  Thanks,

Duh, it would help if I actually paid attention to your lspci output...

The reason I'm thinking about this is that virtfn_add() is only used
when we enable SR-IOV.  If we clear dev->multifunction there, we only
end up with the correct value if we start with SR-IOV disabled, and
then enable it.

If SR-IOV were enabled by the firmware before Linux boots, we wouldn't
go through the virtfn_add() path, and dev->multifunction might still
be wrong.

I'm pretty sure Nishank said there were Cisco boxes that enable SR-IOV
in the firmware, but I don't know how that works.  It looks like we
would disable SR-IOV during enumeration in the path below:

  pci_scan_slot
    pci_scan_single_device
      pci_device_add
        pci_init_capabilities
          pci_iov_init
            sriov_init
              pci_read_config_word(dev, pos + PCI_SRIOV_CTRL, &ctrl)
              if (ctrl & PCI_SRIOV_CTRL_VFE)
                pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, 0)

>From that path, it *looks* like it doesn't really matter whether
SR-IOV is enabled at handoff, because we disable it anyway.

So I'm not sure if I misunderstood Nishank or what.  I think it would
be cool if we could enumerate previously-enabled VFs, but maybe there
are other issues that would make that impossible.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/