| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <vbf36imsb79.fsf@mellanox.com>
Date: Wed, 31 Jul 2019 11:19:41 +0000
From: Vlad Buslov <vladbu@...lanox.com>
To: Lu Baolu <baolu.lu@...ux.intel.com>
CC: Robin Murphy <robin.murphy@....com>,
Vlad Buslov <vladbu@...lanox.com>,
Joerg Roedel <jroedel@...e.de>,
Ran Rozenstein <ranro@...lanox.com>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
Maor Gottlieb <maorg@...lanox.com>
Subject: Re: Failure to recreate virtual functions
On Wed 31 Jul 2019 at 10:29, Lu Baolu <baolu.lu@...ux.intel.com> wrote:
> Hi,
>
> On 7/30/19 7:22 PM, Robin Murphy wrote:
>> On 30/07/2019 05:28, Lu Baolu wrote:
>>> Hi,
>>>
>>> On 7/29/19 6:05 PM, Vlad Buslov wrote:
>>>> On Sat 27 Jul 2019 at 05:15, Lu Baolu<baolu.lu@...ux.intel.com> wrote:
>>>>> Hi Vilad,
>>>>>
>>>>> On 7/27/19 12:30 AM, Vlad Buslov wrote:
>>>>>> Hi Lu Baolu,
>>>>>>
>>>>>> Our mlx5 driver fails to recreate VFs when cmdline includes
>>>>>> "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
>>>>>> Delegate DMA domain to generic iommu". I've bisected the failure to
>>>>>> patch b7297783c2bb ("iommu/vt-d: Remove duplicated code for device
>>>>>> hotplug"). Here is the dmesg log for following case: enable switchdev
>>>>>> mode, set number of VFs to 0, then set it back to any value
>>>>>>> 0.
>>>>>> [ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV:
>>>>>> nvfs(2) mode (1)
>>>>>> [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active
>>>>>> vports(3)
>>>>>> [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
>>>>>> [ 223.663864] pci 0000:81:00.2: enabling Extended Tags
>>>>>> [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52
>>>>>> [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping
>>>>>> [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002)
>>>>>> [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148
>>>>>> [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are
>>>>>> supported, range: 0Mbps to 97656Mbps
>>>>>> [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address
>>>>>> 56:1f:95:e0:51:d6
>>>>>> [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0
>>>>>> [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000
>>>>>> [ 224.041711] pci 0000:81:00.3: enabling Extended Tags
>>>>>> [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53
>>>>>> [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping
>>>>>> [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002)
>>>>>> [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148
>>>>>> [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are
>>>>>> supported, range: 0Mbps to 97656Mbps
>>>>>> [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address
>>>>>> a6:1e:56:0a:d9:f2
>>>>>> [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0
>>>>>> [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active
>>>>>> vports(3) mode(1)
>>>>>> [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV:
>>>>>> nvfs(2) mode (2)
>>>>>> [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
>>>>>> [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 237.447846] ens1f0_0: renamed from eth0
>>>>>> [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active
>>>>>> vports(3)
>>>>>> [ 237.526880] ens1f0_1: renamed from eth1
>>>>>> [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52
>>>>>> [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53
>>>>>> [ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active
>>>>>> vports(3) mode(2)
>>>>>> [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)
>>>>>> StrdSz(2048) RxCqeCmprss(0)
>>>>>> [ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
>>>>>> [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV:
>>>>>> nvfs(2) mode (1)
>>>>>> [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active
>>>>>> vports(3)
>>>>>> [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
>>>>>> [ 258.936258] pci 0000:81:00.2: enabling Extended Tags
>>>>>> [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16
>>>>> It seems that an EBUSY error returned from iommu_group_add_device(). Can
>>>>> you please hack some debug messages in iommu_group_add_device() so that
>>>>> we can know where the EBUSY returns?
>>>>>
>>>>> Best regards,
>>>>> Baolu
>>>> The error code is returned by __iommu_attach_device().
>>>>
>>>
>>> Thanks!
>>>
>>> It looks like the system has already a domain for specific pci bdf
>>> device. Does this VF share the bdf with other devices? Or has been
>>> previously created, and system failed to get chance to remove it?
>>
>> At a glance, it looks like it might be down to intel_iommu_remove_device() not
>> calling dmar_remove_one_dev_info() like the old notifier did. If the group is
>> getting torn down and recreated, but the driver still has a stale pointer to
>> the old default domain cached, which dmar_insert_one_dev_info() finds and
>> returns, that would seem to explain the observed behaviour.
>
> Yes agreed.
>
> Vlad,
>
> Can you please try below change?
>
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index baf21001c339..abffc520fe05 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -5575,6 +5575,8 @@ static void intel_iommu_remove_device(struct device *dev)
> if (!iommu)
> return;
>
> + dmar_remove_one_dev_info(dev);
> +
> iommu_group_remove_device(dev);
>
> iommu_device_unlink(&iommu->iommu, dev);
>
> Best regards,
> Baolu
Hi Baolu,
This patch fixes the issue for me.
Thanks,
Vlad
Powered by blists - more mailing lists