lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list] 
Message-ID: <vbf8sskwyiv.fsf@mellanox.com>
Date:   Fri, 26 Jul 2019 16:30:53 +0000
From:   Vlad Buslov <vladbu@...lanox.com>
To:     Lu Baolu <baolu.lu@...ux.intel.com>
CC:     Joerg Roedel <jroedel@...e.de>, Maor Gottlieb <maorg@...lanox.com>,
        Ran Rozenstein <ranro@...lanox.com>,
        "iommu@...ts.linux-foundation.org" <iommu@...ts.linux-foundation.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Failure to recreate virtual functions

Hi Lu Baolu,

Our mlx5 driver fails to recreate VFs when cmdline includes
"intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to
patch b7297783c2bb ("iommu/vt-d: Remove duplicated code for device
hotplug"). Here is the dmesg log for following case: enable switchdev
mode, set number of VFs to 0, then set it back to any value
>0.

[  223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1)
[  223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
[  223.663864] pci 0000:81:00.2: enabling Extended Tags
[  223.665143] pci 0000:81:00.2: Adding to iommu group 52
[  223.665215] pci 0000:81:00.2: Using iommu direct mapping
[  223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002)
[  223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148
[  223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6
[  224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0
[  224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000
[  224.041711] pci 0000:81:00.3: enabling Extended Tags
[  224.043660] pci 0000:81:00.3: Adding to iommu group 53
[  224.043738] pci 0000:81:00.3: Using iommu direct mapping
[  224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002)
[  224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148
[  224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2
[  224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0
[  236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1)
[  236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2)
[  237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
[  237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  237.447846] ens1f0_0: renamed from eth0
[  237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  237.526880] ens1f0_1: renamed from eth1
[  248.953873] pci 0000:81:00.2: Removing from iommu group 52
[  248.954114] pci 0000:81:00.3: Removing from iommu group 53
[  249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2)
[  250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
[  258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1)
[  258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
[  258.936258] pci 0000:81:00.2: enabling Extended Tags
[  258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16
[  258.938053] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002)
[  258.938196] mlx5_core 0000:81:00.2: firmware version: 16.26.148
[  258.938229] mlx5_core 0000:81:00.2: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting
[  258.938315] mlx5_core 0000:81:00.2: init_one:1308:(pid 265): mlx5_load_one failed with error code -12
[  258.938540] mlx5_core: probe of 0000:81:00.2 failed with error -12
[  258.938597] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000
[  258.938657] pci 0000:81:00.3: enabling Extended Tags
[  258.939431] pci 0000:81:00.3: Failed to add to iommu group 52: -16
[  258.939928] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002)
[  258.940039] mlx5_core 0000:81:00.3: firmware version: 16.26.148
[  258.940071] mlx5_core 0000:81:00.3: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting
[  258.940158] mlx5_core 0000:81:00.3: init_one:1308:(pid 265): mlx5_load_one failed with error code -12
[  258.940400] mlx5_core: probe of 0000:81:00.3 failed with error -12


On previous patch 0e31a7266508 ("iommu/vt-d: Remove startup parameter
from device_def_domain_type()") in the series same sequence of actions
doesn't trigger any iommu errors:

[  164.252254] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1)
[  164.288724] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  164.394839] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
[  164.394938] pci 0000:81:00.2: enabling Extended Tags
[  164.396087] pci 0000:81:00.2: Adding to iommu group 52
[  164.396154] pci 0000:81:00.2: Using iommu direct mapping
[  164.396679] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002)
[  164.396803] mlx5_core 0000:81:00.2: firmware version: 16.26.148
[  164.619320] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  164.625754] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  164.625922] mlx5_core 0000:81:00.2: Assigned random MAC address 5e:1e:9b:ca:c8:e5
[  164.739694] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0
[  164.774637] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000
[  164.774709] pci 0000:81:00.3: enabling Extended Tags
[  164.775816] pci 0000:81:00.3: Adding to iommu group 53
[  164.775886] pci 0000:81:00.3: Using iommu direct mapping
[  164.776610] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002)
[  164.776734] mlx5_core 0000:81:00.3: firmware version: 16.26.148
[  164.999360] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  165.007118] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  165.007327] mlx5_core 0000:81:00.3: Assigned random MAC address 82:4a:7a:5f:81:55
[  165.123927] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0
[  172.063665] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1)
[  172.103306] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2)
[  173.033033] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  173.091605] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  173.129258] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
[  173.129863] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  173.203879] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  173.204002] ens1f0_0: renamed from eth1
[  173.289454] ens1f0_1: renamed from eth0
[  186.720692] pci 0000:81:00.2: Removing from iommu group 52
[  186.720994] pci 0000:81:00.3: Removing from iommu group 53
[  187.771549] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2)
[  188.141758] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  188.394072] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0
[  191.116400] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1)
[  191.128965] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[  191.235151] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000
[  191.235250] pci 0000:81:00.2: enabling Extended Tags
[  191.236463] pci 0000:81:00.2: Adding to iommu group 52
[  191.236531] pci 0000:81:00.2: Using iommu direct mapping
[  191.237037] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002)
[  191.237161] mlx5_core 0000:81:00.2: firmware version: 16.26.148
[  191.457369] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  191.463355] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  191.463509] mlx5_core 0000:81:00.2: Assigned random MAC address e6:f2:0c:b4:e3:2e
[  191.572884] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0
[  191.608592] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000
[  191.608664] pci 0000:81:00.3: enabling Extended Tags
[  191.609434] pci 0000:81:00.3: Adding to iommu group 53
[  191.609466] pci 0000:81:00.3: Using iommu direct mapping
[  191.609760] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002)
[  191.609862] mlx5_core 0000:81:00.3: firmware version: 16.26.148
[  191.826324] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[  191.832558] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  191.832730] mlx5_core 0000:81:00.3: Assigned random MAC address a2:dc:76:30:18:6c
[  191.949625] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0

Thanks,
Vlad

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ