netdev - Re: [RFC PATCH kernel] Revert "net/mlx4_core: Set UAR page size to 4KB regardless of system page size"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <56E91A78.5000700@ozlabs.ru>
Date:	Wed, 16 Mar 2016 19:34:00 +1100
From:	Alexey Kardashevskiy <aik@...abs.ru>
To:	Eli Cohen <eli@...lanox.com>
Cc:	Or Gerlitz <gerlitz.or@...il.com>, Huy Nguyen <huyn@...lanox.com>,
	Doug Ledford <dledford@...hat.com>,
	Eugenia Emantayev <eugenia@...lanox.com>,
	Hal Rosenstock <hal.rosenstock@...il.com>,
	Sean Hefty <sean.hefty@...el.com>,
	Yishai Hadas <yishaih@...lanox.com>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
	Linux Netdev List <netdev@...r.kernel.org>,
	Paul Mackerras <paulus@...ba.org>,
	Carol L Soto <clsoto@...ibm.com>,
	Yevgeny Petrilin <yevgenyp@...lanox.com>,
	Brian J King <bjking1@...ibm.com>
Subject: Re: [RFC PATCH kernel] Revert "net/mlx4_core: Set UAR page size to
 4KB regardless of system page size"

On 03/16/2016 05:09 PM, Eli Cohen wrote:
> On Wed, Mar 16, 2016 at 04:49:00PM +1100, Alexey Kardashevskiy wrote:
>> On 03/16/2016 04:10 PM, Eli Cohen wrote:
>>> On Wed, Mar 16, 2016 at 01:07:58PM +1100, Alexey Kardashevskiy wrote:
>>>>
>>>> So with v4.5 as a host, there is no actual distro available today to
>>>> use as a guest in the next 6 months (or whatever it takes to
>>>> backport this partucular patch back there).
>>>>
>>>> You could have added a module parameter to enforce the old behavoir,
>>>> at least...
>>>>
>>>> And sorry but from the original commit log I could not understand
>>>> why exactly all existing guests need to be broken. Could you please
>>>> point me to a piece of documentation describing all this UAR
>>>> bisuness (what is UAR, why 128 UARs are required and for what, etc).
>>>> Thanks.
>>>>
>>>
>>> We are going to send a patch that fixes this using a module parameter.
>>> The patch will be on top of Huy's patch.
>>>
>>> Some background to the problem: mlx4 supported devices require 128 UAR
>>
>> What does UAR stand for?
> User Access Region. It's the way you interface with the hardware.
>>
>>> pages from PCI memory space defined by BAR2-3. Each UAR page can be
>>> any power of 2 value from 4K up to 64K. Before Huy's patch the driver
>>> chose UAR page size to be equal to system page size. Since PowerPC's
>>> page size is 64K this means minimum requirement of UAR pages is not
>>> met (default UAR BAR is 8MB and only half of it is really reserved for
>>> UARs).
>>
>> And what was the downside? afaict the performance was good...
>>
>
> It's not a performance issue. Defining 64KB for a UAR is not required
> and wastes pci memory mapped i/o space.
>
>>
>>> More details can be found in the programmer's manual.
>>
>> Can you please point me to this manual on the website? I tried,
>> honestly, could not find it. Thanks.
>>
> It's not publically available. If you have an FAE that work with your
> company you can ask him how to get the doc.


Oh. ok. It also looks like even with the reverted patch, mlx4 VF does not 
work in a guest:

root@...dbg:~# dhclient eth0
mlx4_en: eth0:   frag:0 - size:1518 prefix:0 stride:1536

mlx4_core 0000:00:00.0: Internal error detected on the communication channel
mlx4_core 0000:00:00.0: device is going to be reset
mlx4_core 0000:00:00.0: VF reset is not needed
mlx4_core 0000:00:00.0: device was reset successfully
mlx4_en 0000:00:00.0: Internal error detected, restarting device
mlx4_core 0000:00:00.0: command 0x5 failed: fw status = 0x1
mlx4_core 0000:00:00.0: Failed to close slave function
mlx4_core 0000:00:00.0: Detected virtual function - running in slave mode
mlx4_core 0000:00:00.0: Sending reset


mlx4_core 0000:00:00.0: slave is currently in the middle of FLR - Deferring 
probe
mlx4_core 0000:00:00.0: mlx4_restart_one: ERROR: mlx4_load_one failed, 
pci_name=0000:00:00.0, err=-517
mlx4_core 0000:00:00.0: mlx4_restart_one was ended, ret=-517

root@...dbg:~# ifconfig -a
lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@...dbg:~# lspci -v
00:00.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
         Subsystem: IBM Device 61b0
         Physical Slot: C16
         Flags: bus master, fast devsel, latency 0
         Memory at 10120000000 (64-bit, prefetchable) [size=64M]
         Capabilities: [60] Express Endpoint, MSI 00
         Capabilities: [9c] MSI-X: Enable- Count=52 Masked-
         Capabilities: [40] Power Management version 0
         Kernel driver in use: mlx4_core



-- 
Alexey