netdev - Re: [RFC] change the default Kconfig value of mlx5

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <9e2027ca-3533-2a37-6930-5064d3517878@mellanox.com>
Date:   Sun, 23 Apr 2017 10:54:01 +0300
From:   Matan Barak <matanb@...lanox.com>
To:     Ian Kumlien <ian.kumlien@...il.com>,
        Saeed Mahameed <saeedm@....mellanox.co.il>
CC:     Leon Romanovsky <leonro@...lanox.com>,
        David Miller <davem@...emloft.net>,
        Saeed Mahameed <saeedm@...lanox.com>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: Re: [RFC] change the default Kconfig value of mlx5_en

On 22/04/2017 12:28, Ian Kumlien wrote:
> On Sat, Apr 22, 2017 at 3:07 AM, Saeed Mahameed
> <saeedm@....mellanox.co.il> wrote:
>> On Sat, Apr 22, 2017 at 3:47 AM, Ian Kumlien <ian.kumlien@...il.com> wrote:
>>> On Sat, Apr 22, 2017 at 2:34 AM, Saeed Mahameed
>>> <saeedm@....mellanox.co.il> wrote:
>>>> On Sat, Apr 22, 2017 at 2:10 AM, Ian Kumlien <ian.kumlien@...il.com> wrote:
>>>>> Sorry,
>>>>>
>>>>> Back again, fighting cold, hot whiskey has been consumed...
>>>>>
>>>>> Something like this would perhaps be a better solution:
>>>>>
>>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>> b/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>> index 60154a175bd3..fe192e247601 100644
>>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>> @@ -1139,6 +1139,10 @@ static int mlx5_load_one(struct mlx5_core_dev
>>>>> *dev, struct mlx5_priv *priv,
>>>>>
>>>>>  #ifdef CONFIG_MLX5_CORE_EN
>>>>>         mlx5_eswitch_attach(dev->priv.eswitch);
>>>>> +#else
>>>>> +       if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH) {
>>>>> +               dev_info(&pdev->dev, "Ethernet device discovered but
>>>>> support not enabled in kernel.");
>>>>> +       }
>>>>>  #endif
>>>>>
>>>>
>>>> Currently both MLX5_CORE=n and MLX5_CORE_EN=n as a default, the issue
>>>> you are seeing can occur only if you explicitly  set MLX5_CORE=y and
>>>> MLX5_CORE=n, Why would someone do this if he knows he wants Ethernet
>>>> support as well ? IMHO this print is redundant .
>>>
>>> Well, I'm running a prebuilt kernel - which was configured this way,
>>> and since there
>>> is no mlx5_en module and it does state that the link is "Ethernet", it
>>> just looks like the
>>> driver is broken or in some kind of really weird state.
>>>
>>>> Anyway, Are you looking for RDMA support over ethernet (RoCE) ? and
>>>> you are not interested to have ethernet netdev support ?
>>>
>>> ? RDMA is something we'll look at in the future, right now, having the
>>> nics actually
>>> work as nics is a priority ;)
>>>
>>
>> I see, i just wanted to understand your situation :)
>>
>>>> if yes, I think this is something that can be achieved, but the
>>>> question is do we really need this ?
>>>
>>> It's really weird to see the driver load, to see everything register
>>> and have no feedback.
>>>
>>
>> So, in your case you have mlx5 core support without MLX5_CORE_EN which
>> provides the eswitch and netdev functionality in ethernet.
>
> Yes
>
>> But you will still have mlx5_ib register an RDMA interface and
>> theoretically it should work, the only thing you won't see is a
>> netdevice.
>>
>> The weird thing is that you don't see a link up on the RDMA interface,
>> Leon/Matan can you please look into this ? do we really need a netdev
>> to have a functioning RDMA logical link in ethernet ?
>

The RDMA core subsystem listens to netdev events and configure RoCE GIDs
accordingly. It currently relies on a RoCE dev to have an associated 
netdev, as even default GIDs (the equivalent to IPv6 link local GIDs) 
relies on a MAC address that comes from the netdev.
In IB NICs, the case is different. There's no associated netdev for 
that, so compiling MLX5_CORE_EN isn't required.

> The switch we have does support RDMA but the manual is sparse (as in
> nothing really there) wrt enabling/configuring the RDMA bit so something
> might be missing.
>
> I'll try to remember to do the same test when we setup the mellanox switches =)
>
>>> Including no network devices, but if you run the Infiniband commands,
>>> they tell you that
>>> you are connected to Ethernet but that the device is down and disabled.
>>>
>>> To me, down and disabled is not the same as in "Ethernet support is
>>> not included" =)
>>>
>>> Basically, i would hate for someone else to end up in the same
>>> situation since you only
>>> get guides on how to enable infiniband/RDMA but what you really want
>>> to do at that point
>>> is to disable it and see if that gives you your network devices back =)
>>>
>>
>> Yes this is misleading, Maybe your kernel log warning is not so bad
>> after all, but let me dig more into this.
>> I will get back to you next week.
>
> Thanks, I bet that there is better ways to do it, this one was just
> one of the first ones i found =)
>
>>> I have had similar issues with some connectx3 devices while playing at
>>> home but i suspect
>>> that it's just a limitation of OFED packages available for the dist I'm running.