Message-ID: <b82f5d3d-f198-3410-af85-85befc14a2ec@intel.com>
Date:   Mon, 8 May 2017 11:55:25 -0400
From:   Dennis Dalessandro <dennis.dalessandro@...el.com>
To:     Leon Romanovsky <leon@...nel.org>
Cc:     Bart Van Assche <Bart.VanAssche@...disk.com>,
        "jiri@...lanox.com" <jiri@...lanox.com>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "ram.amrani@...ium.com" <ram.amrani@...ium.com>,
        "sagi@...mberg.me" <sagi@...mberg.me>,
        "ogerlitz@...lanox.com" <ogerlitz@...lanox.com>,
        "hch@....de" <hch@....de>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "jgunthorpe@...idianresearch.com" <jgunthorpe@...idianresearch.com>,
        "stephen@...workplumber.org" <stephen@...workplumber.org>,
        "dledford@...hat.com" <dledford@...hat.com>,
        "ariela@...lanox.com" <ariela@...lanox.com>
Subject: Re: [RFC iproute2 0/8] RDMA tool

On 05/04/2017 03:42 PM, Leon Romanovsky wrote:
> On Thu, May 04, 2017 at 03:26:13PM -0400, Dennis Dalessandro wrote:
>> On 05/04/2017 02:45 PM, Leon Romanovsky wrote:
>>> On Thu, May 04, 2017 at 06:30:27PM +0000, Bart Van Assche wrote:
>>>> On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
>>>>> On Thu, May 04, 2017 at 06:10:54PM +0000, Bart Van Assche wrote:
>>>>>> On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
>>>>>>> Following our discussion both in mailing list [1] and at the LPC 2016 [2],
>>>>>>> we would like to propose this RDMA tool to be part of iproute2 package
>>>>>>> and finally improve this situation.
>>>>>>
>>>>>> Hello Leon,
>>>>>>
>>>>>> Although I really appreciate your work: can you clarify why you would like to
>>>>>> add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
>>>>>> for adding RDMA functionality to iproute2 in [1].
>>>>>
>>>>> We are planning to reuse the same infrastructure provided by iproute2,
>>>>> like netlink parsing, access to distributions, same CLI and same standards.
>>>>>
>>>>> Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
>>>>> Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
>>>>> RDMA.
>>>>>
>>>>> I do expect that iproute2 will be installed on every machine with any
>>>>> type of connection, including IB and OPA.
>>>>>
>>>>> So I think that it is enough to be part of that suite and don't invent
>>>>> our own for one specific tool.
>>>>
>>>> Hello Leon,
>>>>
>>>> Sorry but to me that sounds like a weak argument for including RDMA functionality
>>>> in iproute2. There is already a library for communication over netlink sockets,
>>>> namely libnl. Is there functionality that is in iproute2 but not in libnl and
>>>> that is needed for the new tool? If so, have you considered to create a new
>>>> library for that functionality?
>>>
>>> It is not hard to create new tool, the hardest part is to ensure that it is
>>> part of the distributions. Did you count how many months we are trying to
>>> add rdma-core to debian?
>>
>> I do agree that it is a strange pairing and am not really a fan. However at
>> the end of the day it's just a name for a repo/package. If the iproute folks
>> are fine to include rdma in their repo/package, great we can leverage their
>> code for CLI and other common stuff.
>>
>> Now if the interface was something like "ip -FlagForRdma ..." I would object
>> to that, but the interface is "rdma ... " so from users perspective it's
>> different tools. They don't need to care that it was sourced from a common
>> git repo.
>>
>> Just as an aside this already works a bit with OPA:
>>
>>  $ ./rdma link
>> 1/1: hfi1_0/1: ifname NONE cap_mask 0x00410022 lid 0x1 lid_mask_count 0
>> link_layer InfiniBand
>>          phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x1 sm_sl 0
>> state 4: ACTIVE
>>
>> Leon I'll get you more feedback and testing, I've just been really bogged
>> down this week, sorry.
>
> Thanks Denny,
>
> Before you are starting to test it, can you please provide your feedback
> on my initial questions? Usability and need of sysfs.
> ----
> This is initial phase to understand if user experience for this tool fits
> RDMA and netdev communities exepectations. Also I would like to get feedback
> if it is really worth to provide legacy sysfs for old kernels, or maybe I should
> implement netlink from the beginning and abandon sysfs completely.
> -----

For the initial phase I think you did the right thing by reading sysfs. 
I like the ability for the tool to be compatible with legacy kernels, 
but at the same time I don't know if it's worth the hassle. I won't 
fight too hard either way.

Perhaps we should take a stab and seeing what a dual sysfs/netlink 
interface would look like, and see just how hard and complicated it 
really is. Then we can make that call. You already have the sysfs 
version, and have to do netink anyway, so let's not rip out what's there 
just yet, this is an RFC after all, not like you are asking for this 
exact version to be merged yet.

As far as usability, I think what's here is a great start and we can 
continue to refine. I'm particularly interested in the stats 
capabilities. In particular being able to filter what stats are shown 
and watch as they change over time. RDMA devices have lots of counters 
and stats. A common tool for users to be able to get that data across HW 
types would be a really good thing.

-Denny

Powered by blists - more mailing lists