netdev - Re: [RFC net-next 0/7] Provide an ism layer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <acd2f413-3cd4-495b-ad84-11e511aa3f43@linux.ibm.com>
Date: Mon, 20 Jan 2025 12:45:49 +0100
From: Alexandra Winter <wintera@...ux.ibm.com>
To: dust.li@...ux.alibaba.com, Wenjia Zhang <wenjia@...ux.ibm.com>,
        Jan Karcher <jaka@...ux.ibm.com>, Gerd Bayer <gbayer@...ux.ibm.com>,
        Halil Pasic <pasic@...ux.ibm.com>,
        "D. Wythe" <alibuda@...ux.alibaba.com>,
        Tony Lu <tonylu@...ux.alibaba.com>, Wen Gu <guwen@...ux.alibaba.com>,
        Peter Oberparleiter <oberpar@...ux.ibm.com>,
        David Miller <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>, Eric Dumazet <edumazet@...gle.com>,
        Andrew Lunn <andrew+netdev@...n.ch>
Cc: Julian Ruess <julianr@...ux.ibm.com>,
        Niklas Schnelle <schnelle@...ux.ibm.com>,
        Thorsten Winkler <twinkler@...ux.ibm.com>, netdev@...r.kernel.org,
        linux-s390@...r.kernel.org, Heiko Carstens <hca@...ux.ibm.com>,
        Vasily Gorbik <gor@...ux.ibm.com>,
        Alexander Gordeev
 <agordeev@...ux.ibm.com>,
        Christian Borntraeger <borntraeger@...ux.ibm.com>,
        Sven Schnelle <svens@...ux.ibm.com>, Simon Horman <horms@...nel.org>
Subject: Re: [RFC net-next 0/7] Provide an ism layer



On 18.01.25 16:24, Dust Li wrote:
> On 2025-01-17 12:04:06, Alexandra Winter wrote:
>> I hit the send button to early, sorry about that. 
>> Let me comment on the other proposals from Dust Li as well.
>>
>> On 16.01.25 10:32, Dust Li wrote:
>>> Abstraction of ISM Device Details: I propose we abstract the ISM device
>>> details by providing SMC with helper functions. These functions could
>>> encapsulate ism->ops, making the implementation cleaner and more
>>> intuitive. 
>>
>>
>> Maybe I misunderstand what you mean by helper functions..
>> Why would you encapsulate ism->ops functions in another set of wrappers?
>> I was happy to remove the helper functions in 2/7 and 7/7.
> 
> What I mean is similar to how IB handles it in include/rdma/ib_verbs.h.
> A good example is ib_post_send or ibv_post_send in user space:
> 
> ```c
> static inline int ib_post_send(struct ib_qp *qp,
>                                const struct ib_send_wr *send_wr,
>                                const struct ib_send_wr **bad_send_wr)
> {
>         const struct ib_send_wr *dummy;
> 
>         return qp->device->ops.post_send(qp, send_wr, bad_send_wr ? : &dummy);
> }
> ```
> 
> By following this approach, we can "hide" all the implementations behind
> ism_xxx. Our users (SMC) should only interact with these APIs. The ism->ops
> would then be used by our device implementers (vISM, loopback, etc.). This
> would help make the layers clearer, which is the same approach IB takes.
> 
> The layout would somehow like this:
> 
> | -------------------- |-----------------------------|
> |  ism_register_dmb()  |                             |
> |  ism_move_data()     | <---  API for our users     |
> |  ism_xxx() ...       |                             |
> | -------------------- |-----------------------------|
> |   ism_device_ops     | <---for our implementers    |
> |                      |    (PCI-ISM/loopback, etc)  |
> |----------------------|-----------------------------|
> 
> 
>>
>>
>> This way, the struct ism_device would mainly serve its
>>> implementers, while the upper helper functions offer a streamlined
>>> interface for SMC.
>>
>>
Thanks for the explanations.
Yes, probably makes sense to further decouple the client API from the
device API. I'll give that a try in the next version.


>> I was actually also wondering, whether the clients should access ism_device
>> at all. Or whether they should only use the ism_ops.
> 
> I believe the client should only pass an ism_dev pointer to the ism_xxx()
> helper functions. They should never directly access any of the fields inside
> the ism_dev.
> 
> 
>> I can give that a try in the next version. I think this RFC almost there already.
>> The clients would still need to pass a poitner to ism_dev as a parameter.
>>
>>
>>> Structuring and Naming: I recommend embedding the structure of ism_ops
>>> directly within ism_dev rather than using a pointer. 
>>
>>
>> I think it is a common method to have the const struct xy_ops in the device driver code
>> and then use pointer to register the device with an upper layer.
> 
> Right, If we have many ism_devs for each one ISM type, then using pointer
> should save us some memory.
> 
>> What would be the benefit of duplicating that struct in every ism_dev?
> 
> The main benefit of embedding ism_device_ops within ism_dev is that it
> reduces the dereferencing of an extra pointer. We already have too many
> dereference in the datapath, it is not good for performance :(
> 
> For example:
> 
> rc = smcd->ism->ops->move_data(smcd->ism, dmb_tok, idx, sf, offset,
>                                data, len);
> 
> Best regards,
> Dust
> 

I see your point. I'm not yet convinced. I'll think more about it.