linux-kernel - Re: [PATCH net-next v3 1/2] net/smc: make wr buffer count configurable

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250928103951.6464dfd3.pasic@linux.ibm.com>
Date: Sun, 28 Sep 2025 10:39:51 +0200
From: Halil Pasic <pasic@...ux.ibm.com>
To: Dust Li <dust.li@...ux.alibaba.com>
Cc: Paolo Abeni <pabeni@...hat.com>, Jakub Kicinski <kuba@...nel.org>,
        Simon
 Horman <horms@...nel.org>,
        "D. Wythe" <alibuda@...ux.alibaba.com>,
        Sidraya
 Jayagond <sidraya@...ux.ibm.com>,
        Wenjia Zhang <wenjia@...ux.ibm.com>,
        Mahanta Jambigi <mjambigi@...ux.ibm.com>,
        Tony Lu
 <tonylu@...ux.alibaba.com>, Wen Gu <guwen@...ux.alibaba.com>,
        netdev@...r.kernel.org, linux-doc@...r.kernel.org,
        linux-kernel@...r.kernel.org, linux-rdma@...r.kernel.org,
        linux-s390@...r.kernel.org, Halil Pasic <pasic@...ux.ibm.com>
Subject: Re: [PATCH net-next v3 1/2] net/smc: make wr buffer count
 configurable

On Sun, 28 Sep 2025 10:02:43 +0800
Dust Li <dust.li@...ux.alibaba.com> wrote:

> >Unfortunately I don't quite understand why qp_attr.cap.max_send_wr is 3
> >times the number of send WR buffers we allocate. My understanding
> >is that qp_attr.cap.max_send_wr is about the number of send WQEs.  
> 
> We have at most 2 RDMA Write for 1 RDMA send. So 3 times is necessary.
> That is explained in the original comments. Maybe it's better to keep it.
> 
> ```
> .cap = {
>                 /* include unsolicited rdma_writes as well,
>                  * there are max. 2 RDMA_WRITE per 1 WR_SEND
>                  */

But what are "the unsolicited" rdma_writes? I have heard of
unsolicited receive, where the data is received without
consuming a WR previously put on the RQ on the receiving end, but
the concept of unsolicited rdma_writes eludes me completely.

I guess what you are trying to say, and what I understand is
that we first put the payload into the RMB of the remote, which
may require up 2 RDMA_WRITE operations, probably because we may
cross the end (and start) of the array that hosts the circular
buffer, and then we send a CDC message to update the cursor.

For the latter a  ib_post_send() is used in smc_wr_tx_send()
and AFAICT it consumes a WR from wr_tx_bufs. For the former
we consume a single wr_tx_rdmas which and each wr_tx_rdmas
has 2 WR allocated.

And all those WRs need a WQE. So I guess now I do understand
SMC_WR_BUF_CNT, but I find the comment still confusing like
hell because of these unsolicited rdma_writes.

Thank you for the explanation! It was indeed helpful! Let
me try to come up with a better comment -- unless somebody
manages to explain "unsolicited rdma_writes" to me.

>         .max_send_wr = SMC_WR_BUF_CNT * 3,
>         .max_recv_wr = SMC_WR_BUF_CNT * 3,
>         .max_send_sge = SMC_IB_MAX_SEND_SGE,
>         .max_recv_sge = lnk->wr_rx_sge_cnt,
>         .max_inline_data = 0,
> },
> ```
> 
> >I assume that qp_attr.cap.max_send_wr == qp_attr.cap.max_recv_wr
> >is not something we would want to preserve.  
> 
> IIUC, RDMA Write won't consume any RX wqe on the receive side, so I think
> the .max_recv_wr can be SMC_WR_BUF_CNT if we don't use RDMA_WRITE_IMM.

Maybe we don't want to assume somebody else (another implementation)
would not use immediate data. I'm not sure. But I don't quite understand
the why the relationship between the send and the receive side either.

Regards,
Halil