netdev - Re: [RFC 07/11] Add support for memory registeration verbs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4b392963-c14c-d793-147a-67b08e97ee06@grimberg.me>
Date:   Wed, 14 Sep 2016 12:25:30 +0300
From:   Sagi Grimberg <sagi@...mberg.me>
To:     "Kalderon, Michal" <Michal.Kalderon@...ium.com>,
        Ram Amrani <Ram.Amrani@...gic.com>,
        "dledford@...hat.com" <dledford@...hat.com>,
        "davem@...emloft.net" <davem@...emloft.net>
Cc:     "Yuval.Mintz@...gic.com" <Yuval.Mintz@...gic.com>,
        "Ariel.Elior@...gic.com" <Ariel.Elior@...gic.com>,
        "Michal.Kalderon@...gic.com" <Michal.Kalderon@...gic.com>,
        "rajesh.borundia@...gic.com" <rajesh.borundia@...gic.com>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: [RFC 07/11] Add support for memory registeration verbs


>>> +struct qedr_mr *__qedr_alloc_mr(struct ib_pd *ibpd, int
>>> +max_page_list_len) {
>>> +	struct qedr_pd *pd = get_qedr_pd(ibpd);
>>> +	struct qedr_dev *dev = get_qedr_dev(ibpd->device);
>>> +	struct qedr_mr *mr;
>>> +	int rc = -ENOMEM;
>>> +
>>> +	DP_VERBOSE(dev, QEDR_MSG_MR,
>>> +		   "qedr_alloc_frmr pd = %d max_page_list_len= %d\n", pd-
>>> pd_id,
>>> +		   max_page_list_len);
>>> +
>>> +	mr = kzalloc(sizeof(*mr), GFP_KERNEL);
>>> +	if (!mr)
>>> +		return ERR_PTR(rc);
>>> +
>>> +	mr->dev = dev;
>>> +	mr->type = QEDR_MR_FRMR;
>>> +
>>> +	rc = init_mr_info(dev, &mr->info, max_page_list_len, 1);
>>> +	if (rc)
>>> +		goto err0;
>>> +
>>> +	rc = dev->ops->rdma_alloc_tid(dev->rdma_ctx, &mr->hw_mr.itid);
>>> +	if (rc) {
>>> +		DP_ERR(dev, "roce alloc tid returned an error %d\n", rc);
>>> +		goto err0;
>>> +	}
>>> +
>>> +	/* Index only, 18 bit long, lkey = itid << 8 | key */
>>> +	mr->hw_mr.tid_type = QED_RDMA_TID_FMR;
>>> +	mr->hw_mr.key = 0;
>>> +	mr->hw_mr.pd = pd->pd_id;
>>
>> Do you have a real MR<->PD association in HW? If so, can you point me to the
>> code that binds it? If not, any reason not to expose the local_dma_lkey?
>>
> Yes, we send the pd id to the FW in function qed_rdma_register_tid.

Right, thanks.

> In any case, if we didn't have the association in HW
> Wouldn't the local_dma_lkey be relevant only to dma_mr ? ( code snippet above refers to FMR)

I was just sticking to a location in the code where you associate
MR<->PD...

The local_dma_lkey is a special key that spans the entire memory space
and unlike the notorious dma_mr, its not associated with a PD.

See the code in ib_alloc_pd(), if the device does not support a single
device local dma lkey, the core allocates a dma mr associated with
the pd. If your device has such a key, you can save a dma mr allocation
for each pd in the system.