[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <037148c3-c15b-4859-9b82-8349fcb54d0a@fujitsu.com>
Date: Thu, 9 Nov 2023 02:24:59 +0000
From: "Zhijian Li (Fujitsu)" <lizhijian@...itsu.com>
To: Zhu Yanjun <yanjun.zhu@...ux.dev>,
"zyjzyj2000@...il.com" <zyjzyj2000@...il.com>,
"jgg@...pe.ca" <jgg@...pe.ca>, "leon@...nel.org" <leon@...nel.org>,
"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>
CC: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"rpearsonhpe@...il.com" <rpearsonhpe@...il.com>,
"Daisuke Matsuda (Fujitsu)" <matsuda-daisuke@...itsu.com>,
"bvanassche@....org" <bvanassche@....org>,
"yi.zhang@...hat.com" <yi.zhang@...hat.com>
Subject: Re: [PATCH RFC V2 0/6] rxe_map_mr_sg() fix cleanup and refactor
On 06/11/2023 21:58, Zhu Yanjun wrote:
> 在 2023/11/6 12:07, Zhijian Li (Fujitsu) 写道:
>>
>>
>> On 03/11/2023 21:00, Zhu Yanjun wrote:
>>> 在 2023/11/3 17:55, Li Zhijian 写道:
>>>> I don't collect the Reviewed-by to the patch1-2 this time, since i
>>>> think we can make it better.
>>>>
>>>> Patch1-2: Fix kernel panic[1] and benifit to make srp work again.
>>>> Almost nothing change from V1.
>>>> Patch3-5: cleanups # newly add
>>>> Patch6: make RXE support PAGE_SIZE aligned mr # newly add, but not fully tested
>>>>
>>>> My bad arm64 mechine offten hangs when doing blktests even though i use the
>>>> default siw driver.
>>>>
>>>> - nvme and ULPs(rtrs, iser) always registers 4K mr still don't supported yet.
>>>
>>> Zhijian
>>>
>>> Please read carefully the whole discussion about this problem. You will find a lot of valuable suggestions, especially suggestions from Jason.
>>
>> Okay, i will read it again. If you can tell me which thread, that would be better.
>>
>>
>>>
>>> From the whole discussion, it seems that the root cause is very clear.
>>> We need to fix this prolem. Please do not send this kind of commits again.
>>>
>>
>> Let's think about what's our goal first.
>>
>> - 1) Fix the panic[1] and only support PAGE_SIZE MR
>> - 2) support PAGE_SIZE aligned MR
>> - 3) support any page_size MR.
>>
>> I'm sorry i'm not familiar with the linux MM subsystem. It seem it's safe/correct to access
>> address/memory across pages start from the return of kmap_loca_page(page).
>> In other words, 2) is already native supported, right?
>
> Yes. Please read the comments from Jason, Leon and Bart. They shared a lot of good advice.
I read the whole discussion again, but I believed i still missed a lost.
> From them, we can know the root cause and how to fix this problem.
I don't think i misunderstood the root cause:
RXE splits memory into PAGE_SIZE units in the xarray. As a result, when we extract an address from the xarray,
we should not access address beyond a PAGE_SIZE window.
IIUC, then how to fix it?
- I'm not going to "removing page_size set", it's out of this patch scope.
Feel free to do the cleanup separately.
- I'm not going to fix the NVMe/rtrs etc problems in this patch set when 64K page is enabled.
But RXE will tell its callers explicitly "RXE don't don't support such page_size"
- I didn't state RXE supports PAGE_SIZE aligned page_size MR before refactoring rxe_map_mr_sg(),
because I worry about it was not correct to access address beyond the PAGE_SIZE window.
What I should do next?
Just state "RXE support PAGE_SIZE aligned MR" ? Then patches become
RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE aligned MR
RDMA/rxe: set RXE_PAGE_SIZE_CAP to starting from PAGE_SIZE
Or just keep we have done in the V1
Thanks
>
> Good Luck.
>
> Zhu Yanjun
>
>>
>> I get totally confused now.
>>
>>
>>
>>> Zhu Yanjun
>>>
>>>>
>>>> [1] https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@mail.gmail.com/T/
>>>>
>>>> Li Zhijian (6):
>>>> RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr
>>>> RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE
>>>> RDMA/rxe: remove unused rxe_mr.page_shift
>>>> RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from
>>>> page_list
>>>> RDMA/rxe: cleanup rxe_mr.{page_size,page_shift}
>>>> RDMA/rxe: Support PAGE_SIZE aligned MR
>>>>
>>>> drivers/infiniband/sw/rxe/rxe_mr.c | 80 ++++++++++++++++-----------
>>>> drivers/infiniband/sw/rxe/rxe_param.h | 2 +-
>>>> drivers/infiniband/sw/rxe/rxe_verbs.h | 9 ---
>>>> 3 files changed, 48 insertions(+), 43 deletions(-)
>>>>
>
Powered by blists - more mailing lists