linux-kernel - Re: [bug report] [rdma] RXE ODP test hangs with new DMA map API

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <3016329a-4edd-4550-862f-b298a1b79a39@gmail.com>
Date: Wed, 21 May 2025 22:16:53 +0900
From: Daisuke Matsuda <dskmtsd@...il.com>
To: linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org,
 leon@...nel.org, jgg@...pe.ca, zyjzyj2000@...il.com
Subject: Re: [bug report] [rdma] RXE ODP test hangs with new DMA map API


>    RDMA/umem: Store ODP access mask information in PFN

This one generates a build error. I modified manually and tried running the test.
It also hanged with ERROR:
```
$ ./build/bin/run_tests.py -v -k odp
test_odp_dc_traffic (tests.test_mlx5_dc.DCTest.test_odp_dc_traffic) ... skipped                                                                                                                                                              'Can not run the test over non MLX5 device'
test_devx_rc_qp_odp_traffic (tests.test_mlx5_devx.Mlx5DevxRcTrafficTest.test_dev                                                                                                                                                             x_rc_qp_odp_traffic) ... skipped 'Can not run the test over non MLX5 device'
test_odp_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest.test_odp_mkey_list                                                                                                                                                             _new_api)
Create Mkeys above ODP MR, configure it with memory layout using the new API and                                                                                                                                                              ... skipped 'Could not open mlx5 context (This is not an MLX5 device)'
test_odp_async_prefetch_rc_traffic (tests.test_odp.OdpTestCase.test_odp_async_pr                                                                                                                                                             efetch_rc_traffic) ... skipped 'Advise MR with flags (0) and advice (0) is not s                                                                                                                                                             upported'
test_odp_implicit_async_prefetch_rc_traffic (tests.test_odp.OdpTestCase.test_odp                                                                                                                                                             _implicit_async_prefetch_rc_traffic) ... skipped 'ODP implicit is not supported'
test_odp_implicit_rc_traffic (tests.test_odp.OdpTestCase.test_odp_implicit_rc_tr                                                                                                                                                             affic) ... skipped 'ODP implicit is not supported'
test_odp_implicit_sync_prefetch_rc_traffic (tests.test_odp.OdpTestCase.test_odp_                                                                                                                                                             implicit_sync_prefetch_rc_traffic) ... skipped 'ODP implicit is not supported'
test_odp_prefetch_async_no_page_fault_rc_traffic (tests.test_odp.OdpTestCase.tes                                                                                                                                                             t_odp_prefetch_async_no_page_fault_rc_traffic) ... skipped 'Advise MR with flags                                                                                                                                                              (0) and advice (2) is not supported'
test_odp_prefetch_sync_no_page_fault_rc_traffic (tests.test_odp.OdpTestCase.test                                                                                                                                                             _odp_prefetch_sync_no_page_fault_rc_traffic) ... skipped 'Advise MR with flags (                                                                                                                                                             1) and advice (2) is not supported'
test_odp_qp_ex_rc_atomic_write (tests.test_odp.OdpTestCase.test_odp_qp_ex_rc_ato                                                                                                                                                             mic_write) ... ERROR


```

Here are the stack of the process:
```
[<0>] rxe_ib_invalidate_range+0x3e/0xa0 [rdma_rxe]
[<0>] __mmu_notifier_invalidate_range_start+0x197/0x200
[<0>] unmap_vmas+0x184/0x190
[<0>] vms_clear_ptes+0x12c/0x190
[<0>] vms_complete_munmap_vmas+0x83/0x1d0
[<0>] do_vmi_align_munmap+0x17f/0x1b0
[<0>] do_vmi_munmap+0xd3/0x190
[<0>] __vm_munmap+0xbb/0x190
[<0>] __x64_sys_munmap+0x1b/0x30
[<0>] x64_sys_call+0x1ea8/0x2660
[<0>] do_syscall_64+0x7e/0x170
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
```
I think this one is related to umem mutex.

So it looks there are two problems:
The stuck issue in rxe_ib_invalidate_range() comes from "RDMA/umem: Store ODP access mask information in PFN",
and the stuck issue in uverbs_destroy_ufile_hw() derives from "RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page linkage".

I'd welcome your help in fixing them.

Thanks,
Daisuke

On 2025/05/21 21:48, Daisuke Matsuda wrote:
> Hi,
> 
> After these two patches are merged to the for-next tree, RXE ODP test always hangs:
>    RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page linkage
>    RDMA/umem: Store ODP access mask information in PFN
> cf. https://lore.kernel.org/linux-rdma/cover.1745831017.git.leon@kernel.org/
> 
> Here is the console log:
> ```
> $ ./build/bin/run_tests.py -v -k odp
> test_odp_dc_traffic (tests.test_mlx5_dc.DCTest.test_odp_dc_traffic) ... skipped 'Can not run the test over non MLX5 device'
> test_devx_rc_qp_odp_traffic (tests.test_mlx5_devx.Mlx5DevxRcTrafficTest.test_devx_rc_qp_odp_traffic) ... skipped 'Can not run the test over non MLX5 device'
> test_odp_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest.test_odp_mkey_list_new_api)
> Create Mkeys above ODP MR, configure it with memory layout using the new API and ... skipped 'Could not open mlx5 context (This is not an MLX5 device)'
> test_odp_async_prefetch_rc_traffic (tests.test_odp.OdpTestCase.test_odp_async_prefetch_rc_traffic) ...
> 
> 
> ```
> 
> It looks that the python process is somehow stuck in uverbs_destroy_ufile_hw():
> ```
> $ sudo cat /proc/1845/task/1845/stack
> [<0>] uverbs_destroy_ufile_hw+0x24/0x100 [ib_uverbs]
> [<0>] ib_uverbs_close+0x1b/0xc0 [ib_uverbs]
> [<0>] __fput+0xea/0x2d0
> [<0>] ____fput+0x15/0x20
> [<0>] task_work_run+0x5d/0xa0
> [<0>] do_exit+0x316/0xa50
> [<0>] make_task_dead+0x81/0x160
> [<0>] rewind_stack_and_make_dead+0x16/0x20
> ```
> 
> I am not sure about the root cause but hope we can fix this before the next merge window.
> 
> Thanks,
> Daisuke