linux-kernel - Re: [Bug report] NULL pointer dereference in frwr_unmap

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <89b0c30b-9b7b-4507-ba1f-0493e88c6791@oracle.com>
Date: Sun, 9 Mar 2025 12:40:37 -0700
From: Dai Ngo <dai.ngo@...cle.com>
To: Li Nan <linan666@...weicloud.com>, Chuck Lever <chuck.lever@...cle.com>
Cc: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        linux-nfs@...r.kernel.org, trondmy@...merspace.com, sagi@...mberg.me,
        cel@...nel.org, "wanghai (M)" <wanghai38@...wei.com>,
        yanhaitao2@...wei.com, chengjike.cheng@...wei.com,
        dingming09@...wei.com
Subject: Re: [Bug report] NULL pointer dereference in frwr_unmap_sync()

Hi Nan,

Can you try the attached patch with 6.14-rc4?

This patch adds a spinlock to protect the rl_free_mrs and rl_registered list.
I have seen list corruption in our RDMA testing but the problem is hard to
reproduce so I have not submitted this patch to upstream.

Thanks,
-Dai

On 3/5/25 6:40 PM, Li Nan wrote:
>
>
> 在 2025/3/5 22:02, Chuck Lever 写道:
>> On 3/4/25 9:43 PM, Li Nan wrote:
>>> We found a following problem in kernel 5.10, and the same problem 
>>> should
>>> exist in mainline:
>>>
>>> During NFS mount using 'soft' option over RoCE network, we observed 
>>> kernel
>>> crash with below trace when network issues occur 
>>> (congestion/disconnect):
>>>    nfs: server 10.10.253.211 not responding, timed out
>>>    BUG: kernel NULL pointer dereference, address: 00000000000000a0
>>>    RIP: 0010:frwr_unmap_sync+0x77/0x200 [rpcrdma]
>>>    Call Trace:
>>>     ? __die_body.cold+0x8/0xd
>>>     ? no_context+0x155/0x230
>>>     ? __bad_area_nosemaphore+0x52/0x1a0
>>>     ? exc_page_fault+0x2dc/0x550
>>>     ? asm_exc_page_fault+0x1e/0x30
>>>     ? frwr_unmap_sync+0x77/0x200 [rpcrdma]
>>>     xprt_release+0x9e/0x1a0 [sunrpc]
>>>     rpc_release_resources_task+0xe/0x50 [sunrpc]
>>>     rpc_release_task+0x19/0xa0 [sunrpc]
>>>     rpc_async_schedule+0x29/0x40 [sunrpc]
>>>     process_one_work+0x1b2/0x350
>>>     worker_thread+0x49/0x310
>>>     ? rescuer_thread+0x380/0x380
>>>     kthread+0xfb/0x140
>>>
>>> Problem analysis:
>>> The crash happens in frwr_unmap_sync() when accessing 
>>> req->rl_registered
>>> list, caused by either NULL pointer or accessing freed MR resources.
>>> There's a race condition between:
>>> T1
>>> __ib_process_cq
>>>   wc->wr_cqe->done (frwr_wc_localinv)
>>>    rpcrdma_flush_disconnect
>>>     rpcrdma_force_disconnect
>>>      xprt_force_disconnect
>>>       xprt_autoclose
>>>        xprt_rdma_close
>>>         rpcrdma_xprt_disconnect
>>>          rpcrdma_reqs_reset
>>>           frwr_reset
>>>            rpcrdma_mr_pop(&req->rl_registered)
>>> T2
>>> rpc_async_schedule
>>>   rpc_release_task
>>>    rpc_release_resources_task
>>>     xprt_release
>>>      xprt_rdma_free
>>>       frwr_unmap_sync
>>>        rpcrdma_mr_pop(&req->rl_registered)
>>>                     This problem also exists in function 
>>> rpcrdma_mrs_destroy().
>>>
>>
>> Dai, is this the same as the system test problem you've been looking at?
>>
>
> Thank you for looking into it. Is there a patch that needs to be 
> tested? We
> are happy to help with the testing.
>
View attachment "0001-xprtrdma-add-spinlock-in-rpcrdma_req-to-protect-rl_f.patch" of type "text/plain" (8099 bytes)