linux-kernel - Re: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout request with low probability

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a5cfe4e4-62b8-5cea-c4bc-06d905ffccf1@nvidia.com>
Date:   Sat, 8 Apr 2023 08:04:01 +0000
From:   Chaitanya Kulkarni <chaitanyak@...dia.com>
To:     Lei Lei2 Yin <yinlei2@...ovo.com>,
        "kbusch@...nel.org" <kbusch@...nel.org>,
        "axboe@...com" <axboe@...com>, "hch@....de" <hch@....de>,
        Sagi Grimberg <sagi@...mberg.me>
CC:     "linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "cybeyond@...mail.com" <cybeyond@...mail.com>
Subject: Re: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout
 request with low probability

On 4/8/23 00:44, Lei Lei2 Yin wrote:
>  From 5148d52554d6bcf6134786d40f1c6f9f22e18978 Mon Sep 17 00:00:00 2001
> From: Lei Yin <yinlei2@...ovo.com>
> Date: Thu, 6 Apr 2023 23:39:11 +0800
> Subject: [PATCH v2] nvme: fix double blk_mq_complete_request for timeout
>   request with low probability
>
> When nvme_cancel_tagset traverses all tagsets and executes
> nvme_cancel_request, this request may be executing blk_mq_free_request
> that is called by nvme_rdma_complete_timed_out/nvme_tcp_complete_timed_out.
> When blk_mq_free_request executes to WRITE_ONCE(rq->state, MQ_RQ_IDLE) and
> __blk_mq_free_request(rq), it will cause double blk_mq_complete_request for
> this request, and it will cause a null pointer error in the second
> execution of this function because rq->mq_hctx has set to NULL in first
> execution.
>
> With multipath, by injecting a large number of requests timed out, I have
> reproduced the issue that caused kernel crashes in three versions of the
> kernel(include 5.10.167, 6.2.10 and upstream version which compiled by
> myself). The error stack is as follows:
>
> [ 2777.253091]  <TASK>
> [ 2777.253102]  nvme_failover_req+0x10a/0x120 [nvme_core]
> [ 2777.255302]  blk_complete_reqs+0x3e/0x60
> [ 2777.255726]  __do_softirq+0xb6/0x2ad
> [ 2777.256139]  ? __pfx_smpboot_thread_fn+0x10/0x10
> [ 2777.256556]  run_ksoftirqd+0x28/0x40
> [ 2777.256978]  smpboot_thread_fn+0xdb/0x1d0
> [ 2777.257399]  kthread+0xd7/0x100
> [ 2777.257827]  ? __pfx_kthread+0x10/0x10
> [ 2777.258253]  ret_from_fork+0x29/0x50
> [ 2777.258695]  </TASK>
>
> Signed-off-by: Lei Yin <yinlei2@...ovo.com>
> ---
>

Can you please add blktests for this fix under nvme category ?

Looks good, it also addresses Sagi's comment about

nvmf_complete_timed_out_request().

Reviewed-by: Chaitanya Kulkarni <kch@...dia.com>

-ck