Message-ID: <e6f3946a-dbe7-6b42-e43c-d3f8d705c732@intel.com>
Date:   Fri, 23 Apr 2021 09:18:47 +0300
From:   Adrian Hunter <adrian.hunter@...el.com>
To:     "Asutosh Das (asd)" <asutoshd@...eaurora.org>, cang@...eaurora.org,
        martin.petersen@...cle.com, linux-scsi@...r.kernel.org
Cc:     linux-arm-msm@...r.kernel.org,
        Alim Akhtar <alim.akhtar@...sung.com>,
        Avri Altman <avri.altman@....com>,
        "James E.J. Bottomley" <jejb@...ux.ibm.com>,
        Krzysztof Kozlowski <krzk@...nel.org>,
        Stanley Chu <stanley.chu@...iatek.com>,
        Andy Gross <agross@...nel.org>,
        Bjorn Andersson <bjorn.andersson@...aro.org>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ingo Molnar <mingo@...hat.com>,
        Matthias Brugger <matthias.bgg@...il.com>,
        Lee Jones <lee.jones@...aro.org>,
        Bean Huo <beanhuo@...ron.com>,
        Kiwoong Kim <kwmad.kim@...sung.com>,
        Colin Ian King <colin.king@...onical.com>,
        Wei Yongjun <weiyongjun1@...wei.com>,
        Yue Hu <huyue2@...ong.com>,
        Bart van Assche <bvanassche@....org>,
        "Gustavo A. R. Silva" <gustavoars@...nel.org>,
        Dinghao Liu <dinghao.liu@....edu.cn>,
        Jaegeuk Kim <jaegeuk@...nel.org>,
        Satya Tangirala <satyat@...gle.com>,
        open list <linux-kernel@...r.kernel.org>,
        "moderated list:ARM/SAMSUNG S3C, S5P AND EXYNOS ARM ARCHITECTURES" 
        <linux-arm-kernel@...ts.infradead.org>,
        "open list:ARM/SAMSUNG S3C, S5P AND EXYNOS ARM ARCHITECTURES" 
        <linux-samsung-soc@...r.kernel.org>,
        "moderated list:UNIVERSAL FLASH STORAGE HOST CONTROLLER DRIVER..." 
        <linux-mediatek@...ts.infradead.org>
Subject: Re: [PATCH v20 1/2] scsi: ufs: Enable power management for wlun

On 23/04/21 7:23 am, Adrian Hunter wrote:
> On 22/04/21 7:38 pm, Asutosh Das (asd) wrote:
>> On 4/20/2021 12:42 AM, Adrian Hunter wrote:
>>> On 20/04/21 7:15 am, Adrian Hunter wrote:
>>>> On 20/04/21 12:53 am, Asutosh Das (asd) wrote:
>>>>> On 4/19/2021 11:37 AM, Adrian Hunter wrote:
>>>>>> On 16/04/21 10:49 pm, Asutosh Das wrote:
>>>>>>>
>>>>>>> Co-developed-by: Can Guo <cang@...eaurora.org>
>>>>>>> Signed-off-by: Can Guo <cang@...eaurora.org>
>>>>>>> Signed-off-by: Asutosh Das <asutoshd@...eaurora.org>
>>>>>>> ---
>>>>>>
>>>>>> I came across 3 issues while testing.  See comments below.
>>>>>>
>>>>> Hi Adrian
>>>>> Thanks for the comments.
>>>>>> <SNIP>
>>>>>>
>>>>>>> @@ -5794,7 +5839,7 @@ static void ufshcd_err_handling_unprepare(struct ufs_hba *hba)
>>>>>>>        if (ufshcd_is_clkscaling_supported(hba))
>>>>>>>            ufshcd_clk_scaling_suspend(hba, false);
>>>>>>>        ufshcd_clear_ua_wluns(hba);
>>>>>>
>>>>>> ufshcd_clear_ua_wluns() deadlocks trying to clear UFS_UPIU_RPMB_WLUN
>>>>>> if sdev_rpmb is suspended and sdev_ufs_device is suspending.
>>>>>> e.g. ufshcd_wl_suspend() is waiting on host_sem while ufshcd_err_handler()
>>>>>> is running, at which point sdev_rpmb has already suspended.
>>>>>>
>>>>> Umm, I didn't understand this deadlock.
>>>>> When you say, sdev_rpmb is suspended, does it mean runtime_suspended?
>>>>> sdev_ufs_device is suspending - this can't be runtime_suspending, while ufshcd_err_handling_unprepare is running.
>>>>>
>>>>> If you've a call-stack of this deadlock, please can you share it with me. I'll also try to reproduce this.
>>>>
>>>> Yes it is system suspend. sdev_rpmb has suspended, sdev_ufs_device is waiting on host_sem.
>>>> ufshcd_err_handler() holds host_sem. ufshcd_clear_ua_wlun(UFS_UPIU_RPMB_WLUN) gets stuck.
>>>> I will get some call-stacks.
>>>
>> Hi Adrian,
>>
>> Thanks for the call stacks.
>> From the current information, I can't say for sure why it'd get stuck in blk_queue_enter().
> 
> I presume SCSI is leaving the RPMB WLUN device runtime suspended and consequently the queue status is RPM_SUSPENDED
> 
>>
>> I tried reproducing this issue on my setup yesterday but couldn't.
>> Here's what I did:
>> 1. sdev_rpmb is RPM_SUSPENDED, checked before initiating system suspend
>> 2. sdev_ufs_device is RPM_RESUMED
>> 3. I triggered system suspend (echo mem > /sys/power/state) and scheduled the error handler from ufshcd_wl_suspend().
>> 4. Waited until error handler ran and then ufshcd_wl_suspend() blocks on host_sem.
>> 5. The ufshcd_clear_wa_wlun(UFS_UPIU_RPMB_WLUN) went through fine.
>>
>> Do you've some specific steps to reproduce this or a script, perhaps? If so, please can you share it with me. I will try again.
> 
> I was using a device that gives occasional errors, but I will what see I can do.

I have attached 2 patches to reproduce the issue, and the test script I used.
Note it is using down_timeout() so it doesn't lock up completely.
The kernel messages are:

[   79.385456] ufshcd_wl_suspend: doing ufshcd_schedule_eh_work
[   79.385511] ufshcd_wl_suspend: sleeping 1000 ms
[   79.386916] ufshcd_err_handler: start
[   79.386979] ufshcd_err_handler: got sem
[   79.387053] ufshcd_err_handling_unprepare: start
[   79.387302] ufshcd_clear_ua_wluns: before ufshcd_clear_ua_wlun(hba, UFS_UPIU_RPMB_WLUN)
[   80.435878] ufshcd_wl_suspend: trying to get sem
[   85.683876] ufshcd_wl_suspend: failed to get sem
[   85.683993] ufs_device_wlun 0:0:0:49488: PM: failed to suspend async: error -22
[   85.686901] ufshcd_clear_ua_wluns: after ufshcd_clear_ua_wlun(hba, UFS_UPIU_RPMB_WLUN), ret 0
[   85.687025] ufshcd_err_handling_unprepare: finish
[   85.687090] ufshcd_err_handler: finish 



View attachment "0001-changes-suggested-since-v20.patch" of type "text/x-patch" (3277 bytes)

View attachment "0002-instrumentation-to-reproduce-issue.patch" of type "text/x-patch" (3069 bytes)

Download attachment "go.sh" of type "application/x-shellscript" (316 bytes)

Powered by blists - more mailing lists