linux-kernel - Re: [REQUEST DISCUSS]: speed up SCSI error handle for host with massive devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 30 Mar 2022 17:11:36 +0800
From:   Wenchao Hao <haowenchao@...wei.com>
To:     Hannes Reinecke <hare@...e.de>,
        Steffen Maier <maier@...ux.ibm.com>,
        <linux-scsi@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "James E.J. Bottomley" <jejb@...ux.ibm.com>,
        "Martin K. Petersen" <martin.petersen@...cle.com>,
        Mike Christie <michael.christie@...cle.com>,
        Lee Duncan <lduncan@...e.com>
CC:     Wu Bo <wubo40@...wei.com>, Feilong Lin <linfeilong@...wei.com>,
        <zhangjian013@...wei.com>, Wenchao Hao <haowenchao@...wei.com>
Subject: Re: [REQUEST DISCUSS]: speed up SCSI error handle for host with
 massive devices

On 2022/3/30 2:56, Hannes Reinecke wrote:
> On 3/29/22 14:40, Wenchao Hao wrote:
>> On 2022/3/29 18:56, Steffen Maier wrote:
>>> On 3/29/22 11:06, Wenchao Hao wrote:
>>>> SCSI timeout would call scsi_eh_scmd_add() on some conditions, host would be set
>>>> to SHOST_RECOVERY state. Once host enter SHOST_RECOVERY, IOs submitted to all
>>>> devices in this host would not succeed until the scsi_error_handler() finished.
>>>> The scsi_error_handler() might takes long time to be done, it's unbearable when
>>>> host has massive devices.
>>>>
>>>> I want to ask is anyone applying another error handler flow to address this
>>>> phenomenon?
>>>>
>>>> I think we can move some operations(like scsi get sense, scsi send startunit
>>>> and scsi device reset) out of scsi_unjam_host(), to perform these operations
>>>> without setting host to SHOST_RECOVERY? It would reduce the time of block the
>>>> whole host.
>>>>
>>>> Waiting for your discussion.
>>>
>>> We already have "async" aborts before even entering scsi_eh. So your use case seems to imply that those aborts fail and we enter scsi_eh?
>>>
>>
>> Yes, I mean when scsi_abort_command() failed and scsi_eh_scmd_add() is called.
>>
>>> There's eh_deadline for limiting the time spent in escalation of scsi_eh, and instead directly go to host reset. Would this help?
>>>
>>>
>>
>> The deadline seems not helpful. What we want to see is a single LUN's command error
>> would not stop other LUNs which share the same host. So my plan is to move reset LUN out
>> from scsi_unjam_host() which run with host set to SHOST_RECOVERY.
> 
> Nope. One of the key points of scsi_unjam_host() is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.
> 

I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O
is because we might reset host during scsi_error_handler() and we must wait host's number of
failed command equal to number of busy command then we can wake up scsi_error_handler(). 

If move reset LUN out of scsi_error_handler(), and perform single LUN reset, we only need
stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed,
we can then call in the large scale error handle.

Here is a brief flow:

abort command
   ||
   || failed
   || 
   \/
stop single LUN's I/O (need to wait LUN's failed command number equal to busy command  number)
   ||
   || failed  (according to our statistic, 90% reset LUN would succeed)
   ||
   \/
stop all LUN's I/O (stage to call in origin error handler)
   ||
   ||
   ||
   \/
wait host's failed command number equal to busy command number
   ||
   ||
   ||
   \/
perform target reset, bus reset and host reset and so on. (do not need to reset LUN since it already failed)

As mentioned, we can reduce the time of setting host to SHOST_RECOVERY which would stop all I/Os.

How do you think? I look forward to hearing from you.