linux-kernel - Re: [PATCH] scsi: sd: stop SSD (non-rotational) disks before reboot

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200623204234.GA16156@khazad-dum.debian.net>
Date:   Tue, 23 Jun 2020 17:42:34 -0300
From:   Henrique de Moraes Holschuh <hmh@....eng.br>
To:     Damien Le Moal <Damien.LeMoal@....com>
Cc:     Simon Arlott <simon@...iron.net>,
        "James E.J. Bottomley" <jejb@...ux.ibm.com>,
        "Martin K. Petersen" <martin.petersen@...cle.com>,
        Jonathan Corbet <corbet@....net>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        "linux-scsi@...r.kernel.org" <linux-scsi@...r.kernel.org>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>
Subject: Re: [PATCH] scsi: sd: stop SSD (non-rotational) disks before reboot

On Thu, 18 Jun 2020, Damien Le Moal wrote:
> Are you experiencing data loss or corruption ? If yes, since a clean reboot or
> shutdown issues a synchronize cache to all devices, a corruption would mean that
> your SSD is probably not correctly processing flush cache commands.

Cache flushes do not matter that much when SSDs and sudden power cuts
are involved.  Power cuts at the wrong time harm the FLASH itself, it is
not about still-in-flight data.

Keep in mind that SSDs do a _lot_ of background writing, and power cuts
during a FLASH write or erase can cause from weakened cells, to much
larger damage.  It is possible to harden the chip or the design against
this, but it is *expensive*.  And even if warded off by hardening and no
FLASH damage happens, an erase/program cycle must be done on the whole
erase block to clean up the incomplete program cycle.

Due to this background activity, an unexpected power cut could damage
data *anywhere* in an SSD: it could hit some filesystem area that was
being scrubbed in background by the SSD, or internal SSD metadata.

So, you want that SSD to know it must be quiescent-for-poweroff for
*real* before you allow the system to do anything that could power it
off.

And, as I have found out the hard way years ago, you also want to give
the SSD enough *extra* time to actually quiesce, even if it claims to be
already prepared for poweroff [1].

When you do not follow these rules, well, excellent datacenter-class
SSDs have super-capacitor power banks that actually work.  Most SSDs do
not, although they hopefully came a long way and hopefully modern SSDs
are not as easily to brick as they were reported to be three or four
years ago.

[1] I have long lost the will and energy to pursue this, so *this* is a
throw-away anecdote for anyone that cares: I reported here a few years
ago that many models of *SATA* based SSDs from Crucial/Micron, Samsung
and Intel were complaining (through their SMART attributes) that Linux
was causing unsafe shutdowns.

https://lkml.org/lkml/2017/4/10/1181

TL;DR: wait one *extra* second after the SSD acknowleged the STOP
command as complete before you trust the SSD device is safe to be
powered down (i.e. before reboot, suspend, poweroff/shutdown, and device
removal/detach).  This worked around the issue for every vendor and
model of SSD we tested.

-- 
  Henrique Holschuh