linux-ext4 - Re: [PATCH v2 5/5] dax: handle media errors in dax_do

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1461628381.1421.24.camel@intel.com>
Date:	Mon, 25 Apr 2016 23:53:13 +0000
From:	"Verma, Vishal L" <vishal.l.verma@...el.com>
To:	"david@...morbit.com" <david@...morbit.com>
CC:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
	"hch@...radead.org" <hch@...radead.org>,
	"xfs@....sgi.com" <xfs@....sgi.com>,
	"linux-nvdimm@...1.01.org" <linux-nvdimm@...1.01.org>,
	"jmoyer@...hat.com" <jmoyer@...hat.com>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	"viro@...iv.linux.org.uk" <viro@...iv.linux.org.uk>,
	"axboe@...com" <axboe@...com>,
	"akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	"Wilcox, Matthew R" <matthew.r.wilcox@...el.com>,
	"jack@...e.cz" <jack@...e.cz>
Subject: Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io

On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> 
<>

> > 
> > - It checks badblocks and discovers it's files have lost data
> Lots of hand-waving here. How does the application map a bad
> "sector" to a file without scanning the entire filesystem to find
> the owner of the bad sector?

Yes this was hand-wavey, but we talked about this a bit at LSF..
The idea is that a per-block-device badblocks list is available at
/sys/block/<pmemX>/badblocks. The application (or a suitable yet-to-be-
written library function) does a fiemap to figure out the sectors its
files are using, and correlates the two lists.
We can also look into providing an easier-to-use interface from the
kernel, in the form of an fiemap flag to report only the bad sectors, or
a SEEK_BAD flag..
The application doesn't have to scan the entire filesystem, but
presumably it knows what files it 'owns', and does a fiemap for those.

> 
> > 
> > - It write()s those sectors (possibly converted to file offsets
> > using
> > fiemap)
> >     * This triggers the fallback path, but if the application is
> > doing
> > this level of recovery, it will know the sector is bad, and write
> > the
> > entire sector
> Where does the application find the data that was lost to be able to
> rewrite it?

The data that was lost is gone -- this assumes the application has some
ability to recover using a journal/log or other redundancy - yes, at the
application layer. If it doesn't have this sort of capability, the only
option is to restore files from a backup/mirror.

> 
> > 
> > - Or it replaces the entire file from backup also using write() (not
> > mmap+stores)
> >     * This just frees the fs block, and the next time the block is
> > reallocated by the fs, it will likely be zeroed first, and that will
> > be
> > done through the driver and will clear errors
> There's an implicit assumption that applications will keep redundant
> copies of their data at the /application layer/ and be able to
> automatically repair it? And then there's the implicit assumption
> that it will unlink and free the entire file before writing a new
> copy, and that then assumes the the filesystem will zero blocks if
> they get reused to clear errors on that LBA sector mapping before
> they are accessible again to userspace..
> 
> It seems to me that there are a number of assumptions being made
> across multiple layers here. Maybe I've missed something - can you
> point me to the design/architecture description so I can see how
> "app does data recovery itself" dance is supposed to work?

There isn't a document other than the flow in my head :) - but maybe I
could write one up..
I wasn't thinking the application itself maintains and restores from
backup copy of the file.. The application hits either a SIGBUS or EIO
depending on how it accesses the data, and crashes or raises some alarm.
The recovery is then done out-of-band, by a sysadmin or such (i.e.
delete the file, replace with a known good copy, restart application).

To summarize, the two cases we want to handle are:
1. Application has inbuilt recovery:
  - hits badblock
  - figures out it is able to recover the data
  - handles SIGBUS or EIO
  - does a (sector aligned) write() to restore the data
2. Application doesn't have any inbuilt recovery mechanism
  - hits badblock
  - gets SIGBUS (or EIO) and crashes
  - Sysadmin restores file from backup

Case 1 is handled by either a fallback to direct_IO from dax_do_io, or
always _actually_ doing direct_IO when we're opened with O_DIRECT in
spite of dax (what Dan suggested). Currently if we're mounted with dax,
all IO O_DIRECT or otherwise will go through dax_do_io.
Case 2 is handled by patch 4 of the series:
    dax: use sb_issue_zerout instead of calling dax_clear_sectors

> 
> Cheers,
> 
> Dave.