Message-ID: <20210816211434.GB12640@magnolia>
Date:   Mon, 16 Aug 2021 14:14:34 -0700
From:   "Darrick J. Wong" <djwong@...nel.org>
To:     linux-xfs@...r.kernel.org, jane.chu@...cle.com,
        willy@...radead.org, tytso@....edu, linux-ext4@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, sandeen@...deen.net
Subject: Re: [PATCHSET 0/2] dax: fix broken pmem poison narrative

On Mon, Aug 16, 2021 at 02:05:18PM -0700, Darrick J. Wong wrote:
> Hi all,
> 
> Our current "advice" to people using persistent memory and FSDAX who
> wish to recover upon receipt of a media error (aka 'hwpoison') event
> from ACPI is to punch-hole that part of the file and then pwrite it,
> which will magically cause the pmem to be reinitialized and the poison
> to be cleared.
> 
> Punching doesn't make any sense at all -- we don't allow userspace to
> allocate from specific parts of the storage, and another writer could
> grab the poisoned range in the meantime.  In other words, the advice is
> seriously overfitted to incidental xfs and ext4 behavior and can
> completely fail.  Worse yet, that concurrent writer now has to deal with
> the poison that it didn't know about, and someone else is trying to fix.
> 
> AFAICT, the only reason why the "punch and write" dance works at all is
> that the XFS and ext4 currently call blkdev_issue_zeroout when
> allocating pmem as part of a pwrite call.  A pwrite without the punch
> won't clear the poison, because pwrite on a DAX file calls
> dax_direct_access to access the memory directly, and dax_direct_access
> is only smart enough to bail out on poisoned pmem.  It does not know how
> to clear it.  Userspace could solve the problem by calling FIEMAP and
> issuing a BLKZEROOUT, but that requires rawio capabilities.
> 
> The whole pmem poison recovery story is is wrong and needs to be
> corrected ASAP before everyone else starts doing this.  Therefore,
> create a dax_zeroinit_range function that filesystems can call to reset
> the contents of the pmem to a known value and clear any state associated
> with the media error.  Then, connect FALLOC_FL_ZERO_RANGE to this new
> function (for DAX files) so that unprivileged userspace has a safe way
> to reset the pmem and clear media errors.

This is a sample copy of a SIGBUS handler that will dump out the siginfo
data, call ZERO_RANGE to clear the poison, and then simulates being
fortunate enough to be able to reconstruct the file contents from
scratch.

Note that I haven't tested this even with simulated pmem because I
cannot figure out how to inject a poison error into the pmem in such a
way that the nvdimm driver records it in the badblocks table.
madvise(HWPOISON) calls the SIGBUS handler, but that code path never
goes outside of the memory manager.

int fd = open(...);
char *data = mmap(fd, ... MAP_SYNC);

static void handle_sigbus(int signal, siginfo_t *info, void *dontcare)
{
	char *buf;
	loff_t err_offset = (char *)info->si_addr - data;
	loff_t err_len = (1ULL << info->si_addr_lsb);
	ssize_t ret;

	printf("RECEIVED SIGBUS (POISON HANDLER)!\n");
	printf("    signal %d\n", info->si_signo);
	printf("    errno %d\n", info->si_errno);
	printf("    addr %p\n", info->si_addr);
	printf("    addr_lsb %d\n", info->si_addr_lsb);

	if (info->si_signo != SIGBUS) {
		printf("    code 0x%x\n", info->si_code);
		return;
	}

	switch (info->si_code) {
	case BUS_ADRALN:
		printf("    code: BUS_ADRALN\n");
		break;
	case BUS_ADRERR:
		printf("    code: BUS_ADRERR\n");
		break;
	case BUS_OBJERR:
		printf("    code: BUS_OBJERR\n");
		break;
	case BUS_MCEERR_AR:
		printf("    code: BUS_MCEERR_AR\n");
		break;
	case BUS_MCEERR_AO:
		printf("    code: BUS_MCEERR_AO\n");
		break;
	default:
		printf("    code 0x%x\n", info->si_code);
		break;
	}

	printf("    err_offset %lld\n", (unsigned long long)err_offset);
	printf("    err_len %lld\n", (unsigned long long)err_len);

	if (info->si_code != BUS_MCEERR_AR)
		return;

	/* clear poison and reset pmem to initial value */
	ret = fallocate(fd, FALLOC_FL_ZERO_RANGE, err_offset, err_len);
	if (ret) {
		perror("fallocate");
		exit(9);
	}

	/* simulate being lucky enough to be able to reconstruct the data */
	buf = malloc(err_len);
	if (!buf) {
		perror("malloc pwrite buf");
		exit(10);
	}

	memset(buf, 0x59, err_len);

	ret = pwrite(fd, buf, err_len, err_offset);
	if (ret < 0) {
		perror("pwrite");
		exit(11);
	}
	if (ret != err_len) {
		fprintf(stderr, "short write %zd bytes, wanted %lld\n",
				ret, (long long)err_len);
		exit(12);
	}

	free(buf);
}

--D

> 
> If you're going to start using this mess, you probably ought to just
> pull from my git trees, which are linked below.
> 
> This is an extraordinary way to destroy everything.  Enjoy!
> Comments and questions are, as always, welcome.
> 
> --D
> 
> kernel git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=dax-zeroinit-clear-poison-5.15
> ---
>  fs/dax.c            |   72 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  fs/ext4/extents.c   |   19 +++++++++++++
>  fs/xfs/xfs_file.c   |   20 ++++++++++++++
>  include/linux/dax.h |    7 +++++
>  4 files changed, 118 insertions(+)
> 

Powered by blists - more mailing lists