linux-kernel - Re: [fstests PATCH v3] generic: add test for DAX MAP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20171116212815.GA27448@linux.intel.com>
Date:   Thu, 16 Nov 2017 14:28:15 -0700
From:   Ross Zwisler <ross.zwisler@...ux.intel.com>
To:     Dave Chinner <david@...morbit.com>
Cc:     Ross Zwisler <ross.zwisler@...ux.intel.com>,
        Amir Goldstein <amir73il@...il.com>,
        Josef Bacik <jbacik@...com>, Eryu Guan <eguan@...hat.com>,
        fstests <fstests@...r.kernel.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Alasdair Kergon <agk@...hat.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Jan Kara <jack@...e.cz>, Mike Snitzer <snitzer@...hat.com>,
        Shaohua Li <shli@...nel.org>, dm-devel@...hat.com,
        linux-nvdimm@...ts.01.org, linux-raid@...r.kernel.org,
        linux-fsdevel <linux-fsdevel@...r.kernel.org>,
        Ext4 <linux-ext4@...r.kernel.org>,
        linux-xfs <linux-xfs@...r.kernel.org>,
        Christoph Hellwig <hch@...radead.org>
Subject: Re: [fstests PATCH v3] generic: add test for DAX MAP_SYNC support

On Thu, Oct 26, 2017 at 08:56:38AM +1100, Dave Chinner wrote:
> On Wed, Oct 25, 2017 at 02:47:04PM -0600, Ross Zwisler wrote:
> > Add a test that exercises DAX's new MAP_SYNC flag.
> > 
> > This test creates a file and writes to it via an mmap(), but never syncs
> > via fsync/msync.  This process is tracked via dm-log-writes, then replayed.
> > 
> > If MAP_SYNC is working the dm-log-writes replay will show the test file
> > with 1MiB of on-media block allocations.  This is because each allocating
> > page fault included an implicit metadata sync.  If MAP_SYNC isn't working
> > (which you can test by fiddling with the parameters to mmap()) the file
> > will be smaller or missing entirely.
> > 
> > Note that dm-log-writes doesn't track the data that we write via the
> > mmap(), so we can't do any data integrity checking.  We can only verify
> > that the metadata writes for the page faults happened.
> > 
> > Signed-off-by: Ross Zwisler <ross.zwisler@...ux.intel.com>
> .....
> > --- /dev/null
> > +++ b/src/t_map_sync.c
> > @@ -0,0 +1,92 @@
> > +#include <errno.h>
> > +#include <fcntl.h>
> > +#include <stdio.h>
> > +#include <stdlib.h>
> > +#include <string.h>
> > +#include <sys/mman.h>
> > +#include <sys/stat.h>
> > +#include <sys/types.h>
> > +#include <unistd.h>
> > +
> > +#define MiB(a) ((a)*1024*1024)
> > +
> > +/*
> > + * These two defines were added to the kernel via commits entitled
> > + * "mm: Define MAP_SYNC and VM_SYNC flags" and
> > + * "mm: introduce MAP_SHARED_VALIDATE, a mechanism to safely define new mmap
> > + * flags", respectively.
> > + */
> > +#ifndef MAP_SYNC
> > +#define MAP_SYNC 0x80000
> > +#endif
> > +
> > +#ifndef MAP_SHARED_VALIDATE
> > +#define MAP_SHARED_VALIDATE 0x3
> > +#endif
> 
> Autoconf rules for detecting supported functionality, please...

Yep, that's better.  As you've suggested down below I'm adding this
functionality to xfs_io instead, and I've added autoconf rules there.

> > +
> > +void err_exit(char *op)
> > +{
> > +	fprintf(stderr, "%s: %s\n", op, strerror(errno));
> > +	exit(1);
> > +}
> > +
> > +void mark_log(char *logwrites_name, char *mark_name)
> > +{
> > +	char command[256];
> > +
> > +	snprintf(command, 256, "dmsetup message %s 0 mark %s",
> > +			logwrites_name, mark_name);
> > +
> > +	if (system(command))
> > +		err_exit("mark_log");
> > +}
> > +
> > +int main(int argc, char *argv[])
> > +{
> > +	int page_size = getpagesize();
> > +	int len = MiB(1);
> > +	int i, fd, err;
> > +	char *data;
> > +
> > +	if (argc < 4) {
> > +		printf("Usage: %s <file> <logwrites_name> <mark_name>\n",
> > +				basename(argv[0]));
> > +		exit(0);
> > +	}
> > +
> > +	fd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
> > +	if (fd < 0)
> > +		err_exit("fd");
> > +
> > +	ftruncate(fd, len);
> > +
> > +	data = mmap(NULL, len, PROT_READ|PROT_WRITE,
> > +			MAP_SHARED_VALIDATE|MAP_SYNC, fd, 0);
> > +	if (data == MAP_FAILED)
> > +		err_exit("mmap");
> 
> As I say to all these sorts of one-off test prgrams: please add the
> new MAP_SYNC flag to xfs_io rather than writing a one-off
> test program to set it and write some data.
> 
> And if we're going to be adding special custom tests just because
> we need to insert dm-log marks, add that functionality to xfs_io,
> too.
> 
> That way we can create complex custom dm logwrite tests without
> needing one-off test programs for them all...

Yep, that was a better path.  I've got things working - need to clean up and
I'll send out soon.

> > +#! /bin/bash
> > +# FS QA Test No. 466
> > +#
> > +# Use md_log_writes to verify that MAP_SYNC actually syncs metadata during
> 
> dm_log_writes?

Fixed.

> > +# We should see $SCRATCH_MNT/test as having 1MiB in block allocations
> > +du -sh $SCRATCH_MNT/test | _filter_scratch | _filter_spaces
> 
> Perhaps stat -c %b $SCRATCH_MNT/test ?

Maybe, but doesn't the output of 'stat -c %b' depend on the block size the
filesystem is using?  I think to use stat I'd have to check both %b and %B,
and account for different block sizes, or do some shell math.  I think it may
be easier to just use du.

Thank you for the review, sorry for the delayed response.