Message-ID: <0100018c40f0e6d8-e9683548-4dde-4320-8ff3-3fd4103c7274-000000@email.amazonses.com>
Date:   Wed, 6 Dec 2023 21:02:58 +0000
From:   John Groves <john@...alactic.com>
To:     Dan Williams <dan.j.williams@...el.com>,
        John Groves <jgroves@...ron.com>,
        John Groves <john@...alactic.com>
Cc:     Vishal Verma <vishal.l.verma@...el.com>,
        Dave Jiang <dave.jiang@...el.com>,
        nvdimm@...ts.linux.dev 
        <nvdimm@...ts.linux.dev>,
        linux-cxl@...r.kernel.org 
        <linux-cxl@...r.kernel.org>,
        linux-kernel@...r.kernel.org 
        <linux-kernel@...r.kernel.org>,
        linux-fsdevel@...r.kernel.org 
        <linux-fsdevel@...r.kernel.org>,
        John Groves <john@...ves.net>
Subject: [PATCH RFC 0/4] dev_dax_iomap: introduce iomap for /dev/dax

From: John Groves <john@...ves.net>

This patch set is not intended to be merged; I'm hoping to get some
clarification as to the correct approach (especialy from Dan). 

This work is related to famfs, which is a dax file system for shared
fabric-attached memory (FAM). Famfs is "coming soon" as an RFC, but
the concept and requirements were presented at LPC 2023. See
https://lpc.events/event/17/contributions/1455/ and
https://www.youtube.com/watch?v=aA_DgO95gLo. My expectation is that
a future (fully working) version of this patch will be folded into the
famfs
patches.

Unlike the current set of fs-dax file systems, famfs does not need a block
(pmem) device, and should really run on a /dev/dax character device since
that's how sharable fabric-attached cxl memory will surface. But
/dev/dax character devices are missing some functionality that is provided
by the block /dev/pmem driver - specifically struct dax_operations pointer
in struct dax_device.

This patch, when CONFIG_DEV_DAX_IOMAP=y, populates dax_dev->ops for
character dax devices. The added operations differ (currently) from
/dev/pmem's dev_dax->ops in that they don't use memremap but instead
provide a physical address in response to the dev_dax->direct_access()
method. 

The dax_operations are direct_access() (which resolves a dax dev offset
to an address), zero_page_range() and recovery_write(). I'm not sure yet
how to test the latter two, but the direct_access() method works in
conjunciton with famfs - but only for mmaped files.

But Posix reads fail. Specifically dax_iomap_iter() calls
dax_copy_to_iter(), which declines to copy the data for some reason in
one of the lower level copy_to_user variants. I've tried to isolate the
reason for the failure with a VM under gdb, but no luck there yet. I'm
wondering if there is some flag or attribute that needs to be applied to
these addresses/pages somehow to allow this to work.

The failing copy_to_user() differs from the same path with pmem fs-dax,
in that pmem does a memremap (which I think generates one contiguous
range, even if the device had more than one range - is this right, and
does this mean it's consuming some of the vmap/vmalloc range?)

I spent some time attempting a memremap, but I haven't figured out the
magic for that. However, I like the simplicity of resolving to phys if
that's not a non-starter for some reason.

I hope this is enough context for a meaningful review and suggestions as to
what a working dev_dax->dax_operations implementation should look like.

Thanks for any tips!


John Groves (4):
  Add add_dax_ops() func for fs-dax to provide dax holder_ops
  Temporary hacks due to linkage issues
  Add dax_operations to /dev/dax struct dax_device
  Add CONFIG_DEV_DAX_IOMAP kernel build parameter

 drivers/dax/Kconfig |   6 ++
 drivers/dax/bus.c   | 155 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/dax/super.c |  16 +++++
 include/linux/dax.h |   5 ++
 4 files changed, 182 insertions(+)

-- 
2.40.1

Powered by blists - more mailing lists