Message-ID: <CAOQ4uxhGX9NVxwsiBMP0q21ZRot6-UA0nGPp1wGNjgmKBjjBBA@mail.gmail.com>
Date:   Tue, 24 Jan 2023 21:06:13 +0200
From:   Amir Goldstein <amir73il@...il.com>
To:     Alexander Larsson <alexl@...hat.com>
Cc:     linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        gscrivan@...hat.com, david@...morbit.com, brauner@...nel.org,
        viro@...iv.linux.org.uk, Vivek Goyal <vgoyal@...hat.com>,
        Miklos Szeredi <miklos@...redi.hu>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
 image filesystem

On Tue, Jan 24, 2023 at 3:13 PM Alexander Larsson <alexl@...hat.com> wrote:
>
> On Tue, 2023-01-24 at 05:24 +0200, Amir Goldstein wrote:
> > On Mon, Jan 23, 2023 at 7:56 PM Alexander Larsson <alexl@...hat.com>
> > wrote:
> > >
> > > On Fri, 2023-01-20 at 21:44 +0200, Amir Goldstein wrote:
> > > > On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
> > > > <alexl@...hat.com>
> > > > wrote:
> > > > >
> > > > > Giuseppe Scrivano and I have recently been working on a new
> > > > > project
> > > > > we
> > > > > call composefs. This is the first time we propose this
> > > > > publically
> > > > > and
> > > > > we would like some feedback on it.
> > > > >
> > > >
> > > > Hi Alexander,
> > > >
> > > > I must say that I am a little bit puzzled by this v3.
> > > > Gao, Christian and myself asked you questions on v2
> > > > that are not mentioned in v3 at all.
> > >
> > > I got lots of good feedback from Dave Chinner on V2 that caused
> > > rather
> > > large changes to simplify the format. So I wanted the new version
> > > with
> > > those changes out to continue that review. I think also having that
> > > simplified version will be helpful for the general discussion.
> > >
> >
> > That's ok.
> > I was not puzzled about why you posted v3.
> > I was puzzled by why you did not mention anything about the
> > alternatives to adding a new filesystem that were discussed on
> > v2 and argue in favor of the new filesystem option.
> > If you post another version, please make sure to include a good
> > explanation for that.
>
> Sure, I will add something to the next version. But like, there was
> already a discussion about this, duplicating that discussion in the v3
> announcement when the v2->v3 changes are unrelated to it doesn't seem
> like it makes a ton of difference.
>
> > > > To sum it up, please do not propose composefs without explaining
> > > > what are the barriers for achieving the exact same outcome with
> > > > the use of a read-only overlayfs with two lower layer -
> > > > uppermost with erofs containing the metadata files, which include
> > > > trusted.overlay.metacopy and trusted.overlay.redirect xattrs that
> > > > refer to the lowermost layer containing the content files.
> > >
> > > So, to be more precise, and so that everyone is on the same page,
> > > lemme
> > > state the two options in full.
> > >
> > > For both options, we have a directory "objects" with content-
> > > addressed
> > > backing files (i.e. files named by sha256). In this directory all
> > > files have fs-verity enabled. Additionally there is an image file
> > > which you downloaded to the system that somehow references the
> > > objects
> > > directory by relative filenames.
> > >
> > > Composefs option:
> > >
> > >  The image file has fs-verity enabled. To use the image, you mount
> > > it
> > >  with options "basedir=objects,digest=$imagedigest".
> > >
> > > Overlayfs option:
> > >
> > >  The image file is a loopback image of a gpt disk with two
> > > partitions,
> > >  one partition contains the dm-verity hashes, and the other
> > > contains
> > >  some read-only filesystem.
> > >
> > >  The read-only filesystem has regular versions of directories and
> > >  symlinks, but for regular files it has sparse files with the
> > > xattrs
> > >  "trusted.overlay.metacopy" and "trusted.overlay.redirect" set, the
> > >  later containing a string like like "/de/adbeef..." referencing a
> > >  backing file in the "objects" directory. In addition, the image
> > > also
> > >  contains overlayfs whiteouts to cover any toplevel filenames from
> > > the
> > >  objects directory that would otherwise appear if objects is used
> > > as
> > >  a lower dir.
> > >
> > >  To use this you loopback mount the file, and use dm-verity to set
> > > up
> > >  the combined partitions, which you then mount somewhere. Then you
> > >  mount an overlayfs with options:
> > >   "metacopy=on,redirect_dir=follow,lowerdir=veritydev:objects"
> > >
> > > I would say both versions of this can work. There are some minor
> > > technical issues with the overlay option:
> > >
> > > * To get actual verification of the backing files you would need to
> > > add support to overlayfs for an "trusted.overlay.digest" xattrs,
> > > with
> > > behaviour similar to composefs.
> > >
> > > * mkfs.erofs doesn't support sparse files (not sure if the kernel
> > > code
> > > does), which means it is not a good option for the backing all
> > > these
> > > sparse files. Squashfs seems to support this though, so that is an
> > > option.
> > >
> >
> > Fair enough.
> > Wasn't expecting for things to work without any changes.
> > Let's first agree that these alone are not a good enough reason to
> > introduce a new filesystem.
> > Let's move on..
>
> Yeah.
>
> > > However, the main issue I have with the overlayfs approach is that
> > > it
> > > is sort of clumsy and over-complex. Basically, the composefs
> > > approach
> > > is laser focused on read-only images, whereas the overlayfs
> > > approach
> > > just chains together technologies that happen to work, but also do
> > > a
> > > lot of other stuff. The result is that it is more work to use it,
> > > it
> > > uses more kernel objects (mounts, dm devices, loopbacks) and it has
> >
> > Up to this point, it's just hand waving, and a bit annoying if I am
> > being honest.
> > overlayfs+metacopy feature were created for the containers use case
> > for very similar set of requirements - they do not just "happen to
> > work"
> > for the same use case.
> > Please stick to technical arguments when arguing in favor of the new
> > "laser focused" filesystem option.
> >
> > > worse performance.
> > >
> > > To measure performance I created a largish image (2.6 GB centos9
> > > rootfs) and mounted it via composefs, as well as overlay-over-
> > > squashfs,
> > > both backed by the same objects directory (on xfs).
> > >
> > > If I clear all caches between each run, a `ls -lR` run on composefs
> > > runs in around 700 msec:
> > >
> > > # hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-
> > > mount"
> > > Benchmark 1: ls -lR cfs-mount
> > >   Time (mean ± σ):     701.0 ms ±  21.9 ms    [User: 153.6 ms,
> > > System: 373.3 ms]
> > >   Range (min … max):   662.3 ms … 725.3 ms    10 runs
> > >
> > > Whereas same with overlayfs takes almost four times as long:
> >
> > No it is not overlayfs, it is overlayfs+squashfs, please stick to
> > facts.
> > As Gao wrote, squashfs does not optimize directory lookup.
> > You can run a test with ext4 for POC as Gao suggested.
> > I am sure that mkfs.erofs sparse file support can be added if needed.
>
> New measurements follow, they now include also erofs over loopback,
> although that isn't strictly fair, because that image is much larger
> due to the fact that it didn't store the files sparsely. It also
> includes a version where the topmost lower is directly on the backing
> xfs (i.e. not via loopback). I attached the scripts used to create the
> images and do the profiling in case anyone wants to reproduce.
>
> Here are the results (on x86-64, xfs base fs):
>
> overlayfs + loopback squashfs - uncached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):      2.483 s ±  0.029 s    [User: 0.167 s, System: 1.656 s]
>   Range (min … max):    2.427 s …  2.530 s    10 runs
>
> overlayfs + loopback squashfs - cached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):     429.2 ms ±   4.6 ms    [User: 123.6 ms, System: 295.0 ms]
>   Range (min … max):   421.2 ms … 435.3 ms    10 runs
>
> overlayfs + loopback ext4 - uncached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):      4.332 s ±  0.060 s    [User: 0.204 s, System: 3.150 s]
>   Range (min … max):    4.261 s …  4.442 s    10 runs
>
> overlayfs + loopback ext4 - cached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):     528.3 ms ±   4.0 ms    [User: 143.4 ms, System: 381.2 ms]
>   Range (min … max):   521.1 ms … 536.4 ms    10 runs
>
> overlayfs + loopback erofs - uncached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):      3.045 s ±  0.127 s    [User: 0.198 s, System: 1.129 s]
>   Range (min … max):    2.926 s …  3.338 s    10 runs
>
> overlayfs + loopback erofs - cached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):     516.9 ms ±   5.7 ms    [User: 139.4 ms, System: 374.0 ms]
>   Range (min … max):   503.6 ms … 521.9 ms    10 runs
>
> overlayfs + direct - uncached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):      2.562 s ±  0.028 s    [User: 0.199 s, System: 1.129 s]
>   Range (min … max):    2.497 s …  2.585 s    10 runs
>
> overlayfs + direct - cached
> Benchmark 1: ls -lR mnt-ovl
>   Time (mean ± σ):     524.5 ms ±   1.6 ms    [User: 148.7 ms, System: 372.2 ms]
>   Range (min … max):   522.8 ms … 527.8 ms    10 runs
>
> composefs - uncached
> Benchmark 1: ls -lR mnt-fs
>   Time (mean ± σ):     681.4 ms ±  14.1 ms    [User: 154.4 ms, System: 369.9 ms]
>   Range (min … max):   652.5 ms … 703.2 ms    10 runs
>
> composefs - cached
> Benchmark 1: ls -lR mnt-fs
>   Time (mean ± σ):     390.8 ms ±   4.7 ms    [User: 144.7 ms, System: 243.7 ms]
>   Range (min … max):   382.8 ms … 399.1 ms    10 runs
>
> For the uncached case, composefs is still almost four times faster than
> the fastest overlay combo (squashfs), and the non-squashfs versions are
> strictly slower. For the cached case the difference is less (10%) but
> with similar order of performance.
>
> For size comparison, here are the resulting images:
>
> 8.6M large.composefs
> 2.5G large.erofs
> 200M large.ext4
> 2.6M large.squashfs
>

Nice.
Clearly, mkfs.ext4 and mkfs.erofs are not optimized for space.
Note that Android has make_ext4fs which can create a compact
ro ext4 image without a journal.
Found this project that builds it outside of Android, but did not test:
https://github.com/iglunix/make_ext4fs

> > > # hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-
> > > mount"
> > > Benchmark 1: ls -lR ovl-mount
> > >   Time (mean ± σ):      2.738 s ±  0.029 s    [User: 0.176 s,
> > > System: 1.688 s]
> > >   Range (min … max):    2.699 s …  2.787 s    10 runs
> > >
> > > With page cache between runs the difference is smaller, but still
> > > there:
> >
> > It is the dentry cache that mostly matters for this test and please
> > use hyerfine -w 1 to warmup dentry cache for correct measurement
> > of warm cache lookup.
>
> I'm not sure why the dentry cache case would be more important?
> Starting a new container will very often not have cached the image.
>
> To me the interesting case is for a new image, but with some existing
> page cache for the backing files directory. That seems to model staring
> a new image in an active container host, but its somewhat hard to test
> that case.
>

ok, you can argue that faster cold cache ls -lR is important
for starting new images.
I think you will be asked to show a real life container use case where
that benchmark really matters.

> > I guess these test runs started with warm cache? but it wasn't
> > mentioned explicitly.
>
> Yes, they were warm (because I ran the previous test before it). But,
> the new profile script explicitly adds -w 1.
>
> > > # hyperfine "ls -lR cfs-mnt"
> > > Benchmark 1: ls -lR cfs-mnt
> > >   Time (mean ± σ):     390.1 ms ±   3.7 ms    [User: 140.9 ms,
> > > System: 247.1 ms]
> > >   Range (min … max):   381.5 ms … 393.9 ms    10 runs
> > >
> > > vs
> > >
> > > # hyperfine -i "ls -lR ovl-mount"
> > > Benchmark 1: ls -lR ovl-mount
> > >   Time (mean ± σ):     431.5 ms ±   1.2 ms    [User: 124.3 ms,
> > > System: 296.9 ms]
> > >   Range (min … max):   429.4 ms … 433.3 ms    10 runs
> > >
> > > This isn't all that strange, as overlayfs does a lot more work for
> > > each lookup, including multiple name lookups as well as several
> > > xattr
> > > lookups, whereas composefs just does a single lookup in a pre-
> > > computed
> >
> > Seriously, "multiple name lookups"?
> > Overlayfs does exactly one lookup for anything but first level
> > subdirs
> > and for sparse files it does the exact same lookup in /objects as
> > composefs.
> > Enough with the hand waving please. Stick to hard facts.
>
> With the discussed layout, in a stat() call on a regular file,
> ovl_lookup() will do lookups on both the sparse file and the backing
> file, whereas cfs_dir_lookup() will just map some page cache pages and
> do a binary search.
>
> Of course if you actually open the file, then cfs_open_file() would do
> the equivalent lookups in /objects. But that is often not what happens,
> for example in "ls -l".
>
> Additionally, these extra lookups will cause extra memory use, as you
> need dentries and inodes for the erofs/squashfs inodes in addition to
> the overlay inodes.
>

I see. composefs is really very optimized for ls -lR.
Now only need to figure out if real users start a container and do ls -lR
without reading many files is a real life use case.

> > > table. But, given that we don't need any of the other features of
> > > overlayfs here, this performance loss seems rather unnecessary.
> > >
> > > I understand that there is a cost to adding more code, but
> > > efficiently
> > > supporting containers and other forms of read-only images is a
> > > pretty
> > > important usecase for Linux these days, and having something
> > > tailored
> > > for that seems pretty useful to me, even considering the code
> > > duplication.
> > >
> > >
> > >
> > > I also understand Cristians worry about stacking filesystem, having
> > > looked a bit more at the overlayfs code. But, since composefs
> > > doesn't
> > > really expose the metadata or vfs structure of the lower
> > > directories it
> > > is much simpler in a fundamental way.
> > >
> >
> > I agree that composefs is simpler than overlayfs and that its
> > security
> > model is simpler, but this is not the relevant question.
> > The question is what are the benefits to the prospect users of
> > composefs
> > that justify this new filesystem driver if overlayfs already
> > implements
> > the needed functionality.
> >
> > The only valid technical argument I could gather from your email is -
> > 10% performance improvement in warm cache ls -lR on a 2.6 GB
> > centos9 rootfs image compared to overlayfs+squashfs.
> >
> > I am not counting the cold cache results until we see results of
> > a modern ro-image fs.
>
> They are all strictly worse than squashfs in the above testing.
>

It's interesting to know why and if an optimized mkfs.erofs
mkfs.ext4 would have done any improvement.

> > Considering that most real life workloads include reading the data
> > and that most of the time inodes and dentries are cached, IMO,
> > the 10% ls -lR improvement is not a good enough reason
> > for a new "laser focused" filesystem driver.
> >
> > Correct me if I am wrong, but isn't the use case of ephemeral
> > containers require that composefs is layered under a writable tmpfs
> > using overlayfs?
> >
> > If that is the case then the warm cache comparison is incorrect
> > as well. To argue for the new filesystem you will need to compare
> > ls -lR of overlay{tmpfs,composefs,xfs} vs. overlay{tmpfs,erofs,xfs}
>
> That very much depends. For the ostree rootfs uscase there would be no
> writable layer, and for containers I'm personally primarily interested
> in "--readonly" containers (i.e. without an writable layer) in my
> current automobile/embedded work. For many container cases however,
> that is true, and no doubt that would make the overhead of overlayfs
> less of a issue.
>
> > Alexander,
> >
> > On a more personal note, I know this discussion has been a bit
> > stormy, but am not trying to fight you.
>
> I'm overall not getting a warm fuzzy feeling from this discussion.
> Getting weird complaints that I'm somehow "stealing" functions or weird
> "who did $foo first" arguments for instance. You haven't personally
> attacked me like that, but some of your comments can feel rather
> pointy, especially in the context of a stormy thread like this. I'm
> just not used to kernel development workflows, so have patience with me
> if I do things wrong.
>

Fair enough.
As long as the things that we discussed are duly
mentioned in future posts, I'll do my best to be less pointy.

> > I think that {mk,}composefs is a wonderful thing that will improve
> > the life of many users.
> > But mount -t composefs vs. mount -t overlayfs is insignificant
> > to those users, so we just need to figure out based on facts
> > and numbers, which is the best technical alternative.
>
> In reality things are never as easy as one thing strictly being
> technically best. There is always a multitude of considerations. Is
> composefs technically better if it uses less memory and performs better
> for a particular usecase? Or is overlayfs technically better because it
> is useful for more usecases and already exists? A judgement needs to be
> made depending on things like complexity/maintainability of the new fs,
> ease of use, measured performance differences, relative importance of
> particular performance measurements, and importance of the specific
> usecase.
>
> It is my belief that the advantages of composefs outweight the cost of
> the code duplication, but I understand the point of view of a
> maintainer of an existing codebase and that saying "no" is often the
> right thing. I will continue to try to argue for my point of view, but
> will try to make it as factual as possible.
>

Improving overlayfs and erofs has additional advantages -
improving performance and size of erofs image may benefit
many other users regardless of the ephemeral containers
use case, so indeed, there are many aspects to consider.

Thanks,
Amir.

Powered by blists - more mailing lists