[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <1d65be2f-6d3a-13c6-4982-66bbb0f9b530@linux.alibaba.com>
Date: Tue, 24 Jan 2023 22:40:39 +0800
From: Gao Xiang <hsiangkao@...ux.alibaba.com>
To: Alexander Larsson <alexl@...hat.com>,
Amir Goldstein <amir73il@...il.com>
Cc: linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
gscrivan@...hat.com, david@...morbit.com, brauner@...nel.org,
viro@...iv.linux.org.uk, Vivek Goyal <vgoyal@...hat.com>,
Miklos Szeredi <miklos@...redi.hu>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
image filesystem
On 2023/1/24 21:10, Alexander Larsson wrote:
> On Tue, 2023-01-24 at 05:24 +0200, Amir Goldstein wrote:
>> On Mon, Jan 23, 2023 at 7:56 PM Alexander Larsson <alexl@...hat.com>
...
>>
>> No it is not overlayfs, it is overlayfs+squashfs, please stick to
>> facts.
>> As Gao wrote, squashfs does not optimize directory lookup.
>> You can run a test with ext4 for POC as Gao suggested.
>> I am sure that mkfs.erofs sparse file support can be added if needed.
>
> New measurements follow, they now include also erofs over loopback,
> although that isn't strictly fair, because that image is much larger
> due to the fact that it didn't store the files sparsely. It also
> includes a version where the topmost lower is directly on the backing
> xfs (i.e. not via loopback). I attached the scripts used to create the
> images and do the profiling in case anyone wants to reproduce.
>
> Here are the results (on x86-64, xfs base fs):
>
> overlayfs + loopback squashfs - uncached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 2.483 s ± 0.029 s [User: 0.167 s, System: 1.656 s]
> Range (min … max): 2.427 s … 2.530 s 10 runs
>
> overlayfs + loopback squashfs - cached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 429.2 ms ± 4.6 ms [User: 123.6 ms, System: 295.0 ms]
> Range (min … max): 421.2 ms … 435.3 ms 10 runs
>
> overlayfs + loopback ext4 - uncached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 4.332 s ± 0.060 s [User: 0.204 s, System: 3.150 s]
> Range (min … max): 4.261 s … 4.442 s 10 runs
>
> overlayfs + loopback ext4 - cached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 528.3 ms ± 4.0 ms [User: 143.4 ms, System: 381.2 ms]
> Range (min … max): 521.1 ms … 536.4 ms 10 runs
>
> overlayfs + loopback erofs - uncached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 3.045 s ± 0.127 s [User: 0.198 s, System: 1.129 s]
> Range (min … max): 2.926 s … 3.338 s 10 runs
>
> overlayfs + loopback erofs - cached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 516.9 ms ± 5.7 ms [User: 139.4 ms, System: 374.0 ms]
> Range (min … max): 503.6 ms … 521.9 ms 10 runs
>
> overlayfs + direct - uncached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 2.562 s ± 0.028 s [User: 0.199 s, System: 1.129 s]
> Range (min … max): 2.497 s … 2.585 s 10 runs
>
> overlayfs + direct - cached
> Benchmark 1: ls -lR mnt-ovl
> Time (mean ± σ): 524.5 ms ± 1.6 ms [User: 148.7 ms, System: 372.2 ms]
> Range (min … max): 522.8 ms … 527.8 ms 10 runs
>
> composefs - uncached
> Benchmark 1: ls -lR mnt-fs
> Time (mean ± σ): 681.4 ms ± 14.1 ms [User: 154.4 ms, System: 369.9 ms]
> Range (min … max): 652.5 ms … 703.2 ms 10 runs
>
> composefs - cached
> Benchmark 1: ls -lR mnt-fs
> Time (mean ± σ): 390.8 ms ± 4.7 ms [User: 144.7 ms, System: 243.7 ms]
> Range (min … max): 382.8 ms … 399.1 ms 10 runs
>
> For the uncached case, composefs is still almost four times faster than
> the fastest overlay combo (squashfs), and the non-squashfs versions are
> strictly slower. For the cached case the difference is less (10%) but
> with similar order of performance.
>
> For size comparison, here are the resulting images:
>
> 8.6M large.composefs
> 2.5G large.erofs
> 200M large.ext4
> 2.6M large.squashfs
Ok, I have to say I'm a bit surprised by these results. Just a wild guess,
`ls -lR` is a seq-like access, so that compressed data (assumed that you
use it) is benefited from it. I cannot think of a proper cause before
looking into more. EROFS is impacted since EROFS on-disk inodes are not
arranged together with the current mkfs.erofs implemenetation (it's just
a userspace implementation details, if people really care about it, I
will refine the implementation), and I will also implement such sparse
files later so that all on-disk inodes won't be impacted as well (I'm on
vacation, but I will try my best).
From the overall results, I don't really know what's the most bottleneck
point honestly:
maybe just like what you said -- due to overlayfs overhead;
or maybe a bottleneck of loopback device.
so it's much better to show some results of "ls -lR" without overlayfs
stacked too.
IMHO, Amir's main point is always [1]
"w.r.t overlayfs, I am not even sure that anything needs to be modified
in the driver.
overlayfs already supports "metacopy" feature which means that an upper
layer could be composed in a way that the file content would be read
from an arbitrary path in lower fs, e.g. objects/cc/XXX. "
I think there is nothing wrong with it (except for fsverity). From the
results, such functionality indeed can already be achieved by overlayfs
+ some localfs with some user-space adaption. And it was not mentioned
in RFC and v2.
So without fs-verity requirement, currently your proposal is mainly
resolving a performance issue of an exist in-kernel approach (except for
unprivileged mounts). It's much better to describe in the cover letter
-- The original problem, why overlayfs + (localfs or FUSE for metadata)
doesn't meet the requirements. That makes much sense compared with the
current cover letter.
Thanks,
Gao Xiang
[1] https://lore.kernel.org/r/CAOQ4uxh34udueT-+Toef6TmTtyLjFUnSJs=882DH=HxADX8pKw@mail.gmail.com/
Powered by blists - more mailing lists