linux-kernel - Re: [RFC PATCH 00/14] Virtual Swap Space

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAKEwX=M3do_7SJGKwfZQ8vOSQN4aM1ZU04Q3E99CW=UTCkUMOQ@mail.gmail.com>
Date: Tue, 8 Apr 2025 09:25:57 -0700
From: Nhat Pham <nphamcs@...il.com>
To: Johannes Weiner <hannes@...xchg.org>
Cc: Usama Arif <usamaarif642@...il.com>, linux-mm@...ck.org, akpm@...ux-foundation.org, 
	hughd@...gle.com, yosry.ahmed@...ux.dev, mhocko@...nel.org, 
	roman.gushchin@...ux.dev, shakeel.butt@...ux.dev, muchun.song@...ux.dev, 
	len.brown@...el.com, chengming.zhou@...ux.dev, kasong@...cent.com, 
	chrisl@...nel.org, huang.ying.caritas@...il.com, ryan.roberts@....com, 
	viro@...iv.linux.org.uk, baohua@...nel.org, osalvador@...e.de, 
	lorenzo.stoakes@...cle.com, christophe.leroy@...roup.eu, pavel@...nel.org, 
	kernel-team@...a.com, linux-kernel@...r.kernel.org, cgroups@...r.kernel.org, 
	linux-pm@...r.kernel.org
Subject: Re: [RFC PATCH 00/14] Virtual Swap Space

On Tue, Apr 8, 2025 at 8:45 AM Johannes Weiner <hannes@...xchg.org> wrote:
>
> On Tue, Apr 08, 2025 at 02:04:06PM +0100, Usama Arif wrote:
> >
> >
> > On 08/04/2025 00:42, Nhat Pham wrote:
> > >
> > > V. Benchmarking
> > >
> > > As a proof of concept, I run the prototype through some simple
> > > benchmarks:
> > >
> > > 1. usemem: 16 threads, 2G each, memory.max = 16G
> > >
> > > I benchmarked the following usemem commands:
> > >
> > > time usemem --init-time -w -O -s 10 -n 16 2g
> > >
> > > Baseline:
> > > real: 33.96s
> > > user: 25.31s
> > > sys: 341.09s
> > > average throughput: 111295.45 KB/s
> > > average free time: 2079258.68 usecs
> > >
> > > New Design:
> > > real: 35.87s
> > > user: 25.15s
> > > sys: 373.01s
> > > average throughput: 106965.46 KB/s
> > > average free time: 3192465.62 usecs
> > >
> > > To root cause this regression, I ran perf on the usemem program, as
> > > well as on the following stress-ng program:
> > >
> > > perf record -ag -e cycles -G perf_cg -- ./stress-ng/stress-ng  --pageswap $(nproc) --pageswap-ops 100000
> > >
> > > and observed the (predicted) increase in lock contention on swap cache
> > > accesses. This regression is alleviated if I put together the
> > > following hack: limit the virtual swap space to a sufficient size for
> > > the benchmark, range partition the swap-related data structures (swap
> > > cache, zswap tree, etc.) based on the limit, and distribute the
> > > allocation of virtual swap slotss among these partitions (on a per-CPU
> > > basis):
> > >
> > > real: 34.94s
> > > user: 25.28s
> > > sys: 360.25s
> > > average throughput: 108181.15 KB/s
> > > average free time: 2680890.24 usecs
> > >
> > > As mentioned above, I will implement proper dynamic swap range
> > > partitioning in a follow up work.
> > >
> > > 2. Kernel building: zswap enabled, 52 workers (one per processor),
> > > memory.max = 3G.
> > >
> > > Baseline:
> > > real: 183.55s
> > > user: 5119.01s
> > > sys: 655.16s
> > >
> > > New Design:
> > > real: mean: 184.5s
> > > user: mean: 5117.4s
> > > sys: mean: 695.23s
> > >
> > > New Design (Static Partition)
> > > real: 183.95s
> > > user: 5119.29s
> > > sys: 664.24s
> > >
> >
> > Hi Nhat,
> >
> > Thanks for the patches! I have glanced over a couple of them, but this was the main question that came to my mind.
> >
> > Just wanted to check if you had a look at the memory regression during these benchmarks?
> >
> > Also what is sizeof(swp_desc)? Maybe we can calculate the memory overhead as sizeof(swp_desc) * swap size/PAGE_SIZE?
> >
> > For a 64G swap that is filled with private anon pages, the overhead in MB might be (sizeof(swp_desc) in bytes * 16M) - 16M (zerobitmap) - 16M*8 (swap map)?
> >
> > This looks like a sizeable memory regression?
>
> One thing to keep in mind is that the swap descriptor is currently
> blatantly explicit, and many conversions and optimizations have not
> been done yet. There are some tradeoffs made here regarding code
> reviewability, but I agree it makes it hard to see what this would
> look like fully realized.
>
> I think what's really missing is an analysis of what the goal is and
> what the overhead will be then.
>
> The swapin path currently consults the swapcache, then the zeromap,
> then zswap, and finally the backend. The external swap_cgroup array is
> consulted to determine who to charge for the new page.
>
> With vswap, the descriptor is looked up and resolves to a type,
> location, cgroup ownership, a refcount. This means it replaces the
> swapcache, the zeromap, the cgroup map, and largely the swap_map.
>
> Nhat was not quite sure yet if the swap_map can be a single bit per
> entry or two bits to represent bad slots. In any case, it's a large
> reduction in static swap space overhead, and eliminates the tricky
> swap count continuation code.

You're right. I haven't touched the swapfile swap map and the zeromap
bitmap at all, primarily because it's non-functional change
(optimization only). It also adds more ifdefs to the final codebase :)

In the next version, I can tag on one patch to:

1. remove zeromap bitmap. This one is pretty much straightforward -
we're not using it at all.

2. Swap map reduction. I'm like 70% sure we don't need SWAP_MAP_BAD
state. With the vswap reverse map and the swapfile inuse counters, we
should be able to convert the swapmap into a pure bitmap. If we can't,
then it's 2 bits per physical swapfiles.