linux-kernel - Re: [PATCH v4 0/9] mm: workingset reporting

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJj2-QEaLTasfQgb=VFfnbOmkcXU3kw2VbsNummNEq0V3b9jdw@mail.gmail.com>
Date: Wed, 29 Jan 2025 18:02:26 -0800
From: Yuanchu Xie <yuanchu@...gle.com>
To: SeongJae Park <sj@...nel.org>
Cc: David Hildenbrand <david@...hat.com>, "Aneesh Kumar K.V" <aneesh.kumar@...ux.ibm.com>, 
	Khalid Aziz <khalid.aziz@...cle.com>, Henry Huang <henry.hj@...group.com>, 
	Yu Zhao <yuzhao@...gle.com>, Dan Williams <dan.j.williams@...el.com>, 
	Gregory Price <gregory.price@...verge.com>, Huang Ying <ying.huang@...el.com>, 
	Lance Yang <ioworker0@...il.com>, Randy Dunlap <rdunlap@...radead.org>, 
	Muhammad Usama Anjum <usama.anjum@...labora.com>, Tejun Heo <tj@...nel.org>, 
	Michal Koutný <mkoutny@...e.com>, 
	Jonathan Corbet <corbet@....net>, Greg Kroah-Hartman <gregkh@...uxfoundation.org>, 
	"Rafael J. Wysocki" <rafael@...nel.org>, "Michael S. Tsirkin" <mst@...hat.com>, Jason Wang <jasowang@...hat.com>, 
	Xuan Zhuo <xuanzhuo@...ux.alibaba.com>, Eugenio Pérez <eperezma@...hat.com>, 
	Michal Hocko <mhocko@...nel.org>, Roman Gushchin <roman.gushchin@...ux.dev>, 
	Shakeel Butt <shakeel.butt@...ux.dev>, Muchun Song <muchun.song@...ux.dev>, 
	Mike Rapoport <rppt@...nel.org>, Shuah Khan <shuah@...nel.org>, 
	Christian Brauner <brauner@...nel.org>, Daniel Watson <ozzloy@...h.do>, cgroups@...r.kernel.org, 
	linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org, 
	virtualization@...ts.linux.dev, linux-mm@...ck.org, 
	linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v4 0/9] mm: workingset reporting

On Wed, Dec 11, 2024 at 11:53 AM SeongJae Park <sj@...nel.org> wrote:
>
> On Fri, 6 Dec 2024 11:57:55 -0800 Yuanchu Xie <yuanchu@...gle.com> wrote:
>
> > Thanks for the response Johannes. Some replies inline.
> >
> > On Tue, Nov 26, 2024 at 11:26\u202fPM Johannes Weiner <hannes@...xchg.org> wrote:
> > >
> > > On Tue, Nov 26, 2024 at 06:57:19PM -0800, Yuanchu Xie wrote:
> > > > This patch series provides workingset reporting of user pages in
> > > > lruvecs, of which coldness can be tracked by accessed bits and fd
> > > > references. However, the concept of workingset applies generically to
> > > > all types of memory, which could be kernel slab caches, discardable
> > > > userspace caches (databases), or CXL.mem. Therefore, data sources might
> > > > come from slab shrinkers, device drivers, or the userspace.
> > > > Another interesting idea might be hugepage workingset, so that we can
> > > > measure the proportion of hugepages backing cold memory. However, with
> > > > architectures like arm, there may be too many hugepage sizes leading to
> > > > a combinatorial explosion when exporting stats to the userspace.
> > > > Nonetheless, the kernel should provide a set of workingset interfaces
> > > > that is generic enough to accommodate the various use cases, and extensible
> > > > to potential future use cases.
> > >
> > > Doesn't DAMON already provide this information?
> > >
> > > CCing SJ.
> > Thanks for the CC. DAMON was really good at visualizing the memory
> > access frequencies last time I tried it out!
>
> Thank you for this kind acknowledgement, Yuanchu!
>
> > For server use cases,
> > DAMON would benefit from integrations with cgroups.  The key then would be a
> > standard interface for exporting a cgroup's working set to the user.
>
> I show two ways to make DAMON supports cgroups for now.  First way is making
> another DAMON operations set implementation for cgroups.  I shared a rough idea
> for this before, probably on kernel summit.  But I haven't had a chance to
> prioritize this so far.  Please let me know if you need more details.  The
> second way is extending DAMOS filter to provide more detailed statistics per
> DAMON-region, and adding another DAMOS action that does nothing but only
> accounting the detailed statistics.  Using the new DAMOS action, users will be
> able to know how much of specific DAMON-found regions are filtered out by the
> given filter.  Because we have DAMOS filter type for cgroups, we can know how
> much of workingset (or, warm memory) belongs to specific groups.  This can be
> applied to not only cgroups, but for any DAMOS filter types that exist (e.g.,
> anonymous page, young page).
>
> I believe the second way is simpler to implement while providing information
> that sufficient for most possible use cases.  I was anyway planning to do this.
For a container orchestrator like kubernetes, the node agents need to
be able to gather the working set stats at a per-job level. Some jobs
can create sub-hierarchies as well, so it's important that we have
hierarchical stats.

Do you think it's a good idea to integrate DAMON to provide some
aggregate stats in a memory controller file? With the DAMOS cgroup
filter, there can be some kind of interface that a DAMOS action or the
damo tool could call into. I feel that would be a straightforward and
integrated way to support cgroups.

Yuanchu