Message-ID: <CAMvvPS4imbSprPPZ3dtPBK2i3DZPffYn3d4rASMoNwcTeHCWqg@mail.gmail.com>
Date: Fri, 20 Jun 2025 16:47:26 -0500
From: Bijan Tabatabai <bijan311@...il.com>
To: SeongJae Park <sj@...nel.org>
Cc: damon@...ts.linux.dev, linux-mm@...ck.org, linux-kernel@...r.kernel.org, 
	akpm@...ux-foundation.org, david@...hat.com, ziy@...dia.com, 
	matthew.brost@...el.com, joshua.hahnjy@...il.com, rakie.kim@...com, 
	byungchul@...com, gourry@...rry.net, ying.huang@...ux.alibaba.com, 
	apopple@...dia.com, bijantabatab@...ron.com, venkataravis@...ron.com, 
	emirakhur@...ron.com, ajayjoshi@...ron.com, vtavarespetr@...ron.com
Subject: Re: [RFC PATCH v2 0/2] mm/damon/paddr: Allow interleaving in
 migrate_{hot,cold} actions

Hi SeongJae,

On Fri, Jun 20, 2025 at 3:21 PM SeongJae Park <sj@...nel.org> wrote:
>
> Hi Bijan,
>
> On Fri, 20 Jun 2025 13:04:56 -0500 Bijan Tabatabai <bijan311@...il.com> wrote:
>
> [...]
> > As a toy example, imagine some application that uses 75% of the local
> > bandwidth. Assuming sufficient capacity, when running alone, we want to
> > keep that application's data in local memory. However, if a second
> > instance of that application begins, using the same amount of bandwidth,
> > it would be best to interleave the data of both processes to alleviate the
> > bandwidth pressure from the local node. Likewise, when one of the processes
> > ends, the data should be moves back to local memory.
> >
> > We imagine there would be a userspace application that would monitor system
> > performance characteristics, such as bandwidth utilization or memory access
> > latency, and uses that information to tune the interleave weights. Others
> > seem to have come to a similar conclusion in previous discussions [3].
> > We are currently working on a userspace program that does this, but it is
> > not quite ready to be published yet.
>
> So, at least in this toy example, we have user-space control.  Then, I think we
> could decouple DAMON and weighted interleaving, and ask the usr-space tool to
> be the connection between those.  That is, extend DAMOS_MIGRATE_{HOT,COLD} to
> let users specify migration target nodes and their weights.  And ask the
> user-space tool to periodically read weighted interleaving parameters that
> could be auto-tuned, and update DAMOS_MIGRATE_{HOT,COLD} parameters
> accordingly.  Actually the user-space tool on this example is making the
> weights by itself, so this should be easy work to do?
>
> Also, even for general use case, I think such user-space intervention is not
> too much request.  Please let me know if I'm wrong.

You are correct. The userspace tool would be coming up with the
weights, so it would not be hard for it to write those weights to two
places. I coupled the weights used in DAMON and weighted interleaving
for this revision and the previous because I could not think of a use
case where you would want to use different weights for allocation time
and migration, so it felt silly to have two different places with the
same data. However, I don't feel too strongly about this, so I'm
willing to defer to your judgement.

Also, our userspace tool updates these weights somewhat frequently,
several times per minute, when it detects a change in the bandwidth
utilization of the system to calibrate the interleave ratio. I am
concerned about how frequent changes to the scheme via the sysfs
interface will affect the effectiveness of DAMON's page sampling. From
what I understand, updates to the sysfs aren't saved until the user
writes to some sysfs file to commit them, then the damon context is
recreated from scratch. Would this throw away all the previous
sampling work done and work splitting and merging regions? I am not
super familiar with how the sysfs interface interacts with the rest of
the system, so this concern might be entirely unfounded, but I would
appreciate some clarification here.

[...]
> >
> > Questions for Reviewers
> > =======================
> > 1. Are you happy with the changes to the DAMON sysfs interface?
>
> I'm happy with it for RFC level implementation.  And in my opinion, you now
> proved this is a good idea.  For next steps toward mainline landing, I'd like
> to suggest below interface change.
>
> Let's allow users specify DAMOS_MIGRATE_{HOT,COLD} target nodes and weights
> using only DAMON interface.  And let the user-space tool do the synchronization
> with weighted interleaving or other required works.
>
> This may require writing not small amount of code, especially for DAMON sysfs
> interface.  I think it is doable, though.  If you don't mind, I'd like to
> quickly make a prototype and share with you.
>
> What do you think?

That sounds good to me! Having a prototype from you for the sysfs
interface would certainly be helpful, but if you're busy, I can take a
pass at it as well.

> > 2. Setting an interleave weight to 0 is currently not allowed. This makes
> >    sense when the weights are only used for allocation. Does it make sense
> >    to allow 0 weights now?
>
> I have no opinion, and would like to let mempolicy folks make voices.  But if
> we go on the decoupling approach as I suggested above, we can do this
> discussion in a separate thread :)
>
> [...]
> > Revision History
> > ================
> > Changes from v1
> > (https://lore.kernel.org/linux-mm/20250612181330.31236-1-bijan311@gmail.com/)
> > - Reuse migrate_{hot,cold} actions instead of creating a new action
> > - Remove vaddr implementation
> > - Remove most of the use of mempolicy, instead duplicate the interleave
> >   logic and access interleave weights directly
> > - Write more about the use case in the cover letter
> > - Write about why DAMON was used for this in the cover letter
> > - Add correctness test to the cover letter
> > - Add performance test
>
> Again, thank you for revisioning.  Please bear in mind with me at next steps.
> I believe this work is very promising.

Thank you for your help and feedback!
Bijan

[...]

Powered by blists - more mailing lists