linux-kernel - Re: [PATCH v34 00/13] Introduce Data Access MONitor (DAMON)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20210804074107.3021-1-sjpark@amazon.de>
Date:   Wed,  4 Aug 2021 07:41:07 +0000
From:   SeongJae Park <sj38.park@...il.com>
To:     Andrew Morton <akpm@...ux-foundation.org>
Cc:     SeongJae Park <sj38.park@...il.com>,
        Shakeel Butt <shakeelb@...gle.com>,
        SeongJae Park <sjpark@...zon.de>, Jonathan.Cameron@...wei.com,
        amit@...nel.org, Jonathan Corbet <corbet@....net>,
        David Hildenbrand <david@...hat.com>, dwmw@...zon.com,
        foersleo@...zon.de, Greg Thelen <gthelen@...gle.com>,
        jgowans@...zon.com, mheyne@...zon.de,
        David Rientjes <rientjes@...gle.com>, sieberf@...zon.com,
        Vlastimil Babka <vbabka@...e.cz>, linux-damon@...zon.com,
        Linux MM <linux-mm@...ck.org>, linux-doc@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>, Wei Xu <weixugc@...gle.com>,
        Paul Turner <pjt@...gle.com>, Yu Zhao <yuzhao@...gle.com>,
        Dave Hansen <dave.hansen@...el.com>
Subject: Re: [PATCH v34 00/13] Introduce Data Access MONitor (DAMON)

From: SeongJae Park <sjpark@...zon.de>

Hello Andrew,

On Mon,  2 Aug 2021 08:24:24 +0000 SeongJae Park <sj38.park@...il.com> wrote:

> From: SeongJae Park <sjpark@...zon.de>
> 
> Hello Andrew,
> 
> On Wed, 28 Jul 2021 08:36:43 +0000 SeongJae Park <sj38.park@...il.com> wrote:
> 
> [...]
> > Now all the patches have at least one 'Reviewed-by:' or 'Acked-by:' tags.  We
> > didn't find serious problems since v26[5], which was posted about four months
> > ago. so I'm thinking this patchset has passed the minimum qualification.  If
> > you think there are more things to be done before this patchset is merged in
> > the -mm tree or mainline, please let me know.  If not, Andrew, I'd like you to
> > consider merging this patchset into '-mm' tree.
> 
> I'm wondering if you had a chance to consider that.  If you had the chance but
> this patchset didn't convince you, could you please let me know your concerns
> so that I can make some progress?

Because nearly three weeks passed since this patchset is posted, I considered
rebasing it on the latest -mm tree and posting it as v35.  But, apparently it
makes no much sense because we found nothing to fix or improve.  And, this
version can still cleanly be applied on top of the latest -mm tree.  So,
instead of merely increasing the version number, I'd like to describe why I
believe this need to be merged into the -mm tree and eventually the mainline.

1. Merging this patchset will not bother other developers

Most changes in this patchset are for DAMON-dedicated new source files.  There
is a change[1] for existing files, which makes PG_Idle independent of Idle Page
Tracking, but it is only small.  Therefore, merging this patchset will not
increase the complexity of the other parts or introduce a regression.

2. Merging this patchset will not bother other users

DAMON utilizes a mechanism that designed to minimize and limit the monitoring
overhead.  That said, DAMON can be opt out in the compile time for users who
don't want it.  Even though it is compiled, it does nothing at all unless a
user explicitly asks it to do some works.  Therefore, merging this patchset
will not silently introduce any additional overhead to users.

3. This patchset is deployed to real users

We are currently using DAMON patchset for profiling production workloads, as
described in 'Real-world User Story' section of the cover letter.  It is also
deployed to real users other than us via Amazon Linux[2,3].  A few companies
and several researchers outside Amazon have publicly and/or privately shown
their interests in DAMON.

4. The downstream-only maintenance overhead is significant

Following development works based on DAMON[4,5,6] are also ongoing.  Because
all the works are currently in downstream only, the maintenance overhead is not
small for us.  Once DAMON is upstreamed, the overhead will significantly be
reduced.

5. This patchset is reviewed and apparently is stabilized

Since the first version of DAMON patchset is posted (2020-01-20), it has
evolved a lot.  All patches of this patchset got at least one 'Reviewed-by:' or
'Acked-by:' tag by v31[7], which have posted about seven weeks ago
(2021-06-21).  After that, we found and fixed only minor issues.  We also got a
few more 'Acked-by:' tags.  Since v34, which has posted about three weeks ago,
we found no more issues.  We are also continuously running extensive
DAMON-dedicated tests.  The tests include unit tests, self tests, functional
tests, performance tests, and static code analysis.  Some of those are also
publicly available[8].

[1] https://lore.kernel.org/linux-mm/20210716081449.22187-5-sj38.park@gmail.com/
[2] https://github.com/amazonlinux/linux/tree/amazon-5.4.y/master/mm/damon
[3] https://github.com/amazonlinux/linux/tree/amazon-5.10.y/master/mm/damon
[4] https://lore.kernel.org/linux-mm/20201216084404.23183-1-sjpark@amazon.com/
[5] https://lore.kernel.org/linux-mm/20201216094221.11898-1-sjpark@amazon.com/
[6] https://lore.kernel.org/linux-mm/20210720131309.22073-1-sj38.park@gmail.com/
[7] https://lore.kernel.org/linux-mm/20210621083108.17589-1-sj38.park@gmail.com/
[8] https://github.com/awslabs/damon-tests


If you think above explanation makes sense, please consider merging this into
the -mm tree.  Else, if this doesn't convince you, please let me know your
concerns or what I'm missing, so that I can make some progress.


Thanks,
SeongJae Park

[...]