linux-kernel - Re: [PATCH v20 00/15] Introduce Data Access MONitor (DAMON)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200831112235.2675-1-sjpark@amazon.com>
Date:   Mon, 31 Aug 2020 13:22:35 +0200
From:   SeongJae Park <sjpark@...zon.com>
To:     SeongJae Park <sjpark@...zon.com>
CC:     <Jonathan.Cameron@...wei.com>, <aarcange@...hat.com>,
        <acme@...nel.org>, <alexander.shishkin@...ux.intel.com>,
        <amit@...nel.org>, <benh@...nel.crashing.org>,
        <brendan.d.gregg@...il.com>, <brendanhiggins@...gle.com>,
        <cai@....pw>, <colin.king@...onical.com>, <corbet@....net>,
        <david@...hat.com>, <dwmw@...zon.com>, <fan.du@...el.com>,
        <foersleo@...zon.de>, <gthelen@...gle.com>, <irogers@...gle.com>,
        <jolsa@...hat.com>, <kirill@...temov.name>, <mark.rutland@....com>,
        <mgorman@...e.de>, <minchan@...nel.org>, <mingo@...hat.com>,
        <namhyung@...nel.org>, <peterz@...radead.org>,
        <rdunlap@...radead.org>, <riel@...riel.com>, <rientjes@...gle.com>,
        <rostedt@...dmis.org>, <rppt@...nel.org>, <sblbir@...zon.com>,
        <shakeelb@...gle.com>, <shuah@...nel.org>, <sj38.park@...il.com>,
        <snu@...zon.de>, <vbabka@...e.cz>, <vdavydov.dev@...il.com>,
        <yang.shi@...ux.alibaba.com>, <ying.huang@...el.com>,
        <zgf574564920@...il.com>, <linux-damon@...zon.com>,
        <linux-mm@...ck.org>, <linux-doc@...r.kernel.org>,
        <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v20 00/15] Introduce Data Access MONitor (DAMON)

On Thu, 20 Aug 2020 09:27:38 +0200 SeongJae Park <sjpark@...zon.com> wrote:

> On Mon, 17 Aug 2020 12:51:22 +0200 SeongJae Park <sjpark@...zon.com> wrote:
> 
> > From: SeongJae Park <sjpark@...zon.de>
> > 
> > Changes from Previous Version
> > =============================
> > 
> > - Place 'CREATE_TRACE_POINTS' after '#include' statements (Steven Rostedt)
> > - Support large record file (Alkaid)
> > - Place 'put_pid()' of virtual monitoring targets in 'cleanup' callback
> > - Avoid conflict between concurrent DAMON users
> > - Update evaluation result document
> > 
> > Introduction
> > ============
> > 
> > DAMON is a data access monitoring framework subsystem for the Linux kernel.
> > The core mechanisms of DAMON called 'region based sampling' and 'adaptive
> > regions adjustment' (refer to 'mechanisms.rst' in the 11th patch of this
> > patchset for the detail) make it
> > 
> >  - accurate (The monitored information is useful for DRAM level memory
> >    management. It might not appropriate for Cache-level accuracy, though.),
> >  - light-weight (The monitoring overhead is low enough to be applied online
> >    while making no impact on the performance of the target workloads.), and
> >  - scalable (the upper-bound of the instrumentation overhead is controllable
> >    regardless of the size of target workloads.).
> > 
> > Using this framework, therefore, the kernel's core memory management mechanisms
> > such as reclamation and THP can be optimized for better memory management.  The
> > experimental memory management optimization works that incurring high
> > instrumentation overhead will be able to have another try.  In user space,
> > meanwhile, users who have some special workloads will be able to write
> > personalized tools or applications for deeper understanding and specialized
> > optimizations of their systems.
> 
> DAMON will be presented in the next week LPC[1].  To be prepared for a screen
> sharing error (if I get no such error, I will do a live-demo), I recorded a
> simple demo video.  I would like to share it here to help your easier
> understanding of DAMON.
> 
>     https://youtu.be/l63eqbVBZRY
> 
> [1] https://linuxplumbersconf.org/event/7/contributions/659/

During the session, I introduced the list of future works and asked the
audiences to vote for the priority of the tasks:
https://youtu.be/jOBkKMA0uF0?t=13253

To summarize here, the tasks are (highest priority first):

1. Make current DAMON patchset series merged in the mainline (6 votes)
2. User space interface improvement (4 votes)
 - Multiple monitoring contexts
 - Charging of the monitoring threads' CPU usage
3. Support more address spaces (2 votes)
 - Cgroups, cached pages, specific file-backed pages, swap slots, ...
3. DAMON-based MM optimizations (2 votes)
 - Page reclaim, THP, compaction, NUMA balancing, ...
4. Optimize for special use-cases (1 vote)
 - Page granularity monitoring, accessed-or-not monitoring, ...

So, I'd like to focus on polishing current patchset so that it could be merged
in.  For that, I'd like to ask your more reviews.

While waiting for the reviews, I will start implementing other future features
that received many votes.  The support of multiple monitoring contexts for the
user space would be the first one.  Once the implementation is finished, I will
post it as separated RFC patchset (the user space interface will be compatible
with current one).

Any comment is welcome.


Thanks,
SeongJae Park