linux-kernel - Re: [PATCH 0/8] idle page tracking / working set estimation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 22 Sep 2011 16:13:25 -0700
From:	Andrew Morton <akpm@...gle.com>
To:	Michel Lespinasse <walken@...gle.com>
Cc:	linux-mm@...ck.org, linux-kernel@...r.kernel.org,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	Dave Hansen <dave@...ux.vnet.ibm.com>,
	Andrea Arcangeli <aarcange@...hat.com>,
	Rik van Riel <riel@...hat.com>,
	Johannes Weiner <jweiner@...hat.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Hugh Dickins <hughd@...gle.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Michael Wolf <mjwolf@...ibm.com>,
	Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [PATCH 0/8] idle page tracking / working set estimation

On Fri, 16 Sep 2011 20:39:05 -0700
Michel Lespinasse <walken@...gle.com> wrote:

> Please comment on the following patches (which are against the v3.0 kernel).
> We are using these to collect memory utilization statistics for each cgroup
> accross many machines, and optimize job placement accordingly.

Please consider updating /proc/kpageflags with the three new page
flags.  If "yes": update.  If "no": explain/justify.

Which prompts the obvious: the whole feature could have been mostly
implemented in userspace, using kpageflags.  Some additional kernel
support would presumably be needed, but I'm not sure how much.

If you haven't already done so, please sketch down what that
infrastructure would look like and have a think about which approach is
preferable?

What bugs me a bit about the proposal is its cgroups-centricity.  The
question "how much memory is my application really using" comes up
again and again.  It predates cgroups.  One way to answer that question
is to force a massive amount of swapout on the entire machine, then let
the system recover and take a look at your app's RSS two minutes later.
This is very lame.

It's a legitimate requirement, and the kstaled infrastructure puts a
lot of things in place to answer it well.  But as far as I can tell it
doesn't quite get over the line.  Then again, maybe it _does_ get
there: put the application into a memcg all of its own, just for
instrumentation purposes and then use kstaled to monitor it?

<later> OK, I'm surprised to discover that kstaled is doing a physical
scan and not a virtual one.  I assume it works, but I don't know why. 
But it makes the above requirement harder, methinks.

How does all this code get along with hugepages, btw?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/