linux-kernel - Re: [PATCH 0/13] maps: pagemap, kpagemap, and related cleanups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20070412185723.5a5f0443.akpm@linux-foundation.org>
Date:	Thu, 12 Apr 2007 18:57:23 -0700
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	Nick Piggin <nickpiggin@...oo.com.au>
Cc:	William Lee Irwin III <wli@...omorphy.com>,
	Matt Mackall <mpm@...enic.com>, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 0/13] maps: pagemap, kpagemap, and related cleanups

On Fri, 13 Apr 2007 11:42:29 +1000 Nick Piggin <nickpiggin@...oo.com.au> wrote:

> Andrew Morton wrote:
> > On Fri, 13 Apr 2007 11:14:20 +1000 Nick Piggin <nickpiggin@...oo.com.au> wrote:
> > 
> > 
> >>Andrew Morton wrote:
> 
> >>>It *will* be viable.  If the application wants to know if a page is dirty,
> >>>it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's
> >>>numerical offset when inspecting fields in /proc/kpagemap.  If correctly
> >>>designed, such a monitoring application will be able to report upon page
> >>>flags which we haven't even thought up yet.
> >>
> >>Ooh, you wanted a _runtime_ mapping of flags, yeah then I guess that works.
> >>Still seems like a basically hit and miss affair to just use flags. What if
> >>you want to know the process mapping a page? With systemtap or something you
> >>could walk the rmap structures. What if you want to look at pages along the
> >>LRU list rather than per-pfn? What about connecting pages to inodes?
> > 
> > 
> > Well hang on.  This isn't a tool for understanding kernel behaviour.  It's
> > a tool for understanding applciation behaviour.
> > 
> > So one doesn't ask "who is mapping that page" - that's a kernel developer
> > thing.
> > 
> > Instead, one says "what pages are being used by my application", then, for
> 
> That includes unmapped pagecache being used by my application, doesn't it?
> Maybe that's too hard to do via /proc so we forget about it...

Yes, harder.  I'm hoping that sampling of /proc/pid/io can be used to
determine pagecache use sufficiently accurately.  I know of one large
hosting company who are using it ("BTW, we are making great use of
taskstats!!  Its GREAT!")

> 
> > each of those pages "what is that page's state".  So the first step is to
> > collect all the pfns from /proc/$(pidof my-application)/pagemap and then to
> > use those pfns to look the individual pages up in /proc/kpagemap.
> 
> OK I realise you could do it that way, but systemtap can definitely be
> used as a tool for understanding application behaviour in the context of
> the kernel, I think? The purpose for it is so that various little bits
> of deep kernel internals do not have to be exposed on a case by case basis.
> 
> If kprobes is simply crappy and doesn't work properly for this, then I
> could accept that. I'm not someone trying to get this info. So why can't
> it be used? (not just for kpagemap, but for clear_refs and all that gunk
> too).
> 
>  > If you really want to know "who is using page 123435" then you'd need to
>  > search /proc/*/pagemap.  There are possibly legitimate reasons why an
>  > application developer would want to at least pertially perform such an
>  > operation ("who am I sharing with"), but I doubt if it's the common case.
> 
> Maybe. How about LRU? Reclaim performance is bad, and you want to work out
> which pages keep going off the end of it, or which pages keep getting
> written out via it, or who's pages are on the active list, forcing mine
> out.

I guess we have static analysis versus dynamic.  The interfaces which Matt
is proposing are suited to answering the question "what is my memory being
used for" (static).  They're unlikely to be useful for answering the question
"what's happening in the VM" (dynamic).  Systemtap is probably better for the
dynamic analysis.

I guess one could generate an answer to the static question with systemtap,
by accumulating running counts across the application lifetime and then
snapshotting them.  Sounds hard though.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/