linux-kernel - Re: [PATCH 1/1] pagemap clear_refs: modify to specify anon or mapped vma clearing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <7928e7bd0907301221v73b050e4hbc91bf08ddf07248@mail.gmail.com>
Date:	Thu, 30 Jul 2009 12:21:23 -0700
From:	Moussa Ba <moussa.a.ba@...il.com>
To:	Mel Gorman <mel@....ul.ie>
Cc:	Andrew Morton <akpm@...ux-foundation.org>,
	linux-kernel@...r.kernel.org, rientjes@...gle.com,
	xiyou.wangcong@...il.com, adobriyan@...il.com, mpm@...enic.com,
	yinghan@...gle.com, npiggin@...e.de, jaredeh@...il.com
Subject: Re: [PATCH 1/1] pagemap clear_refs: modify to specify anon or mapped 
	vma clearing

Mel,

here are two scripts that make use of the interface to "measure"
Referenced anon vs. mapped pages.

save the awk script to refs.awk  and run the bash script as follows

./refscount PID secs

PID is the process you want to monitor while secs is the number of
seconds to sleep before clearing the refs.    The output after x secs
is a the total number of refs before and immediately after clearing.
Here is an example run using firefox as I was writing this email.

/usr/lib/firefox-3.0.3/firefox
12:10:58 Mapped = 12856 kB Anon = 27896 kB
12:10:58 Mapped = 760 kB Anon = 4 kB
12:11:08 Mapped = 5884 kB Anon = 20340 kB
.
.
.
.
12:16:19 Mapped = 476 kB Anon = 0 kB
12:16:29 Mapped = 7016 kB Anon = 13128 kB
12:16:29 Mapped = 476 kB Anon = 0 kB
12:16:39 Mapped = 9276 kB Anon = 21588 kB
12:16:39 Mapped = 476 kB Anon = 0 kB
12:16:49 Mapped = 10288 kB Anon = 16000 kB
12:16:49 Mapped = 716 kB Anon = 12 kB

BEGIN BASH SCRIPT
--------------------------------
#!/bin/bash

echo `cat /proc/$1/cmdline`
#Intial state of the process
cat /proc/$1/smaps | awk -f refs.awk

while [ true ]
do
    echo 1 > /proc/$1/clear_refs
    # Immidiately after clearing
    cat /proc/$1/smaps | awk -f refs.awk
    sleep $2
    # after x seconds
    cat /proc/$1/smaps | awk -f refs.awk
done
---------------------------
END BASH SCRIPT

BEGIN AWK SCRIPT
-----------------------

BEGIN {refsanon = 0;refsmapped=0;}

{
    if (NF == 6)  {
	if ($4 == "00:00")
	    type = 1;
	else
	    type = 2;
    }

    if ($1 == "Referenced:") {
	if (type == 1)
	    refsanon += $2;
	if (type == 2)
	    refsmapped += $2;
	type = 0;
    }

}
END {printf("%s ",strftime("%H:%M:%S"));  print "Mapped = " refsmapped
" kB Anon = " refsanon " kB";}

---------------------------
END AWK SCRIPT


On Wed, Jul 29, 2009 at 7:58 AM, Mel Gorman<mel@....ul.ie> wrote:
> On Tue, Jul 28, 2009 at 05:00:54PM -0700, Moussa Ba wrote:
>> The patch makes the clear_refs more versatile in adding the option to
>> select anonymous pages or file backed pages for clearing.  This
>> addition has a measurable impact on user space application performance
>> as it decreases the number of pagewalks in scenarios where one is only
>> interested in a specific type of page (anonymous or file mapped).
>>
>
> I think what Andrew might be looking for is a repeat of the use-case for
> using clear_refs at all and what the additional usecase is that makes this
> patch beneficial. To be honest, I had to go digging for a bit to find out
> why clear_refs is used at all and the potential benefits of this patch were
> initially unclear to me although they were obvious to others. I confess I
> haven't read the patch closely to determine if it's behaving as advertised
> or not.
>
> Bonus points for a wee illustration of a script that measures the apparent
> working set of a process using clear_refs, showing the working set for anon
> vs filebacked and the difference of measuring just anon versus the full
> process for example. Such a script could be added to Documentation/vm as
> I didn't spot any example in there already.
>
> Total aside, is there potential to really mess with LRU aging by abusing
> clear_refs a lot, partcularly as clear_refs is writable by the process
> owner? Does it matter?
>
>>
>> On Tue, Jul 28, 2009 at 4:52 PM, Andrew Morton<akpm@...ux-foundation.org> wrote:
>> > On Tue, 28 Jul 2009 15:52:05 -0700
>> > "Moussa A. Ba" <moussa.a.ba@...il.com> wrote:
>> >
>> >> This patch adds anonymous and file backed filters to the clear_refs interface.
>> >> echo 1 > /proc/PID/clear_refs resets the bits on all pages
>> >> echo 2 > /proc/PID/clear_refs resets the bits on anonymous pages only
>> >> echo 3 > /proc/PID/clear_refs resets the bits on file backed pages only
>> >> Any other value is ignored
>> >
>> > The changelog is missing any rationale for making this change.
>> > Originally we were told that it "makes the clear_refs proc interface a
>> > bit more versatile", but that's a bit thin.
>> >
>> > How do we justify making this change to Linux?  What value does it
>> > have?  Can you describe a usage scenario which would help people
>> > understand the usefulness of the change?
>> >
>> > Thanks.
>> >
>>
>
> --
> Mel Gorman
> Part-time Phd Student                          Linux Technology Center
> University of Limerick                         IBM Dublin Software Lab
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/