linux-kernel - Re: [PATCH v5 0/4] idle memory tracking

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 09 Jun 2015 13:56:19 +0530
From:	Raghavendra K T <raghavendra.kt@...ux.vnet.ibm.com>
To:	Andrew Morton <akpm@...ux-foundation.org>
CC:	Vladimir Davydov <vdavydov@...allels.com>,
	Minchan Kim <minchan@...nel.org>,
	Johannes Weiner <hannes@...xchg.org>,
	Michal Hocko <mhocko@...e.cz>,
	Greg Thelen <gthelen@...gle.com>,
	Michel Lespinasse <walken@...gle.com>,
	David Rientjes <rientjes@...gle.com>,
	Pavel Emelyanov <xemul@...allels.com>,
	Cyrill Gorcunov <gorcunov@...nvz.org>,
	Jonathan Corbet <corbet@....net>, linux-api@...r.kernel.org,
	linux-doc@...r.kernel.org, linux-mm@...ck.org,
	cgroups@...r.kernel.org,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v5 0/4] idle memory tracking

On 06/09/2015 01:05 AM, Andrew Morton wrote:
> On Sun, 7 Jun 2015 11:41:15 +0530 Raghavendra KT <raghavendra.kt@...ux.vnet.ibm.com> wrote:
>
>> On Tue, May 12, 2015 at 7:04 PM, Vladimir Davydov
>> <vdavydov@...allels.com> wrote:
>>> Hi,
>>>
>>> This patch set introduces a new user API for tracking user memory pages
>>> that have not been used for a given period of time. The purpose of this
>>> is to provide the userspace with the means of tracking a workload's
>>> working set, i.e. the set of pages that are actively used by the
>>> workload. Knowing the working set size can be useful for partitioning
>>> the system more efficiently, e.g. by tuning memory cgroup limits
>>> appropriately, or for job placement within a compute cluster.
>>>
>>> ---- USE CASES ----
>>>
>>> The unified cgroup hierarchy has memory.low and memory.high knobs, which
>>> are defined as the low and high boundaries for the workload working set
>>> size. However, the working set size of a workload may be unknown or
>>> change in time. With this patch set, one can periodically estimate the
>>> amount of memory unused by each cgroup and tune their memory.low and
>>> memory.high parameters accordingly, therefore optimizing the overall
>>> memory utilization.
>>>
>>
>> Hi Vladimir,
>>
>> Thanks for the patches, I was able test how the series is helpful to determine
>> docker container workingset / idlemem with these patches. (tested on ppc64le
>> after porting to a distro kernel).
>
> And what were the results of your testing?  The more details the
> better, please.
>
>

Hi Andrew,
This is what I had done in my experiment (Theoretical):
1) created a docker container
2)
Ran the python script (example in first patch) provided by Vladimir
to get idle memory in the docker container. This would further help
in analyzing what is the rss docker container would ideally use
and hence we could set the memory limit for the container and we will
know how much we should ideally scale without degrading the performance 
of other containers.

# ~/raghu/idlemmetrack/idlememtrack.py
Setting the idle flag for each page...
Wait until the workload accesses its working set, then press Enter
Counting idle pages..
/sys/fs/cgroup/memory: 9764 KB
[...]
/sys/fs/cgroup/memory/system.slice/docker-[...].scope: 224 K
....

I understand that you might probably want how did the scaling experiment 
with memory limit tuning went after that, but I have not got
that data yet.. :(..

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/