linux-kernel - Re: [PATCH] proc/meminfo: add MemKernel counter

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Wed, 22 May 2019 21:18:36 +0300
From:   Konstantin Khlebnikov <koct9i@...il.com>
To:     Roman Gushchin <guro@...com>
Cc:     Konstantin Khlebnikov <khlebnikov@...dex-team.ru>,
        Michal Hocko <mhocko@...nel.org>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Johannes Weiner <hannes@...xchg.org>
Subject: Re: [PATCH] proc/meminfo: add MemKernel counter

On Wed, May 22, 2019 at 8:04 PM Roman Gushchin <guro@...com> wrote:
>
> On Wed, May 22, 2019 at 07:09:22PM +0300, Konstantin Khlebnikov wrote:
> > On 22.05.2019 18:52, Michal Hocko wrote:
> > > On Wed 22-05-19 17:40:09, Konstantin Khlebnikov wrote:
> > > > Some kinds of kernel allocations are not accounted or not show in meminfo.
> > > > For example vmalloc allocations are tracked but overall size is not shown
> > > > for performance reasons. There is no information about network buffers.
> > > >
> > > > In most cases detailed statistics is not required. At first place we need
> > > > information about overall kernel memory usage regardless of its structure.
> > > >
> > > > This patch estimates kernel memory usage by subtracting known sizes of
> > > > free, anonymous, hugetlb and caches from total memory size: MemKernel =
> > > > MemTotal - MemFree - Buffers - Cached - SwapCached - AnonPages - Hugetlb.
> > >
> > > Why do we need to export something that can be calculated in the
> > > userspace trivially? Also is this really something the number really
> > > meaningful? Say you have a driver that exports memory to the userspace
> > > via mmap but that memory is not accounted. Is this really a kernel
> > > memory?
> > >
> >
> > It may be trivial right now but not fixed.
> > Adding new kinds of memory may change this definition.
>
> Right, and it's what causes me to agree with Michal here, and leave it
> to the userspace calculation.
>
> The real meaning of the counter is the size of the "gray zone",
> basically the memory which we have no clue about.

Well, all kernel memory is a gray zone for normal programmers.
They have direct control only over anon and file-cache.

I want to invent simple metrics for 'system' memory usage.
It's about the same as separation cpu time to user and system.

> If we'll add accounting of some new type of memory, which now in this
> gray zone (say, xfs buffers), we probably should exclude it too.
> And this means that definition of this counter will change.

I'm not very familiar with xfs internals, never digged into it.
I've excluded buffers because this is simply file-cache for block devices.
Filesystems use it as cache for metadata. But userspace has direct access to it.

>
> So IMO the definition is way too implementation-defined to be a part
> of procfs API.
>

Ok. User/kernel memory separation could be redefined in more
abstract manner depending on the data access.