netdev - Re: [PATCH v2] docs: proc.rst: meminfo: briefly describe gaps in memory accounting

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YH7ds1YOAOQt8Mpf@dhcp22.suse.cz>
Date:   Tue, 20 Apr 2021 15:57:07 +0200
From:   Michal Hocko <mhocko@...e.com>
To:     Matthew Wilcox <willy@...radead.org>
Cc:     Mike Rapoport <rppt@...nel.org>, Jonathan Corbet <corbet@....net>,
        Alexey Dobriyan <adobriyan@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Eric Dumazet <eric.dumazet@...il.com>,
        Mike Rapoport <rppt@...ux.ibm.com>, linux-doc@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-mm@...ck.org, netdev@...r.kernel.org
Subject: Re: [PATCH v2] docs: proc.rst: meminfo: briefly describe gaps in
 memory accounting

On Tue 20-04-21 14:24:30, Matthew Wilcox wrote:
> On Tue, Apr 20, 2021 at 03:13:54PM +0300, Mike Rapoport wrote:
> > Add a paragraph that explains that it may happen that the counters in
> > /proc/meminfo do not add up to the overall memory usage.
> 
> ... that is, the sum may be lower because memory is allocated for other
> purposes that is not reported here, right?

yes. Many direct page allocator users are not accounted in any of the
existing counters.

> Is it ever possible for it to be higher?  Maybe due to a race when
> sampling the counters?

Yes likely possible. You will never get an atomic snapshot of all
counters.

> >  Provides information about distribution and utilization of memory.  This
> > -varies by architecture and compile options.  The following is from a
> > -16GB PIII, which has highmem enabled.  You may not have all of these fields.
> > +varies by architecture and compile options. Please note that it may happen
> > +that the memory accounted here does not add up to the overall memory usage
> > +and the difference for some workloads can be substantial. In many cases there
> > +are other means to find out additional memory using subsystem specific
> > +interfaces, for instance /proc/net/sockstat for TCP memory allocations.
> 
> How about just:
> 
> +varies by architecture and compile options.  The memory reported here
> +may not add up to the overall memory usage and the difference for some
> +workloads can be substantial. [...]
> 
> But I'd like to be a bit more explicit about the reason, hence my question
> above to be sure I understand.
> 
> 
> It's also not entirely clear which of the fields in meminfo can be
> usefully summed.  VmallocTotal is larger than MemTotal, for example.

Yes. Many/Most counters cannot be simply sumed up. A trivial example would be
Active/Inactive is a sum of both anona and file. Mlocked will be
accounted in LRU pages and Unevictable. MemAvailable is not really a
counter... 

Usual memory consumption is usually something like LRU pages + Slab
memory + kernel stack + vmalloc used + pcp.

> But I know that KernelStack is allocated through vmalloc these days,
> and I don't know whether VmallocUsed includes KernelStack or whether I
> can sum them.  Similarly, is Mlocked a subset of Unevictable?
> 
> There is some attempt at explaining how these numbers fit together, but
> it's outdated, and doesn't include Mlocked, Unevictable or KernelStack

Agreed there is a lot of tribal knowledge or even misconceptions flying
around and it will take much more work to put everything into shape.
This is only one tiny step forward.
-- 
Michal Hocko
SUSE Labs