linux-kernel - [PATCH v2 00/13] Improvements to memory use

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20231012062359.1616786-1-irogers@google.com>
Date:   Wed, 11 Oct 2023 23:23:46 -0700
From:   Ian Rogers <irogers@...gle.com>
To:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Arnaldo Carvalho de Melo <acme@...nel.org>,
        Mark Rutland <mark.rutland@....com>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Jiri Olsa <jolsa@...nel.org>,
        Namhyung Kim <namhyung@...nel.org>,
        Ian Rogers <irogers@...gle.com>,
        Adrian Hunter <adrian.hunter@...el.com>,
        Nick Terrell <terrelln@...com>,
        Kan Liang <kan.liang@...ux.intel.com>,
        Song Liu <song@...nel.org>,
        Sandipan Das <sandipan.das@....com>,
        Anshuman Khandual <anshuman.khandual@....com>,
        James Clark <james.clark@....com>,
        Liam Howlett <liam.howlett@...cle.com>,
        Miguel Ojeda <ojeda@...nel.org>, Leo Yan <leo.yan@...aro.org>,
        German Gomez <german.gomez@....com>,
        Ravi Bangoria <ravi.bangoria@....com>,
        Artem Savkov <asavkov@...hat.com>,
        Athira Rajeev <atrajeev@...ux.vnet.ibm.com>,
        Andi Kleen <ak@...ux.intel.com>, linux-kernel@...r.kernel.org,
        linux-perf-users@...r.kernel.org, bpf@...r.kernel.org
Subject: [PATCH v2 00/13] Improvements to memory use

Fix memory leaks detected by address/leak sanitizer affecting LBR
call-graphs, perf mem and BPF offcpu.

Make branch_type_stat in callchain_list optional as it is large and
not always necessary - in particular it isn't used by perf top.

Make the allocations of zstd streams, kernel symbols and event copies
lazier in order to save memory in cases like perf record.

Handle the thread exit event and have it remove the thread from the
threads set in machine. Don't do this for perf report as it causes a
regression for task lists, which assume threads are never removed from
the machine's set, and offcpu events, that may sythensize samples for
threads that have exited.

The overall effect is to reduce memory consumption significantly for
perf top - with call graphs enabled running longer before 1GB of
memory is consumed. For a perf record of 'true', the memory
consumption goes from 39912kb max resident to 20820kb max resident -
nearly halved.

v2: Add additional memory fixes on top of initial LBR and rc check
    fixes.

Ian Rogers (13):
  perf machine: Avoid out of bounds LBR memory read
  libperf rc_check: Make implicit enabling work for GCC
  perf hist: Add missing puts to hist__account_cycles
  perf threads: Remove unused dead thread list
  perf offcpu: Add missed btf_free
  perf callchain: Make display use of branch_type_stat const
  perf callchain: Make brtype_stat in callchain_list optional
  perf callchain: Minor layout changes to callchain_list
  perf mem_info: Add and use map_symbol__exit and addr_map_symbol__exit
  perf record: Lazy load kernel symbols
  libperf: Lazily allocate mmap event copy
  perf mmap: Lazily initialize zstd streams
  perf machine thread: Remove exited threads by default

 tools/lib/perf/include/internal/mmap.h     |  2 +-
 tools/lib/perf/include/internal/rc_check.h |  6 ++-
 tools/lib/perf/mmap.c                      |  9 ++++
 tools/perf/builtin-inject.c                |  4 ++
 tools/perf/builtin-record.c                |  2 +
 tools/perf/builtin-report.c                |  7 +++
 tools/perf/util/Build                      |  1 +
 tools/perf/util/bpf_off_cpu.c              | 10 ++--
 tools/perf/util/branch.c                   |  4 +-
 tools/perf/util/branch.h                   |  4 +-
 tools/perf/util/callchain.c                | 62 ++++++++++++++--------
 tools/perf/util/callchain.h                | 18 +++----
 tools/perf/util/compress.h                 |  1 +
 tools/perf/util/event.c                    |  4 +-
 tools/perf/util/hist.c                     | 16 +++---
 tools/perf/util/machine.c                  | 39 +++++++-------
 tools/perf/util/machine.h                  |  1 -
 tools/perf/util/map_symbol.c               | 15 ++++++
 tools/perf/util/map_symbol.h               |  4 ++
 tools/perf/util/mmap.c                     |  5 +-
 tools/perf/util/mmap.h                     |  1 -
 tools/perf/util/symbol.c                   |  5 +-
 tools/perf/util/symbol_conf.h              |  4 +-
 tools/perf/util/thread.h                   | 14 +++++
 tools/perf/util/zstd.c                     | 61 +++++++++++----------
 25 files changed, 196 insertions(+), 103 deletions(-)
 create mode 100644 tools/perf/util/map_symbol.c

-- 
2.42.0.609.gbb76f46606-goog