Date:   Mon, 15 Mar 2021 16:45:55 +0000
From:   "Denys Zagorui -X (dzagorui - GLOBALLOGIC INC at Cisco)" 
        <dzagorui@...co.com>
To:     Jiri Olsa <jolsa@...hat.com>
CC:     "peterz@...radead.org" <peterz@...radead.org>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "acme@...nel.org" <acme@...nel.org>,
        "mark.rutland@....com" <mark.rutland@....com>,
        "alexander.shishkin@...ux.intel.com" 
        <alexander.shishkin@...ux.intel.com>,
        "namhyung@...nel.org" <namhyung@...nel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] perf: build reproducibility improvements

> Makefile.config:1026: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel
> cp: '/home/jolsa/kernel/linux-perf/tools/perf/Documentation/tips.txt' and 'Documentation/tips.txt' are the same file
>  BISON    util/parse-events-bison.c
> bison: unrecognized option '--file-prefix-map=='

I thought that this flag was added in v3.6.3 because in git history next tag after corresponding bison patch was v3.6.3. But this is not true.

> hum, do we actualy want this? I think we want the exact path
> we used for compilation, no? what's the benefit?
...
> same here, we want to be sure to use the python path
> from the exact build laction no?

This patch makes perf build more deterministic. This means that if we build perf on two different
build machines from exactly the same sources we will have absolutely identical binaries. To achieve
this absolute paths should not be stored in resulting binary. That is why i tried to determine those paths
in runtime instead of storing them in binary compile time.
There is ongoing project
https://reproducible-builds.org/reports/2021-02/ project
Kernel already achieved this
https://www.kernel.org/doc/html/latest/kbuild/reproducible-builds.html

This patch doesn't make perf 100% reproducible. There is one more known issue with pmu event ordering
https://bugzilla.opensuse.org/show_bug.cgi?id=1180882

________________________________________
From: Jiri Olsa <jolsa@...hat.com>
Sent: 15 March 2021 14:28
To: Denys Zagorui -X (dzagorui - GLOBALLOGIC INC at Cisco)
Cc: peterz@...radead.org; mingo@...hat.com; acme@...nel.org; mark.rutland@....com; alexander.shishkin@...ux.intel.com; namhyung@...nel.org; linux-kernel@...r.kernel.org
Subject: Re: [PATCH] perf: build reproducibility improvements

On Fri, Mar 12, 2021 at 03:17:00PM +0000, Denys Zagorui wrote:
> This patch helps to make perf build more reproducible
>
> It seems there is some need to have an ability to invoke
> perf from build directory without installation
> (84cfac7f05e1: perf tools: Set and pass DOCDIR to builtin-report.c)
> DOCDIR contains an absolute path to kernel source directory.
> In such case this path can be determined in runtime by using
> /proc/self/exe link. In case of building perf with O=
> Documentation/tips.txt can be copied to output directory.
>
> There is also python binding test where PYTHONPATH is used to store
> absolute path to python/perf.so library. This path can be
> also determined in runtime.
>
> bison stores full paths in generated files. This can be
> remapped by using --file-prefix-map option that is available
> starting from version 3.6.3.
>
> Signed-off-by: Denys Zagorui <dzagorui@...co.com>

hi,
I'm getting build error:

[jolsa@...va perf]$ make JOBS=1
  BUILD:   Doing 'make -j1' parallel build
Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h'
diff -u tools/include/uapi/linux/kvm.h include/uapi/linux/kvm.h
Makefile.config:1026: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel
cp: '/home/jolsa/kernel/linux-perf/tools/perf/Documentation/tips.txt' and 'Documentation/tips.txt' are the same file
  BISON    util/parse-events-bison.c
bison: unrecognized option '--file-prefix-map=='
Try 'bison --help' for more information.
make[4]: *** [util/Build:211: util/parse-events-bison.c] Error 1
make[3]: *** [/home/jolsa/kernel/linux-perf/tools/build/Makefile.build:139: util] Error 2
make[2]: *** [Makefile.perf:653: perf-in.o] Error 2
make[1]: *** [Makefile.perf:236: sub-make] Error 2
make: *** [Makefile:70: all] Error 2


SNIP

> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> index f6e609673de2..2bebff69b064 100644
> --- a/tools/perf/Makefile.perf
> +++ b/tools/perf/Makefile.perf
> @@ -627,6 +627,9 @@ $(OUTPUT)python/perf.so: $(PYTHON_EXT_SRCS) $(PYTHON_EXT_DEPS) $(LIBTRACEEVENT_D
>         --quiet build_ext; \
>       cp $(PYTHON_EXTBUILD_LIB)perf*.so $(OUTPUT)python/
>
> +_dummy := $(shell [ -d '$(OUTPUT)Documentation' ] || mkdir -p '$(OUTPUT)Documentation' && \
> +             cp '$(srcdir)/Documentation/tips.txt' '$(OUTPUT)Documentation/')
> +
>  please_set_SHELL_PATH_to_a_more_modern_shell:
>       $(Q)$$(:)
>
> diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> index 2a845d6cac09..d9441055357e 100644
> --- a/tools/perf/builtin-report.c
> +++ b/tools/perf/builtin-report.c
> @@ -610,12 +610,27 @@ static int report__browse_hists(struct report *rep)
>       struct perf_session *session = rep->session;
>       struct evlist *evlist = session->evlist;
>       const char *help = perf_tip(system_path(TIPDIR));
> +     char *exec_path;
> +     char *docdir;
>
>       if (help == NULL) {
>               /* fallback for people who don't install perf ;-) */
> -             help = perf_tip(DOCDIR);
> -             if (help == NULL)
> -                     help = "Cannot load tips.txt file, please install perf!";
> +             exec_path = get_exec_abs_path();
> +             if (exec_path == NULL || asprintf(&docdir, "%sDocumentation", exec_path) < 0) {
> +                     docdir = NULL;
> +                     help = "Not enough memory or some other internal error occurred!";
> +             }

hum, do we actualy want this? I think we want the exact path
we used for compilation, no? what's the benefit?

> +
> +             if (docdir != NULL) {
> +                     help = perf_tip(docdir);
> +                     if (help == NULL)
> +                             help = "Cannot load tips.txt file, please install perf!";
> +             }
> +
> +             if (exec_path)
> +                     free(exec_path);
> +             if (docdir)
> +                     free(docdir);
>       }
>
>       switch (use_browser) {
> diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
> index 650aec19d490..a20098dcdbc4 100644
> --- a/tools/perf/tests/Build
> +++ b/tools/perf/tests/Build
> @@ -98,5 +98,5 @@ perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
>  endif
>
>  CFLAGS_attr.o         += -DBINDIR="BUILD_STR($(bindir_SQ))" -DPYTHON="BUILD_STR($(PYTHON_WORD))"
> -CFLAGS_python-use.o   += -DPYTHONPATH="BUILD_STR($(OUTPUT)python)" -DPYTHON="BUILD_STR($(PYTHON_WORD))"
> +CFLAGS_python-use.o   += -DPYTHON="BUILD_STR($(PYTHON_WORD))"
>  CFLAGS_dwarf-unwind.o += -fno-optimize-sibling-calls
> diff --git a/tools/perf/tests/python-use.c b/tools/perf/tests/python-use.c
> index 98c6d474aa6f..8b4865c90d5d 100644
> --- a/tools/perf/tests/python-use.c
> +++ b/tools/perf/tests/python-use.c
> @@ -8,16 +8,28 @@
>  #include <linux/compiler.h>
>  #include "tests.h"
>  #include "util/debug.h"
> +#include <subcmd/exec-cmd.h>
>
>  int test__python_use(struct test *test __maybe_unused, int subtest __maybe_unused)
>  {
>       char *cmd;
>       int ret;
> +     char *exec_path = NULL;
> +     char *pythonpath;
> +
> +     exec_path = get_exec_abs_path();
> +     if (exec_path == NULL)
> +             return -1;
> +
> +     if (asprintf(&pythonpath, "%spython", exec_path) < 0)
> +             return -1;

same here, we want to be sure to use the python path
from the exact build laction no?

thanks,
jirka

Powered by blists - more mailing lists