linux-kernel - Re: Intermittent perf build failures

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20161011205927.GA24518@krava>
Date:   Tue, 11 Oct 2016 22:59:27 +0200
From:   Jiri Olsa <jolsa@...hat.com>
To:     Laura Abbott <labbott@...hat.com>
Cc:     Andi Kleen <andi@...stfloor.org>,
        Arnaldo Carvalho de Melo <acme@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Ingo Molnar <mingo@...hat.com>
Subject: Re: Intermittent perf build failures

On Tue, Oct 11, 2016 at 01:43:36PM -0700, Laura Abbott wrote:
> Hi,
> 
> While building today's Fedora rawhide kernel, there was a failure
> building perf with -j4 [1]:
> 
> 
>   gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -D"BUILD_STR(s)=#s"   -c -o plugin_scsi.o plugin_scsi.c
>    ld  -r -o plugin_function-in.o  plugin_function.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_jbd2.so plugin_jbd2-in.o
>   gcc -Wp,-MD,./.plugin_cfg80211.o.d,-MT,plugin_cfg80211.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -D"BUILD_STR(s)=#s"   -c -o plugin_cfg80211.o plugin_cfg80211.c
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_hrtimer.so plugin_hrtimer-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_kmem.so plugin_kmem-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_kvm.so plugin_kvm-in.o
>    ld  -r -o plugin_xen-in.o  plugin_xen.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_mac80211.so plugin_mac80211-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_sched_switch.so plugin_sched_switch-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_function.so plugin_function-in.o
>    ld  -r -o plugin_cfg80211-in.o  plugin_cfg80211.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_xen.so plugin_xen-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -shared -nostartfiles -o plugin_cfg80211.so plugin_cfg80211-in.o
>   gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC  -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include  '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"'  -D_GNU_SOURCE -D"BUILD_STR(s)=#s"   -c -o plugin_scsi.o plugin_scsi.c
>    ld  -r -o plugin_scsi-in.o  plugin_scsi.o
> plugin_scsi.o: file not recognized: File truncated
> make[3]: *** [/builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/build/Makefile.build:140: plugin_scsi-in.o] Error 1
> make[2]: *** [Makefile:189: plugin_scsi-in.o] Error 2
> make[1]: *** [Makefile.perf:570: libtraceevent_plugins] Error 2
> make: *** [Makefile:109: install-traceevent-plugins] Error 2
> 
> The gcc call to compile plugin_scsi.o seems to be invoked twice which would
> account for the ld failing. Unfortunately, this failure seems to be
> intermittent and I've only been able to reproduce it once outside of
> the Fedora build environment so bisection doesn't seem feasible.

hi,
yep, looks like build race.. but -j4 should not be that bad ;-)

some questions:
  - how often do you see this?
  - is it always plugin_scsi.o failing?
  - hows the build invoked?
  - could you please attach full build log?

thanks,
jirka

> 
> The build invocation for this was changed in
> 0c3b7e42616f ("tools build: Add support for host programs format") so
> I'm suspicious of that commit. I'm reverting that commit for today's
> build but I'm not sure how much that's going to indicate given the
> intermittent nature of the failure.
> 
> Any ideas?
> 
> Thanks,
> Laura
> 
> 
> [1] https://kojipkgs.fedoraproject.org//work/tasks/415/16050415/build.log