netdev - Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20200520052604.o53ca2kyzd7sd3g3@ast-mbp.dhcp.thefacebook.com>
Date:   Tue, 19 May 2020 22:26:04 -0700
From:   Alexei Starovoitov <alexei.starovoitov@...il.com>
To:     Alan Maguire <alan.maguire@...cle.com>
Cc:     shuah@...nel.org, ast@...nel.org, daniel@...earbox.net,
        andriin@...com, bpf@...r.kernel.org, kafai@...com,
        songliubraving@...com, yhs@...com, john.fastabend@...il.com,
        kpsingh@...omium.org, linux-kselftest@...r.kernel.org,
        netdev@...r.kernel.org
Subject: Re: [PATCH bpf-next] selftests/bpf: add general instructions for
 test execution

On Tue, May 19, 2020 at 10:34:25PM +0100, Alan Maguire wrote:
> On Tue, 19 May 2020, Alexei Starovoitov wrote:
> 
> > On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> > > Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> > > are used, pahole is recent (>=1.16) and config matches the specified
> > > config file as closely as possible.  Document all of this in the general
> > > README.rst file.  Also note how to work around timeout failures.
> > > 
> > > Signed-off-by: Alan Maguire <alan.maguire@...cle.com>
> > > ---
> > >  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
> > >  1 file changed, 46 insertions(+)
> > > 
> > > diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> > > index 0f67f1b..b00eebb 100644
> > > --- a/tools/testing/selftests/bpf/README.rst
> > > +++ b/tools/testing/selftests/bpf/README.rst
> > > @@ -1,6 +1,52 @@
> > >  ==================
> > >  BPF Selftest Notes
> > >  ==================
> > > +First verify the built kernel config options match the config options
> > > +specified in the config file in this directory.  Test failures for
> > > +unknown helpers, inability to find BTF etc will be observed otherwise.
> > > +
> > > +To ensure the maximum number of tests pass, it is best to use the latest
> > > +trunk LLVM/clang, i.e.
> > > +
> > > +git clone https://github.com/llvm/llvm-project
> > > +
> > > +Build/install trunk LLVM:
> > > +
> > > +.. code-block:: bash
> > > +  git clone https://github.com/llvm/llvm-project
> > > +  cd llvm-project
> > > +  mkdir build/llvm
> > > +  cd build/llvm
> > > +  cmake ../../llvm/
> > > +  make
> > > +  sudo make install
> > > +  cd ../../
> > > +
> > > +Build/install trunk clang:
> > > +
> > > +.. code-block:: bash
> > > +  mkdir -p build/clang
> > > +  cd build/clang
> > > +  cmake ../../clang
> > > +  make
> > > +  sudo make install
> > > +
> > 
> > these instructions are obsolete and partially incorrect.
> > May be refer to Documentation/bpf/bpf_devel_QA.rst instead?
> >
> 
> Sure; looks like there are up-to-date sections there on
> running BPF selftests and building LLVM manually.  Perhaps
> I should add the notes about pahole etc there too?

yes. please.
Could you mention distros that have fresh pahole?
Otherwise users will have an impression that pahole needs
to be build from scratch as well. Which is not the case.

> I should also have noted that without an up-to-date iproute2
> failures will be observed also.

Would be good to highlight which tests will fail with old iproute2.
I'm not sure which version is necessary.
What is 'up-to-date' ?
The tests I run before applying work for me and I rebuild iproute2
every year or so :)

> > > +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> > > +version 16 or later is also required for BTF function
> > > +support. pahole can be built from the source at
> > > +
> > > +https://github.com/acmel/dwarves
> > > +
> > > +It is often available in "dwarves/libdwarves" packages also,
> > > +but be aware that versions prior to 1.16 will fail with
> > > +errors that functions cannot be found in BTF.
> > > +
> > > +When running selftests, the default timeout of 45 seconds
> > > +can be exceeded by some tests.  We can override the default
> > > +timeout via a "settings" file; for example:
> > > +
> > > +.. code-block:: bash
> > > +  echo "timeout=120" > tools/testing/selftests/bpf/settings
> > 
> > Is it really the case?
> > I've never seen anything like this.
> > 
> 
> When running via "make run_tests" on baremetal systems I
> see test timeouts pretty consistently; e.g. from a bpf tree test
> run yesterday:
> 
> not ok 6 selftests: bpf: test_progs # TIMEOUT
> not ok 31 selftests: bpf: test_tunnel.sh # TIMEOUT
> not ok 38 selftests: bpf: test_lwt_ip_encap.sh # TIMEOUT
> not ok 40 selftests: bpf: test_tc_tunnel.sh # TIMEOUT
> not ok 42 selftests: bpf: test_xdping.sh # TIMEOUT
> not ok 43 selftests: bpf: test_bpftool_build.sh # TIMEOUT
> 
> These will only occur if running via "make run_tests",
> so running tests individually would not trigger these
> failures.

If timeout is necessary it's better to fix it in the git
instead of requiring users to tweak their environment.