Message-ID: <MWHPR13MB0895FB798A05EAC455B61679FDF60@MWHPR13MB0895.namprd13.prod.outlook.com>
Date:   Tue, 17 Mar 2020 00:24:26 +0000
From:   "Bird, Tim" <Tim.Bird@...y.com>
To:     shuah <shuah@...nel.org>, Shuah Khan <skhan@...uxfoundation.org>
CC:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: cross-compiling vm, confusion over output directory

> -----Original Message-----
> From: shuah <shuah@...nel.org>
> 
> Hi Tim,
> 
> You are using an ancient email address for me.
> 
> Use either shuah@...nel.org or skhan@...uxfoundation.org
Sorry about that.  I have now fixed my contact info for you
in my email client.

> 
> On 3/13/20 4:14 PM, Bird, Tim wrote:
> > I was working on fixing the cross-compilation for the selftests/vm tests.
> > Currently, there are two issues in my testing:
> >
> > 1) problem: required library missing from some cross-compile environments:
> > tools/testing/selftests/vm/mlock-random-test.c requires libcap
> > be installed.  The target line for mlock-random-test in
> > tools/testing/selftests/vm/Makefile looks like this:
> >
> > $(OUTPUT)/mlock-random-test: LDLIBS += -lcap
> 
> You can use TEST_GEN_PROGS and define LDFLAGS for lcap
The example you have for seccomp uses a single LDFLAGS before
the include of ../lib.mk.

The vm/Makefile has different libraries needed for different programs:
 userfaultfd need libpthread
 mlock-random-test needs libcap
 (something) needs librt

Do you want me to add all these to a single LDFLAGS line, like so?  
LDFLAGS= -lrt -lpthread -lcap

It won't hurt anything, but it seems like overkill.
> 
> >
> > and mlock-random-test.c has this include line:
> > #include <sys/capability.h>
> >
> > this is confusing, since this is different from the header file
> > linux/capability.h.  It is associated with the capability library (libcap)
> > and not the kernel.  In any event, on some distros and in some
> > cross-compile SDKs the package containing these files is not installed
> > by default.
> >
> > Once this library is installed, things progress farther.  Using an Ubuntu
> > system, you can install the cross version of this library (for arm64) by doing:
> > $ sudo apt install libcap-dev:arm64
> >
> > 1) solution:
> > I would like to add some meta-data about this build dependency, by putting
> > something in the settings file as a hint to CI build systems.  Specifically, I'd like to
> > create the file 'tools/testing/selftests/vm/settings', with the content:
> > NEED_LIB=cap
> >
> > We already use settings for other meta-data about a test (right now, just a
> > non-default timeout value), but I don't want to create a new file or syntax
> > for this build dependency data.
> >
> > Let me know what you think.
> >
> > I may follow up with some script in the kernel source tree to check these
> > dependencies, independent of any CI system.  I have such a script in Fuego
> > that I could submit, but it would need some work to fit into the kernel build
> > flow for kselftest.  The goal would be to provide a nicely formatted warning,
> > with a recommendation for a package install.  But that's more work than
> > I think is needed right now just to let developers know there's a build dependency
> > here.
> >
> > 2) problem: reference to source-relative header file
> > the Makefile for vm uses a relative path for include directories.
> > Specifically, it has the line:
> > CFLAGS = -Wall -I ../../../../../usr/include $(EXTRA_CFLAGS)
> >
> > I believe this needs to reference kernel include files from the
> > output directory, not the source directory.
> >
> > With the relative include directory path, the program userfaultfd.c
> > gets compilation error like this:
> >
> > userfaultfd.c:267:21: error: 'UFFD_API_RANGE_IOCTLS_BASIC' undeclared here (not in a function)
> >    .expected_ioctls = UFFD_API_RANGE_IOCTLS_BASIC,
> >                       ^
> > userfaultfd.c: In function 'uffd_poll_thread':
> > userfaultfd.c:529:8: error: 'UFFD_EVENT_FORK' undeclared (first use in this function)
> >     case UFFD_EVENT_FORK:
> >          ^
> > userfaultfd.c:529:8: note: each undeclared identifier is reported only once for each function it appears in
> > userfaultfd.c:531:18: error: 'union <anonymous>' has no member named 'fork'
> >      uffd = msg.arg.fork.ufd;
> >                    ^
> >
> > 2) incomplete solution:
> > I originally changed this line to read:
> > CFLAGS = -Wall -I $(KBUILD_OUTPUT)/usr/include $(EXTRA_CFLAGS)
> >
> 
> Take a look at seccomp patch I sent out. Use the approach used
> in this patch.
> 
> https://lore.kernel.org/linux-kselftest/20200313212404.24552-1-skhan@linuxfoundation.org/T/#u

I took a look at this, and if I'm reading it correctly, the CFLAGS needs to be
set before 'include ../lib.mk'

But it can't be, because the CFLAGS needs OUTPUT to be set, and OUTPUT
is set by lib.mk.

I'm going to play around with this here and see if I can come up with something
that works.
> 
> > This works when the output directory is specified using KBUILD_OUTPUT,
> > but not when the output directory is specified using O=
> > I'm not sure what happens when the output directory is specified
> > with a non-source-tree current working directory.
> >
> > In any event, while researching a proper solution to this, I found
> > the following in tools/testing/selftests/Makefile:
> >
> > If compiling with ifneq ($(O),)
> >      BUILD := $(O)
> > else
> >      ifneq ($(KBUILD_OUTPUT),)
> >          BUILD := $(KBUILD_OUTPUT)/kselftest
> >      else
> >          BUILD := $(shell pwd)
> >          DEFAULT_INSTALL_HDR_PATH := 1
> >      endif
> > endif
> >
> > This doesn't seem right.  It looks like the selftests Makefile treats a directory
> > passed in using O= different from one specified using KBUILD_OUTPUT
> > or the current working directory.
> > In the KBUILD_OUTPUT case, you get an extra 'kselftest' directory layer
> > that you don't get for the other two.
> 
> Are you using
> 
> git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git
> kernelci branch

I wasn't, but I am now.
> 
> This is fixed and that is the very first patch I sent.
> 
> I also applied the patch to
> 
> git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
> branch

I'm now using linux-next master and the linux-kselftest kernelci branch
for my references.

With both of these  I'm getting an error with scripts/Makefile.build. (see below)

KBUILD_OUTPUT is set to
    /fuego-rw/buildzone/ren1.kernelci_one.Functional.kselftest_compile_install-poky-aarch64/kbuild
(this is a directory 'kbuild' inside the kernel source tree)

The command is:
    make V=1 ARCH=arm64 TARGETS="vm" kselftest-install
Here is the make execution output:
------
make -C /fuego-rw/buildzone/ren1.next_one.Functional.kselftest_compile_install-poky-aarch64/kbuild -f /fuego-rw/buildzone/ren1.next_one.Functional.kselftest_compile_install-poky-aarch64/Makefile kselftest-install
make[1]: Entering directory '/fuego-rw/buildzone/ren1.next_one.Functional.kselftest_compile_install-poky-aarch64/kbuild'
test -e include/generated/autoconf.h -a -e include/config/auto.conf || (		\
echo >&2;							\
echo >&2 "  ERROR: Kernel configuration is invalid.";		\
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix it.";	\
echo >&2 ;							\
/bin/false)
make -C ../tools/testing/selftests install
make --no-builtin-rules INSTALL_HDR_PATH=$BUILD/usr \
	ARCH=arm64 -C ../../.. headers_install
make -f ../scripts/Makefile.build obj=scripts/basic
make[4]: ../scripts/Makefile.build: No such file or directory
make[4]: *** No rule to make target '../scripts/Makefile.build'.  Stop.
Makefile:504: recipe for target 'scripts_basic' failed
make[3]: *** [scripts_basic] Error 2
Makefile:151: recipe for target 'khdr' failed
make[2]: *** [khdr] Error 2
/fuego-rw/buildzone/ren1.next_one.Functional.kselftest_compile_install-poky-aarch64/Makefile:1228: recipe for target 'kselftest-install' failed
make[1]: *** [kselftest-install] Error 2
make[1]: Leaving directory '/fuego-rw/buildzone/ren1.next_one.Functional.kselftest_compile_install-poky-aarch64/kbuild'
Makefile:180: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2-----

There's still something wonky with using KBUILD_OUTPUT
with kselftest (at least with my configuration).

Note that I switched over to using the top-level Makefile target 'kselftest-install'.

I am investigating and will report back when I figure out what the issue is.

Thanks,
 -- Tim

Powered by blists - more mailing lists