linux-kernel - Re: [RFC PATCH 7/7] tools: add skeleton code for userland testing of VMA logic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Thu, 27 Jun 2024 20:31:07 +0100
From: Lorenzo Stoakes <lstoakes@...il.com>
To: "Liam R. Howlett" <Liam.Howlett@...cle.com>,
	Kees Cook <kees@...nel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org, Vlastimil Babka <vbabka@...e.cz>,
	Matthew Wilcox <willy@...radead.org>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	Christian Brauner <brauner@...nel.org>, Jan Kara <jack@...e.cz>,
	Eric Biederman <ebiederm@...ssion.com>,
	Suren Baghdasaryan <surenb@...gle.com>
Subject: Re: [RFC PATCH 7/7] tools: add skeleton code for userland testing of
 VMA logic

On Thu, Jun 27, 2024 at 02:25:36PM -0400, Liam R. Howlett wrote:
> * Kees Cook <kees@...nel.org> [240627 12:58]:
> > On Thu, Jun 27, 2024 at 11:39:32AM +0100, Lorenzo Stoakes wrote:
> > > Establish a new userland VMA unit testing implementation under
> > > tools/testing which utilises existing logic providing maple tree support in
> > > userland utilising the now-shared code previously exclusive to radix tree
> > > testing.
> > >
> > > This provides fundamental VMA operations whose API is defined in mm/vma.h,
> > > while stubbing out superfluous functionality.
> > >
> > > This exists as a proof-of-concept, with the test implementation functional
> > > and sufficient to allow userland compilation of vma.c, but containing only
> > > cursory tests to demonstrate basic functionality.
> >
> > Interesting! Why do you want to have this in userspace instead of just
> > wiring up what you have here to KUnit so testing can be performed by
> > existing CI systems that are running all the KUnit tests?
>
> The primary reason we did the maple tree testing in userspace was for
> speed of testing.  We don't need to build the kernel, but a subset of
> APIs.  Debugging problems is also much quicker since we can instrument
> and rebuild, iterate down faster.  Tracing every call to the maple tree
> on boot alone is massive.
>
> It is also difficult to verify the vma correctness without exposing APIs
> we don't want exported (or, I guess, parse proc files..).  On my side, I
> have a module for testing the overall interface while I have more tests
> on the userspace side that poke and prod on internal states, and
> userspace rcu testing is possible.  I expect the same issues on the vma
> side.
>
> Adding tests can also be made very efficient with tracepoints dumping
> something to add to an array, for example.
>
> Finally, you have ultimate control on what other functions return (or
> do) - so you can fail allocations to test error paths, for example.  Or
> set the external function to fail after N allocations.  This comes in
> handy when a syzbot reports a failed allocation at line X caused a
> crash.
>
> This has worked out phenomenally on the maple tree side.  I've been able
> to record boot failures and import them, syzbot tests, and fuzzer tests.
> The result is a huge list of tests that allowed me to rewrite my node
> replacement algorithm and have it just work, once it passed the
> collected tests.
>
> I haven't used kunit as much as I have userspace testing, so I cannot
> say if all of these points are not possible, but I didn't see a way to
> test races like I do with rcu in userspace.
>
> Thanks,
> Liam

Liam's response is excellent, and obviously I agree
wholeheartedly. Additionally, I'm not really experienced with kunit, but
surely it's implemented as a kernel module somehow? If so, these interfaces
are largely not exported so it wouldn't be functional as a unit test.

And also as Liam says, it'd be very difficult to test this stuff _in_ the
kernel without unwanted side-effects triggering and it'd be very difficult
to isolate or mock components we don't want to play a role (for instance -
rlimits that we might not be able to control).

But overall (again as Liam says) the performance benefit, flexibility and
ability to recreate things at a whim are huge.

And the fact maple tree (which forms a HUGE part of these VMA operations)
and related radix tree and other shims/stubs already exist means that it
wasn't anywhere near as huge a task to implement this as it would be
otherwise.