linux-kernel - [PATCH 0/2] modules: allow error injection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250122-modules-error-injection-v1-0-910590a04fd5@samsung.com>
Date: Wed, 22 Jan 2025 14:11:12 +0100
From: Daniel Gomez <da.gomez@...sung.com>
To: Luis Chamberlain <mcgrof@...nel.org>, Petr Pavlu <petr.pavlu@...e.com>,
	Sami Tolvanen <samitolvanen@...gle.com>, Alexei Starovoitov
	<ast@...nel.org>, Daniel Borkmann <daniel@...earbox.net>, Andrii Nakryiko
	<andrii@...nel.org>, Martin KaFai Lau <martin.lau@...ux.dev>, Eduard
	Zingerman <eddyz87@...il.com>, Song Liu <song@...nel.org>, Yonghong Song
	<yonghong.song@...ux.dev>, John Fastabend <john.fastabend@...il.com>, "KP
 Singh" <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, Hao Luo
	<haoluo@...gle.com>, Jiri Olsa <jolsa@...nel.org>, Nathan Chancellor
	<nathan@...nel.org>, Nick Desaulniers <ndesaulniers@...gle.com>, "Bill
 Wendling" <morbo@...gle.com>, Justin Stitt <justinstitt@...gle.com>
CC: <linux-modules@...r.kernel.org>, <linux-kernel@...r.kernel.org>,
	<bpf@...r.kernel.org>, <llvm@...ts.linux.dev>,
	<iovisor-dev@...ts.iovisor.org>, <gost.dev@...sung.com>, Daniel Gomez
	<da.gomez@...sung.com>
Subject: [PATCH 0/2] modules: allow error injection

Allow error injection during module initialization for testing.

This adds ALLOW_ERROR_INJECTION() annotation for complete_formation(),
do_init_module() and module_enable_rodata_ro_after_init() so we can test
the error path in these functions.

In addition, add moderr support, an eBPF based tool to inject module
errors in any of the above functions for a specific module name.

The following example is added to the moderr commit. Adding it also here
for convenience:

    Example: Inject error -22 to module_enable_rodata_ro_after_init for
    brd module:

    sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
    --error=-22 --trace
    Monitoring module error injection... Hit Ctrl-C to end.
    MODULE     ERROR FUNCTION
    brd        -22   module_enable_rodata_after_init()

    Kernel messages:
    [   89.463690] brd: module loaded
    [   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,
    ro_after_init data might still be writable

The tool is based on libbpf and is located under tools/bpf.

Here some specific questions for discussion.

* I've noticed that some users of ALLOW_ERROR_INJECTION() in the
kernel add the option to fail behind a Kconfig (Fault-injection
framework, FAULT_INJECTION) in lib/Kconfig.debug, such as FAILSLAB or
FAIL_PAGE_ALLOC. Shall we add similar condition for error injection
in modules?

* What other functions may people be interested in adding to the list of
suported for error injection?

* I found a more generic error injection tool (inject.py) [1] in
IOVisor/BCC (Ccing IOVisor mailing list as well). The tool seems to
be maintained and supports error injection based on a specified call
chain and optional predicates. However, it relies on the old and
deprecated Python eBPF/BCC infrastructure. Since there are other users
of ALLOW_ERROR_INJECTION(), I'm curious: is there a different generic
tool to generate error injection in the kernel already converted to
libbpf? Do users of ALLOW_ERROR_INJECTION() still depend on inject.py
tool?

[1] https://github.com/iovisor/bcc/blob/master/tools/inject.py

Additionally, would it make sense to develop an in-tree tool with
libbpf support for this purpose? This approach might require rewriting
inject.py in libbpf and/or extending moderr to include inject.py
features (probably replace and rework some of the current functionality
in moderr).

moderr uses an enum to detect which kprobe function to override the
return value. I think adding bpf_get_stack()/bpf_get_stackid() support
would avoid the need for hardcoding the kprobe detection mechanism.
This approach could also be used for navigation of the call stack as in
inject.py. Are there any alternative methods to achieve this that avoid
relying on enumerating the fuctions to probe?

Signed-off-by: Daniel Gomez <da.gomez@...sung.com>
---
Daniel Gomez (2):
      module: allow for module error injection
      moderr: add module error injection tool

 kernel/module/main.c          |   3 +
 kernel/module/strict_rwx.c    |   2 +
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 8 files changed, 515 insertions(+), 3 deletions(-)
---
base-commit: 232f121837ad8b1c21cc80f2c8842a4090c5a2a0
change-id: 20250121-modules-error-injection-176c525791b0

Best regards,
-- 
Daniel Gomez <da.gomez@...sung.com>