linux-kernel - uprobe causing a process to crash on file modification

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <d26368538d5f4504b9ca239b14328d33@crowdstrike.com>
Date: Fri, 9 Aug 2024 14:16:54 +0000
From: Rahul Shah <rahul.shah@...wdstrike.com>
To: "bpf@...r.kernel.org" <bpf@...r.kernel.org>
CC: Marco Vedovati <marco.vedovati@...wdstrike.com>,
        Kelly Martin
	<kelly.martin@...wdstrike.com>,
        "andrii@...nel.org" <andrii@...nel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-perf-users@...r.kernel.org" <linux-perf-users@...r.kernel.org>,
        "linux-trace-kernel@...r.kernel.org" <linux-trace-kernel@...r.kernel.org>,
        Rahul Shah <rahul.shah@...wdstrike.com>
Subject: uprobe causing a process to crash on file modification

We have noticed an issue when a uprobe is attached to a valid file offset, and the content of that file is modified (with no inode change).
Executing from a modified file while the uprobe is attached may cause a process to crash. The crash happens if the probed offset is invalid for the new file content.

Here's a simple scenario for an executable leading to consistent crashes (but something similar can happen for DSO too):

1. Create a simple binary B1 that has symbol S1 at offset O1. The binary B1 has inode INO_1.
2. Start a new process P1 and attach an uprobe to this binary at B1:S1:O1
3. Run the binary B1 and the uprobe is fired as expected.
4. Continue to run the process P1 (a.k.a the uprobe is still attached to B1:S1:O1)
5. Create a new binary B2 that no longer has the symbol S1 at offset O1. The binary B2 has inode INO_2
6. Copy the content of binary INO_2 to INO_1 using cp command. As a result the original binary B1 with INO_1  has new contents and no longer has symbol S1 at offset O1. The key here is it’s the same inode.
7. The original process P1 still has an uprobe attached to B1:S1:O1.
8. Re-run the binary B1 and it seg faults immediately. In short any B1 become completely unusable.
9. Detach the uprobe.
10. Re-run the binary B1 and all good

Sample gdb output where it crashes (Notice the (bad))

0x0000555555555154 in _fini ()
(gdb) x/20i _fini
   0x555555555148 <_fini>:	repz int3
   0x55555555514a <_fini+2>:	(bad)
   0x55555555514b <_fini+3>:	cli
   0x55555555514c <_fini+4>:	sub    $0x8,%rsp
   0x555555555150 <_fini+8>:	add    $0x8,%rsp
=> 0x555555555154 <_fini+12>:	ret

Syslogs output:
Aug  8 21:33:00 rshahvm2 kernel: [50553.686400] hello_world[30144]: segfault at f7d3e9c8 ip 00007f9701b40144 sp 00000000f7d3e9c8 error 4 in libfoo.so[7f9701b40000+1000]
Aug  8 21:33:00 rshahvm2 kernel: [50553.686409] Code: 0f 1e fa 55 48 89 e5 90 90 90 5d c3 f3 0f 1e fa 55 48 89 e5 e8 1d ff ff ff 5d c3 00 00 00 f3 cc 1e fa 48 83 ec 08 48 83 c4 08 <c3> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

What we saw is that when a file (executable or a DSO) is mapped in a process memory, the kernel installs all uprobes registered for that file's inode, no matter if the file has been changed since the initial uprobe registration.

If uprobes are installed at a offset that's mo more valid, this behavior can make all new application completely unusable.

Just wondering if there is any fix in the kernel for this of if anyone else has observed similar behavior.

Thanks,
Rahul