| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <eb171bee-1b59-822f-4816-2adb7da5161f@huaweicloud.com>
Date: Wed, 11 Oct 2023 14:48:32 +0800
From: Hou Tao <houtao@...weicloud.com>
To: Hsin-Wei Hung <hsinweih@....edu>
Cc: Alexei Starovoitov <ast@...nel.org>,
Daniel Borkmann <daniel@...earbox.net>, Andrii Nakryiko <andrii@...nel.org>,
Martin KaFai Lau <kafai@...com>, Song Liu <songliubraving@...com>,
Yonghong Song <yhs@...com>, John Fastabend <john.fastabend@...il.com>,
KP Singh <kpsingh@...nel.org>, Network Development <netdev@...r.kernel.org>,
bpf <bpf@...r.kernel.org>, Kumar Kartikeya Dwivedi <memxor@...il.com>
Subject: Re: Possible kernel memory leak in bpf_timer
Hi,
On 10/11/2023 2:16 PM, Hou Tao wrote:
> Hi,
>
> On 10/11/2023 12:39 PM, Hsin-Wei Hung wrote:
>> On Sat, Oct 7, 2023 at 7:46 PM Hou Tao <houtao@...weicloud.com> wrote:
>>> Hi,
>>>
>>> On 9/27/2023 1:32 PM, Hsin-Wei Hung wrote:
>>>> Hi,
>>>>
>>>> We found a potential memory leak in bpf_timer in v5.15.26 using a
>>>> customized syzkaller for fuzzing bpf runtime. It can happen when
>>>> an arraymap is being released. An entry that has been checked by
>>>> bpf_timer_cancel_and_free() can again be initialized by bpf_timer_init().
>>>> Since both paths are almost identical between v5.15 and net-next,
>>>> I suspect this problem still exists. Below are kmemleak report and
>>>> some additional printks I inserted.
>>>>
>>>> [ 1364.081694] array_map_free_timers map:0xffffc900005a9000
>>>> [ 1364.081730] ____bpf_timer_init map:0xffffc900005a9000
>>>> timer:0xffff888001ab4080
>>>>
>>>> *no bpf_timer_cancel_and_free that will kfree struct bpf_hrtimer*
>>>> at 0xffff888001ab4080 is called
>>> I think the kmemleak happened as follows:
>>>
>>> bpf_timer_init()
>>> lock timer->lock
>>> read timer->timer as NULL
>>> read map->usercnt != 0
>>>
>>> bpf_map_put_uref()
>>> // map->usercnt = 0
>>> atomic_dec_and_test(map->usercnt)
>>> array_map_free_timers()
>>> // just return and lead to mem leak
>>> find timer->timer is NULL
>>>
>>> t = bpf_map_kmalloc_node()
>>> timer->timer = t
>>> unlock timer->lock
>>>
>>> Could you please try the attached patch to check whether the kmemleak
>>> problem has been fixed ?
>>>
>> Hi,
>>
>> Sorry for the late reply to this thread.
>>
>> KASAN is complaining about double-free/invalid-free in the kfree after
>> applying the patch. There are some cases that jump to "out" before the
>> bpf_hrtimer is allocated or when the bpf_hrtimer is already allocated.
> My bad. Didn't carefully test the patch before posting the patch. Could
> you please apply the modification below to the patch and try it again ?
>
> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
> index bcbd47436a19..c72e28d0ce86 100644
> --- a/kernel/bpf/helpers.c
> +++ b/kernel/bpf/helpers.c
> @@ -1175,6 +1175,7 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern
> *, timer, struct bpf_map *, map
> __bpf_spin_lock_irqsave(&timer->lock);
> t = timer->timer;
> if (t) {
> + t = NULL;
> ret = -EBUSY;
> goto out;
> }
Sorry again. After pressed the send button, I realize the modification
is still not right. The following modification will work.
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index bcbd47436a19..2fd916e0d964 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -1156,7 +1156,7 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern
*, timer, struct bpf_map *, map
u64, flags)
{
clockid_t clockid = flags & (MAX_CLOCKS - 1);
- struct bpf_hrtimer *t;
+ struct bpf_hrtimer *t = NULL;
int ret = 0;
BUILD_BUG_ON(MAX_CLOCKS != 16);
@@ -1173,8 +1173,7 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern
*, timer, struct bpf_map *, map
clockid != CLOCK_BOOTTIME))
return -EINVAL;
__bpf_spin_lock_irqsave(&timer->lock);
- t = timer->timer;
- if (t) {
+ if (timer->timer) {
ret = -EBUSY;
goto out;
}
>
>
>> I am still trying to have a standalone working POC. I think a key to
>> trigger this memory leak is to 1) have a large array map 2) a bpf
>> program init a timer in a small-index entry and then 3) release the
>> map.
> Yes. And I still think my guess about how the kmemleak happens is correct.
>
>> -Amery
>>
>>
>>>> [ 1383.907869] kmemleak: 1 new suspected memory leaks (see
>>>> /sys/kernel/debug/kmemleak)
>>>> BUG: memory leak
>>>> unreferenced object 0xffff888001ab4080 (size 96):
>>>> comm "sshd", pid 279, jiffies 4295233126 (age 29.952s)
>>>> hex dump (first 32 bytes):
>>>> 80 40 ab 01 80 88 ff ff 00 00 00 00 00 00 00 00 .@..............
>>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>> backtrace:
>>>> [<000000009d018da0>] bpf_map_kmalloc_node+0x89/0x1a0
>>>> [<00000000ebcb33fc>] bpf_timer_init+0x177/0x320
>>>> [<00000000fb7e90bf>] 0xffffffffc02a0358
>>>> [<000000000c89ec4f>] __cgroup_bpf_run_filter_skb+0xcbf/0x1110
>>>> [<00000000fd663fc0>] ip_finish_output+0x13d/0x1f0
>>>> [<00000000acb3205c>] ip_output+0x19b/0x310
>>>> [<000000006b584375>] __ip_queue_xmit+0x182e/0x1ed0
>>>> [<00000000b921b07e>] __tcp_transmit_skb+0x2b65/0x37f0
>>>> [<0000000026104b23>] tcp_write_xmit+0xf19/0x6290
>>>> [<000000006dc71bc5>] __tcp_push_pending_frames+0xaf/0x390
>>>> [<00000000251b364a>] tcp_push+0x452/0x6d0
>>>> [<000000008522b7d3>] tcp_sendmsg_locked+0x2567/0x3030
>>>> [<0000000038c644d2>] tcp_sendmsg+0x30/0x50
>>>> [<000000009fe3413f>] inet_sendmsg+0xba/0x140
>>>> [<0000000034d78039>] sock_sendmsg+0x13d/0x190
>>>> [<00000000f55b8db6>] sock_write_iter+0x296/0x3d0
>>>>
>>>>
>>>> Thanks,
>>>> Hsin-Wei (Amery)
>>>>
>>>>
>>>> .
>> .
>
>
> .
Powered by blists - more mailing lists