linux-kernel - Re: Warnings/memory corruption in perf intel events

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:	Fri, 22 Jan 2016 09:53:15 -0800
From:	Stephane Eranian <eranian@...gle.com>
To:	Sasha Levin <sasha.levin@...cle.com>
Cc:	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>,
	Arnaldo Carvalho de Melo <acme@...nel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	"H. Peter Anvin" <hpa@...or.com>,
	LKML <linux-kernel@...r.kernel.org>,
	"x86@...nel.org" <x86@...nel.org>
Subject: Re: Warnings/memory corruption in perf intel events

On Fri, Jan 22, 2016 at 4:36 AM, Sasha Levin <sasha.levin@...cle.com> wrote:
> On 08/10/2015 07:25 AM, Peter Zijlstra wrote:
>> On Sun, Aug 09, 2015 at 03:46:48PM -0400, Sasha Levin wrote:
>>> > Hi all,
>>> >
>>> > While fuzzing with trinity inside a KVM tools guest running -next I've stumbled on the following:
>>
>>> > [424256.911563] ==================================================================
>>> > [424256.913989] BUG: KASan: use after free in intel_get_event_constraints+0xdb0/0xf90 at addr ffff8801741a70e9
>>> > [424256.917044] Read of size 1 by task trinity-c162/20397
>>> > [424256.918696] =============================================================================
>>> > [424256.921309] BUG kmalloc-2048 (Tainted: G        W      ): kasan: bad access detected
>>
>> Quite the puzzle that, and I'm not entirely sure I see how. The WARNs
>> preceding this aren't giving me much confidence either, I've yet to find
>> a way for them to happen.
>>
>> That said, the only dynamically allocated memory here is managed on CPU
>> hotplug. And we appear to consistently return NOTIFY_BAD if an
>> allocation there fails, which should preclude the CPU hotplug from
>> completing and therefore preclude events from forming on that CPU.
>>
>> We do however appear to fail to NULL all our pointers, and even miss one
>> kfree() in the error paths there (the constraint_list allocation fail,
>> fails to free the shared_regs one).
>>
>> Rework that code such that we kfree() all and always NULL our pointers
>> after free.
>
> I suspect that that patch didn't help, I'm seeing traces like this:
>
> [ 1362.573349] WARNING: CPU: 3 PID: 13908 at arch/x86/kernel/cpu/perf_event.c:1345 x86_pmu_del+0x3e1/0x7f0()
> [ 1362.574566] Modules linked in:

But that's a different problem here. It is not about the constraints
anymore but rather related to the events.
Here we are trying to delete an event which is not there anymore.

> [ 1362.575017] CPU: 3 PID: 13908 Comm: syz-executor Not tainted 4.4.0-next-20160121-sasha-00020-g5e5e971-dirty #2811
> [ 1362.576318]  1ffff10038db1e65 00000000a4877b67 ffff8801c6d8f3a8 ffffffff8344c8c1
> [ 1362.577342]  0000000041b58ab3 ffffffff8f9763f5 ffffffff8344c7f6 ffff8800c33e0bc8
> [ 1362.578371]  ffff8800c33e0bd0 00000000a4877b67 00000000a4877b67 0000000000000003
> [ 1362.579217] Call Trace:
> [ 1362.579536]  [<ffffffff8344c8c1>] dump_stack+0xcb/0x14a
> [ 1362.580093]  [<ffffffff8344c7f6>] ? _atomic_dec_and_lock+0x106/0x106
> [ 1362.580769]  [<ffffffff813bbda1>] warn_slowpath_common+0xe1/0x160
> [ 1362.581410]  [<ffffffff811ce121>] ? x86_pmu_del+0x3e1/0x7f0
> [ 1362.599438]  [<ffffffff813bc049>] warn_slowpath_null+0x29/0x30
> [ 1362.600063]  [<ffffffff811ce121>] x86_pmu_del+0x3e1/0x7f0
> [ 1362.600657]  [<ffffffff816bacfb>] event_sched_out+0x5ab/0x1480
> [ 1362.601275]  [<ffffffff816ba750>] ? perf_pmu_enable+0x1c0/0x1c0
> [ 1362.602016]  [<ffffffff816bc797>] ? __perf_remove_from_context+0x137/0x380
> [ 1362.602854]  [<ffffffff816bc7a5>] __perf_remove_from_context+0x145/0x380
> [ 1362.603675]  [<ffffffff816bc660>] ? __perf_event_disable+0x8a0/0x8a0
> [ 1362.604463]  [<ffffffff816a30c0>] ? free_ctx+0x70/0x70
> [ 1362.605104]  [<ffffffff816a320b>] remote_function+0x14b/0x200
> [ 1362.605848]  [<ffffffff815ab368>] generic_exec_single+0x308/0x5d0
> [ 1362.606632]  [<ffffffff815ab060>] ? cpumask_next+0xd0/0xd0
> [ 1362.607359]  [<ffffffff834d0329>] ? check_preemption_disabled+0x39/0x270
> [ 1362.608147]  [<ffffffff816bc660>] ? __perf_event_disable+0x8a0/0x8a0
> [ 1362.608816]  [<ffffffff815ab992>] smp_call_function_single+0x122/0x4c0
> [ 1362.624201]  [<ffffffff816a1086>] task_function_call+0x156/0x1a0
> [ 1362.627453]  [<ffffffff816aa367>] event_function_call+0x1a7/0x310
> [ 1362.630435]  [<ffffffff816aa89c>] perf_remove_from_context+0x10c/0x1b0
> [ 1362.633337]  [<ffffffff816ca938>] put_event+0x318/0x510
> [ 1362.637005]  [<ffffffff816cab8c>] perf_release+0x3c/0x60
> [ 1362.649756]  [<ffffffff818e7ee5>] __fput+0x255/0x870
> [ 1362.650347]  [<ffffffff818e8595>] ____fput+0x15/0x20
> [ 1362.651060]  [<ffffffff8142ca66>] task_work_run+0x136/0x240
> [ 1362.651716]  [<ffffffff813c7ffb>] do_exit+0x86b/0x1850
> [ 1362.652369]  [<ffffffff813c7790>] ? mm_update_next_owner+0x6f0/0x6f0
> [ 1362.653167]  [<ffffffff813e5edf>] ? __dequeue_signal+0x32f/0x730
> [ 1362.653924]  [<ffffffff813c913c>] do_group_exit+0xec/0x390
> [ 1362.654609]  [<ffffffff813f2f6a>] get_signal+0x5aa/0x1cb0
> [ 1362.655276]  [<ffffffff8117aaed>] do_signal+0x8d/0x20d0
> [ 1362.660705]  [<ffffffff8100557b>] exit_to_usermode_loop+0x1bb/0x270
> [ 1362.661510]  [<ffffffff81006ded>] syscall_return_slowpath+0x3fd/0x590
> [ 1362.662358]  [<ffffffff8d058aa2>] int_ret_from_sys_call+0x25/0x9f
>
>
> Thanks,
> Sasha