Date:   Tue, 24 Jan 2017 14:37:28 +0100
From:   Dmitry Vyukov <dvyukov@...gle.com>
To:     Josh Poimboeuf <jpoimboe@...hat.com>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>,
        "H. Peter Anvin" <hpa@...or.com>,
        "x86@...nel.org" <x86@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        syzkaller <syzkaller@...glegroups.com>
Subject: Re: x86: WARNING: kernel stack frame pointer in kworker has bad value

On Mon, Jan 23, 2017 at 11:50 PM, Josh Poimboeuf <jpoimboe@...hat.com> wrote:
> On Mon, Jan 23, 2017 at 07:43:18PM +0100, Dmitry Vyukov wrote:
>> Hello,
>>
>> I am getting the following warnings while running syzkaller fuzzer on
>> 7a308bb3016f57e5be11a677d15b821536419d36:
>>
>> WARNING: kernel stack frame pointer at ffff880065f7fef8 in
>> kworker/1:3:21075 has bad value ffffffff81598080
>> unwind stack type:0 next_sp:          (null) mask:2 graph_idx:0
>> ...
>>
>> Here are 4 examples:
>>
>> https://gist.githubusercontent.com/dvyukov/7258fca11e5c3db6e8f8fb684e6168ff/raw/2ca14f3c7294c7e466611f8ecfb3072c676a657c/gistfile1.txt
>> https://gist.githubusercontent.com/dvyukov/cc442b144ae712f0e1bdeb065085e36a/raw/f713cef7e9689914fc94838a0a7d4192224fb6d2/gistfile1.txt
>> https://gist.githubusercontent.com/dvyukov/9058cce2647e3d37157c283c6400498c/raw/7a8ec8d71b9e95eeb24156073eeef96ba61d95ea/gistfile1.txt
>> https://gist.githubusercontent.com/dvyukov/b899dfa72d611b511a674b63676921f3/raw/90203ed26b6674dedba8c1c16dd76309ef21fbea/gistfile1.txt
>>
>> Unfortunately this is not reproducible, but there is a stack dump, so
>> maybe it's enough to figure out the root cause.
>
> This seems like some kind of stack corruption.  Here are the stack
> traces:
>
>> https://gist.githubusercontent.com/dvyukov/7258fca11e5c3db6e8f8fb684e6168ff/raw/2ca14f3c7294c7e466611f8ecfb3072c676a657c/gistfile1.txt
>
>   ffff8800520eef68: ffffffff812b148e (__save_stack_trace+0x6e/0xd0)
>   ffff8800520eefe8: ffffffff812b1506 (save_stack_trace+0x16/0x20)
>   ffff8800520eeff8: ffffffff81a0c843 (save_stack+0x43/0xd0)
>   ffff8800520ef228: ffffffff81a0d17f (kasan_slab_free+0x6f/0xb0)
>   ffff8800520ef250: ffffffff81a0a4e1 (kmem_cache_free+0x71/0x240)
>   ffff8800520ef280: ffffffff822924d4 (ioc_release_fn+0x2e4/0x380)
>   ffff8800520ef550: ffffffff814b4900 (process_one_work+0xbd0/0x1c10)
>   ffff8800520ef9a8: ffffffff814b5b63 (worker_thread+0x223/0x1990)
>   ffff8800520eff00: ffffffff814ce526 (kthread+0x326/0x3f0)
>
>> https://gist.githubusercontent.com/dvyukov/cc442b144ae712f0e1bdeb065085e36a/raw/f713cef7e9689914fc94838a0a7d4192224fb6d2/gistfile1.txt
>
>   ffff88005b40ef68: ffffffff812b260e (__save_stack_trace+0x6e/0xd0)
>   ffff88005b40efe8: ffffffff812b2686 (save_stack_trace+0x16/0x20)
>   ffff88005b40eff8: ffffffff81a0e8c3 (save_stack+0x43/0xd0)
>   ffff88005b40f228: ffffffff81a0f1ff (kasan_slab_free+0x6f/0xb0)
>   ffff88005b40f250: ffffffff81a0c561 (kmem_cache_free+0x71/0x240)
>   ffff88005b40f280: ffffffff822943b4 (ioc_release_fn+0x2e4/0x380)
>   ffff88005b40f550: ffffffff814b6980 (process_one_work+0xbd0/0x1c10)
>   ffff88005b40f9a8: ffffffff814b7be3 (worker_thread+0x223/0x1990)
>   ffff88005b40ff00: ffffffff814d05a6 (kthread+0x326/0x3f0)
>
>> https://gist.githubusercontent.com/dvyukov/9058cce2647e3d37157c283c6400498c/raw/7a8ec8d71b9e95eeb24156073eeef96ba61d95ea/gistfile1.txt
>
>   ffff880065f7ef68: ffffffff812b260e (__save_stack_trace+0x6e/0xd0)
>   ffff880065f7efe8: ffffffff812b2686 (save_stack_trace+0x16/0x20)
>   ffff880065f7eff8: ffffffff81a0e8c3 (save_stack+0x43/0xd0)
>   ffff880065f7f228: ffffffff81a0f1ff (kasan_slab_free+0x6f/0xb0)
>   ffff880065f7f250: ffffffff81a0c561 (kmem_cache_free+0x71/0x240)
>   ffff880065f7f280: ffffffff822943b4 (ioc_release_fn+0x2e4/0x380)
>   ffff880065f7f550: ffffffff814b6980 (process_one_work+0xbd0/0x1c10)
>   ffff880065f7f9a8: ffffffff814b7be3 (worker_thread+0x223/0x1990)
>   ffff880065f7ff00: ffffffff814d05a6 (kthread+0x326/0x3f0)
>
>> https://gist.githubusercontent.com/dvyukov/b899dfa72d611b511a674b63676921f3/raw/90203ed26b6674dedba8c1c16dd76309ef21fbea/gistfile1.txt
>
>   ffff880065e57220: ffffffff812b148e (__save_stack_trace+0x6e/0xd0)
>   ffff880065e572a0: ffffffff812b1506 (save_stack_trace+0x16/0x20)
>   ffff880065e572b0: ffffffff81a0c843 (save_stack+0x43/0xd0)
>   ffff880065e574e0: ffffffff81a0cb0a (kasan_kmalloc+0xaa/0xd0)
>   ffff880065e57520: ffffffff81a0d102 (kasan_slab_alloc+0x12/0x20)
>   ffff880065e57530: ffffffff81a08704 (kmem_cache_alloc+0xf4/0x680)
>   ffff880065e57598: ffffffff823293b3 (ida_pre_get+0x1f3/0x3b0)
>   ffff880065e57720: ffffffff823296d8 (ida_simple_get+0x168/0x320)
>   ffff880065e578e8: ffffffff814ad926 (create_worker+0x96/0x620)
>   ffff880065e579a8: ffffffff814b68ad (worker_thread+0xf6d/0x1990)
>   ffff880065e57f00: ffffffff814ce526 (kthread+0x326/0x3f0)
>
> In all four cases, the frame pointer pushed by worker_thread() is
> somehow changed to be the address at the end of
> debug_check_no_locks_freed():
>
>   ffff880065f7fef8: ffffffff81598080 (debug_check_no_locks_freed+0x5c0/0x5c0)
>
> Can you please share the following:
>
> - The output of "grep ffffffff81598080 System.map"
> - The .config file


Hi,

Thanks for looking at this.

I've attached my config.
Unfortunately I don't have that System.map/vmlinux, I've rebased my
kernel and rebuilt it. But here is what after
debug_check_no_locks_freed how:

vmlinux:ffffffff815956a0 0000000000000467 T lockdep_count_backward_deps
vmlinux:ffffffff81595b10 00000000000005b6 T debug_check_no_locks_freed
vmlinux:ffffffff815960d0 0000000000003428 t __lock_acquire


But If you suspect a corruption, then I would suggest to stop spending
more time on this now. At least until I have more data.
This happened while I stress tested KVM, and I've seen a low rate of
what looks like random memory corruptions (e.g. corrupted slab
freelists and other unexplainable things). So let's write it off onto
KVM for now.

Download attachment ".config" of type "application/octet-stream" (116341 bytes)

Powered by blists - more mailing lists