Message-ID: <1598510.AHGpDp18sh@blindfold>
Date:   Tue, 26 Sep 2017 23:51:31 +0200
From:   Richard Weinberger <richard@...ma-star.at>
To:     Alexei Starovoitov <alexei.starovoitov@...il.com>
Cc:     ast@...nel.org, daniel@...earbox.net, netdev@...r.kernel.org,
        linux-kernel@...r.kernel.org, jpoimboe@...hat.com, mingo@...nel.org
Subject: Re: WARNING: kernel stack frame pointer at ffff880156a5fea0 in bash:2103 has bad value 00007ffec7d87e50

Alexei,

CC'ing Josh and Ingo.

Am Dienstag, 26. September 2017, 06:09:02 CEST schrieb Alexei Starovoitov:
> On Mon, Sep 25, 2017 at 11:23:31PM +0200, Richard Weinberger wrote:
> > Hi!
> > 
> > While playing with bcc's opensnoop tool on Linux 4.14-rc2 I managed to
> > trigger this splat:
> > 
> > [  297.629773] WARNING: kernel stack frame pointer at ffff880156a5fea0 in
> > bash:2103 has bad value 00007ffec7d87e50
> > [  297.629777] unwind stack type:0 next_sp:          (null) mask:0x6
> > graph_idx:0
> > [  297.629783] ffff88015b207ae0: ffff88015b207b68 (0xffff88015b207b68)
> > [  297.629790] ffff88015b207ae8: ffffffffb163c00e
> > (__save_stack_trace+0x6e/
> > 0xd0)
> > [  297.629792] ffff88015b207af0: 0000000000000000 ...
> > [  297.629795] ffff88015b207af8: ffff880156a58000 (0xffff880156a58000)
> > [  297.629799] ffff88015b207b00: ffff880156a60000 (0xffff880156a60000)
> > [  297.629800] ffff88015b207b08: 0000000000000000 ...
> > [  297.629803] ffff88015b207b10: 0000000000000006 (0x6)
> > [  297.629806] ffff88015b207b18: ffff880151b02700 (0xffff880151b02700)
> > [  297.629809] ffff88015b207b20: 0000010100000000 (0x10100000000)
> > [  297.629812] ffff88015b207b28: ffff880156a5fea0 (0xffff880156a5fea0)
> > [  297.629815] ffff88015b207b30: ffff88015b207ae0 (0xffff88015b207ae0)
> > [  297.629818] ffff88015b207b38: ffffffffc0050282 (0xffffffffc0050282)
> > [  297.629819] ffff88015b207b40: 0000000000000000 ...
> > [  297.629822] ffff88015b207b48: 0000000001000000 (0x1000000)
> > [  297.629825] ffff88015b207b50: ffff880157b98280 (0xffff880157b98280)
> > [  297.629828] ffff88015b207b58: ffff880157b98380 (0xffff880157b98380)
> > [  297.629831] ffff88015b207b60: ffff88015ad2b500 (0xffff88015ad2b500)
> > [  297.629834] ffff88015b207b68: ffff88015b207b78 (0xffff88015b207b78)
> > [  297.629838] ffff88015b207b70: ffffffffb163c086
> > (save_stack_trace+0x16/0x20) [  297.629841] ffff88015b207b78:
> > ffff88015b207da8 (0xffff88015b207da8) [  297.629847] ffff88015b207b80:
> > ffffffffb18a8ed6 (save_stack+0x46/0xd0) [  297.629850] ffff88015b207b88:
> > 000000400000000c (0x400000000c)
> > [  297.629852] ffff88015b207b90: ffff88015b207ba0 (0xffff88015b207ba0)
> > [  297.629855] ffff88015b207b98: ffff880100000000 (0xffff880100000000)
> > [  297.629859] ffff88015b207ba0: ffffffffb163c086
> > (save_stack_trace+0x16/0x20) [  297.629864] ffff88015b207ba8:
> > ffffffffb18a8ed6 (save_stack+0x46/0xd0) [  297.629868] ffff88015b207bb0:
> > ffffffffb18a9752 (kasan_slab_free+0x72/0xc0)
> Thanks for the report!
> I'm not sure I understand what's going on here.
> It seems you have kasan enabled and it's trying to do save_stack()
> and something crashing?
> I don't see any bpf related helpers in the stack trace.
> Which architecture is this? and .config ?
> Is bpf jit enabled? If so, make sure that net.core.bpf_jit_kallsyms=1

I found some time to dig a little further.
It seems to happen only when CONFIG_DEBUG_SPINLOCK is enabled, please see the 
attached .config. The JIT is off.
KAsan is also not involved at all, the regular stack saving machinery from the 
trace framework initiates the stack unwinder.

The issue arises as soon as in pre_handler_kretprobe() raw_spin_lock_irqsave() 
is being called.
It happens on all releases that have commit c32c47c68a0a ("x86/unwind: Warn on 
bad frame pointer").
Interestingly it does not happen when I run 
samples/kprobes/kretprobe_example.ko. So, BPF must be involved somehow.

Here is another variant of the warning, it matches the attached .config:

[   42.729039] WARNING: kernel stack frame pointer at ffff99ef4076bea0 in 
opensnoop:2008 has bad value 0000000000000008
[   42.729041] unwind stack type:0 next_sp:          (null) mask:0x2 
graph_idx:0
[   42.729042] ffff99ef4076bcb0: ffff99ef4076bd38 (0xffff99ef4076bd38)
[   42.729044] ffff99ef4076bcb8: ffffffffac42781e (__save_stack_trace+0x6e/
0xd0)
[   42.729044] ffff99ef4076bcc0: 0000000000000000 ...
[   42.729045] ffff99ef4076bcc8: ffff99ef40768000 (0xffff99ef40768000)
[   42.729045] ffff99ef4076bcd0: ffff99ef4076c000 (0xffff99ef4076c000)
[   42.729045] ffff99ef4076bcd8: 0000000000000000 ...
[   42.729046] ffff99ef4076bce0: 0000000000000002 (0x2)
[   42.729046] ffff99ef4076bce8: ffff8a1c39163fc0 (0xffff8a1c39163fc0)
[   42.729047] ffff99ef4076bcf0: 0000000100000000 (0x100000000)
[   42.729047] ffff99ef4076bcf8: ffff99ef4076bea0 (0xffff99ef4076bea0)
[   42.729048] ffff99ef4076bd00: ffff99ef4076bcb0 (0xffff99ef4076bcb0)
[   42.729048] ffff99ef4076bd08: ffffffffc00b302f (0xffffffffc00b302f)
[   42.729048] ffff99ef4076bd10: 0000000000000000 ...
[   42.729049] ffff99ef4076bd18: ffff8a1c39163fc0 (0xffff8a1c39163fc0)
[   42.729049] ffff99ef4076bd20: 0000000000000000 ...
[   42.729052] ffff99ef4076bd28: ffffffffadb9ccc0 (lock_classes
+0x55500/0x29fec0)
[   42.729052] ffff99ef4076bd30: 0000000000000000 ...
[   42.729052] ffff99ef4076bd38: ffff99ef4076bd48 (0xffff99ef4076bd48)
[   42.729053] ffff99ef4076bd40: ffffffffac427896 (save_stack_trace+0x16/0x20)
[   42.729054] ffff99ef4076bd48: ffff99ef4076bd98 (0xffff99ef4076bd98)
[   42.729055] ffff99ef4076bd50: ffffffffac4a18d5 (__lock_acquire.isra.
34+0x525/0x700)
[   42.729055] ffff99ef4076bd58: 0000000000000000 ...
[   42.729055] ffff99ef4076bd68: ffff99ef00000411 (0xffff99ef00000411)
[   42.729056] ffff99ef4076bd70: 0000000000000046 (0x46)
[   42.729056] ffff99ef4076bd78: 0000000000000000 ...
[   42.729057] ffff99ef4076bd98: ffff99ef4076be00 (0xffff99ef4076be00)
[   42.729057] ffff99ef4076bda0: ffffffffac4a224a (lock_acquire+0xca/0x170)
[   42.729059] ffff99ef4076bda8: ffffffffac50a2cd (pre_handler_kretprobe+0x3d/
0x1b0)
[   42.729059] ffff99ef4076bdb0: 0000000100000000 (0x100000000)
[   42.729060] ffff99ef4076bdb8: ffff8a1c00000000 (0xffff8a1c00000000)
[   42.729063] ffff99ef4076bdc0: 0000000000000046 (0x46)
[   42.729063] ffff99ef4076bdc8: 00000001ac47ee61 (0x1ac47ee61)
[   42.729064] ffff99ef4076bdd0: ffff8a1c37b0e0d0 (0xffff8a1c37b0e0d0)
[   42.729064] ffff99ef4076bdd8: ffff8a1c37b0e0b8 (0xffff8a1c37b0e0b8)
[   42.729067] ffff99ef4076bde0: 0000000000000082 (0x82)
[   42.729067] ffff99ef4076bde8: ffff8a1c37b0e0b8 (0xffff8a1c37b0e0b8)
[   42.729067] ffff99ef4076bdf0: ffff99ef4076beb0 (0xffff99ef4076beb0)
[   42.729068] ffff99ef4076bdf8: ffff8a1c39163fc0 (0xffff8a1c39163fc0)
[   42.729068] ffff99ef4076be00: ffff99ef4076be28 (0xffff99ef4076be28)
[   42.729070] ffff99ef4076be08: fffffffface13e56 (_raw_spin_lock_irqsave
+0x46/0x60)
[   42.729071] ffff99ef4076be10: ffffffffac50a2cd (pre_handler_kretprobe+0x3d/
0x1b0)
[   42.729072] ffff99ef4076be18: ffff8a1c37b0e010 (0xffff8a1c37b0e010)
[   42.729072] ffff99ef4076be20: ffff8a1c37b0e010 (0xffff8a1c37b0e010)
[   42.729073] ffff99ef4076be28: ffff99ef4076be60 (0xffff99ef4076be60)
[   42.729074] ffff99ef4076be30: ffffffffac50a2cd (pre_handler_kretprobe+0x3d/
0x1b0)
[   42.729074] ffff99ef4076be38: ffff8a1c37b0e010 (0xffff8a1c37b0e010)
[   42.729074] ffff99ef4076be40: ffff8a1c38cc1780 (0xffff8a1c38cc1780)
[   42.729075] ffff99ef4076be48: ffff99ef4076beb0 (0xffff99ef4076beb0)
[   42.729075] ffff99ef4076be50: 000055b4ef12d1b0 (0x55b4ef12d1b0)
[   42.729076] ffff99ef4076be58: 000055b4ee9920a0 (0x55b4ee9920a0)
[   42.729076] ffff99ef4076be60: ffff99ef4076be88 (0xffff99ef4076be88)
[   42.729077] ffff99ef4076be68: ffffffffac509f6a (opt_pre_handler+0x3a/0x60)
[   42.729078] ffff99ef4076be70: 0000000000000246 (0x246)
[   42.729078] ffff99ef4076be78: 000055b4ef12cd70 (0x55b4ef12cd70)
[   42.729079] ffff99ef4076be80: 0000000000000001 (0x1)
[   42.729079] ffff99ef4076be88: ffff99ef4076bea0 (0xffff99ef4076bea0)
[   42.729080] ffff99ef4076be90: ffffffffac442721 (optimized_callback
+0x81/0x90)
[   42.729081] ffff99ef4076be98: 000055b4ef134d50 (0x55b4ef134d50)
[   42.729081] ffff99ef4076bea0: 0000000000000008 (0x8)
[   42.729082] ffff99ef4076bea8: ffffffffc00b302f (0xffffffffc00b302f)
[   42.729082] ffff99ef4076beb0: 000055b4ee9920a0 (0x55b4ee9920a0)
[   42.729083] ffff99ef4076beb8: 000055b4ef12d1b0 (0x55b4ef12d1b0)
[   42.729083] ffff99ef4076bec0: 0000000000000001 (0x1)
[   42.729084] ffff99ef4076bec8: 000055b4ef12cd70 (0x55b4ef12cd70)
[   42.729084] ffff99ef4076bed0: 0000000000000008 (0x8)
[   42.729084] ffff99ef4076bed8: 000055b4ef134d50 (0x55b4ef134d50)
[   42.729085] ffff99ef4076bee0: ffff8a1c39163fc0 (0xffff8a1c39163fc0)
[   42.729085] ffff99ef4076bee8: 0000000000000000 ...
[   42.729086] ffff99ef4076bef0: 0000000000000001 (0x1)
[   42.729086] ffff99ef4076bef8: 0000000000000008 (0x8)
[   42.729086] ffff99ef4076bf00: 0000000000000002 (0x2)
[   42.729087] ffff99ef4076bf08: 0000000000000000 ...
[   42.729087] ffff99ef4076bf10: 00000000000001b6 (0x1b6)
[   42.729087] ffff99ef4076bf18: 0000000000000000 ...
[   42.729088] ffff99ef4076bf20: 000055b4ef12d1b0 (0x55b4ef12d1b0)
[   42.729088] ffff99ef4076bf28: ffffffffffffffff (0xffffffffffffffff)
[   42.729090] ffff99ef4076bf30: ffffffffac5c5031 (SyS_open+0x1/0x20)
[   42.729090] ffff99ef4076bf38: 0000000000000010 (0x10)
[   42.729090] ffff99ef4076bf40: 0000000000000293 (0x293)
[   42.729091] ffff99ef4076bf48: ffff99ef4076bf50 (0xffff99ef4076bf50)
[   42.729092] ffff99ef4076bf50: fffffffface13f77 (entry_SYSCALL_64_fastpath
+0x1a/0xaa)
[   42.729092] ffff99ef4076bf58: 0000000000000026 (0x26)
[   42.729093] ffff99ef4076bf60: 00007f276f5e2600 (0x7f276f5e2600)
[   42.729093] ffff99ef4076bf68: 0000000000000001 (0x1)
[   42.729094] ffff99ef4076bf70: 0000000000000026 (0x26)
[   42.729094] ffff99ef4076bf78: 000055b4ef1035d0 (0x55b4ef1035d0)
[   42.729094] ffff99ef4076bf80: 0000000000000026 (0x26)
[   42.729095] ffff99ef4076bf88: 0000000000000246 (0x246)
[   42.729095] ffff99ef4076bf90: 0000000000000000 ...
[   42.729095] ffff99ef4076bf98: 0000000000000001 (0x1)
[   42.729096] ffff99ef4076bfa0: 0000000000000008 (0x8)
[   42.729096] ffff99ef4076bfa8: ffffffffffffffda (0xffffffffffffffda)
[   42.729097] ffff99ef4076bfb0: 00007f276f3234e0 (0x7f276f3234e0)
[   42.729097] ffff99ef4076bfb8: 00000000000001b6 (0x1b6)
[   42.729097] ffff99ef4076bfc0: 0000000000000000 ...
[   42.729098] ffff99ef4076bfc8: 000055b4ef12d1b0 (0x55b4ef12d1b0)
[   42.729098] ffff99ef4076bfd0: 0000000000000002 (0x2)
[   42.729099] ffff99ef4076bfd8: 00007f276f3234e0 (0x7f276f3234e0)
[   42.729099] ffff99ef4076bfe0: 0000000000000033 (0x33)
[   42.729100] ffff99ef4076bfe8: 0000000000000246 (0x246)
[   42.729100] ffff99ef4076bff0: 00007ffd98082448 (0x7ffd98082448)
[   42.729100] ffff99ef4076bff8: 000000000000002b (0x2b)

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y
View attachment "bpf_splat.config" of type "text/x-mpsub" (118048 bytes)

Powered by blists - more mailing lists