linux-kernel - Re: [per_cpu_ptr_to_phys] PANIC: early exception 0x0d IP 10:ffffffffa892f15f error 0 cr2 0xffff88001fbff000

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Thu, 03 May 2018 13:45:43 +0000
From:   Pavel Tatashin <pasha.tatashin@...cle.com>
To:     Fengguang Wu <fengguang.wu@...el.com>
Cc:     Dennis Zhou <dennisszhou@...il.com>,
        Linux Memory Management List <linux-mm@...ck.org>,
        Tejun Heo <tj@...nel.org>, Christoph Lameter <cl@...ux.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Josef Bacik <jbacik@...com>,
        LKML <linux-kernel@...r.kernel.org>, LKP <lkp@...org>
Subject: Re: [per_cpu_ptr_to_phys] PANIC: early exception 0x0d IP
 10:ffffffffa892f15f error 0 cr2 0xffff88001fbff000

Hi Fengguang,

My root cause for the problem was correct. You are now finding either a
different problem that has the same signature, or what is more likely the
same issue but simply was not introduced by my change: my change reduced
number of pre-initialized pages because we init them on demand with my
work, but we could run out of them even before my change, and because of
KASLR we never know how much is needed to be pre-initialized.

Could you please test if my patch fixes the issue?

http://ozlabs.org/~akpm/mmots/broken-out/mm-access-to-uninitialized-struct-page.patch

Thank you,
Pavel
On Wed, May 2, 2018 at 8:44 AM Fengguang Wu <fengguang.wu@...el.com> wrote:

> Hi all,

> On Wed, Apr 18, 2018 at 06:38:25PM -0500, Dennis Zhou wrote:
> >Hi,
> >
> >On Wed, Apr 18, 2018 at 09:55:53PM +0800, Fengguang Wu wrote:
> >>
> >> Hello,
> >>
> >> FYI here is a slightly different boot error in mainline kernel
4.17.0-rc1.
> >> It also dates back to v4.16 .

> Now I find 2 more occurrances in v4.15 kernel.

> Here are the statistics:

>          kernel      count     error-id
>          v4.15:          2     RIP:per_cpu_ptr_to_phys
>          v4.16:         12     RIP:per_cpu_ptr_to_phys
>          v4.16:          1
BUG:KASAN:null-ptr-deref-in-per_cpu_ptr_to_phys
>          v4.16-rc7:      2     RIP:per_cpu_ptr_to_phys
>          v4.17-rc1:    217     RIP:per_cpu_ptr_to_phys
>          v4.17-rc1:      5
BUG:KASAN:null-ptr-deref-in-per_cpu_ptr_to_phys
>          v4.17-rc2:     46     RIP:per_cpu_ptr_to_phys
>          v4.17-rc2:     15
BUG:KASAN:null-ptr-deref-in-per_cpu_ptr_to_phys
>          v4.17-rc3:     12     RIP:per_cpu_ptr_to_phys

> >> It occurs in 4 out of 4 boots.
> >>
> >> [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages:
128873
> >> [    0.000000] Kernel command line: root=/dev/ram0 hung_task_panic=1
debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1
nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0
earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw
link=/kbuild-tests/run-queue/kvm/x86_64-randconfig-a0-04172313/linux-devel:devel-hourly-2018041714:60cc43fc888428bb2f18f08997432d426a243338/.vmlinuz-60cc43fc888428bb2f18f08997432d426a243338-20180418000325-19:yocto-lkp-nhm-dp2-4
branch=linux-devel/devel-hourly-2018041714
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a0-04172313/gcc-7/60cc43fc888428bb2f18f08997432d426a243338/vmlinuz-4.17.0-rc1
drbd.minor_count=8 rcuperf.shutdown=0
> >> [    0.000000] sysrq: sysrq always enabled.
> >> [    0.000000] Dentry cache hash table entries: 65536 (order: 7,
524288 bytes)
> >> [    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144
bytes)
> >> PANIC: early exception 0x0d IP 10:ffffffffa892f15f error 0 cr2
0xffff88001fbff000
> >> [    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G                T
4.17.0-rc1 #238
> >> [    0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.10.2-1 04/01/2014
> >> [    0.000000] RIP: 0010:per_cpu_ptr_to_phys+0x16a/0x298:
> >>                                              __section_mem_map_addr at
include/linux/mmzone.h:1188
> >>                                               (inlined by)
per_cpu_ptr_to_phys at mm/percpu.c:1849
> >> [    0.000000] RSP: 0000:ffffffffab407e50 EFLAGS: 00010046 ORIG_RAX:
0000000000000000
> >> [    0.000000] RAX: dffffc0000000000 RBX: ffff88001f17c340 RCX:
000000000000000f
> >> [    0.000000] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffffffffacfbf580
> >> [    0.000000] RBP: ffffffffab40d000 R08: fffffbfff57c4eca R09:
0000000000000000
> >> [    0.000000] R10: ffff880015421000 R11: fffffbfff57c4ec9 R12:
0000000000000000
> >> [    0.000000] R13: ffff88001fb03ff8 R14: ffff88001fc051c0 R15:
0000000000000000
> >> [    0.000000] FS:  0000000000000000(0000) GS:ffffffffab4c5000(0000)
knlGS:0000000000000000
> >> [    0.000000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> [    0.000000] CR2: ffff88001fbff000 CR3: 000000001a06c000 CR4:
00000000000006b0
> >> [    0.000000] Call Trace:
> >> [    0.000000]  setup_cpu_entry_areas+0x7b/0x27b:
> >>                                              setup_cpu_entry_area at
arch/x86/mm/cpu_entry_area.c:104
> >>                                               (inlined by)
setup_cpu_entry_areas at arch/x86/mm/cpu_entry_area.c:177
> >> [    0.000000]  trap_init+0xb/0x13d:
> >>                                              trap_init at
arch/x86/kernel/traps.c:949
> >> [    0.000000]  start_kernel+0x2a5/0x91d:
> >>                                              mm_init at init/main.c:519
> >>                                               (inlined by)
start_kernel at init/main.c:589
> >> [    0.000000]  ? thread_stack_cache_init+0x6/0x6
> >> [    0.000000]  ? memcpy_orig+0x16/0x110:
> >>                                              memcpy_orig at
arch/x86/lib/memcpy_64.S:77
> >> [    0.000000]  ? x86_family+0x5/0x1d:
> >>                                              x86_family at
arch/x86/lib/cpu.c:8
> >> [    0.000000]  ? load_ucode_bsp+0x42/0x13e:
> >>                                              load_ucode_bsp at
arch/x86/kernel/cpu/microcode/core.c:183
> >> [    0.000000]  secondary_startup_64+0xa5/0xb0:
> >>                                              secondary_startup_64 at
arch/x86/kernel/head_64.S:242
> >> [    0.000000] Code: 78 06 00 49 8b 45 00 48 85 c0 74 a5 49 c1 ec 28
41 81 e4 e0 0f 00 00 49 01 c4 4c 89 e2 48 b8 00 00 00 00 00 fc ff df 48 c1
ea 03 <80> 3c 02 00 74 08 4c 89 e7 e8 63 78 06 00 49 8b 04 24 81 e5 ff
> >> BUG: kernel hang in boot stage
> >>
> >
> >I spent some time bisecting this one and it seemse to be an intermittent
> >issue starting with this commit for me:
> >c9e97a1997, mm: initialize pages on demand during boot. The prior
> >commit, 3a2d7fa8a3, did not run into this issue after 10+ boots.

> That commit is post-4.16, so probably not the root cause.

> >I don't have that much time right now, nor the expertise with this code.
> >Pavel could you take a look at this?

> Thanks,
> Fengguang