linux-kernel - Re: commit 444d13ff10f introduced boot failure on s390x

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160815191253.GA10262@packer-debian-8-amd64.digitalocean.com>
Date:	Mon, 15 Aug 2016 15:12:53 -0400
From:	Jessica Yu <jeyu@...hat.com>
To:	Eryu Guan <eguan@...hat.com>
Cc:	live-patching@...r.kernel.org, linux-kernel@...r.kernel.org,
	Rusty Russell <rusty@...tcorp.com.au>,
	Heiko Carstens <heiko.carstens@...ibm.com>,
	Martin Schwidefsky <schwidefsky@...ibm.com>
Subject: Re: commit 444d13ff10f introduced boot failure on s390x

+++ Jessica Yu [10/08/16 18:58 -0400]:
>+++ Eryu Guan [10/08/16 23:21 +0800]:
>>Hi,
>>
>>I hit boot failure on s390x host starting from 4.8-rc1 kernel, 4.7
>>kernel works fine. And I bisected to this commit 444d13ff10fb
>>
>>   commit 444d13ff10fb13bc3e64859c3cf9ce43dcfeb075
>>   Author: Jessica Yu <jeyu@...hat.com>
>>   Date:   Wed Jul 27 12:06:21 2016 +0930
>>
>>       modules: add ro_after_init support
>>
>>       Add ro_after_init support for modules by adding a new page-aligned section
>>       in the module layout (after rodata) for ro_after_init data and enabling RO
>>       protection for that section after module init runs.
>>
>>       Signed-off-by: Jessica Yu <jeyu@...hat.com>
>>       Acked-by: Kees Cook <keescook@...omium.org>
>>       Signed-off-by: Rusty Russell <rusty@...tcorp.com.au>
>>
>>and I've only hit this panic on s390x hosts. Console log is appended at
>>the end of email.
>>
>>Thanks,
>>Eryu
>
>Hi Eryu, thanks for reporting this. It's a bit difficult to tell from
>the stacktrace alone what's really going on, so I'll attempt to
>reproduce this on a 4.8-rc1 kernel once I get my hands on an s390x
>system and report back.

[ CC'ing Heiko and Martin ]

So this panic is related to some recent changes to set_memory_{ro,rw}
on s390x, see commit e8a97e42 "s390/pageattr: allow kernel page table
splitting." The new implementation of set_memory_{ro,rw} on s390 isn't
handling the case when numpages is 0.

Recall the general layout of a module:
    [text] [rodata] [ro-after-init] [writable data]

Normally a module's ro after init section sits between rodata and
writable data. When a module doesn't have a ro after init section,
set_memory_ro gets called with the first page-aligned addr after
rodata, but with numpages = 0. However in this case since
set_memory_ro isn't handling the case when numpages is 0, it
incorrectly ends up walking the page table anyway and ends up setting
a normally writable page to ro. Adding a simple numpages == 0 check
to set_memory_{ro,rw} and returning fixes the panic.

Jessica

>>[    2.050197] device-mapper: uevent: version 1.0.3
>>[    2.050370] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-d
>>evel@...hat.com
>>[    2.057615] Unable to handle kernel pointer dereference in virtual kernel add
>>ress space
>>[    2.057619] Failing address: 000003ff8001d000 TEID: 000003ff8001d407
>>[    2.057620] Fault in home space mode while using kernel ASCE.
>>[    2.057622] AS:0000000000a7c007 R3:000000007c974007 S:000000007cc24800 P:0000
>>00000239b21d
>>[    2.057665] Oops: 0004 ilc:3 [#1] SMP
>>[    2.057667] Modules linked in: dm_mod
>>[    2.057670] CPU: 0 PID: 399 Comm: modprobe Not tainted 4.7.0+ #7
>>[    2.057672] Hardware name: IBM              2827 H43              400
>>     (z/VM)
>>[    2.057673] task: 000000007cccd100 ti: 0000000002324000 task.ti: 000000000232
>>4000
>>[    2.057675] Krnl PSW : 0704c00180000000 000000000043a5c8 (__list_add_rcu+0x50
>>/0xa8)
>>[    2.057683]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:
>>0 EA:3
>>Krnl GPRS: 0000000000000006 000000000098b278 000003ff80117208 000000000098b278
>>[    2.057685]            000003ff8001df08 00000000001c913c 0000000000000002 000
>>000008001e55c
>>[    2.057686]            0000000000000560 0000000002327e00 000003ff80117218 000
>>003ff80117208
>>[    2.057687]            000000000098b278 000003ff8001df08 0000000002327cc0 000
>>0000002327c90
>>[    2.057696] Krnl Code: 000000000043a5b6: e3d0b0000024        stg     %r13,0(%
>>r11)
>>          000000000043a5bc: e3c0b0080024       stg     %r12,8(%r11)
>>         #000000000043a5c2: e3b0c0000024       stg     %r11,0(%r12)
>>         >000000000043a5c8: e3b0d0080024       stg     %r11,8(%r13)
>>          000000000043a5ce: e340f0b80004       lg      %r4,184(%r15)
>>          000000000043a5d4: ebbff0a00004       lmg     %r11,%r15,160(%r15)
>>          000000000043a5da: 07f4               bcr     15,%r4
>>          000000000043a5dc: e34040080004       lg      %r4,8(%r4)
>>[    2.057706] Call Trace:
>>[    2.057708] ([<0000000002327cc0>] 0x2327cc0)
>>[    2.057714] ([<00000000001c98d0>] load_module+0x8e0/0x1870)
>>[    2.057715] ([<00000000001caa74>] SyS_finit_module+0xb4/0xf0)
>>[    2.057720] ([<00000000006678b6>] system_call+0xd6/0x264)
>>[    2.057721] Last Breaking-Event-Address:
>>[    2.057722]  [<00000000001c98ca>] load_module+0x8da/0x1870
>>[    2.057723]
>>[    2.057724] Kernel panic - not syncing: Fatal exception: panic_on_oops