linux-kernel - Re: Panic on ppc64le using kernel 5.13.0-rc3

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+QYu4oFGkP1G+9TqGtqffgOEBycSMKtFbV-1X+kL4NeTyEegg@mail.gmail.com>
Date:   Thu, 10 Jun 2021 17:14:24 +0200
From:   Bruno Goncalves <bgoncalv@...hat.com>
To:     Rasmus Villemoes <linux@...musvillemoes.dk>
Cc:     linux-kernel@...r.kernel.org, CKI Project <cki-project@...hat.com>
Subject: Re: Panic on ppc64le using kernel 5.13.0-rc3

On Thu, Jun 10, 2021 at 3:02 PM Rasmus Villemoes
<linux@...musvillemoes.dk> wrote:
>
> On 10/06/2021 13.47, Bruno Goncalves wrote:
> > Hello,
> >
> > We've observed in some cases kernel panic when trying to boot on
> > ppc64le using a kernel based on 5.13.0-rc3. We are not sure if it
> > could be related to patch
> > https://lore.kernel.org/lkml/20210313212528.2956377-2-linux@rasmusvillemoes.dk/
> >
>
> Thanks for the report. It's possible, but I'll need some help from you
> to get more info.
>
> First, can you send me the .config?

The .config is on
https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2021/06/09/317881801/build_ppc64le_redhat:1332368174/kernel-block-ppc64le-d3f02e52f5548006f04358d407bbb7fe51255c41.config

>
> >
> > [    1.516075] wait_for_initramfs() called before rootfs_initcalls
>
> This is likely because you have CONFIG_UEVENT_HELPER_PATH set to some
> non-empty path (/sbin/hotplug perhaps). This did get reported once before:
>

CONFIG_UEVENT_HELPER_PATH is not set. In the .config we have "#
CONFIG_UEVENT_HELPER is not set"


> https://lore.kernel.org/lkml/45556f52-cd2f-5512-ba65-81e4acee21ff@rasmusvillemoes.dk/
>
> I think I should go and prepare a patch that moves the
> usermodehelper_enable() call to after initramfs unpacking has been
> initiated.
>
> But until then, can you check if you do have UEVENT_HELPER_PATH set, and
> if so, does changing it to the empty string make a change wrt this crash?
>
>
> > [    1.559757] PCI: CLS 128 bytes, default 128
> > [    1.560150] Trying to unpack rootfs image as initramfs...
>
> OK, so now we got to populate_rootfs() and have kicked off a worker to
> do the unpacking. Meanwhile, PID1 goes on to do other initcalls.
>
> ...
>
> > [    1.764430] Initramfs unpacking failed: no cpio magic
>
> Whoa, that's not good. Did something scramble over the initramfs memory
> while it was being unpacked? It's been .2 seconds since the start of the
> unpacking, so it's unlikely the very beginning of the initramfs is corrupt.
>
> Can you try booting with initramfs_async=0 on the command line and see
> if the kernel still crashes?

We are not able to reproduce it 100% of the time, but sure I can try
with this option and see what happens.

We've also seen:
Initramfs unpacking failed: junk within compressed archive

This can be seen on the other 2 console logs that I provided the link to.

Bruno
>
> > [    1.766204] Freeing initrd memory: 18176K
> ...
>
>
> > [    1.787649] Run /init as init process
> > [    1.787793] Failed to execute /init (error -2)
> > [    1.787801] Run /sbin/init as init process
> > [    1.787842] Run /etc/init as init process
> > [    1.787880] Run /bin/init as init process
> > [    1.787921] Run /bin/sh as init process
> > [    1.787978] Kernel panic - not syncing: No working init found.  Try
>
> Yeah, well, this is expected when unpacking the initramfs failed.
>
> So I think the problem is the "no cpio magic", i.e. the initramfs got
> corrupted somehow. But I don't have any idea why that would happen -
> freeing the initramfs memory only happens after unpacking is done
> (naturally...).
>
> Rasmus
>