linux-kernel - Re: test 1: was: Re: A Comparison of printk between upstream and linux-rt-devel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Zsjd0WUNIU8_kprt@debarbos-thinkpadt14sgen2i.remote.csb>
Date: Fri, 23 Aug 2024 15:06:57 -0400
From: Derek Barbosa <debarbos@...hat.com>
To: Petr Mladek <pmladek@...e.com>
Cc: pmaldek@...e.com, williams@...hat.com, john.ogness@...utronix.de,
	tglx@...utronix.de, linux-rt-users@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: Re: test 1: was: Re: A Comparison of printk between upstream and
 linux-rt-devel

Hi,

On Fri, Aug 23, 2024 at 12:19:13PM +0200, Petr Mladek wrote:
> Could you please also share the kernel command line? I can't find it
> anywhere.
> 
> Especially I am interested whether it:
> 
>   + wanted to show backtraces on all CPUs via "panic_print" parameter.

Sure, here's the output of cat /proc/cmdline on the stock kernel

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc3.20240814git6b0f8db921ab.32.eln142.x86_64 
root=/dev/mapper/cs_rt--qe--06-root ro 
crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M 
resume=/dev/mapper/cs_rt--qe--06-swap rd.lvm.lv=cs_rt-qe-06/root rd.lvm.lv=cs_rt-qe-06/swap 
console=ttyS0,115200

>   + did a crashdump or a reboot.

Yes, with kexec-tools and kdump configured, we succesfully booted into the
kdump kernel and rebooted every time as well as generated vmcores.

In fact, examining said vmvcore-dmesg.txt log, we are able to see the final
trace printed. It is simply through the serial console that we are unable to.

I will attach the vmcore-dmesg.txt. I also just did another run of
console_blast, capturing the output from invocation to reboot.

>   + used also another console (graphics).

No, I solely used a serial console at ttyS0 at 115200 baud.

> Do you miss the backtrace from the panic-CPU or non-panic-CPUs or
> both?

I am not 100% certain. I included the two attachments, which may have your
answer.

> The dump of the backtraces on non-panic-CPUs might have been affected
> by the regression fixed earlier this week via
> https://lore.kernel.org/r/20240812072703.339690-1-takakura@valinux.co.jp

Excellent. Is this included in the latest rt-devel tree? I can build that kernel
and run it through the gambit of tests again.

> Did the system reboot in the end?
> Or does it got stuck somewhere?

The system always reboots.

> It is great to see that the serial console driver transformed into
> the new nbcon console is so reliable.

I agree. I have been throwing quite a lot at it (panic, nmis, panic-on-nmis,
multiple nmis + printk spammer) and it seems to be holding up
fairly well :-^)

> Anyway, thanks a lot for the testing and sharing the results.

No worries. Thanks for all you do.

> PS: I still have to think about the other results. But they seem to
>     be less surprising. I am most curious about the so bad behavior
>     of the stock kernel in the first test. I hope that we did not
>     break something in the patch handling the legacy consoles.

If you have any other questions, please let me know. I would be happy to re-do
these tests with different kernels, configs + other variables and controls.

Cheers,

View attachment "console_blast_serial_output.txt" of type "text/plain" (156615 bytes)

View attachment "vmcore-dmesg.txt" of type "text/plain" (1082181 bytes)