linux-kernel - Re: [RFC PATCH] kdump: Add support for crashkernel=auto

Open Source and information security mailing list archives

Message-ID: <20220131113351.49a47caf@rhtmp>
Date:   Mon, 31 Jan 2022 11:33:51 +0100
From:   Philipp Rudo <prudo@...hat.com>
To:     Petr Tesařík <ptesarik@...e.cz>
Cc:     Tiezhu Yang <yangtiezhu@...ngson.cn>, Baoquan He <bhe@...hat.com>,
        Vivek Goyal <vgoyal@...hat.com>,
        Dave Young <dyoung@...hat.com>,
        Jonathan Corbet <corbet@....net>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Xuefeng Li <lixuefeng@...ngson.cn>, kexec@...ts.infradead.org,
        linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH] kdump: Add support for crashkernel=auto

Hi,

On Fri, 28 Jan 2022 11:31:49 +0100
Petr Tesařík <ptesarik@...e.cz> wrote:

> Hi Tiezhu Yang,
> 
> On Jan 28, 2022 at 02:20 Tiezhu Yang wrote:
> >[...]
> > Hi Petr,
> > 
> > Thank you for your reply.
> > 
> > This is a RFC patch, the initial aim of this patch is to discuss what is 
> > the proper way to support crashkernel=auto.  
> 
> Well, the point I'm trying to make is that crashkernel=auto cannot be 
> implemented. Your code would have to know what happens in the future, 
> and AFAIK time travel has not been discovered yet. ;-)
>
> A better approach is to make a very large allocation initially, e.g. 
> half of available RAM. The remaining RAM should still be big enough to 
> start booting the system. Later, when a kdump user-space service knows 
> what it wants to load, it can shrink the reservation by writing a lower 
> value into /sys/kernel/kexec_crash_size.

Even this approach doesn't work in every situation. For example it
requires that the system has at least twice the RAM it requires to
safely boot. That's not always given for e.g minimalistic VMs or
embedded systems.
Furthermore the memory requirement can also change during runtime due
to, e.g. workload spikes, device hot plug, moving the dump target from
an un-encrypted to an encrypted disk, etc.. So even when your user-space
program can exactly calculate the memory requirement at the moment
it loads kdump it might be too little at the moment the system panics.
In order for it to work the user-space would constantly need to monitor
how much memory is needed and adjust the requirement. But that would
also require to increase the reservation during runtime which would be
extremely expensive (if possible at all).

All in all I support Petr that time travel is the only proper solution
for implementing crashkernel=auto. But once we have time travel I
would prefer to use the gained knowledge to fix the bug that triggered
the panic rather than calculating the memory requirement for kdump.

> The alternative approach does not need any changes to the kernel, except 
> maybe adding something like "crashkernel=max".

A slightly different approach is for the user-space tool to simply set
the crashkernel= parameter on the kernel commandline for the next boot.
This also works for memory restrained systems. Needs a reboot though...

> > A moment ago, I find the following patch, it is more flexible, but it is 
> > not merged into the upstream kernel now.
> > 
> > kernel/crash_core: Add crashkernel=auto for vmcore creation
> > 
> > https://lore.kernel.org/lkml/20210223174153.72802-1-saeed.mirzamohammadi@oracle.com/ 

The patch was ultimately rejected by Linus

https://lore.kernel.org/linux-mm/20210507010432.IN24PudKT%25akpm@linux-foundation.org/

Thanks
Philipp

> >   
> >>  
> >>> [...]
> >>> diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> >>> index 256cf6d..32c51e2 100644
> >>> --- a/kernel/crash_core.c
> >>> +++ b/kernel/crash_core.c
> >>> @@ -252,6 +252,26 @@ static int __init __parse_crashkernel(char 
> >>> *cmdline,
> >>>       if (suffix)
> >>>           return parse_crashkernel_suffix(ck_cmdline, crash_size,
> >>>                   suffix);
> >>> +
> >>> +    if (strncmp(ck_cmdline, "auto", 4) == 0) {
> >>> +#if defined(CONFIG_X86_64) || defined(CONFIG_S390)
> >>> +        ck_cmdline = "1G-4G:160M,4G-64G:192M,64G-1T:256M,1T-:512M";
> >>> +#elif defined(CONFIG_ARM64)
> >>> +        ck_cmdline = "2G-:448M";
> >>> +#elif defined(CONFIG_PPC64)
> >>> +        char *fadump_cmdline;
> >>> +
> >>> +        fadump_cmdline = get_last_crashkernel(cmdline, "fadump=", 
> >>> NULL);
> >>> +        fadump_cmdline = fadump_cmdline ?
> >>> +                fadump_cmdline + strlen("fadump=") : NULL;
> >>> +        if (!fadump_cmdline || (strncmp(fadump_cmdline, "off", 3) == 
> >>> 0))
> >>> +            ck_cmdline =
> >>> "2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G";
> >>> +        else
> >>> +            ck_cmdline =
> >>> "4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"; 
> >>>
> >>>
> >>> +#endif
> >>> +        pr_info("Using crashkernel=auto, the size chosen is a best
> >>> effort estimation.\n");
> >>> +    }
> >>> +  
> >>
> >> How did you even arrive at the above numbers?  
> > 
> > Memory requirements for kdump:
> > 
> > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/supported-kdump-configurations-and-targets_managing-monitoring-and-updating-the-kernel#memory-requirements-for-kdump_supported-kdump-configurations-and-targets 
> > 
> > 
> > I've done some research on  
> >> this topic recently (ie. during the last 7 years or so). My x86_64
> >> system with 8G RAM running openSUSE Leap 15.3 seems needs 188M for
> >> saving to the local disk, and 203M to save over the network (using
> >> SFTP). My PPC64 LPAR with 16G RAM running latest Beta of SLES 15 SP4
> >> needs 587M, i.e. with the above numbers it may run out of memory while
> >> saving the dump.
> >>
> >> Since this is not the first time, I'm trying to explain things, I've
> >> written a blog post now:
> >>
> >> https://sigillatum.tesarici.cz/2022-01-27-whats-wrong-with-crashkernel-auto.html 
> >>
> >>  
> > 
> > Thank you, this is useful.
> > 
> > Thanks,
> > Tiezhu
> >   
> >>
> >> HTH
> >> Petr Tesarik  
> > 
> > 
> > _______________________________________________
> > kexec mailing list
> > kexec@...ts.infradead.org
> > http://lists.infradead.org/mailman/listinfo/kexec  
> 
> _______________________________________________
> kexec mailing list
> kexec@...ts.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives