Message-ID: <ZXGyANwELvFKpysH@tiehlicka>
Date:   Thu, 7 Dec 2023 12:52:32 +0100
From:   Michal Hocko <mhocko@...e.com>
To:     Philipp Rudo <prudo@...hat.com>
Cc:     Baoquan He <bhe@...hat.com>, Donald Dutile <ddutile@...hat.com>,
        Jiri Bohac <jbohac@...e.cz>, Pingfan Liu <piliu@...hat.com>,
        Tao Liu <ltao@...hat.com>, Vivek Goyal <vgoyal@...hat.com>,
        Dave Young <dyoung@...hat.com>, kexec@...ts.infradead.org,
        linux-kernel@...r.kernel.org,
        David Hildenbrand <dhildenb@...hat.com>
Subject: Re: [PATCH 0/4] kdump: crashkernel reservation from CMA

On Thu 07-12-23 12:13:14, Philipp Rudo wrote:
> On Thu, 7 Dec 2023 09:55:20 +0100
> Michal Hocko <mhocko@...e.com> wrote:
> 
> > On Thu 07-12-23 12:23:13, Baoquan He wrote:
> > [...]
> > > We can't guarantee how swift the DMA transfer could be in the cma, case,
> > > it will be a venture.  
> > 
> > We can't guarantee this of course but AFAIK the DMA shouldn't take
> > minutes, right? While not perfect, waiting for some time before jumping
> > into the crash kernel should be acceptable from user POV and it should
> > work around most of those potential lingering programmed DMA transfers.
> 
> I don't think that simply waiting is acceptable. For one it doesn't
> guarantee that there is no corruption (please also see below) but only
> reduces its probability. Furthermore, how long would you wait?

I would like to talk to storage experts to have some ballpark idea about
worst case scenario but waiting for 1 minutes shouldn't terribly
influence downtime and remember this is an opt-in feature. If that
doesn't fit your use case, do not use it.

> Thing is that users don't only want to reduce the memory usage but also
> the downtime of kdump. In the end I'm afraid that "simply waiting" will
> make things unnecessarily more complex without really solving any issue.

I am not sure I see the added complexity. Something as simple as
__crash_kexec:
	if (crashk_cma_cnt) 
		mdelay(TIMEOUT)

should do the trick.

> > So I guess what we would like to hear from you as kdump maintainers is
> > this. Is it absolutely imperative that these issue must be proven
> > impossible or is a best effort approach something worth investing time
> > into? Because if the requirement is an absolute guarantee then I simply
> > do not see any feasible way to achieve the goal of reusable memory.
> > 
> > Let me reiterate that the existing reservation mechanism is showing its
> > limits for production systems and I strongly believe this is something
> > that needs addressing because crash dumps are very often the only tool
> > to investigate complex issues.
> 
> Because having a crash dump is so important I want a prove that no
> legal operation can corrupt the crashkernel memory. The easiest way to
> achieve this is by simply keeping the two memory regions fully
> separated like it is today. In theory it should also be possible to
> prevent any kind of page pinning in the shared crashkernel memory. But
> I don't know which side effect this has for mm. Such an idea needs to
> be discussed on the mm mailing list first.

I do not see that as a feasible option. That would require to migrate
memory on any gup user that might end up sending data over DMA.

> Finally, let me question whether the whole approach actually solves
> anything. For me the difficulty in determining the correct crashkernel
> memory is only a symptom. The real problem is that most developers
> don't expect their code to run outside their typical environment.
> Especially not in an memory constraint environment like kdump. But that
> problem won't be solved by throwing more memory at it as this
> additional memory will eventually run out as well. In the end we are
> back at the point where we are today but with more memory.

I disagree with you here. While the kernel is really willing to cache
objects into memory I do not think that any particular subsystem is
super eager to waste memory.

The thing we should keep in mind is that the memory sitting aside is not
used in majority of time. Crashes (luckily/hopefully) do not happen very
often. And I can really see why people are reluctant to waste it. Every
MB of memory has an operational price tag on it. And let's just be
really honest, a simple reboot without a crash dump is very likely
a cheaper option than wasting a productive memory as long as the issue
happens very seldom.
 
> Finally finally, one tip. Next time a customer complaints about how
> much memory the crashkernel "wastes" ask them how much one day of down
> time for one machine costs them and how much memory they could buy for
> that money. After that calculation I'm pretty sure that an additional
> 100M of crashkernel memory becomes much more tempting.

Exactly and that is why a simple reboot would be a preferred option than
configuring kdump and invest admin time to keep testing configuration
after every (major) upgrade to make sure the existing setup still works.
>From my experience crashdump availability hugely improves chances to get
underlying crash diagnosed and bug solved so it is also in our interest
to encourage kdump deployments as much as possible.

Now I do get your concerns about potential issues and I fully recognize
the pain you have gone through when debugging these subtle issues in the
past but let's not forget that perfect is an enemy of good and that a
best effort solution might be better than crash dumps at all.

At the end, let me just ask a theoretical question. With the experience
you have gained would you nack the kexec support if it was proposed now
just because of all the potential problems it might have?
-- 
Michal Hocko
SUSE Labs

Powered by blists - more mailing lists