linux-kernel - Re: kdump: crash_kexec()-smp_send

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 24 Oct 2011 10:07:19 -0700
From:	ebiederm@...ssion.com (Eric W. Biederman)
To:	Américo Wang <xiyou.wangcong@...il.com>
Cc:	holzheu@...ux.vnet.ibm.com, Vivek Goyal <vgoyal@...hat.com>,
	akpm@...ux-foundation.org, schwidefsky@...ibm.com,
	heiko.carstens@...ibm.com, kexec@...ts.infradead.org,
	linux-kernel@...r.kernel.org
Subject: Re: kdump: crash_kexec()-smp_send_stop() race in panic

Américo Wang <xiyou.wangcong@...il.com> writes:

> On Mon, Oct 24, 2011 at 11:14 PM, Eric W. Biederman
> <ebiederm@...ssion.com> wrote:
>> Michael Holzheu <holzheu@...ux.vnet.ibm.com> writes:
>>
>>> Hello Vivek,
>>>
>>> In our tests we ran into the following scenario:
>>>
>>> Two CPUs have called panic at the same time. The first CPU called
>>> crash_kexec() and the second CPU called smp_send_stop() in panic()
>>> before crash_kexec() finished on the first CPU. So the second CPU
>>> stopped the first CPU and therefore kdump failed.
>>>
>>> 1st CPU:
>>> panic()->crash_kexec()->mutex_trylock(&kexec_mutex)-> do kdump
>>>
>>> 2nd CPU:
>>> panic()->crash_kexec()->kexec_mutex already held by 1st CPU
>>>        ->smp_send_stop()-> stop CPU 1 (stop kdump)
>>>
>>> How should we fix this problem? One possibility could be to do
>>> smp_send_stop() before we call crash_kexec().
>>>
>>> What do you think?
>>
>> smp_send_stop is insufficiently reliable to be used before crash_kexec.
>>
>> My first reaction would be to test oops_in_progress and wait until
>> oops_in_progress == 1 before calling smp_send_stop.
>>
>
> +1
>
> One of my colleague mentioned the same problem with me inside
> RH, given the fact that the race condition window is small, it would
> not be easy to reproduce this scenario.

As for reproducing it I have a hunch you could hack up something
horrible with smp_call_function and kprobes.


On a little more reflection we can't wait until oops_in_progress goes
to 1 before calling smp_send_stop.  Because if crash_kexec is not
involved nothing we will never call smp_send_stop. 

So my second thought is to introduce another atomic variable
panic_in_progress, visible only in panic.  The cpu that sets
increments panic_in_progress can call smp_send_stop.  The rest of
the cpus can just go into a busy wait.  That should stop nasty
fights about who is going to come out of smp_send_stop first.

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/