linux-kernel - Re: [PATCH net-next v2 0/2] kernel: add support to collect hardware logs in crash recovery kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <87tvt1ecg9.fsf@xmission.com>
Date:   Tue, 27 Mar 2018 10:59:50 -0500
From:   ebiederm@...ssion.com (Eric W. Biederman)
To:     Rahul Lakkireddy <rahul.lakkireddy@...lsio.com>
Cc:     "netdev\@vger.kernel.org" <netdev@...r.kernel.org>,
        "linux-fsdevel\@vger.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "kexec\@lists.infradead.org" <kexec@...ts.infradead.org>,
        "linux-kernel\@vger.kernel.org" <linux-kernel@...r.kernel.org>,
        "davem\@davemloft.net" <davem@...emloft.net>,
        "viro\@zeniv.linux.org.uk" <viro@...iv.linux.org.uk>,
        "stephen\@networkplumber.org" <stephen@...workplumber.org>,
        "akpm\@linux-foundation.org" <akpm@...ux-foundation.org>,
        "torvalds\@linux-foundation.org" <torvalds@...ux-foundation.org>,
        Ganesh GR <ganeshgr@...lsio.com>,
        Nirranjan Kirubaharan <nirranjan@...lsio.com>,
        Indranil Choudhury <indranil@...lsio.com>
Subject: Re: [PATCH net-next v2 0/2] kernel: add support to collect hardware logs in crash recovery kernel

Rahul Lakkireddy <rahul.lakkireddy@...lsio.com> writes:

> On Tuesday, March 03/27/18, 2018 at 18:47:34 +0530, Eric W. Biederman wrote:
>> Rahul Lakkireddy <rahul.lakkireddy@...lsio.com> writes:
>> 
>> > On Saturday, March 03/24/18, 2018 at 20:50:52 +0530, Eric W. Biederman wrote:
>> >> 
>> >> Rahul Lakkireddy <rahul.lakkireddy@...lsio.com> writes:
>> >> 
>> >> > On production servers running variety of workloads over time, kernel
>> >> > panic can happen sporadically after days or even months. It is
>> >> > important to collect as much debug logs as possible to root cause
>> >> > and fix the problem, that may not be easy to reproduce. Snapshot of
>> >> > underlying hardware/firmware state (like register dump, firmware
>> >> > logs, adapter memory, etc.), at the time of kernel panic will be very
>> >> > helpful while debugging the culprit device driver.
>> >> >
>> >> > This series of patches add new generic framework that enable device
>> >> > drivers to collect device specific snapshot of the hardware/firmware
>> >> > state of the underlying device in the crash recovery kernel. In crash
>> >> > recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/
>> >> > directory, which is copied by user space scripts for post-analysis.
>> >> >
>> >> > A kernel module crashdd is newly added. In crash recovery kernel,
>> >> > crashdd exposes /sys/kernel/crashdd/ directory containing device
>> >> > specific hardware/firmware logs.
>> >> 
>> >> Have you looked at instead of adding a sysfs file adding the dumps
>> >> as additional elf notes in /proc/vmcore?
>> >> 
>> >
>> > I see the crash recovery kernel's memory is not present in any of the
>> > the PT_LOAD headers.  So, makedumpfile is not collecting the dumps
>> > that are in crash recovery kernel's memory.
>> >
>> > Also, are you suggesting exporting the dumps themselves as PT_NOTE
>> > instead?  I'll look into doing it this way.
>> 
>> Yes.  I was suggesting exporting the dumps themselves as PT_NOTE
>> in /proc/vmcore.  I think that will allow makedumpfile to collect
>> your new information without modification.
>> 
>
> If I export the dumps themselves as PT_NOTE in /proc/vmcore, can the 
> crash tool work without modification; i.e can crash tool extract these
> notes?

I believe crash would need to be taught about these notes.   This is
something new.

However "readelf -a random_elf_file" does display elf notes, and elf
notes in general are not hard to extract.

What I expect from an enconding in ELF core dump format is a way to
captuer the data, a way to encode the data, and a way to transport the
data to the people who care.  Analysis tools are easy enough after the
fact.

Eric