linux-kernel - Re: [QUESTION] Is there a better way to get ftrace dump on guest?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160628212652.295cbb21@gandalf.local.home>
Date:	Tue, 28 Jun 2016 21:26:52 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Minchan Kim <minchan@...nel.org>
Cc:	Rabin Vincent <rabin@....in>, Namhyung Kim <namhyung@...nel.org>,
	<kvm@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	anderson@...hat.com
Subject: Re: [QUESTION] Is there a better way to get ftrace dump on guest?

On Wed, 29 Jun 2016 09:57:41 +0900
Minchan Kim <minchan@...nel.org> wrote:

> Hello,
> 
> On Tue, Jun 28, 2016 at 06:46:34PM +0200, Rabin Vincent wrote:
> > On Tue, Jun 28, 2016 at 03:33:18PM +0900, Namhyung Kim wrote:  
> > > On Tue, Jun 28, 2016 at 3:25 PM, Namhyung Kim <namhyung@...nel.org> wrote:  
> > > > I'm running some guest machines for kernel development.  For debugging
> > > > purpose, I use lots of trace_printk() since it's faster than normal
> > > > printk().  When kernel crash happens the trace buffer is printed on
> > > > console (I set ftrace_dump_on_oops) but it takes too much time.  I
> > > > don't want to reduce the size of ring buffer as I want to collect the
> > > > debug info as much as possible.  And I also want to see trace from all
> > > > cpu so 'ftrace_dump_on_oop = 2' is not an option.
> > > >
> > > > I know the kexec/kdump (and the crash tool) can dump and analyze the
> > > > trace buffer later.  But it's cumbersome to do it everytime and more
> > > > importantly, I don't want to spend the memory for the crashkernel.  
> > 
> > Assuming you're using QEMU:
> > 
> > QEMU has a dump-guest-memory command which can be used to dump the
> > guest's entire memory to an ELF which can be loaded by the crash utility
> > to extract the trace buffer.  This doesn't require kexec/kdump or any
> > other support from the guest kernel.  
> 
> Thanks for the hint. It's surely handy rather than kexec/kdump.
> 
> A question is that it's possible to capture guest's entire memory
> when guest kernel is oops?
> I mean I don't want to capture alive guest but get snapshot image
> when guest kernel encounters BUG_ON and see event trace from the
> image.
> 
> Anyway, I tried crashtool and load trace.so but failed to load
> extension module 'trace.so' because read_string failed in
> ftrace_get_event_type_name of trace.c.
> Does it work with recent kernel?
> 
> My kernel is 4.7.0-rc4-mm1.

It probably needs another update. I usually send patches to David
Anderson for updates. Fujitsu started that work and was maintaining it
for a while, but I don't think they are anymore. I have no problem
maintaining the trace.so module.

If I get time tomorrow, I'll see if I can get it up to date again.

-- Steve


> 
> > 
> > It's apparently even possible to run QEMU with the guest memory in a
> > file and load that to crash directly, although this is not something
> > I've had a chance to try out myself:
> > 
> > https://github.com/crash-utility/crash/commit/89ed9d0a7f7da4578294a492c1ad857244ce7352