Date:	Tue, 26 May 2009 22:27:43 -0400 (EDT)
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
cc:	LKML <linux-kernel@...r.kernel.org>, Ingo Molnar <mingo@...e.hu>
Subject: Re: [2.6.30-rcX] Some ftrace debug option is consuming memory?


[ added Ingo ]


On Mon, 25 May 2009, Tetsuo Handa wrote:

> I was running a VM with 1024MB of RAM.
> 
> I changed it to 512MB of RAM, and now I'm experiencing out of memory error.
> 
> [    2.815999] scsi: ***** BusLogic SCSI Driver Version 2.1.16 of 18 July 2002 *****
> [    2.818347] scsi: Copyright 1995-1998 by Leonard N. Zubkoff <lnz@...delion.com>
> [    2.822849] Kernel panic - not syncing: Out of memory and no killable processes...
> [    2.822857] 
> [    2.824546] Pid: 1, comm: swapper Not tainted 2.6.30-rc7 #1
> [    2.825677] Call Trace:
> [    2.826424]  [<c015105d>] ? printk+0x1d/0x30
> [    2.827136]  [<c014fd48>] panic+0x48/0x130
> [    2.827832]  [<c01c5b29>] __out_of_memory+0x109/0x110
> [    2.828665]  [<c01c5ca1>] out_of_memory+0xb1/0x100
> [    2.829772]  [<c01c8df3>] __alloc_pages_internal+0x5b3/0x750
> [    2.830686]  [<c01c902b>] __get_free_pages+0x9b/0xa0
> [    2.831517]  [<c0200e49>] __kmalloc+0x79/0x360
> [    2.832258]  [<c03feb19>] scsi_host_alloc+0x59/0x3d0
> [    2.833066]  [<c0411122>] ? BusLogic_AnnounceDriver+0x42/0x50
> [    2.834044]  [<c13d3b9b>] BusLogic_init+0x6bb/0xa50
> [    2.834839]  [<c03c78c9>] ? driver_register+0xf9/0x160
> [    2.835693]  [<c015105d>] ? printk+0x1d/0x30
> [    2.836406]  [<c13cfe00>] ? ide_cdrom_init+0x0/0x20
> [    2.837253]  [<c13d34e0>] ? BusLogic_init+0x0/0xa50
> [    2.838050]  [<c0101058>] do_one_initcall+0x48/0x260
> [    2.838862]  [<c05a39bc>] ? __schedule+0x62c/0xad0
> [    2.839668]  [<c0188c36>] ? validate_chain+0x3f6/0x510
> [    2.840501]  [<c018a92c>] ? __lock_acquire+0x29c/0x830
> [    2.842191]  [<c01fcdf9>] ? check_bytes_and_report+0x29/0xe0
> [    2.843135]  [<c01fcf53>] ? check_pad_bytes+0xa3/0xb0
> [    2.843960]  [<c01fd252>] ? check_object+0x1a2/0x2d0
> [    2.844780]  [<c034810b>] ? ida_get_new_above+0x23b/0x260
> [    2.846429]  [<c018a92c>] ? __lock_acquire+0x29c/0x830
> [    2.847291]  [<c010cd93>] ? sched_clock+0x23/0xc0
> [    2.848057]  [<c0186753>] ? lock_release_holdtime+0x73/0xb0
> [    2.848961]  [<c027b982>] ? proc_register+0xf2/0x210
> [    2.850429]  [<c018b9d7>] ? __lock_release+0x47/0x70
> [    2.851522]  [<c027b982>] ? proc_register+0xf2/0x210
> [    2.852333]  [<c03659ad>] ? _raw_spin_unlock+0xcd/0x150
> [    2.853149]  [<c05a8222>] ? _spin_unlock+0x22/0x30
> [    2.854555]  [<c027b982>] ? proc_register+0xf2/0x210
> [    2.855379]  [<c027beaa>] ? create_proc_entry+0xaa/0x130
> [    2.856241]  [<c01a65e0>] ? register_irq_proc+0xd0/0x100
> [    2.857113]  [<c139ce9a>] do_initcalls+0x2a/0x40
> [    2.858413]  [<c139cf00>] ? kernel_init+0x0/0xc0
> [    2.859183]  [<c139cecc>] do_basic_setup+0x1c/0x20
> [    2.859962]  [<c139cf55>] kernel_init+0x55/0xc0
> [    2.860702]  [<c0104707>] kernel_thread_helper+0x7/0x10
> 
> 2.6.29.4 can boot with 512MB of RAM.
> 2.6.30-rc1 and later cannot boot with 512MB of RAM if some debug options are
> enabled.
> 
> It seems to me that some debug option is consuming a lot of memory (enough to
> trigger BusLogic driver's initialization failure).
> 
> Config which can boot with 512MB of RAM is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.30-rc7-ok
> 
> Config which cannot boot with 512MB of RAM is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.30-rc7-ng
> 
> $ diff config-2.6.30-rc7-ok config-2.6.30-rc7-ng
> 4c4
> < # Mon May 25 12:33:54 2009
> ---
> > # Mon May 25 13:28:39 2009
> 1519a1520,1521
> > CONFIG_NOP_TRACER=y
> > CONFIG_HAVE_FTRACE_NMI_ENTER=y
> 1525a1528,1531
> > CONFIG_TRACER_MAX_TRACE=y
> > CONFIG_RING_BUFFER=y
> > CONFIG_FTRACE_NMI_ENTER=y
> > CONFIG_TRACING=y
> 1531,1535c1537,1542
> < # CONFIG_FUNCTION_TRACER is not set
> < # CONFIG_IRQSOFF_TRACER is not set
> < # CONFIG_SYSPROF_TRACER is not set
> < # CONFIG_SCHED_TRACER is not set
> < # CONFIG_CONTEXT_SWITCH_TRACER is not set
> ---
> > CONFIG_FUNCTION_TRACER=y
> > CONFIG_FUNCTION_GRAPH_TRACER=y
> > CONFIG_IRQSOFF_TRACER=y
> > CONFIG_SYSPROF_TRACER=y
> > CONFIG_SCHED_TRACER=y
> > CONFIG_CONTEXT_SWITCH_TRACER=y
> 1538,1541c1545,1551
> < # CONFIG_BOOT_TRACER is not set
> < # CONFIG_TRACE_BRANCH_PROFILING is not set
> < # CONFIG_POWER_TRACER is not set
> < # CONFIG_STACK_TRACER is not set
> ---
> > CONFIG_BOOT_TRACER=y


> > CONFIG_TRACE_BRANCH_PROFILING=y
> > CONFIG_PROFILE_ALL_BRANCHES=y
> > CONFIG_TRACING_BRANCHES=y
> > CONFIG_BRANCH_TRACER=y

I should have noticed this before. I bet if you disable the branch tracers 
you will boot again. The branch tracers makes a table of _all_ branches in 
the kernel. But the change we did between 29 and 30 is that we also make 
it per cpu. On big boxes the cache line hits were so bad that the 
interrupts would take so long that they would live lock the machine.

You configured 2 CPUS so you doubled the size of that table.

To test if this is your issue you can try changing NR_CPUS to 1, or just 
disabled the BRANCH_PROFILERs.

-- Steve



> > CONFIG_POWER_TRACER=y
> > CONFIG_STACK_TRACER=y
> 1544a1555,1557
> > CONFIG_DYNAMIC_FTRACE=y
> > CONFIG_FTRACE_MCOUNT_RECORD=y
> > # CONFIG_FTRACE_STARTUP_TEST is not set
> 1695c1708
> < # CONFIG_BINARY_PRINTF is not set
> ---
> > CONFIG_BINARY_PRINTF=y
> 
> Regards.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists