Date:	Sun, 31 May 2009 09:15:08 +0200
From:	Rambaldi <rambaldi@...all.nl>
To:	Jeremy Fitzhardinge <jeremy@...p.org>
CC:	linux-kernel@...r.kernel.org,
	Xen-devel <xen-devel@...ts.xensource.com>
Subject: Re: kernel BUG at arch/x86/xen/multicalls.c:204!

Jeremy Fitzhardinge wrote:
> Rambaldi wrote:
>> Hi,
>>
>> while booting the 2.6.29-gentoo-r5 kernel as xen domU guest I got the
>> following kernel BUG.
>>
>> I have not tried a vanilla 2.6.29 kernel, but can if requested,
>> .config is also available on reguest.
>>   
> 
> That would be useful, particularly if you enable the various debugging
> options:
> CONFIG_DEBUG_KERNEL
> CONFIG_FRAME_POINTER
> 
> It's not clear to me what's failing.  Are there any messages on the Xen
> concole (xm dmesg)?
> 
>    J
> 
Hi Jeremy,

thanks for looking at it.

I tried with a vanilla 2.6.29.4 kernel. I added CONFIG_FRAME_POINTER. CONFIG_DEBUG_KERNEL was already set.
I got the same result as with the gentoo kernel and in both cases not a single line in the xen console.

Any other options or kernels I can try?

tia

R


Started domain
                                 Linux version 2.6.29.4 (root@...ula) (gcc version 4.3.2 (Gentoo 4.3.2-r3 p1.6,
pie-10.1.5) ) #1 SMP Sun May 31 08:53:04 CEST 2009
Command line: root=/dev/xvda1 ro xencons=hvc0  xen_fbfront.video=4,1152,864
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000000ea8000 (usable)
 Xen: 0000000000ea8000 - 0000000001635000 (reserved)
 Xen: 0000000001635000 - 00000000f1400000 (usable)
DMI not present or invalid.
last_pfn = 0xf1400 max_arch_pfn = 0x100000000
init_memory_mapping: 0000000000000000-00000000f1400000
last_map_addr: f1400000 end: f1400000
(5 early reservations) ==> bootmem [0000000000 - 00f1400000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0001635000 - 0001644000]   XEN PAGETABLES ==> [0001635000 - 0001644000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0000200000 - 0000ea7f18]    TEXT DATA BSS ==> [0000200000 - 0000ea7f18]
  #4 [0001644000 - 0001dc1000]          PGTABLE ==> [0001644000 - 0001dc1000]
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000000 -> 0x000000a0
    0: 0x00000100 -> 0x00000ea8
    0: 0x00001635 -> 0x000f1400
SMP: Allowing 8 CPUs, 0 hotplug CPUs
No local APIC present
Allocating PCI resources starting at f2000000 (gap: f1400000:ec00000)
NR_CPUS:4096 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
PERCPU: Allocating 73728 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 969379
Kernel command line: root=/dev/xvda1 ro xencons=hvc0  xen_fbfront.video=4,1152,864
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Detected 2499.998 MHz processor.
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Checking aperture...
No AGP bridge found
Memory: 3856500k/3952640k available (2342k kernel code, 8116k absent, 87764k reserved, 2269k data, 480k init)
installing Xen timer for CPU 0
Calibrating delay loop (skipped), value calculated using timer frequency.. 4999.99 BogoMIPS (lpj=24999980)
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Freeing SMP alternatives: 27k freed
cpu 0 spinlock event irq 1
installing Xen timer for CPU 1
cpu 1 spinlock event irq 7
Initializing CPU#1
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 2
cpu 2 spinlock event irq 13
Initializing CPU#2
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 3
cpu 3 spinlock event irq 19
Initializing CPU#3
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 4
cpu 4 spinlock event irq 25
Initializing CPU#4
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 5
cpu 5 spinlock event irq 31
Initializing CPU#5
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 6
cpu 6 spinlock event irq 37
Initializing CPU#6
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
installing Xen timer for CPU 7
cpu 7 spinlock event irq 43
Initializing CPU#7
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Brought up 8 CPUs
net_namespace: 1824 bytes
Booting paravirtualized kernel on Xen
Xen version: 3.3.1 (preserve-AD)
xor: automatically using best checksumming function: generic_sse
   generic_sse:  2589.600 MB/sec
xor: using function: generic_sse (2589.600 MB/sec)
Grant table initialized
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
xen_balloon: Initialising balloon driver.
SCSI subsystem initialized
PCI: System does not support PCI
PCI: System does not support PCI
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
NET: Registered protocol family 1
platform rtc_cmos: registered platform RTC device (no PNP device found)
msgmni has been set to 7720
alg: No test for stdrng (krng)
async_tx: api initialized (sync-only)
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
Console: switching to colour frame buffer device 144x54
console [tty0] enabled
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
loop: module loaded
Initialising Xen virtual ethernet driver.
blkfront: xvda: barriers enabled
 xvda: xvda1 xvda2
Driver 'sd' needs updating - please use bus_type methods
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
blkfront: xvdb: barriers enabled
input: PC Speaker as /class/input/input0
 xvdb: unknown partition table
input: Xen Virtual Keyboard as /class/input/input1
input: Xen Virtual Pointer as /class/input/input2
rtc_cmos: probe of rtc_cmos failed with error -16
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid10 personality registered for level 10
raid6: int64x1   2218 MB/s
raid6: int64x2   2752 MB/s
raid6: int64x4   2133 MB/s
raid6: int64x8   1871 MB/s
raid6: sse2x1    2953 MB/s
raid6: sse2x2    5642 MB/s
raid6: sse2x4    6250 MB/s
raid6: using algorithm sse2x4 (6250 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: multipath personality registered for level -4
device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@...hat.com
cpuidle: using governor ladder
cpuidle: using governor menu
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
TCP cubic registered
NET: Registered protocol family 17
IO APIC resources could be not be allocated.
registered taskstats version 1
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly on device 202:1.
Freeing unused kernel memory: 480k freed
------------[ cut here ]------------
kernel BUG at arch/x86/xen/multicalls.c:204!
invalid opcode: 0000 [#1] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 959, comm: bash Not tainted 2.6.29.4 #1
RIP: e030:[<ffffffff8020c012>]  [<ffffffff8020c012>] __xen_mc_entry+0x94/0xb6
RSP: e02b:ffff8800ef11f398  EFLAGS: 00010006
RAX: 0000000000000218 RBX: ffff880001df7070 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000001000 RDI: 0000000000000000
RBP: ffff8800ef11f3c8 R08: 0000000000000040 R09: 0000000000000040
R10: 00000000000e95d4 R11: 0000000000000001 R12: 0000000000000218
R13: 00007f0ac5d3b315 R14: 0000000000000000 R15: ffff8800efee89d8
FS:  00007f0ac5d3a6f0(0000) GS:ffffffff80c25000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0ac5d3b315 CR3: 00000000efeef000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 959, threadinfo ffff8800ef11e000, task ffff8800ef0e75e0)
Stack:
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000200 ffff8800ef11f3f8 ffff8800ef11f628 ffffffff8020ab5e
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffff8020ab5e>] xen_flush_tlb_others+0xc3/0x165
 [<ffffffff8021c016>] flush_tlb_page+0xaa/0xb9
 [<ffffffff8020c08f>] ? pte_pfn_to_mfn+0x24/0x51
 [<ffffffff80277194>] do_wp_page+0x6a8/0x7d4
 [<ffffffff80278a80>] handle_mm_fault+0x7b0/0x876
 [<ffffffff80446c79>] ? _spin_unlock_irqrestore+0x18/0x1c
 [<ffffffff802244c7>] do_page_fault+0x60d/0xa0e
 [<ffffffff8044581f>] ? __wait_on_bit+0x6a/0x79
 [<ffffffff802abc0f>] ? sync_buffer+0x0/0x3f
 [<ffffffff802abc0f>] ? sync_buffer+0x0/0x3f
 [<ffffffff80445898>] ? out_of_line_wait_on_bit+0x6a/0x77
 [<ffffffff8024c19e>] ? wake_bit_function+0x0/0x2a
 [<ffffffff803145f7>] ? kobject_get+0x1a/0x22
 [<ffffffff8028f52b>] ? cdev_get+0x43/0x5b
 [<ffffffff8028f54f>] ? exact_lock+0xc/0x14
 [<ffffffff80446e35>] ? lock_kernel+0x27/0x38
 [<ffffffff80446e75>] ? unlock_kernel+0x2f/0x32
 [<ffffffff80345521>] ? memory_open+0xd9/0xe0
 [<ffffffff8028f810>] ? chrdev_open+0x162/0x181
 [<ffffffff8028f6ae>] ? chrdev_open+0x0/0x181
 [<ffffffff8028b6b3>] ? __dentry_open+0x16b/0x25c
 [<ffffffff8028b871>] ? nameidata_to_filp+0x41/0x53
 [<ffffffff80297a30>] ? do_filp_open+0x418/0x840
 [<ffffffff80273d48>] ? mod_zone_page_state+0xa6/0xb0
 [<ffffffff80446c28>] ? _spin_lock+0x9/0xc
 [<ffffffff8028854e>] ? kmem_cache_free+0xf0/0x100
 [<ffffffff80446f85>] page_fault+0x25/0x30
Code: 48 c1 e8 03 8d 14 c5 00 00 00 00 8b 8b 00 14 00 00 8d 41 01 89 83 00 14 00 00 42 8d 04 22 89 83 04 14 00 00 3d 00
02 00 00 76 04 <0f> 0b eb fe 48 83 c4 20 89 d0 48 8d 94 03 00 10 00 00 89 c8 48
RIP  [<ffffffff8020c012>] __xen_mc_entry+0x94/0xb6
 RSP <ffff8800ef11f398>
---[ end trace 35f44aea30d5e8cf ]---
------------[ cut here ]------------
kernel BUG at arch/x86/xen/multicalls.c:190!
invalid opcode: 0000 [#2] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 959, comm: bash Tainted: G      D    2.6.29.4 #1
RIP: e030:[<ffffffff8020bfad>]  [<ffffffff8020bfad>] __xen_mc_entry+0x2f/0xb6
RSP: e02b:ffff8800ef11ef68  EFLAGS: 00010006
RAX: 0000000000000218 RBX: ffff880001df7070 RCX: 0000000000000000
RDX: ffff880081215000 RSI: 0000000000201000 RDI: 0000000000000018
RBP: ffff8800ef11ef98 R08: 0000000000000000 R09: 00000000000028ff
R10: ffff8800ef11f0c8 R11: 0000000000000006 R12: 0000000000000018
R13: ffff8800f0499001 R14: ffff8800f04990a0 R15: 0000000000000004
FS:  00007f0ac5d3a6f0(0000) GS:ffffffff80c25000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0ac5d3b315 CR3: 0000000000201000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 959, threadinfo ffff8800ef11e000, task ffff8800ef0e75e0)
Stack:
 ffff8800ef11ef88 ffffffff80446c79 0000000000000000 ffffffff805458c0
 00000000002005fc 0000000000201000 ffff8800ef11efe8 ffffffff8020b02e
 0000000000000000 ffffffff80213276 0000000080000001 0000000000000000
Call Trace:
 [<ffffffff80446c79>] ? _spin_unlock_irqrestore+0x18/0x1c
 [<ffffffff8020b02e>] __xen_write_cr3+0x6b/0xe7
 [<ffffffff80213276>] ? do_softirq+0x8a/0x8f
 [<ffffffff8020b0fc>] xen_write_cr3+0x52/0xb1
 [<ffffffff8021bf38>] leave_mm+0x42/0x45
 [<ffffffff8020ca20>] xen_exit_mmap+0x53/0x1dc
 [<ffffffff8020922a>] ? _stext+0x22a/0x1000
 [<ffffffff8020922a>] ? _stext+0x22a/0x1000
 [<ffffffff8027c868>] exit_mmap+0x4e/0x138
 [<ffffffff80237548>] mmput+0x2b/0xb5
 [<ffffffff8023acb0>] exit_mm+0x101/0x10c
 [<ffffffff8023c727>] do_exit+0x187/0x745
 [<ffffffff80239d42>] ? release_console_sem+0x199/0x1ce
 [<ffffffff8021433c>] oops_end+0x99/0x9e
 [<ffffffff802144f5>] die+0x55/0x5e
 [<ffffffff802123ff>] do_trap+0x110/0x11f
 [<ffffffff802127be>] do_invalid_op+0x98/0xa1
 [<ffffffff8020c012>] ? __xen_mc_entry+0x94/0xb6
 [<ffffffff8021175b>] invalid_op+0x1b/0x20
 [<ffffffff8020c012>] ? __xen_mc_entry+0x94/0xb6
 [<ffffffff8020bfdd>] ? __xen_mc_entry+0x5f/0xb6
 [<ffffffff8020ab5e>] xen_flush_tlb_others+0xc3/0x165
 [<ffffffff8021c016>] flush_tlb_page+0xaa/0xb9
 [<ffffffff8020c08f>] ? pte_pfn_to_mfn+0x24/0x51
 [<ffffffff80277194>] do_wp_page+0x6a8/0x7d4
 [<ffffffff80278a80>] handle_mm_fault+0x7b0/0x876
 [<ffffffff80446c79>] ? _spin_unlock_irqrestore+0x18/0x1c
 [<ffffffff802244c7>] do_page_fault+0x60d/0xa0e
 [<ffffffff8044581f>] ? __wait_on_bit+0x6a/0x79
 [<ffffffff802abc0f>] ? sync_buffer+0x0/0x3f
 [<ffffffff802abc0f>] ? sync_buffer+0x0/0x3f
 [<ffffffff80445898>] ? out_of_line_wait_on_bit+0x6a/0x77
 [<ffffffff8024c19e>] ? wake_bit_function+0x0/0x2a
 [<ffffffff803145f7>] ? kobject_get+0x1a/0x22
 [<ffffffff8028f52b>] ? cdev_get+0x43/0x5b
 [<ffffffff8028f54f>] ? exact_lock+0xc/0x14
 [<ffffffff80446e35>] ? lock_kernel+0x27/0x38
 [<ffffffff80446e75>] ? unlock_kernel+0x2f/0x32
 [<ffffffff80345521>] ? memory_open+0xd9/0xe0
 [<ffffffff8028f810>] ? chrdev_open+0x162/0x181
 [<ffffffff8028f6ae>] ? chrdev_open+0x0/0x181
 [<ffffffff8028b6b3>] ? __dentry_open+0x16b/0x25c
 [<ffffffff8028b871>] ? nameidata_to_filp+0x41/0x53
 [<ffffffff80297a30>] ? do_filp_open+0x418/0x840
 [<ffffffff80273d48>] ? mod_zone_page_state+0xa6/0xb0
 [<ffffffff80446c28>] ? _spin_lock+0x9/0xc
 [<ffffffff8028854e>] ? kmem_cache_free+0xf0/0x100
 [<ffffffff80446f85>] page_fault+0x25/0x30
Code: 70 20 be 80 48 89 e5 41 54 53 49 89 fc 48 83 ec 20 65 48 8b 14 25 08 00 00 00 48 8d 1c 10 8b 83 04 14 00 00 3d 00
02 00 00 76 04 <0f> 0b eb fe 83 bb 00 14 00 00 20 74 1e 89 c0 48 83 c0 07 48 c1
RIP  [<ffffffff8020bfad>] __xen_mc_entry+0x2f/0xb6
 RSP <ffff8800ef11ef68>
---[ end trace 35f44aea30d5e8d0 ]---
Fixing recursive fault but reboot is needed!
------------[ cut here ]------------
kernel BUG at arch/x86/xen/multicalls.c:190!
invalid opcode: 0000 [#3] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 959, comm: bash Tainted: G      D    2.6.29.4 #1
RIP: e030:[<ffffffff8020bfad>]  [<ffffffff8020bfad>] __xen_mc_entry+0x2f/0xb6
RSP: e02b:ffff8800f0647ce0  EFLAGS: 00010006
RAX: 0000000000000218 RBX: ffff880001df7070 RCX: ffff8800ef0e7b18
RDX: ffff880081215000 RSI: ffff8800f0645a78 RDI: 0000000000000000
RBP: ffff8800f0647d10 R08: 0000000000987b41 R09: 000000008876db1a
R10: ffff8800ef11ebe8 R11: ffff8800ef11ec08 R12: 0000000000000000
R13: ffff8800ef0e7bf8 R14: ffff8800f0645a78 R15: ffff880001e03880
FS:  00007f0ac5d3a6f0(0000) GS:ffffffff80c25000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0ac5d3b315 CR3: 0000000000201000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 959, threadinfo ffff8800ef11e000, task ffff8800ef0e75e0)
Stack:
 ffff8800f0647d60 ffffffff8020dc08 ffff8800f0487030 0000000100000010
 0000000000000000 ffff8800f0645a78 ffff8800f0647d50 ffffffff8020ad96
 ffff880001dfd8c0 ffff880001e05e70 0000000000000001 ffff8800f062cec8
Call Trace:
Code: 70 20 be 80 48 89 e5 41 54 53 49 89 fc 48 83 ec 20 65 48 8b 14 25 08 00 00 00 48 8d 1c 10 8b 83 04 14 00 00 3d 00
02 00 00 76 04 <0f> 0b eb fe 83 bb 00 14 00 00 20 74 1e 89 c0 48 83 c0 07 48 c1
RIP  [<ffffffff8020bfad>] __xen_mc_entry+0x2f/0xb6
 RSP <ffff8800f0647ce0>
---[ end trace 35f44aea30d5e8d1 ]---
Fixing recursive fault but reboot is needed!






View attachment "config-linux-2.6.29.4" of type "text/plain" (44838 bytes)

Powered by blists - more mailing lists