| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20090105121409.GA5783@alice>
Date: Mon, 5 Jan 2009 13:14:09 +0100
From: Eric Sesterhenn <snakebyte@....de>
To: "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc: Kamalesh Babulal <kamalesh@...ux.vnet.ibm.com>,
linux-kernel@...r.kernel.org, josh@...edesktop.org,
dipankar@...ibm.com
Subject: Re: [BUG] NULL pointer deref with rcutorture
* Paul E. McKenney (paulmck@...ux.vnet.ibm.com) wrote:
> On Mon, Jan 05, 2009 at 12:38:55AM +0100, Eric Sesterhenn wrote:
> > hi,
> >
> > * Paul E. McKenney (paulmck@...ux.vnet.ibm.com) wrote:
> > > On Sun, Jan 04, 2009 at 03:57:26PM +0100, Eric Sesterhenn wrote:
> > > >
> > > > Just tell me what i need to do, I am not really familiar with ftrace.
> > > > I am only able to test 2.6.28-04980-gb58602a, since current -git is not
> > > > able to boot on this box :|
> > >
> > > Very cool!
> > >
> > > The idea is to have __call_rcu() in kernel/rcutree.c record the
> > > address of the callback (argument "head") and the function (argument
> > > "func"). In rcu_do_batch(), just before invoking list->func(list),
> > > also record the address of the callback ("list") and the function
> > > (again, "func").
> > >
> > > The new ftrace package has some mechanisms for doing this, but there is
> > > always the old-fashioned way of using printk(), for example in
> > > rcu_do_batch():
> > >
> > > prefetch(next);
> > > if (rcu_dump_callbacks)
> > > printk("rcu_head=%p, func=%p\n", list, func);
> > > list->func(list);
> > >
> > > Initialize rcu_dump_callbacks to zero, then use a small kernel module
> > > (or some such) to set it to one just before running your test.
> >
> > i did it via the ugly printk and captured it via netconsole
>
> Good stuff!!!
>
> > for the box with rcutree you can find the log here:
> > http://www.cccmz.de/~snakebyte/200.log
> > The interesting part seems start at 155.858923, thats when I load
> > the rcutorture module
>
> When I download this, I see an 82.100963 immediately followed by an
> 167.894017, no 155.858923. Do I have the right log?
>
> > with http://www.cccmz.de/~snakebyte/200_1.log i had another try,
> > actually saw a rcu_do_batch: rcu_head=d1907720, func=(null) in the traces,
> > box went into unresponsive mode afterwards
>
> Interesting. The original function is 0xd1902ad0, passed to __call_rcu()
> at 157.364214 -- would you be able to tell me what function that
> corresponds to? (Looks to me like a module, perhaps rcutorture?)
I did the log more than once, guess this was the time from the one
i uploaded earlier with the same name
> Whatever function it is, the rcu_head definitely got corrupted some
> time during the 8 milliseconds or so that the callback was waiting for
> a grace period. However, it did just fine being invoked several times
> beforehand -- this is a very popular RCU callback function, it appears.
>
> My guess is that the ensuing CPU 0 stall messages are due to CPU 0
> having a heart attack trying to execute at address 0.
Could the popular rcu function be registered by rcutorture, but when
we remove the module the callback is no longer valid? I can compile
a kernel just fine and with other stress tests i did not see any oops so
far.
> Are you swapping over NFS, or is the netpoll_send_udp() just garbage on
> the stack?
Swap is locally on a disc, nfs is configured but not used for swap.
> > And just in case it helps somehow: http://www.cccmz.de/~snakebyte/200_3.log
>
> Hmmm... The main recent change to kernel/rcutorture.c is the addition
> of a reboot notifier, so that rcutorture can shut itself down gracefully
> should the system go down while rcutorture is still running. This panic
> occurs in rcu_torture_stutter(), which is shown below:
>
> static int
> rcu_torture_stutter(void *arg)
> {
> VERBOSE_PRINTK_STRING("rcu_torture_stutter task started");
> do {
> schedule_timeout_interruptible(stutter * HZ);
> stutter_pause_test = 1;
> if (!kthread_should_stop() && !fullstop)
> schedule_timeout_interruptible(stutter * HZ);
> stutter_pause_test = 0;
> } while (!kthread_should_stop() && !fullstop);
> VERBOSE_PRINTK_STRING("rcu_torture_stutter task stopping");
> return 0;
> }
>
> I don't see any reason why this function would transfer control to
> location 0x60, though I do see a bug in the new shutdown code. I
> don't expect this to make any difference, but a patch is included
> nevertheless.
Sadly it doesnt.
> > for the one with rcupreemt you can find the log here:
> > http://www.cccmz.de/~snakebyte/201.log
> > rcutorture is loaded at 87.312399
>
> Hmmm... Offset 0x60 seems to be a common thread. This lets
> rcu_torture_ops off the hook, since it isn't that large.
>
> Anyway, could you give the attached patch a go, even though I cannot see
> how it would help? ;-)
This is with the patch on the preemt rcu box (the 201.logs)
[ 593.404728] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4
stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5
irqreader=1
[ 601.000124] BUG: unable to handle kernel paging request at d0af8a2d
[ 601.000431] IP: [<d0af8a2d>] 0xd0af8a2d
[ 601.000696] *pde = 0fbb4067 *pte = 00000000
[ 601.000946] Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
[ 601.001027] last sysfs file: /sys/block/ram9/range
[ 601.001027] Modules linked in: [last unloaded: rcutorture]
[ 601.001027]
[ 601.001027] Pid: 5021, comm: rcu_torture_fak Tainted: G W
(2.6.28-05692-g7d3b56b-dirty #169) System Name
[ 601.001027] EIP: 0060:[<d0af8a2d>] EFLAGS: 00010246 CPU: 0
[ 601.001027] EIP is at 0xd0af8a2d
[ 601.001027] EAX: 00000000 EBX: c98acfc4 ECX: c04f2c53 EDX: c98acf74
[ 601.001027] ESI: d0af8c0a EDI: 00000000 EBP: c98acfbc ESP: c98acfbc
[ 601.001027] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 601.001027] Process rcu_torture_fak (pid: 5021, ti=c98ac000
task=cbb64100 task.ti=c98ac000)
[ 601.001027] Stack:
[ 601.001027] c98acfd0 d0af8c89 74816929 000024a7 00000000 c98acfe0
c0136bcc c0136b8e
[ 601.001027] 00000000 00000000 c0103643 c9897ea4 00000000 00000000
00000000 00001301
[ 601.001027] 00008169
[ 601.001027] Call Trace:
[ 601.001027] [<c0136bcc>] ? kthread+0x3e/0x66
[ 601.001027] [<c0136b8e>] ? kthread+0x0/0x66
[ 601.001027] [<c0103643>] ? kernel_thread_helper+0x7/0x10
[ 601.001027] Code: Bad EIP value.
[ 601.001027] EIP: [<d0af8a2d>] 0xd0af8a2d SS:ESP 0068:c98acfbc
[ 601.001027] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 601.006764] BUG: unable to handle kernel paging request at d0af8a2d
[ 601.007023] IP: [<d0af8a2d>] 0xd0af8a2d
[ 601.007265] *pde = 0fbb4067 *pte = 00000000
[ 601.007511] Oops: 0000 [#2] PREEMPT DEBUG_PAGEALLOC
[ 601.007716] last sysfs file: /sys/block/ram9/range
[ 601.007716] Modules linked in: [last unloaded: rcutorture]
[ 601.007716]
[ 601.007716] Pid: 5020, comm: rcu_torture_fak Tainted: G D W
(2.6.28-05692-g7d3b56b-dirty #169) System Name
[ 601.007716] EIP: 0060:[<d0af8a2d>] EFLAGS: 00010296 CPU: 0
[ 601.007716] EIP is at 0xd0af8a2d
[ 601.007716] EAX: 00000000 EBX: c98adfc4 ECX: c04f2c53 EDX: c98adf74
[ 601.007716] ESI: d0af8c0a EDI: 00000000 EBP: c98adfbc ESP: c98adfbc
[ 601.007716] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 601.007716] Process rcu_torture_fak (pid: 5020, ti=c98ad000
task=cbb62700 task.ti=c98ad000)
[ 601.007716] Stack:
[ 601.007716] c98adfd0 d0af8c89 e5bc239c 000024ab 00000000 c98adfe0
c0136bcc c0136b8e
[ 601.007716] 00000000 00000000 c0103643 c9897ea4 00000000 00000000
00000000 00000201
[ 601.007716] 00000674
[ 601.007716] Call Trace:
[ 601.007716] [<c0136bcc>] ? kthread+0x3e/0x66
[ 601.007716] [<c0136b8e>] ? kthread+0x0/0x66
[ 601.007716] [<c0103643>] ? kernel_thread_helper+0x7/0x10
[ 601.007716] Code: Bad EIP value.
[ 601.007716] EIP: [<d0af8a2d>] 0xd0af8a2d SS:ESP 0068:c98adfbc
[ 601.007716] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 601.013281] BUG: unable to handle kernel paging request at d0af8a2d
[ 601.013542] IP: [<d0af8a2d>] 0xd0af8a2d
[ 601.013785] *pde = 0fbb4067 *pte = 00000000
[ 601.014032] Oops: 0000 [#3] PREEMPT DEBUG_PAGEALLOC
[ 601.014239] last sysfs file: /sys/block/ram9/range
[ 601.014239] Modules linked in: [last unloaded: rcutorture]
[ 601.014239]
[ 601.014239] Pid: 5019, comm: rcu_torture_fak Tainted: G D W
(2.6.28-05692-g7d3b56b-dirty #169) System Name
[ 601.014239] EIP: 0060:[<d0af8a2d>] EFLAGS: 00010286 CPU: 0
[ 601.014239] EIP is at 0xd0af8a2d
[ 601.014239] EAX: 00000000 EBX: c98b2fc4 ECX: c04f2c53 EDX: c98b2f74
[ 601.014239] ESI: d0af8c0a EDI: 00000000 EBP: c98b2fbc ESP: c98b2fbc
[ 601.014239] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 601.014239] Process rcu_torture_fak (pid: 5019, ti=c98b2000
task=c9804e00 task.ti=c98b2000)
[ 601.014239] Stack:
[ 601.014239] c98b2fd0 d0af8c89 61a1df8f 000024a5 00000000 c98b2fe0
c0136bcc c0136b8e
[ 601.014239] 00000000 00000000 c0103643 c9897ea4 00000000 00000000
00000000 000007bf
[ 601.014239] 00000012
[ 601.014239] Call Trace:
[ 601.014239] [<c0136bcc>] ? kthread+0x3e/0x66
[ 601.014239] [<c0136b8e>] ? kthread+0x0/0x66
[ 601.014239] [<c0103643>] ? kernel_thread_helper+0x7/0x10
[ 601.014239] Code: Bad EIP value.
[ 601.014239] EIP: [<d0af8a2d>] 0xd0af8a2d SS:ESP 0068:c98b2fbc
[ 601.014239] ---[ end trace 4eaa2a86a8e2da22 ]---
then i added the rcutree patch (Fix rcutree grace-period-latency bug on
small systems) and got
[ 65.968568] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4
stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5
irqreader=1
[ 68.872082] BUG: unable to handle kernel NULL pointer dereference at
(null)
[ 68.872455] IP: [<(null)>] (null)
[ 68.872653] *pde = 00000000
[ 68.872849] Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
[ 68.873027] last sysfs file: /sys/block/ram9/range
[ 68.873027] Modules linked in: [last unloaded: rcutorture]
[ 68.873027]
[ 68.873027] Pid: 5023, comm: bash Tainted: G W
(2.6.28-05692-g7d3b56b-dirty #169) System Name
[ 68.873027] EIP: 0060:[<00000000>] EFLAGS: 00010246 CPU: 0
[ 68.873027] EIP is at 0x0
[ 68.873027] EAX: d0afd0b8 EBX: 00000000 ECX: c01612a6 EDX: 00000006
[ 68.873027] ESI: d0afd0b8 EDI: 0000001c EBP: c0b03fe0 ESP: c0b03fd4
[ 68.873027] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[ 68.873027] Process bash (pid: 5023, ti=c0b03000 task=c98d1a00
task.ti=ce824000)
[ 68.873027] Stack:
[ 68.873027] c01612ad 00000200 00000001 c0b03ff8 c012aa97 0000000a
ce824e98 00000046
[ 68.873027] c012aa28 ce824ea8 c01042c2
[ 68.873027] Call Trace:
[ 68.873027] [<c01612ad>] ? rcu_process_callbacks+0x65/0x79
[ 68.873027] [<c012aa97>] ? __do_softirq+0x6f/0xf6
[ 68.873027] [<c012aa28>] ? __do_softirq+0x0/0xf6
[ 68.873027] <IRQ> <0> [<c012a9a5>] ? irq_exit+0x40/0x7c
[ 68.873027] [<c0110ce1>] ? smp_apic_timer_interrupt+0x68/0x73
[ 68.873027] [<c0103521>] ? apic_timer_interrupt+0x2d/0x34
[ 68.873027] [<c07a024c>] ? _spin_lock+0x56/0x5d
[ 68.873027] [<c07a0000>] ? _write_unlock_bh+0x28/0x37
[ 68.873027] [<c017ad50>] ? do_wp_page+0x38b/0x453
[ 68.873027] [<c017be9e>] ? handle_mm_fault+0x4e8/0x54d
[ 68.873027] [<c0139da3>] ? down_read_trylock+0x3d/0x47
[ 68.873027] [<c07a25d3>] ? do_page_fault+0x2dc/0x601
[ 68.873027] [<c07a22f7>] ? do_page_fault+0x0/0x601
[ 68.873027] [<c07a0bd7>] ? error_code+0x6f/0x74
[ 68.873027] Code: Bad EIP value.
[ 68.873027] EIP: [<00000000>] 0x0 SS:ESP 0068:c0b03fd4
[ 68.880266] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 68.880394] Kernel panic - not syncing: Fatal exception in interrupt
Tried again without the rcutree patch and got this one,
guess its just a timing thing wheter the NULL pointer deref or the other
appears
[ 99.643517] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4
stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5
irqreader=1
[ 104.149086] BUG: unable to handle kernel NULL pointer dereference at
(null)
[ 104.149418] IP: [<(null)>] (null)
[ 104.149615] *pde = 00000000
[ 104.149813] Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
[ 104.150029] last sysfs file: /sys/block/ram9/range
[ 104.150029] Modules linked in: [last unloaded: rcutorture]
[ 104.150029]
[ 104.150029] Pid: 5030, comm: udevd Tainted: G W
(2.6.28-05692-g7d3b56b-dirty #169) System Name
[ 104.150029] EIP: 0060:[<00000000>] EFLAGS: 00010246 CPU: 0
[ 104.150029] EIP is at 0x0
[ 104.150029] EAX: d0afcc38 EBX: 00000000 ECX: c01612a6 EDX: 00000006
[ 104.150029] ESI: d0afcc38 EDI: 0000001c EBP: c0b03fe0 ESP: c0b03fd4
[ 104.150029] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[ 104.150029] Process udevd (pid: 5030, ti=c0b03000 task=c98cb400
task.ti=c98b1000)
[ 104.150029] Stack:
[ 104.150029] c01612ad 00000200 00000001 c0b03ff8 c012aa97 0000000a
c98b1e40 00000046
[ 104.150029] c012aa28 c98b1e50 c01042c2
[ 104.150029] Call Trace:
[ 104.150029] [<c01612ad>] ? rcu_process_callbacks+0x65/0x79
[ 104.150029] [<c012aa97>] ? __do_softirq+0x6f/0xf6
[ 104.150029] [<c012aa28>] ? __do_softirq+0x0/0xf6
[ 104.150029] <IRQ> <0> [<c012a9a5>] ? irq_exit+0x40/0x7c
[ 104.150029] [<c0110ce1>] ? smp_apic_timer_interrupt+0x68/0x73
[ 104.150029] [<c0103521>] ? apic_timer_interrupt+0x2d/0x34
[ 104.150029] [<c016e63f>] ? find_lock_page+0x5f/0x63
[ 104.150029] [<c0170122>] ? filemap_fault+0x9f/0x391
[ 104.150029] [<c017a6e8>] ? __do_fault+0x40/0x31d
[ 104.150029] [<c017bc30>] ? handle_mm_fault+0x27a/0x54d
[ 104.150029] [<c0139da3>] ? down_read_trylock+0x3d/0x47
[ 104.150029] [<c07a25d3>] ? do_page_fault+0x2dc/0x601
[ 104.150029] [<c07a22f7>] ? do_page_fault+0x0/0x601
[ 104.150029] [<c07a0bd7>] ? error_code+0x6f/0x74
[ 104.150029] Code: Bad EIP value.
[ 104.150029] EIP: [<00000000>] 0x0 SS:ESP 0068:c0b03fd4
[ 104.157313] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 104.157441] Kernel panic - not syncing: Fatal exception in interrupt
I enabled classic rcu and pulled current -git (without both patches) and got these:
[ 67.424425] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4
stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5
irqreader=1
[ 70.205083] BUG: unable to handle kernel NULL pointer dereference at
(null)
[ 70.205416] IP: [<(null)>] (null)
[ 70.205605] *pde = 00000000
[ 70.205798] Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
[ 70.206026] last sysfs file: /sys/block/ram9/range
[ 70.206026] Modules linked in: [last unloaded: rcutorture]
[ 70.206026]
[ 70.206026] Pid: 1726, comm: udevd Tainted: G W
(2.6.28-05716-gfe0bdec #170) System Name
[ 70.206026] EIP: 0060:[<00000000>] EFLAGS: 00010202 CPU: 0
[ 70.206026] EIP is at 0x0
[ 70.206026] EAX: d0afc018 EBX: c10697c0 ECX: c0161905 EDX: d0afc018
[ 70.206026] ESI: 00000000 EDI: 00000001 EBP: c0b0bfd8 ESP: c0b0bfc4
[ 70.206026] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[ 70.206026] Process udevd (pid: 1726, ti=c0b0b000 task=cee10000
task.ti=ce80b000)
[ 70.206026] Stack:
[ 70.206026] c016197c 00000202 00000200 00000001 0000001c c0b0bfe0
c0161a00 c0b0bff8
[ 70.206026] c012acca 0000000a ce80bec4 00000046 c012ac5b ce80bed4
c01042c2
[ 70.206026] Call Trace:
[ 70.206026] [<c016197c>] ? __rcu_process_callbacks+0x141/0x1ae
[ 70.206026] [<c0161a00>] ? rcu_process_callbacks+0x17/0x28
[ 70.206026] [<c012acca>] ? __do_softirq+0x6f/0x100
[ 70.206026] [<c012ac5b>] ? __do_softirq+0x0/0x100
[ 70.206026] <IRQ> <0> [<c012abdd>] ? irq_exit+0x40/0x77
[ 70.206026] [<c0110ce9>] ? smp_apic_timer_interrupt+0x68/0x73
[ 70.206026] [<c0103521>] ? apic_timer_interrupt+0x2d/0x34
[ 70.206026] [<c014007b>] ? tick_do_periodic_broadcast+0xf/0x3a
[ 70.206026] [<c018f325>] ? kmem_cache_free+0xbd/0xc5
[ 70.206026] [<c019a45d>] ? putname+0x28/0x33
[ 70.206026] [<c019a45d>] ? putname+0x28/0x33
[ 70.206026] [<c019b7a0>] ? do_rmdir+0xba/0xc3
[ 70.206026] [<c04e8b50>] ? trace_hardirqs_on_thunk+0xc/0x10
[ 70.206026] [<c0102ecf>] ? sysenter_exit+0xf/0x16
[ 70.206026] [<c01448f1>] ? trace_hardirqs_on_caller+0x17/0x15a
[ 70.206026] [<c019b7f2>] ? sys_rmdir+0x15/0x17
[ 70.206026] [<c0102ea1>] ? sysenter_do_call+0x12/0x31
[ 70.206026] Code: Bad EIP value.
[ 70.206026] EIP: [<00000000>] 0x0 SS:ESP 0068:c0b0bfc4
[ 70.213814] ---[ end trace 4eaa2a86a8e2da22 ]---
[ 70.213934] Kernel panic - not syncing: Fatal exception in interrupt
On the other box with tree rcu and the rcutorture patch (200.logs):
[ 168.408126] BUG: unable to handle kernel NULL pointer dereference at
00000029
[ 168.408433] IP: [<d1902ebf>] 0xd1902ebf
[ 168.408614] *pde = 00000000
[ 168.408778] Oops: 0000 [#1] DEBUG_PAGEALLOC
[ 168.408978] last sysfs file: /sys/block/sda/size
[ 168.409164] Modules linked in: nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc ipv6 fuse unix [last unloaded: rcutorture]
[ 168.410099]
[ 168.410197] Pid: 4154, comm: rcu_torture_rea Not tainted (2.6.28 #83)
[ 168.410332] EIP: 0060:[<d1902ebf>] EFLAGS: 00010202 CPU: 0
[ 168.410516] EIP is at 0xd1902ebf
[ 168.410628] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000
[ 168.410760] ESI: 00000000 EDI: 00000000 EBP: c7d52fd0 ESP: c7d52f84
[ 168.410992] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 168.411120] Process rcu_torture_rea (pid: 4154, ti=c7d52000
task=c7b73710 task.ti=c7d52000)
[ 168.411302] Stack:
[ 168.411398] 00000002 00000000 c0936a24 c7b48f8c 00000001 d1903040
00000000 c0936000
[ 168.411981] d1902ff7 5f756372 74726f74 5f657275 00616572 0000103a
2e11c1af 00000dd1
[ 168.412021] 00000000 d1902e50 00000000 c7d52fe0 c013d81a c013d7e0
00000000 00000000
[ 168.412021] Call Trace:
[ 168.412021] [<c013d81a>] ? kthread+0x3a/0x70
[ 168.412021] [<c013d7e0>] ? kthread+0x0/0x70
[ 168.412021] [<c0103cf3>] ? kernel_thread_helper+0x7/0x14
[ 168.412021] Code: 28 85 f6 74 1c 8d 45 bc c7 45 c8 40 30 90 d1 c7 45
cc 00 00 00 00 e8 91 03 83 ee 90 a1 00 5e 90 d1 8b 1d a4 59 90 d1 85 db
74 12 <8b> 48 28 85 c9 74 0b 8b 55 bc 85 d2 0f 84 19 01 00 00 8d b4 26
[ 168.412021] EIP: [<d1902ebf>] 0xd1902ebf SS:ESP 0068:c7d52f84
[ 168.418864] ---[ end trace 4bbd74c2c853213b ]---
[ 168.429775] __call_rcu: rcu_head=c3f47f24, func=c01b5a70
[ 168.429958] __call_rcu: rcu_head=c7bbecac, func=c0143580
[ 168.430129] __call_rcu: rcu_head=c7dbeea0, func=c01a6ae0
[ 168.431826] __call_rcu: rcu_head=c7de9328, func=c01a6ae0
[ 168.432089] rcu_do_batch: rcu_head=c3f47f24, func=c01b5a70
[ 168.432869] __call_rcu: rcu_head=c994fe18, func=c01b9c20
[ 168.436064] rcu_do_batch: rcu_head=c7bbecac, func=c0143580
[ 168.436226] rcu_do_batch: rcu_head=c7dbeea0, func=c01a6ae0
[ 168.444046] rcu_do_batch: rcu_head=c7de9328, func=c01a6ae0
[ 168.444193] rcu_do_batch: rcu_head=c994fe18, func=c01b9c20
[ 168.445866] BUG: unable to handle kernel NULL pointer dereference at
00000029
[ 168.446126] IP: [<d1902ebf>] 0xd1902ebf
[ 168.446315] *pde = 00000000
[ 168.446488] Oops: 0000 [#2] DEBUG_PAGEALLOC
[ 168.446692] last sysfs file: /sys/block/sda/size
[ 168.446883] Modules linked in: nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc ipv6 fuse unix [last unloaded: rcutorture]
[ 168.447818]
[ 168.447916] Pid: 4153, comm: rcu_torture_rea Tainted: G D
(2.6.28 #83)
[ 168.448022] EIP: 0060:[<d1902ebf>] EFLAGS: 00010202 CPU: 0
[ 168.448022] EIP is at 0xd1902ebf
[ 168.448022] EAX: 00000001 EBX: 00000001 ECX: 00000000 EDX: 00000000
[ 168.448022] ESI: 00000000 EDI: 00000000 EBP: c7b48fd0 ESP: c7b48f84
[ 168.448022] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 168.448022] Process rcu_torture_rea (pid: 4153, ti=c7b48000
task=c7b5c710 task.ti=c7b48000)
[ 168.448022] Stack:
[ 168.448022] 00000002 00000000 c7d52f8c c7de5f74 00000001 d1903040
00000000 c0936000
[ 168.448022] d1902ff7 5f756372 74726f74 5f657275 00616572 00001039
b87c4524 00000ca3
[ 168.448022] 00000000 d1902e50 00000000 c7b48fe0 c013d81a c013d7e0
00000000 00000000
[ 168.448022] Call Trace:
[ 168.448022] [<c013d81a>] ? kthread+0x3a/0x70
[ 168.448022] [<c013d7e0>] ? kthread+0x0/0x70
[ 168.448022] [<c0103cf3>] ? kernel_thread_helper+0x7/0x14
[ 168.448022] Code: 28 85 f6 74 1c 8d 45 bc c7 45 c8 40 30 90 d1 c7 45
cc 00 00 00 00 e8 91 03 83 ee 90 a1 00 5e 90 d1 8b 1d a4 5[ 168.457352]
BUG: unable to handle kernel NULL pointer dereference at 0000001d
[ 168.457603] IP: [<d1903163>] 0xd1903163
[ 168.457776] *pde = 00000000
[ 168.457942] Oops: 0000 [#3] DEBUG_PAGEALLOC
[ 168.458141] last sysfs file: /sys/block/sda/size
[ 168.458323] Modules linked in: nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc ipv6 fuse unix [last unloaded: rcutorture]
[ 168.459254]
[ 168.459350] Pid: 4151, comm: rcu_torture_fak Tainted: G D
(2.6.28 #83)
[ 168.459523] EIP: 0060:[<d1903163>] EFLAGS: 00010287 CPU: 0
[ 168.459706] EIP is at 0xd1903163
[ 168.459818] EAX: 00000001 EBX: c7dcafc0 ECX: 35502ab8 EDX: 00000021
[ 168.459949] ESI: cccccccd EDI: 00000000 EBP: c7dcafd0 ESP: c7dcafb8
[ 168.460080] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 168.460262] Process rcu_torture_fak (pid: 4151, ti=c7dca000
task=c7dcb710 task.ti=c7dca000)
[ 168.460442] Stack:
[ 168.460485] c7dbadd8 00000000 01f30e05 00002699 00000000 d19030f0
c7dcafe0 c013d81a
[ 168.460485] c013d7e0 00000000 00000000 c0103cf3 c7ca9e70 00000000
00000000 00000000
[ 168.460485] 20df0493 0a860000
[ 168.460485] Call Trace:
[ 168.460485] [<c013d81a>] ? kthread+0x3a/0x70
[ 168.460485] [<c013d7e0>] ? kthread+0x0/0x70
[ 168.460485] [<c0103cf3>] ? kernel_thread_helper+0x7/0x14
[ 168.460485] Code: c1 f7 e6 c1 ea 03 8d 14 92 01 d2 29 d1 8d 41 01 e8
23 83 ca ee 89 d8 e8 1c f7 ff ff 25 ff 03 00 00 e8 f2 c7 99 ee a1 00 5e
90 d1 <ff> 50 1c e8 75 fc ff ff e8 d0 a4 83 ee 85 c0 8d b6 00 00 00 00
[ 168.460485] EIP: [<d1903163>] 0xd1903163 SS:ESP 0068:c7dcafb8
[ 168.467531] ---[ end trace 4bbd74c2c853213b ]---
[ 168.476855] BUG: unable to handle kernel NULL pointer dereference at
00000004
[ 168.477134] IP: [<d1903233>] 0xd1903233
[ 168.477322] *pde = 00000000
[ 168.477485] Oops: 0002 [#4] DEBUG_PAGEALLOC
[ 168.477680] last sysfs file: /sys/block/sda/size
[ 168.477857] Modules linked in: nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc ipv6 fuse unix [last unloaded: rcutorture]
[ 168.478774]
[ 168.478871] Pid: 4147, comm: rcu_torture_wri Tainted: G D
(2.6.28 #83)
[ 168.479045] EIP: 0060:[<d1903233>] EFLAGS: 00010202 CPU: 0
[ 168.479228] EIP is at 0xd1903233
[ 168.479340] EAX: 00000000 EBX: d19073fc ECX: 00000001 EDX: 00000000
[ 168.479472] ESI: 0000000a EDI: 00000000 EBP: c7e05fd0 ESP: c7e05fc0
[ 168.479604] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 168.479787] Process rcu_torture_wri (pid: 4147, ti=c7e05000
task=c994e710 task.ti=c7e05000)
[ 168.479968] Stack:
[ 168.480015] 00000000 00000000 00000000 d19031f0 c7e05fe0 c013d81a
c013d7e0 00000000
[ 168.480015] 00000000 c0103cf3 c7ca9e70 00000000 00000000 00000000
7574726f 725f6572
[ 168.480015] Call Trace:
[ 168.480015] [<c013d81a>] ? kthread+0x3a/0x70
[ 168.480015] [<c013d7e0>] ? kthread+0x0/0x70
[ 168.480015] [<c0103cf3>] ? kernel_thread_helper+0x7/0x14
[ 168.480015] Code: 01 00 00 ba 13 00 00 00 be 0a 00 00 00 a1 00 20 83
c0 e8 a1 21 82 ee e9 a0 00 00 00 8d 74 26 00 ff 05 6c 78 90 d1 8b 43 04
8b 13 <89> 42 04 89 10 b8 cc 59 90 d1 89 1b 89 5b 04 e8 19 a3 ca ee 83
[ 168.480015] EIP: [<d1903233>] 0xd1903233 SS:ESP 0068:c7e05fc0
[ 168.486680] ---[ end trace 4bbd74c2c853213b ]---
[ 168.486798] Kernel panic - not syncing: Fatal exception in interrupt
9 90 d1 85 db
74 12 <8b> 48 28 85 c9 74 0b 8b 55 bc 85 d2 0f 84 19 01 00 00 8d b4 26
[ 168.448022] EIP: [<d1902ebf>] 0xd1902ebf SS:ESP 0068:c7b48f84
[ 168.456684] ---[ end trace 4bbd74c2c853213b ]---
Greetings, Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists