linux-kernel - Re: Strange block/scsi/workqueue issue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <1302617212.2661.14.camel@dolmen>
Date:	Tue, 12 Apr 2011 15:06:52 +0100
From:	Steven Whitehouse <swhiteho@...hat.com>
To:	James Bottomley <James.Bottomley@...senPartnership.com>
Cc:	Tejun Heo <tj@...nel.org>, linux-kernel@...r.kernel.org,
	Jens Axboe <jaxboe@...ionio.com>
Subject: Re: Strange block/scsi/workqueue issue

Hi,

On Tue, 2011-04-12 at 08:42 -0500, James Bottomley wrote:
> On Tue, 2011-04-12 at 09:42 +0100, Steven Whitehouse wrote:
> > #2 On its own:
> 
> Right, that's how it's supposed to be applied
> 
> > scsi 0:0:32:0: Enclosure         DP       BACKPLANE        1.07 PQ: 0 ANSI: 5
> > scsi 0:2:0:0: Direct-Access     DELL     PERC 6/i         1.22 PQ: 0 ANSI: 5
> > scsi 0:2:1:0: Direct-Access     DELL     PERC 6/i         1.22 PQ: 0 ANSI: 5
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> > IP: [<ffffffff810ab4d3>] __queue_work+0x403/0x460
> > PGD 0 
> > Oops: 0000 [#1] PREEMPT SMP 
> > last sysfs file: 
> > CPU 0 
> > Modules linked in:
> > 
> > Pid: 3, comm: ksoftirqd/0 Not tainted 2.6.39-rc2+ #190 Dell Inc. PowerEdge R710/0N047H
> > RIP: 0010:[<ffffffff810ab4d3>]  [<ffffffff810ab4d3>] __queue_work+0x403/0x460
> > RSP: 0018:ffff8800c90abb20  EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: ffff8800c56a8c58 RCX: ffff8800c56a8c60
> > RDX: ffff8800c56a8c60 RSI: 0000000000000000 RDI: ffff8800cb00e388
> > RBP: ffff8800c90abb70 R08: 0000000000000900 R09: 0000000000000000
> > R10: 0000000000000001 R11: 0000000000000001 R12: ffff8800cb00e340
> > R13: 0000000000000000 R14: ffff8800cb1d6700 R15: 0000000000000000
> > FS:  0000000000000000(0000) GS:ffff8800cb000000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000008 CR3: 0000000001c33000 CR4: 00000000000006f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process ksoftirqd/0 (pid: 3, threadinfo ffff8800c90aa000, task ffff8800c90a80c0)
> > Stack:
> >  ffff8800c90abb90 0000000000000086 0000000000000000 0000000000000086
> >  ffff8800c90a80c0 ffff8800c92ebe00 ffff8800c56a8c58 ffff8800c88bac28
> >  ffff8800c90abc30 ffff8800c565c080 ffff8800c90abb80 ffffffff810ab59d
> > Call Trace:
> >  [<ffffffff810ab59d>] queue_work_on+0x1d/0x30
> >  [<ffffffff810ac749>] queue_work+0x29/0x60
> >  [<ffffffff810ac865>] queue_delayed_work+0x25/0x30
> >  [<ffffffff813aff75>] __blk_run_queue+0xb5/0x110
> 
> OK, this is because the patch is incomplete ... it needs an additional
> reference which I missed in run queue ... can you apply the update patch
> below on top?
> 
> Actually, there are probably more places I missed, sigh.
> 
> >  [<ffffffff813b04ed>] blk_run_queue+0x2d/0x50
> >  [<ffffffff8147715a>] scsi_run_queue+0xea/0x3f0
> >  [<ffffffff814710cf>] ? __scsi_put_command+0x5f/0xa0
> >  [<ffffffff814793cd>] scsi_next_command+0x3d/0x60
> >  [<ffffffff81479602>] scsi_io_completion+0x1b2/0x630
> >  [<ffffffff81470cd7>] scsi_finish_command+0xc7/0x130
> >  [<ffffffff8147936f>] scsi_softirq_done+0x13f/0x160
> >  [<ffffffff813b8252>] blk_done_softirq+0xa2/0xc0
> >  [<ffffffff810970ac>] __do_softirq+0xdc/0x290
> >  [<ffffffff81097395>] run_ksoftirqd+0x135/0x250
> >  [<ffffffff81097260>] ? __do_softirq+0x290/0x290
> >  [<ffffffff810b1f16>] kthread+0xa6/0xb0
> >  [<ffffffff816870e4>] kernel_thread_helper+0x4/0x10
> >  [<ffffffff8107fe1f>] ? finish_task_switch+0x6f/0x110
> >  [<ffffffff8167ec56>] ? _raw_spin_unlock_irq+0x46/0x70
> >  [<ffffffff8167f018>] ? retint_restore_args+0x13/0x13
> >  [<ffffffff810b1e70>] ? __init_kthread_worker+0x70/0x70
> >  [<ffffffff816870e0>] ? gs_change+0x13/0x13
> > Code: 8b 44 24 48 49 8d 7c 24 48 48 83 e8 08 31 f6 eb 27 0f 1f 80 00 00 00 00 48 8b 00 49 89 c0 41 80 e0 00 a8 04 48 89 f0 49 0f 45 c0 
> >  8b 40 08 f6 00 10 74 14 48 8d 41 f8 48 8b 48 08 48 8d 50 08 
> > RIP  [<ffffffff810ab4d3>] __queue_work+0x403/0x460
> >  RSP <ffff8800c90abb20>
> > CR2: 0000000000000008
> > ---[ end trace 52fce2e9afb9bf59 ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> > 
> > These are both on the same base kernel as yesterday (i.e. Linus tree
> > from this time yesterday morning) plus a few GFS2 patches, but they
> > only affect the GFS2 module which is not loaded.
> > 
> > I don't do anything special to trigger the problem - it happens of its
> > own accord during normal boot. The initscripts are those from f14.
> 
> That's useful ... at least it's not hard to reproduce ... it's just I
> don't see anything like this in my boot sequence.
> 
> James
> 
> ---
> 
> Index: linux-2.6/block/blk-core.c
> ===================================================================
> --- linux-2.6.orig/block/blk-core.c
> +++ linux-2.6/block/blk-core.c
> @@ -328,7 +328,7 @@ void __blk_run_queue(struct request_queu
>  	if (!force_kblockd && !queue_flag_test_and_set(QUEUE_FLAG_REENTER, q)) {
>  		q->request_fn(q);
>  		queue_flag_clear(QUEUE_FLAG_REENTER, q);
> -	} else
> +	} else if (!blk_get_queue(q))
>  		queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
>  }
>  EXPORT_SYMBOL(__blk_run_queue);
> 
> 

Messages attached - getting closer I think, but still not quite there.
It is very odd that so far (at least I've seen no other reports) that I
seem to be the only one who hits this. I wonder what is different about
my kernel/hardware...

Steve.



View attachment "james3.txt" of type "text/plain" (27081 bytes)