linux-kernel - Re: Strange block/scsi/workqueue issue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1302627097.2661.25.camel@dolmen>
Date:	Tue, 12 Apr 2011 17:51:37 +0100
From:	Steven Whitehouse <swhiteho@...hat.com>
To:	James Bottomley <James.Bottomley@...senPartnership.com>
Cc:	Tejun Heo <tj@...nel.org>, linux-kernel@...r.kernel.org,
	Jens Axboe <jaxboe@...ionio.com>
Subject: Re: Strange block/scsi/workqueue issue

Hi,

On Tue, 2011-04-12 at 11:27 -0500, James Bottomley wrote:
> On Tue, 2011-04-12 at 17:04 +0100, Steven Whitehouse wrote:
> > Hi,
> > 
> > On Tue, 2011-04-12 at 10:14 -0500, James Bottomley wrote:
> > > On Tue, 2011-04-12 at 15:06 +0100, Steven Whitehouse wrote:
> > > > Messages attached - getting closer I think, but still not quite there.
> > > > It is very odd that so far (at least I've seen no other reports) that I
> > > > seem to be the only one who hits this. I wonder what is different about
> > > > my kernel/hardware...
> > > 
> > > 
> > > > csi: killing requests for dead queue
> > > > scsi: killing requests for dead queue
> > > > BUG: unable to handle kernel scsi: killing requests for dead queue
> > > > NULL pointer dereference at 0000000000000028
> > > > IP: [<ffffffff813b34ed>] blk_peek_request+0x1ad/0x220
> > > 
> > > Hmm, I don't see where this is ... it's a null pointer deref offset by
> > > 0x28 bytes ... but I can't see where ... can you resolve this to a line?
> > > 
> > > James
> > > 
> > > 
> > 
> > So assuming I've calculated this correctly, it should be just about....
> > 
> > /home/steve/linux-2.6/include/linux/rcupdate.h:677
> >     6de8:       a8 08                   test   $0x8,%al
> >     6dea:       0f 84 65 ff ff ff       je     6d55 <blk_peek_request+0xf5>
> >     6df0:       e8 00 00 00 00          callq  6df5 <blk_peek_request+0x195>
> >                         6df1: R_X86_64_PC32     preempt_schedule-0x4
> >     6df5:       e9 5b ff ff ff          jmpq   6d55 <blk_peek_request+0xf5>
> >     6dfa:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
> > __elv_next_request():
> > /home/steve/linux-2.6/block/blk.h:65
> >                 if (!list_empty(&q->queue_head)) {
> >                         rq = list_entry_rq(q->queue_head.next);
> >                         return rq;
> >                 }
> > 
> >                 if (!q->elevator->ops->elevator_dispatch_fn(q, 0))
> >     6e00:       49 8b 44 24 18          mov    0x18(%r12),%rax
> >     6e05:       31 f6                   xor    %esi,%esi
> >     6e07:       48 8b 00                mov    (%rax),%rax
> >     6e0a:       4c 89 e7                mov    %r12,%rdi                              <---- here!
> >     6e0d:       ff 50 28                callq  *0x28(%rax)
> >     6e10:       85 c0                   test   %eax,%eax
> >     6e12:       0f 85 f4 fe ff ff       jne    6d0c <blk_peek_request+0xac>
> >     6e18:       45 31 ed                xor    %r13d,%r13d
> >     6e1b:       e9 70 ff ff ff          jmpq   6d90 <blk_peek_request+0x130>
> > blk_peek_request():
> > /home/steve/linux-2.6/block/blk-core.c:1912
> >                                  * so that we don't add it again
> >                                  */
> > 
> > The offset I got was 6e0d, but I guess thats the start of the following instruction
> > by the time the fault is logged. Also I've attached my current .config just in case
> > that might be a help is reproducing it,
> 
> Good work!  it's simpler if you compile with -g ... then the gdb line
> command will just tell you all of this.  However, it seems clear that
> elevator->ops is NULL (elevator_dispatch_fn is 0x28 offset into that in
> a 64 bit system).
> 
> That's because blk_cleanup_queue called elevator_exit, so we need an out
> in the peek case.
> 
> So unwrapping the onion, this is the next level
> 
> James
> 
> ---
> 
> Index: linux-2.6/block/blk.h
> ===================================================================
> --- linux-2.6.orig/block/blk.h
> +++ linux-2.6/block/blk.h
> @@ -62,7 +62,7 @@ static inline struct request *__elv_next
>  			return rq;
>  		}
>  
> -		if (!q->elevator->ops->elevator_dispatch_fn(q, 0))
> +		if (!q->elevator->ops || !q->elevator->ops->elevator_dispatch_fn(q, 0))
>  			return NULL;
>  	}
>  }
> 
> 

Still not quite there, but looking more hopeful now,

Steve.


View attachment "james4.txt" of type "text/plain" (39826 bytes)