linux-kernel - Re: Changes to Linux/SCSI target mode infrastructure for v2.6.28

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <1228265226.6229.127.camel@haakon2.linux-iscsi.org>
Date:	Tue, 02 Dec 2008 16:47:06 -0800
From:	"Nicholas A. Bellinger" <nab@...ux-iscsi.org>
To:	Mike Anderson <andmike@...ux.vnet.ibm.com>
Cc:	Tejun Heo <tj@...nel.org>,
	FUJITA Tomonori <fujita.tomonori@....ntt.co.jp>,
	Mike Christie <michaelc@...wisc.edu>,
	Christoph Hellwig <hch@....de>,
	James Bottomley <James.Bottomley@...senPartnership.com>,
	Andrew Morton <akpm@...l.org>,
	Alan Stern <stern@...land.harvard.edu>,
	Hannes Reinecke <hare@...e.de>,
	Boaz Harrosh <bharrosh@...asas.com>,
	Jens Axboe <jens.axboe@...cle.com>,
	linux-scsi <linux-scsi@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>,
	"Linux-iSCSI.org Target Dev" 
	<linux-iscsi-target-dev@...glegroups.com>
Subject: Re: Changes to Linux/SCSI target mode infrastructure for v2.6.28

On Tue, 2008-12-02 at 15:18 -0800, Nicholas A. Bellinger wrote:
> On Tue, 2008-12-02 at 01:13 -0800, Mike Anderson wrote:
> > Nicholas A. Bellinger <nab@...ux-iscsi.org> wrote:
> > > On Tue, 2008-12-02 at 00:30 -0800, Nicholas A. Bellinger wrote:
> > > > On Mon, 2008-12-01 at 22:40 -0800, Mike Anderson wrote:
> > > > > Nicholas A. Bellinger <nab@...ux-iscsi.org> wrote:
> > > > > > On Tue, 2008-12-02 at 13:18 +0900, Tejun Heo wrote:
> > > > > > > 
> > > > > > > >>> The other one is a BUG_ON in blk/blk-timeout.c:177 in blk_add_timeout()
> > > > > > > >>> that happens after a few hundred MB of READ_10 traffic, which also
> > > > > > > >>> appears to pass through elv_dequeue_request() at some point:
> > > > > > > >>>
> > > > > > > >>> http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-2.png
> > > > > > > >>> http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-4.png
> > > > > > >
> > > > > > > Hmmm... this means blk_add_timer() is being called after the request
> > > > > > > is already completed.
> > > > > 
> > > > > or is it possible since elv_dequeue_request BUG_ON check of queuelist did
> > > > > not trigger a request is on the queuelist with a timeout_list not empty.
> > > > > 
> > > > > It would be interesting for a debug run to change the
> > > > > "BUG_ON(!list_empty(&req->timeout_list))" in blk_add_timer to print out
> > > > > the cmd_flags plus req->atomic_flags and also add a
> > > > > "BUG_ON(!list_empty(&rq->timeout_list))" to elv_insert to ensure a request
> > > > > is never added to the queuelist with a timeout_list not empty.
> > > > > 
> > > > 
> > > > Ok, so blk_dump_rq_flags() is now being called in
> > > > block/blk-timeout.c:blk_add_timer() for the case
> > > > BUG_ON(list_empty(&req->timeout_list)) case:
> > > > 
> > > > http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-6.png
> > > > 
> > > > Hmm, the outputted "sector " range is definately is bogus, as the only
> > > > READ_10 that have been sent are at LBA offset 0 for 8 * 512 byte sectors
> > > > for the partition table during Open/iSCSI LUN scanning.
> > > > 
> > > 
> > > Also, the following code from block/blk-core.c:blk_dump_rq_flags():
> > > 
> > >         if (blk_pc_request(rq)) {
> > >                 printk(KERN_INFO "  cdb: ");
> > >                 for (bit = 0; bit < BLK_MAX_CDB; bit++)
> > >                         printk("%02x ", rq->cmd[bit]);
> > >                 printk("\n");
> > >         }
> > > 
> > > is not printing out the copied CDB in struct request->cmd[], which makes
> > > me think the struct request->cmd_flags (that blk_pc_request() is
> > > checking) are also bogus when blk_add_timer() is being called..
> > > 
> > Based on the output from 2.6.28-rc6-oops-6.png the flags value of 82c21
> > looks like
> > (__REQ_ALLOCED,__REQ_ELVPRIV,__REQ_DONTPREP,__REQ_STARTED,__REQ_SORTED,__REQ_RW),
> > but it is late so someone maybe should check my shift counts.
> > 
> > The type is also REQ_TYPE_FS so the blk_pc cdb: info will not be printed.
> > 
> > I will talk to you more in the morning.
> > 
> 
> Ok, after double checking the original patch again, I noticed that
> struct request was getting released with __blk_put_request() from the
> struct request->end_io() caller context in pscsi_req_done(), and again
> after the original target mode CDB was acknowledged from the fabric in
> pscsi_free_task() with blk_put_request()..  Not good.
> 
> Anyways, I removed the extra blk_put_request() in pscsi_free_task() and
> am running some badblocks tests now.  Things are obviously looking much
> better.
> 

Ok, passing badblocks tests now and looking stable on v2.6.28-rc7 using
Tejun's patch, here the commit diff for the LIO-Core v3.0 PSCSI changes:

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=1b14b5e1fc9a7074322b1015bb86eca2a8ef4560

Now the question becomes, how can we get rid of scsi_req_map_sg() usage
(and struct bio in between) all together..?

--nab


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/