linux-kernel - RE: scsi-mq V2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <94D0CD8314A33A4D9D801C0FE68B402958B9742E@G9W0745.americas.hpqcorp.net>
Date:	Thu, 10 Jul 2014 21:10:57 +0000
From:	"Elliott, Robert (Server Storage)" <Elliott@...com>
To:	Jeff Moyer <jmoyer@...hat.com>
CC:	Christoph Hellwig <hch@...radead.org>,
	Jens Axboe <axboe@...nel.dk>,
	"dgilbert@...erlog.com" <dgilbert@...erlog.com>,
	James Bottomley <James.Bottomley@...senPartnership.com>,
	Bart Van Assche <bvanassche@...ionio.com>,
	Benjamin LaHaise <bcrl@...ck.org>,
	"linux-scsi@...r.kernel.org" <linux-scsi@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: scsi-mq V2



> -----Original Message-----
> From: Jeff Moyer [mailto:jmoyer@...hat.com]
> Sent: Thursday, 10 July, 2014 2:14 PM
> To: Elliott, Robert (Server Storage)
> Cc: Christoph Hellwig; Jens Axboe; dgilbert@...erlog.com; James Bottomley;
> Bart Van Assche; Benjamin LaHaise; linux-scsi@...r.kernel.org; linux-
> kernel@...r.kernel.org
> Subject: Re: scsi-mq V2
> 
> "Elliott, Robert (Server Storage)" <Elliott@...com> writes:
> 
> >> -----Original Message-----
> >> From: Christoph Hellwig [mailto:hch@...radead.org]
> >> Sent: Thursday, 10 July, 2014 11:15 AM
> >> To: Elliott, Robert (Server Storage)
> >> Cc: Jens Axboe; dgilbert@...erlog.com; James Bottomley; Bart Van Assche;
> >> Benjamin LaHaise; linux-scsi@...r.kernel.org; linux-kernel@...r.kernel.org
> >> Subject: Re: scsi-mq V2
> >>
> >> On Thu, Jul 10, 2014 at 09:04:22AM -0700, Christoph Hellwig wrote:
> >> > It's starting to look weird.  I'll prepare another two bisect branches
> >> > around some MM changes, which seems the only other possible candidate.
> >>
> >> I've pushed out scsi-mq.3-bisect-3
> >
> > Good.
> >
> >> and scsi-mq.3-bisect-4 for you.
> >
> > Bad.
> >
> > Note: I had to apply the vdso2c.h patch to build this -rc3 based kernel:
> > diff --git a/arch/x86/vdso/vdso2c.h b/arch/x86/vdso/vdso2c.h
> > index df95a2f..11b65d4 100644
> > --- a/arch/x86/vdso/vdso2c.h
> > +++ b/arch/x86/vdso/vdso2c.h
> > @@ -93,6 +93,9 @@ static void BITSFUNC(copy_section)(struct
> BITSFUNC(fake_sections) *out,
> > 	uint64_t flags = GET_LE(&in->sh_flags);
> >
> > 	bool copy = flags & SHF_ALLOC &&
> > +		(GET_LE(&in->sh_size) ||
> > +		(GET_LE(&in->sh_type) != SHT_RELA &&
> > +		GET_LE(&in->sh_type) != SHT_REL)) &&
> > 		strcmp(name, ".altinstructions") &&
> > 		strcmp(name, ".altinstr_replacement");
> >
> > Results: fio started OK, getting 900K IOPS, but ^C led to 0 IOPS and
> > an fio hang, with one CPU (CPU 0) stuck in io_submit loops.
> 

I added some prints in aio_setup_ring and  ioctx_alloc and
rebooted.  This time it took much longer to hit the problem.  It 
survived dozens of ^Cs.  Running a few minutes, though, IOPS 
eventually dropped.  So, sometimes it happens immediately,
sometimes it takes time to develop.

I will rerun bisect-1 -2 and -3 for longer times to increase
confidence that they didn't just appear good.

On this bisect-4 run, as IOPS started to drop from 900K to 40K, 
I ran perf top when it was at 700K.  You can see io_submit times
creeping up.

  4.30%  [kernel]            [k] do_io_submit
  4.29%  [kernel]            [k] _raw_spin_lock_irqsave
  3.88%  libaio.so.1.0.1     [.] io_submit
  3.55%  [kernel]            [k] system_call
  3.34%  [kernel]            [k] put_compound_page
  3.11%  [kernel]            [k] io_submit_one
  3.06%  [kernel]            [k] system_call_after_swapgs
  2.89%  [kernel]            [k] copy_user_generic_string
  2.45%  [kernel]            [k] lookup_ioctx
  2.16%  [kernel]            [k] apic_timer_interrupt
  2.00%  [kernel]            [k] _raw_spin_lock
  1.97%  [scsi_debug]        [k] sdebug_q_cmd_hrt_complete
  1.84%  [kernel]            [k] __get_page_tail
  1.74%  [kernel]            [k] do_blockdev_direct_IO
  1.68%  [kernel]            [k] blk_flush_plug_list
  1.41%  [kernel]            [k] _raw_spin_unlock_irqrestore
  1.24%  [scsi_debug]        [k] schedule_resp

finally settling like before:
 14.15%  [kernel]                    [k] do_io_submit
 13.61%  libaio.so.1.0.1             [.] io_submit
 11.81%  [kernel]                    [k] system_call
 10.11%  [kernel]                    [k] system_call_after_swapgs
  8.59%  [kernel]                    [k] io_submit_one
  8.56%  [kernel]                    [k] copy_user_generic_string
  7.96%  [kernel]                    [k] lookup_ioctx
  5.33%  [kernel]                    [k] blk_flush_plug_list
  3.11%  [kernel]                    [k] blk_finish_plug
  2.84%  [kernel]                    [k] sysret_check
  2.63%  fio                         [.] fio_libaio_commit
  2.27%  [kernel]                    [k] blk_start_plug
  1.17%  [kernel]                    [k] SyS_io_submit

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/