linux-kernel - Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20131119213051.GP11434@dastard>
Date:	Wed, 20 Nov 2013 08:30:51 +1100
From:	Dave Chinner <david@...morbit.com>
To:	Jens Axboe <axboe@...nel.dk>
Cc:	Christoph Hellwig <hch@...radead.org>, linux-kernel@...r.kernel.org
Subject: Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on
 virtio devices

On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote:
> On Tue, Nov 19 2013, Christoph Hellwig wrote:
> > On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote:
> > > I have no idea if it's related to the above hang, but either way
> > > breaking iostat is a major regression....
> > 
> > Both of them are most likely due to the conversion of virtio_blk
> > to the blk-mq code.
> > 
> > I've not seen the hang in my heavy xfstests testing, but that was a
> > slightly different codebase than what finally got in, so I'll try
> > to reproduce it once I get some spare QA cycles.
> 
> Thanks! Dave, if you still have it in that state, can you dump the
> contents of /sys/block/<devs>/mq/ for the device/devices that are hung?

# find /sys/block/vdb/mq -type f -print -exec cat {} \;
/sys/block/vdb/mq/0/run
1313835
/sys/block/vdb/mq/0/cpu0/completed
546857 207203
/sys/block/vdb/mq/0/cpu0/rq_list
CTX pending:
/sys/block/vdb/mq/0/cpu0/merged
0
/sys/block/vdb/mq/0/cpu0/dispatched
257733 496352
/sys/block/vdb/mq/0/cpu1/completed
547714 200741
/sys/block/vdb/mq/0/cpu1/rq_list
CTX pending:
/sys/block/vdb/mq/0/cpu1/merged
0
/sys/block/vdb/mq/0/cpu1/dispatched
262114 486381
/sys/block/vdb/mq/0/tags
nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
nr_free=0, nr_reserved=1
  cpu00: nr_free=0
  cpu01: nr_free=0
/sys/block/vdb/mq/0/ipi_redirect
0
/sys/block/vdb/mq/0/queued
1384795
/sys/block/vdb/mq/0/dispatched
       0        194596
       1        1022691
       2        68540
       4        10138
       8        7686
      16        10234
      32        0
      64        0
     128        0
     256        0
/sys/block/vdb/mq/0/pending
HCTX pending:
#

It looks like it's hung because of this:

/sys/block/vdb/mq/0/tags
nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
nr_free=0, nr_reserved=1
  cpu00: nr_free=0
  cpu01: nr_free=0

No free tags in teh pool, nor any free tags in the per-cpu
magazines. Perhaps there is a leak somewhere?

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/