linux-kernel - RE: NVMe Poll CQ on timeout

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Tue, 15 May 2018 13:58:32 +0000
From:   Bharat Kumar Gogada <bharatku@...inx.com>
To:     Keith Busch <keith.busch@...ux.intel.com>
CC:     "linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "keith.busch@...el.com" <keith.busch@...el.com>,
        "axboe@...com" <axboe@...com>, "hch@....de" <hch@....de>
Subject: RE: NVMe Poll CQ on timeout

> I recall we did observe issues like this when legacy interrupts were used, so
> the driver does try to use MSI/MSIx if possible.
> 
> The nvme_timeout() is called from the block layer when the driver didn't
> provide a completion within the timeout (default is 30 seconds for IO,
> 60 seconds for admin).
> 
> This message you're seeing means the device did indeed post a completion
> queue entry for the timed out command, but the driver believes it was never
> notified via interrupt to check the completion queue.
> 
> This means either one of two things happened: the interrupt was raised prior
> to the completion queue entry being written, or the interrupt was never
> raised in the first place.
> 
> It might be possible to determine which if you can read the values from
> /proc/irq/<irq#>/spurious and see if the "last_unhandled" aligns with the
> expected completion time.
> 
Thanks keith. We are seeing the condition for transactions greater than 256 KB.
We did try increase IO timeout to 60sec but we still see issue. 

We do see spurious interrupts as following:
count 53224
unhandled 15890
last_unhandled 4294917520 ms

If there are spurious interrupts, isn't the EP handler called more times 
and this might help EP driver to process pending completions. (Because as per the code in 4.14 EP
driver isn't checking any interrupt status register, it starts processing completion queues immediately
in interrupt handler)

If we have spurious why do we still see completion polled ?

Regards,
Bharat


> > > Hi,
> > >
> > > We are testing NVMe cards on ARM64 platform, the card uses legacy
> > > interrupts.
> > > Intermittently we are hitting following case in drivers/nvme/host/pci.c
> > >        /*
> > >          * Did we miss an interrupt?
> > >          */
> > >         if (__nvme_poll(nvmeq, req->tag)) {
> > >                 dev_warn(dev->ctrl.device,
> > >                          "I/O %d QID %d timeout, completion polled\n",
> > >                          req->tag, nvmeq->qid);
> > >                 return BLK_EH_HANDLED;
> > >         }
> > >
> > > Can anyone tell when does nvme_timeout gets invoked ?
> > > What does "Did we miss an interrupt mean" ? Does it mean host
> > > missing to service a interrupt raised by EP card ?
> > >
> > > Regards,
> > > Bharat