linux-kernel - Re: [regression/bisected] 4.19 cycle boot time IO stalls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <178d6adc-2a79-99dc-daa8-fb0ed9f0afd1@kernel.dk>
Date:   Wed, 5 Sep 2018 07:39:08 -0600
From:   Jens Axboe <axboe@...nel.dk>
To:     Mike Galbraith <efault@....de>, LKML <linux-kernel@...r.kernel.org>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        "James E.J. Bottomley" <jejb@...ux.vnet.ibm.com>,
        "Martin K. Petersen" <martin.petersen@...cle.com>
Subject: Re: [regression/bisected] 4.19 cycle boot time IO stalls

On 9/5/18 2:55 AM, Mike Galbraith wrote:
> Greetings,
> 
> I've been seeing $subject, decided to take the time to try to bisect
> the little bugger.  The hangs are not 100% repeatable, and while
> bisection with a 5 boot go/nogo threshold seemed to go smoothly, it
> ended up fingering a merge commit (sigh).
> 
> Box has an SSD (unused only by windows 10 box came with) and 3 spinning
> rust buckets that I normally use with BFQ via a udev rule, but CFQ does
> the same, so scheduler is seemingly irrelevant.  However, in 7 crash
> dumps, all of which look about like the data below the bisect log,
> there is something relevant, namely the hung 'tlp' task (powersaving
> script of some sort for laptops according to the package description). 
> That knob twiddling script is present/hung in all, making me a tad
> suspicious, and indeed, testing with the final (bad) kernel, all I have
> to do to eliminate hangs is to remove the 'tlp' package.  Verified via
> remove, 5 boots work fine, reinstall, 2 of 5 hang, remove again, 10 of
> 10 work fine, reinstall, 2 in a row hang.
> 
> Seems pretty certain that tlp script is what inspires bug to raise its
> ugly head.  WRT bisection result itself, munged merge seems far less
> likely than a false negative having knocked bisection off course.
> 
> 72f02ba66bd83b54054da20eae550123de84da6f is the first bad commit

I bet it's the host busy change from Ming, which I already
reported as being the culprit for another test failure I had. For
some reason it's not merged yet, nudge nudge Martin. You can test
by reverting:

commit 328728630d9f2bf14b82ca30b5e47489beefe361
Author: Ming Lei <ming.lei@...hat.com>
Date:   Sun Jun 24 22:03:27 2018 +0800

    scsi: core: avoid host-wide host_busy counter for scsi_mq

which was slightly modified by 265d59aacbce, so you'll want to
yank that one first.

BTW, that suse email for me hasn't worked in 12 years :-)

-- 
Jens Axboe