linux-kernel - RE: Boot regression (was "Re: [PATCH] genhd: Do not hold event lock when scheduling workqueue elements")

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <MWHPR03MB2669B4FFC900983C8C357A77BF440@MWHPR03MB2669.namprd03.prod.outlook.com>
Date:   Fri, 10 Feb 2017 14:49:28 +0000
From:   Dexuan Cui <decui@...rosoft.com>
To:     "hch@....de" <hch@....de>
CC:     Jens Axboe <axboe@...nel.dk>,
        Bart Van Assche <Bart.VanAssche@...disk.com>,
        "hare@...e.com" <hare@...e.com>, "hare@...e.de" <hare@...e.de>,
        "Martin K. Petersen" <martin.petersen@...cle.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "jth@...nel.org" <jth@...nel.org>, Christoph Hellwig <hch@....de>,
        Nick Meier <Nick.Meier@...rosoft.com>,
        "Alex Ng (LIS)" <alexng@...rosoft.com>,
        Long Li <longli@...rosoft.com>,
        "Adrian Suhov (Cloudbase Solutions SRL)" <v-adsuho@...rosoft.com>,
        "Chris Valean (Cloudbase Solutions SRL)" <v-chvale@...rosoft.com>
Subject: RE: Boot regression (was "Re: [PATCH] genhd: Do not hold event lock
 when scheduling workqueue elements")

> From: hch@....de [mailto:hch@....de]
>
> Hi Dexuan,
>
> I've spent some time with the logs and looking over the code and
> couldn't find any smoking gun.  I start to wonder if it might just
> be a timing issue?

Thanks a lot for digging into the issue!

I'm not sure if it's a timing issue, but the fact is that the VM hangs every time
due to the issue, when it tries to boot up.
We tried both UP VM and SMP VM, and they always failed to boot up.

> Can you try one or two things for me:

I did the following tests with
1 CPU,
IDE controller 0 with the VM's image,
IDE controller 1 with no disk,
no SCSI controller, and
no CD drive.

>  1) run with the blk-mq I/O path for scsi by either enabling it a boot /
>     module load time with the scsi_mod.use_blk_mq=Y option, or at compile
>     time by enabling the CONFIG_SCSI_MQ_DEFAULT option.  If that fails
>     with the commit a blk-mq run before the commit would also be useful.

With the scsi_mod.use_blk_mq=Y kernel parameter, the VM can boot fine!
And with =N (the default), the VM hits the issue.

>  2) if possible run a VM config without the virtual CD-ROM drive -
>     a lot of the scsi log chatter is about handling timeouts on the
>     CD drive, so that might be able to isolate issues a bit better.
Please see the attached logs.
I got the good.log by reverting the bad commit. It's so long so I only
reserve the first part of the log.

> Note that I'll be offline from this afternoon European time until Sunday
> night as I'm out in the mountains at a lodge without internet access,
> but this issue will be my priority once back.

Have a nice trip! :-)

Thanks,
-- Dexuan

Download attachment "good-bad-scsi-logs.zip" of type "application/x-zip-compressed" (80096 bytes)