linux-kernel - Re: Weird delay introduced in v6.16-rc only, possible regression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <69ea6e7a-a6f9-4199-9dc4-01b37092795f@gmx.com>
Date: Wed, 2 Jul 2025 07:37:02 +0930
From: Qu Wenruo <quwenruo.btrfs@....com>
To: Jens Axboe <axboe@...nel.dk>,
 "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
 LKML <linux-kernel@...r.kernel.org>
Subject: Re: Weird delay introduced in v6.16-rc only, possible regression



在 2025/7/2 02:00, Jens Axboe 写道:
> On 6/29/25 3:45 PM, Qu Wenruo wrote:
>> Hi,
>>
>> Recently I'm hitting a very weird delay when doing development inside a x86_64 VM.
>>
>> The dmesg shows the delay (10+ sec) between virtio blk and device-mapper:
>>
>> [    3.651377] virtio_blk virtio4: 10/0/0 default/read/poll queues
>> [    3.653075] virtio_scsi virtio2: 10/0/0 default/read/poll queues
>> [    3.670269] virtio_blk virtio4: [vda] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>> [    3.672096] scsi host6: Virtio SCSI HBA
>> [    3.708452]  vda: vda1 vda2
>> [    3.711073] virtio_blk virtio5: 10/0/0 default/read/poll queues
>> [    3.729879] virtio_blk virtio5: [vdb] 167772160 512-byte logical blocks (85.9 GB/80.0 GiB)
>> [    3.737535] virtio_blk virtio8: 10/0/0 default/read/poll queues
>> [    3.747045] virtio_blk virtio8: [vdc] 83886080 512-byte logical blocks (42.9 GB/40.0 GiB)
>> [   17.453833] device-mapper: uevent: version 1.0.3
>> [   17.455689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@...ts.linux.dev
>> :: performing fsck on '/dev/os/root'
>> /dev/mapper/os-root: clean, 240299/1048576 files, 3372218/4194304 blocks
>> :: mounting '/dev/os/root' on real root
>> [   17.871671] EXT4-fs (dm-0): mounted filesystem 00a85626-d289-4817-8183-ee828e221f76 r/w with ordered data mode. Quota mode: none.
>>
>> The VM is running kernel based on upstream commit 78f4e737a53e ("Merge tag 'for-6.16/dm-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm"), with a lot of extra btrfs patches.
>>
>>
>> The v6.15 kernel from Archlinux is totally fine without any delay.
>>
>> The v6.16-rc kernel may have some different configs, but the config is used for a long long time, way before v6.15, so it looks like it's something in the v6.16 cycle causing problems.
>>
>> I can definitely do a bisection, but any clue would be appreciated.
> 
> Probably a good idea to go ahead with a bisect to help pin it down.
> 

BTW, a little more digging shows it's the `udevadm settle` causing the 
long delay in the initramfs.

The rootfs is an ext4 on a LVM lv, so initramfs is required to mount the 
rootfs.

So it may not be the block/dm layer causing the problem.

Thanks,
Qu