linux-ext4 - Re: unexpected sync delays in dpkg for small pre-allocated files on ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <578648BE.8020708@siemens.com>
Date:	Wed, 13 Jul 2016 15:57:18 +0200
From:	Gernot Hillier <gernot.hillier@...mens.com>
To:	"Theodore Ts'o" <tytso@....edu>,
	Dave Chinner <david@...morbit.com>, linux-scsi@...r.kernel.org,
	MPT-FusionLinux.pdl@...adcom.com, linux-ext4@...r.kernel.org,
	sathya.prakash@...adcom.com, chaitra.basappa@...adcom.com,
	suganath-prabu.subramani@...adcom.com
Cc:	SCHILD HENNING Z003BPMR <henning.schild@...mens.com>
Subject: Re: unexpected sync delays in dpkg for small pre-allocated files on
 ext4

Dear Ted,

On 01.06.2016 16:12, Theodore Ts'o wrote:
> On Wed, Jun 01, 2016 at 03:17:14PM +0200, Gernot Hillier wrote:
>> I repeated the discussed tests and found comparable results on this machine:
>>
>> - 3 seconds dpkg install time on ext3 vs. 80 seconds for ext4
>>   on same partition for same package
>> - 40 ms for fallocate+write+sync_file_range for writing a few bytes
>> - 15 ms for write+fdatasync vs. 45 ms for BLKZEROOUT on raw device
>>
>> So this seems to be not bound to one specific disk+controller setup, but
>> it still can't be a common problem affecting many people as then we
>> would see more reports about it...

Sorry for 2nd follow-up and the long delay (caused by vacation +
illness), but we went over the whole discussion again and can now likely
confirm your assumption of a bad SCSI WRITE SAME implementation on our
hardware.

We found that by issuing "echo 0 >
/sys/devices/.../scsi_disk/0:0:0:0/max_write_same_blocks" we can also
fix our issue, reducing dpkg installation time for e.g. linux-headers
from minutes to seconds.

Did you already have time to look into my old btrace from June, 2nd, did
it help somehow?

If not, please find updated btraces from two machines, for slow and fast
case, on each one below.

Does this provide enough details to blacklist our devices for SCSI WRITE
SAME?

> OK, so let's try to get common baseline, shall we?  I'm using as my
> test package:
> 
> http://debug.mirrors.debian.org/debian-debug/pool/main/e/e2fsprogs/e2fsprogs-dbgsym_1.43-2_amd64.deb

This time, I decided to use a test package with a bit more files to make
the issue more obvious:

http://snapshot.debian.org/archive/debian/20160712T042309Z/pool/main/m/manpages/manpages_4.06-1_all.deb

# md5sum manpages_4.06-1_all.deb
251aadf9a0117cac8248343a9f09d74b  manpages_4.06-1_all.deb

On machine 1, a Supermicro X8DTH board with Xeon E5520 CPU, equipped
with a SEAGATE ST973451SS disk connected to an LSI SAS2008 controller
(mpt3sas.ko):

(If you do this at home, please note that this will overwrite your local
manpages, so you need to go back to original package afterwards!)

# /usr/bin/time dpkg --no-triggers --force-depends --unpack
manpages_4.06-1_all.deb
[...]
0.13user 0.07system 0:03.38elapsed 6%CPU (0avgtext+0avgdata
15132maxresident)k
0inputs+4040outputs (0major+6066minor)pagefaults 0swaps

# echo 0 >
/sys/devices/pci0000:00/0000:00:09.0/0000:05:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/max_write_same_blocks

# /usr/bin/time dpkg --no-triggers --force-depends --unpack
manpages_4.06-1_all.deb
[...]
0.11user 0.08system 0:00.55elapsed 35%CPU (0avgtext+0avgdata
15168maxresident)k
0inputs+4056outputs (0major+6219minor)pagefaults 0swaps

So disabling SCSI WRITE SAME reduced installation time for a package
with ~300 files from 3.5 s to 1 s.

On machine 2, a Supermicro X9DR3-F board with Xeon E5-2620, SEAGATE
ST300MM0006 disk connected to an Intel C606 SAS controller, default
installation time for manpages_4.06-1_all.deb was 9s, dropping to 1s
after "echo 0 > .../max_write_same_blocks".

For "real" packages like linux-headers, disabling SCSI WRITE SAME
reduced installation time from minutes to seconds.

(For sure, I repeated those measurements several times with switching
back and forth between both cases, rebooting etc.)

I followed your instructions, assured that btrace output ends up on
tmpfs and recorded four new btraces. As installation of manpages.deb
results in a 350k btrace (40k compressed), I decided to provide URLs
instead of attachments this time:

https://github.com/gernot-h/slow-dpkg/blob/master/btrace-machine1-write_same.out.bz2?raw=true
-> btrace for default (slow) case on machine1 (i.e. after reboot or
   "echo 65535 > .../max_write_same_blocks"

https://github.com/gernot-h/slow-dpkg/blob/master/btrace-machine1-no_write_same.out.bz2?raw=true
-> btrace for fast case on machine1 (i.e. after "echo 0 >
   .../max_write_same_blocks"

https://github.com/gernot-h/slow-dpkg/blob/master/btrace-machine2-write_same.out.bz2?raw=true
-> btrace for default (slow) case on machine2 (i.e. after reboot or
   "echo 65535 > .../max_write_same_blocks"

https://github.com/gernot-h/slow-dpkg/blob/master/btrace-machine2-no_write_same.out.bz2?raw=true
-> btrace for fast case on machine1 (i.e. after "echo 0 >
   .../max_write_same_blocks"

Thanks again and please let me know if you need any additional traces,
test results, measurements, bug report, etc.!

-- 
With kind regards,
Gernot Hillier

Siemens AG, Corporate Technology
Corporate Competence Center Embedded Linux

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html