lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Thu, 8 Dec 2016 14:14:01 +0100 (CET) From: Renaud Mariana <rmariana@...ine.net> To: Sven Joachim <svenjoac@....de> Cc: Andreas Dilger <adilger@...ger.ca>, Ext4 Developers List <linux-ext4@...r.kernel.org>, debian-dpkg@...ts.debian.org Subject: Re: HUGE slowdown when doing dpkg with ext4 over nbd > Add debian-dpkg mailing list to CC. > > On Dec 7, 2016, at 10:58 AM, Andreas Dilger <adilger@...ger.ca> wrote: >> >> On Dec 7, 2016, at 2:52 AM, Renaud Mariana <rmariana@...ine.net> wrote: >>> >>> Here are my answers, hope it will help solve this issue, thanks. >>> >>> Recap: >>> dpkg kibana on ext4 over a nbd device takes 10 minutes >>> with xfs it's only 30s. >>> with ext4 no extends only 30s. >>> >>> >>> kernels : >>> 4.5.7 has this issue as older kernel like 4.4.34 >>> The issue is also when nbd client & server run on same host >>> >>> >>> How small are the files? >>> here is the histogram of file sizes : http://pasteboard.co/6HC3nKyk2.png >>> We can see 5000 files around 512 Bytes. >> >> Definitely there is no value to use fallocate for 512-byte files, or any >> of the files that can be written in a single write() syscall. I'd expect >> any reasonable tool to be using a write buffer of at least 2-4MB these >> days to get good performance, so writes below the buffer size shouldn't >> use fallocate() at all. It should be noted that the latest dpkg (1.18.15) only uses fallocate for files which are at least 16 KiB in size[1], so it would be nice if Renaud could recheck with that version, or cherry-pick the patch into whatever version he uses. >>> dpkg using fallocate() ? >>> Yes, there are 16044 calls by the same process >>> what are these uninitialized extents ? >> >> Uninitialized extents are preallocated ranges of a file on disk that will >> read back as zero, but are not necessarily zero-filled at allocation time. >> For large files that are written randomly (or written slowly and may have >> contention from other writers) fallocate() + uninitialized extents will >> preallocate the space for the file so that it is (largely) contiguous on >> disk and overwrites will not result in random block allocation. Cheers, Sven 1. https://anonscm.debian.org/cgit/dpkg/dpkg.git/commit/?id=a971ad91437af8880cad4703695dcf12ee45959b Thanks Sven for this link This slowdown is related to an old issue : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=824636 but amplified by the nbd layer (tcp massive zero writes) I quote the fix: Ftrace showed that the delay is caused by sb_issue_zeroout() in ext4_ext_zeroout() called by ext4_ext_convert_to_initialized(). This call chain is initially triggered by fallocate(). this can be disabled using the max_zeroout parameter. echo 0 > /sys/fs/ext4/sdb2/extent_max_zeroout_kb in my case echo 3 > /sys/fs/ext4/nbd0/extent_max_zeroout_kb seems sufficient for dpkg and may be other intensive fallocate applications ? dpkg time is then reduced to an acceptable time of 20s. Is there any recommendations / warnings with this value extent_max_zeroout_kb = 3 ? Cheers Renaud -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists