linux-ext4 - Re: HUGE slowdown when doing dpkg with ext4 over nbd

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <452CC006-0197-45C0-BA00-CDA0F8132965@dilger.ca>
Date:   Wed, 7 Dec 2016 11:12:32 -0700
From:   Andreas Dilger <adilger@...ger.ca>
To:     Renaud Mariana <rmariana@...ine.net>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: HUGE slowdown when doing dpkg with ext4 over nbd

On Dec 7, 2016, at 10:58 AM, Andreas Dilger <adilger@...ger.ca> wrote:
> 
> On Dec 7, 2016, at 2:52 AM, Renaud Mariana <rmariana@...ine.net> wrote:
>> 
>> Here are my answers, hope it will help solve this issue, thanks.
>> 
>> Recap:
>> dpkg kibana on ext4 over a nbd device takes 10 minutes
>> with xfs it's only 30s.
>> with ext4 no extends only 30s.
>> 
>> 
>> kernels :
>> 4.5.7 has this issue as older kernel like 4.4.34
>> The issue is also when nbd client & server run on same host
>> 
>> 
>> How small are the files?
>> here is the histogram of file sizes : http://pasteboard.co/6HC3nKyk2.png
>> We can see 5000 files around 512 Bytes.
> 
> Definitely there is no value to use fallocate for 512-byte files, or any
> of the files that can be written in a single write() syscall.  I'd expect
> any reasonable tool to be using a write buffer of at least 2-4MB these
> days to get good performance, so writes below the buffer size shouldn't
> use fallocate() at all.
> 
>> dpkg using fallocate() ?
>> Yes, there are 16044 calls by the same process
>> what are these uninitialized extents ?
> 
> Uninitialized extents are preallocated ranges of a file on disk that will
> read back as zero, but are not necessarily zero-filled at allocation time.
> For large files that are written randomly (or written slowly and may have
> contention from other writers) fallocate() + uninitialized extents will
> preallocate the space for the file so that it is (largely) contiguous on
> disk and overwrites will not result in random block allocation.

It would probably be worthwhile to file a ticket in the Debian BTS:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=dpkg

with your information, so that there is something to submit a patch against.

Cheers, Andreas






Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)