linux-kernel - Re: [fuse-devel] Horrible mmap write performance (kernel writeback issue?)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:	Wed, 16 Mar 2016 10:44:06 +0100
From:	Miklos Szeredi <miklos@...redi.hu>
To:	Jakob Unterwurzacher <jakobunt@...il.com>
Cc:	fuse-devel <fuse-devel@...ts.sourceforge.net>,
	Linux-Fsdevel <linux-fsdevel@...r.kernel.org>,
	Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [fuse-devel] Horrible mmap write performance (kernel writeback issue?)

On Tue, Mar 15, 2016 at 8:55 AM, Jakob Unterwurzacher
<jakobunt@...il.com> wrote:
> Just for anybody finding this thread: This still happens in v4.4, it
> just took longer to trigger.
>
> I have posted more details to linux-kernel (copy-pasted below),
> http://thread.gmane.org/gmane.linux.kernel/2132944

Okay, so you can reproduce this relatively quickly.   Can you try "git
bisect" to find exactly which commit is responsible?

Thanks,
Miklos

>
> -------- copy of the email to linux-kernel -------------------
>
> 2016-01-22 21:10:59
>
> I have noticed an annoying regression that was introduced in 4.2 and is
> still there in 4.4. mmap writes to FUSE filesystems are throttled down
> to basically zero.
>
> Reproducer: https://github.com/rfjakob/mmapwrite , testing against encfs:
>
> $ mmapwrite /tmp/encfs-mnt/foo
>   1 .................................................. 107.01 MB/s
>   2 .................................................. 101.98 MB/s
> [...]
>  68 .................................................. 106.79 MB/s
>  69 .................................................. 105.09 MB/s
>  70 ..................................................   2.02 MB/s
>  71 ..................................................   1.77 MB/s
>  72 ..................................................   0.42 MB/s
>  73 .................................... (hangs)
>
> I have tested kernels from 4.0 and this seems to have been introduced in
> 4.2:
>
> 4.0 ....... 140MB/s permanent
> 4.1 ....... 140MB/s permanent
> 4.2 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~5GB
> 4.3 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~1.5GB
> 4.4-rc4 ... 100MB/s at the start, slowly ramps down, 0.3MB/s after ~2GB
> 4.4 ....... 100MB/s at the start, sudden slowdown after ~3GB
>
> Is there a way to disable the throttling? Or at least exempt FUSE until
> there is a proper fix?
>
> Thanks,
> Jakob
>
>
>
>
> On 17.12.2015 00:26, Jakob Unterwurzacher wrote:
>> This seems to be fixed in v4.4-rc5-18-gedb42dc. mmap writes now proceed
>> at solid 100MB/s with full CPU saturation.
>>
>> Thanks,
>> Jakob
>>
>> On Mon, Dec 14, 2015 at 9:06 AM, Jakob Unterwurzacher
>> <jakobunt@...il.com <mailto:jakobunt@...il.com>> wrote:
>>
>>     I am the developer of https://github.com/rfjakob/gocryptfs (an
>>     encrypted overlay filesystem like EncFS)
>>     and have ported xfstests over for regression testing
>>     ( https://github.com/rfjakob/fuse-xfstests ).
>>
>>     xfstests generic/074 is how I noticed that mmap write performance
>>     plummeted when Fedora upgraded my kernel to 4.2.5.
>>     It used to complete in 10 minutes and now it will probably take days.
>>     I am on kernel 4.4-rc1 now and still seeing the same issue.
>>
>>     It looks like the kernel at some point the kernel writeback
>>     mechanism gets stuck (or throttles?)
>>
>>     Testing against encfs:
>>
>>     ./mmapwrite /tmp/e2/foo          #
>>     https://github.com/rfjakob/mmapwrite .
>>     ....................................................................................................
>>     98.91 MB/s
>>     ....................................................................................................
>>     93.74 MB/s
>>     ....................................................................................................
>>     103.89 MB/s
>>     ....................................................................................................
>>     100.20 MB/s
>>     ....................................................................................................
>>     104.03 MB/s
>>     ....................................................................................................
>>     98.06 MB/s
>>     ....................................................................................................
>>     10.17 MB/s
>>     ....................................................................................................
>>     9.50 MB/s
>>     .................................. (hangs)
>>
>>     At this point no write requests are submitted to encfs and the
>>     mmapwrite process is
>>     stuck in the kernel in balance_dirty_pages.isra.22.
>>
>>     Bisecting this will be a pain, I would appreciate any suggestions.
>>
>>     Note that this seems to affect every FUSE filesystem, also ntfs-3g.
>>
>>     Best regards,
>>     Jakob
>>
>>
>
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
> --
> fuse-devel mailing list
> To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel