linux-kernel - Re: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150901091535.GA21329@altium.nl>
Date:	Tue, 1 Sep 2015 11:15:35 +0200
From:	Dick Streefland <dick.streefland@...ium.nl>
To:	Erik Cumps <erik.cumps@...turnus.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Unexpected slow block device write IO performance compared to
 uncached, unsynced direct IO using stock kernels

On Thursday 2015-06-18 09:35, Erik Cumps wrote:
| On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <erik.cumps@...turnus.com> wrote:
| > The context is a 16 GB 32-bit intel debian workstation, using an ext4
| > filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
| > recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
| > machines. The host system (so not the virual machines) shows sporadic
| > extremely slow write performance (around 4 megabytes per second).
| > However, if we use the debian 3.2.0 kernel this problem does not
| > manifest itself.
[...]
| Actually, it is the *synchronous*, direct IO that matches the expected
| raw write performance of the device.
| 
| The "regular IO" test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
|     dd if=ramdisk_file of=test_file bs=1M count=100
| 
| The direct IO test is doing roughly this:
| 
|     echo 3 > /proc/sys/vm/drop_caches
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
|     dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

I'm seeing this as well here on a number of new Dell Optiplex 7020
machines and one older Optiplex 780, all with 8GB RAM and running
Ubuntu 14.04 in 32-bit mode.

A simple dd command shows the problem:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 7.02392 s, 1.5 MB/s

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest oflag=sync,direct
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.535397 s, 19.6 MB/s

In my case, running:

  echo 3 > /proc/sys/vm/drop_caches

will restore the normal speed for a limited time:

$ dd bs=1M count=10 if=/dev/zero of=/tmp/ddtest
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0123759 s, 847 MB/s

There is an old Ubuntu bug report describing the same issue:

  https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294

-- 
Dick
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/