lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:	Mon, 28 Jul 2014 14:51:39 -0700
From:	Tim Chen <tim.c.chen@...ux.intel.com>
To:	Al Viro <viro@...iv.linux.org.uk>
Cc:	suruchi <suruchi.a.kadu@...el.com>, ak <ak@...ux.intel.com>,
	Dave Hansen <dave.hansen@...el.com>,
	Matthew Wilcox <willy@...ux.intel.com>,
	"hubert.nueckel" <hubert.nueckel@...el.com>,
	beth.marsh-prime@...el.com, "doug.nelson" <doug.nelson@...el.com>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	linux-fsdevel <linux-fsdevel@...r.kernel.org>
Subject: [Regression 3.16-rc kernel] -55% reduction in throughput for OLTP
 benchmark

Al,

We saw a large regression (-55%) in a standard OLTP benchmark
between 3.15 kernel and 3.16-rc4 kernel.

One issue was caused by the changes in async direct IO in the 
direct IO patch series between commit f6c0a192 and 62a8067a 
for the 3.16-rc kernel.  The kernel before this series at 
commit 38583f09 has no such regression.

After some investigations and instrumentation, we saw that the
problem was caused by a lot of io_wait issued from 
do_blockdev_direct_IO for async direct IO writes, which
did not happen for 3.15 kernel.  The benchmark
used to run at 97% cpu utilization now has only
35% cpu utilization with plenty of idle time.

At the end of do_blockdev_direct_IO, 

        if (dio->is_async && retval == 0 && dio->result &&
            ((rw == READ) || (dio->result == sdio.size)))
                retval = -EIOCBQUEUED;

        if (retval != -EIOCBQUEUED)
                dio_await_completion(dio);

we saw for async writes, the check
for (dio->result == sdio.size) failed. The retval was not
set to -EIOCBQUEUED, causing dio_await_completion to be issued
and hence the io_waits.

It is possible that the problem was introduced when dio->result
computation was moved into do_direct_IO in commit 3320c60b.
However, we cannot compile and test this commit separately as 
it is dependent on some later patches in the direct IO patch series
so we have to test the patch series as a whole.

This is quite a large regression for us and we hope that
it can be fixed before 3.16 kernel is released.

Thanks.

Tim

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists