| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 22 Mar 2013 16:32:36 -0700 From: Octavian Purdila <octavian.purdila@...el.com> To: Kent Overstreet <koverstreet@...gle.com> Cc: linux-kernel@...r.kernel.org, linux-aio@...ck.org, linux-s390@...r.kernel.org, bcrl@...ck.org, schwidefsky@...ibm.com, kirill.shutemov@...ux.intel.com, Andi Kleen <ak@...ux.intel.com> Subject: Re: [PATCH] aio: convert the ioctx list to radix tree On Fri, Mar 22, 2013 at 1:36 PM, Kent Overstreet <koverstreet@...gle.com> wrote: > On Fri, Mar 22, 2013 at 08:33:19PM +0200, Octavian Purdila wrote: >> When using a large number of threads performing AIO operations the >> IOCTX list may get a significant number of entries which will cause >> significant overhead. For example, when running this fio script: >> >> rw=randrw; size=256k ;directory=/mnt/fio; ioengine=libaio; iodepth=1 >> blocksize=1024; numjobs=512; thread; loops=100 >> >> on an EXT2 filesystem mounted on top of a ramdisk we can observe up to >> 30% CPU time spent by lookup_ioctx: >> >> 32.51% [guest.kernel] [g] lookup_ioctx >> 9.19% [guest.kernel] [g] __lock_acquire.isra.28 >> 4.40% [guest.kernel] [g] lock_release >> 4.19% [guest.kernel] [g] sched_clock_local >> 3.86% [guest.kernel] [g] local_clock >> 3.68% [guest.kernel] [g] native_sched_clock >> 3.08% [guest.kernel] [g] sched_clock_cpu >> 2.64% [guest.kernel] [g] lock_release_holdtime.part.11 >> 2.60% [guest.kernel] [g] memcpy >> 2.33% [guest.kernel] [g] lock_acquired >> 2.25% [guest.kernel] [g] lock_acquire >> 1.84% [guest.kernel] [g] do_io_submit >> >> This patchs converts the ioctx list to a radix tree. For a performance >> comparison the above FIO script was run on a 2 sockets 8 core >> machine. This are the results for the original list based >> implementation and for the radix tree based implementation: > > The biggest reason the overhead is so high is that the kioctx's > hlist_node shares a cacheline with the refcount. Did you check what just > fixing that does? My aio patch series (in akpm's tree) fixes that. > Hi Kent, Just checked, and I don't see any improvements for this particular workload. > Also, why are you using so many kioctxs? I can't think any good reason > why userspace would want to - you really want to use only one or a few > (potentially one per cpu) so that events can get serviced as soon as a > worker thread is available. > For servers, 1 kioctx per core can easily translate to 16-32 kioctxes. And you probably want to oversubscribe the cores, especially since IO is getting faster these days. So, I think 512 is not such an outrageously large number of kioctxes. > Currently there are applications using many kioctxs to work around the > fact that performance is terrible when you're sharing kioctxs between > threads - but that's fixed in my aio patch series. > > In fact, we want userspace to be using as few kioctxs as they can so we > can benefit from batch completion. > I think that using multiple contexts has its uses, with your great series not for I/O performance anymore :) , but for example for I/O operations management/grouping. Then, there is the case of existing applications, it would be nice to have them perform better without rewriting them. So, I think that this patch is complementary to your work. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists