[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 11 Nov 2008 16:41:04 -0500
From: Jeff Layton <jlayton@...hat.com>
To: Jeff Moyer <jmoyer@...hat.com>
Cc: Jens Axboe <jens.axboe@...cle.com>,
"Vitaly V. Bursov" <vitalyb@...enet.dn.ua>,
linux-kernel@...r.kernel.org, bfields@...ldses.org
Subject: Re: Slow file transfer speeds with CFQ IO scheduler in some cases
On Tue, 11 Nov 2008 14:36:07 -0500
Jeff Moyer <jmoyer@...hat.com> wrote:
> Jens Axboe <jens.axboe@...cle.com> writes:
>
> > OK, that looks better. Can I talk you into just trying this little
> > patch, just to see what kind of performance that yields? Remove the cfq
> > patch first. I would have patched nfsd only, but this is just a quick'n
> > dirty.
>
> I went ahead and gave it a shot. The updated CFQ patch with no I/O
> context sharing does about 40MB/s reading a 1GB file. Backing that
> patch out, and then adding the patch to share io_context's between
> kthreads yields 45MB/s.
>
Here's a quick and dirty patch to make all of the nfsd's have the same
io_context. Comments appreciated -- I'm not that familiar with the IO
scheduling code. If this looks good, I'll clean it up, add some
comments and formally send it to Bruce.
----------------[snip]-------------------
>From dd15b19a0eab3e181a6f76f1421b97950e255b4b Mon Sep 17 00:00:00 2001
From: Jeff Layton <jlayton@...hat.com>
Date: Tue, 11 Nov 2008 15:43:15 -0500
Subject: [PATCH] knfsd: make all nfsd threads share an io_context
This apparently makes the I/O scheduler treat the threads as a group
which helps throughput when sequential I/O is multiplexed over several
nfsd's.
Signed-off-by: Jeff Layton <jlayton@...hat.com>
---
fs/nfsd/nfssvc.c | 27 +++++++++++++++++++++++++++
1 files changed, 27 insertions(+), 0 deletions(-)
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 07e4f5d..6d87f74 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -22,6 +22,7 @@
#include <linux/freezer.h>
#include <linux/fs_struct.h>
#include <linux/kthread.h>
+#include <linux/iocontext.h>
#include <linux/sunrpc/types.h>
#include <linux/sunrpc/stats.h>
@@ -42,6 +43,7 @@ static int nfsd(void *vrqstp);
struct timeval nfssvc_boot;
static atomic_t nfsd_busy;
static unsigned long nfsd_last_call;
+static struct io_context *nfsd_io_context;
static DEFINE_SPINLOCK(nfsd_call_lock);
/*
@@ -173,6 +175,7 @@ static void nfsd_last_thread(struct svc_serv *serv)
nfsd_serv = NULL;
nfsd_racache_shutdown();
nfs4_state_shutdown();
+ nfsd_io_context = NULL;
printk(KERN_WARNING "nfsd: last server has exited, flushing export "
"cache\n");
@@ -398,6 +401,28 @@ update_thread_usage(int busy_threads)
}
/*
+ * should be called while holding nfsd_mutex
+ */
+static void
+nfsd_set_io_context(void)
+{
+ int cpu, node;
+
+ if (!nfsd_io_context) {
+ cpu = get_cpu();
+ node = cpu_to_node(cpu);
+ put_cpu();
+
+ /*
+ * get_io_context can return NULL if the alloc_context fails.
+ * That's not technically fatal here, so we don't bother to
+ * check for it.
+ */
+ nfsd_io_context = get_io_context(GFP_KERNEL, node);
+ } else
+ copy_io_context(¤t->io_context, &nfsd_io_context);
+}
+/*
* This is the NFS server kernel thread
*/
static int
@@ -410,6 +435,8 @@ nfsd(void *vrqstp)
/* Lock module and set up kernel thread */
mutex_lock(&nfsd_mutex);
+ nfsd_set_io_context();
+
/* At this point, the thread shares current->fs
* with the init process. We need to create files with a
* umask of 0 instead of init's umask. */
--
1.5.5.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists