Message-ID: <482D1BCE.3060501@cosmosbay.com>
Date:	Fri, 16 May 2008 07:29:50 +0200
From:	Eric Dumazet <dada1@...mosbay.com>
To:	Rusty Russell <rusty@...tcorp.com.au>
Cc:	Andrew Morton <akpm@...ux-foundation.org>,
	linux kernel <linux-kernel@...r.kernel.org>,
	Mike Travis <travis@....com>
Subject: Re: [PATCH] modules: Use a better scheme for refcounting

Rusty Russell a écrit :
> On Friday 16 May 2008 06:40:37 Eric Dumazet wrote:
>   
>> Current refcounting for modules (done if CONFIG_MODULE_UNLOAD=y)
>> is using a lot of memory.
>>     
>
> Hi Eric,
>
>    I like this patch!  The plan was always to create a proper dynamic per-cpu
> allocator which used the normal per-cpu offsets, but I think module refcounts
> are worthwhile as a special case.
>
>    Any chance I can ask you look at the issue of full dynamic per-cpu
> allocation?  The problem of allocating memory which is laid out precisely
> as the original per-cpu alloc is vexing on NUMA, and probably requires
> reserving virtual address space and remapping into it, but the rewards
> would be maximally-efficient per-cpu accessors, and getting rid of that
> boutique allocator in module.c.
>
>   
You mean using alloc_percpu() ? Problem is that current implementation 
is expensive, since it is using
an extra array of pointers (struct percpu_data). On x86_64, that means 
at least a 200% space increase
over the solution of using 4 bytes in the static percpu zone. We 
probably can change this to dynamic
per-cpu as soon as Mike or Christopher finish their work on new dynamic 
per-cpu implementation ?
> Only minor commentry on the patch itself:
>
>   
>> +#ifdef CONFIG_SMP
>> +       char *refptr;
>> +#else
>>     
>
> void * would seem more natural here (love those gcc'isms)
>
>   
Yes, sure.
>> +#if defined(CONFIG_MODULE_UNLOAD) && defined(CONFIG_SMP)
>> +       void *refptr = NULL;
>> +#endif
>>     
>
> Looks like you can skip this if you assign to mod->refptr directly below:
>
>   
>> +#if defined(CONFIG_MODULE_UNLOAD) && defined(CONFIG_SMP)
>> +       refptr = percpu_modalloc(sizeof(local_t), sizeof(local_t), mod->name);
>> +       if (!refptr)
>> +               goto free_mod;
>> +       mod->refptr = refptr;
>> +#endif
>>     
>
>   
Unfortunatly no. I tried it and failed.
This is the problem that at freeing time, we cannot anymore use mod->refptr,
since we just unmaped mod, using vfree().
You had same problem with percpu, and had to use a temporaty variable on 
stack :)
> And finally:
>
>   
>> +#if defined(CONFIG_MODULE_UNLOAD) && defined(CONFIG_SMP)
>> +       if (refptr)
>> +               percpu_modfree(refptr);
>> +#endif
>>     
>
> This if (refptr) seems redundant.
>
>   
Yes, thanks a lot. Please find an updated patch.
> Thanks!
> Rusty.
>
>
>   
[PATCH] modules: Use a better scheme for refcounting

Current refcounting for modules (done if CONFIG_MODULE_UNLOAD=y)
is using a lot of memory (and disk space)

Each 'struct module' contains an [NR_CPUS] array of full cache lines.

This patch uses existing infrastructure (percpu_modalloc() & 
percpu_modfree())
to allocate percpu space for the refcount storage.

Instead of wasting NR_CPUS*128 bytes (on i386), we now use
num_possible_cpus*sizeof(local_t) bytes.

On a typical distro, where NR_CPUS=8, shiping 2000 modules, we reduce
size of module files by about 2 Mbytes. (1Kb per module)

Instead of having all refcounters in the same memory node - with TLB misses
because of vmalloc() - this new implementation permits to have better
NUMA properties, since each  CPU will use storage on its preferred node,
thanks to percpu storage.

Signed-off-by: Eric Dumazet <dada1@...mosbay.com>
---
 include/linux/module.h |   25 +++++++++++++++---------
 kernel/module.c        |   40 +++++++++++++++++++++++++++++----------
 2 files changed, 46 insertions(+), 19 deletions(-)


View attachment "module.patch" of type "text/plain" (4871 bytes)

Powered by blists - more mailing lists