| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 13 May 2015 10:53:33 -0500
From: nzimmer <nzimmer@....com>
To: Andrew Morton <akpm@...ux-foundation.org>,
Mel Gorman <mgorman@...e.de>
CC: Waiman Long <waiman.long@...com>,
Dave Hansen <dave.hansen@...el.com>,
Scott Norton <scott.norton@...com>,
Daniel J Blueman <daniel@...ascale.com>,
Linux-MM <linux-mm@...ck.org>,
LKML <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] mm: meminit: Finish initialisation of struct pages before
basic setup
I am just noticed a hang on my largest box.
I can only reproduce with large core counts, if I turn down the number
of cpus it doesn't have an issue.
Also as time goes on the amount of time required to initialize pages
goes up.
log_uv48_05121052:[ 177.250385] node 0 initialised, 14950072 pages in 544ms
log_uv48_05121052:[ 177.269629] node 1 initialised, 15990505 pages in 564ms
log_uv48_05121052:[ 177.436047] node 215 initialised, 3600110 pages in
724ms
log_uv48_05121052:[ 177.464056] node 102 initialised, 3604205 pages in
756ms
log_uv48_05121052:[ 178.073822] node 30 initialised, 7732972 pages in
1368ms
log_uv48_05121052:[ 178.082888] node 31 initialised, 7728877 pages in
1372ms
log_uv48_05121052:[ 178.080060] node 29 initialised, 7728877 pages in
1376ms
....
log_uv48_05121052:[ 178.217980] node 197 initialised, 7728877 pages in
1504ms
log_uv48_05121052:[ 178.217851] node 196 initialised, 7732972 pages in
1504ms
log_uv48_05121052:[ 178.219992] node 247 initialised, 7726418 pages in
1504ms
log_uv48_05121052:[ 178.325299] node 3 initialised, 15986409 pages in
1624ms
log_uv48_05121052:[ 178.328455] node 2 initialised, 15990505 pages in
1624ms
log_uv48_05121052:[ 178.383371] node 4 initialised, 15990505 pages in
1680ms
...
log_uv48_05121052:[ 178.438401] node 19 initialised, 15986409 pages in
1728ms
I apologize for the tardiness of this report but I have not been able to
get to the largest boxes reliably.
Hopefully I will have more access this week.
On 05/07/2015 05:09 PM, Andrew Morton wrote:
> On Thu, 7 May 2015 08:25:18 +0100 Mel Gorman <mgorman@...e.de> wrote:
>
>> Waiman Long reported that 24TB machines hit OOM during basic setup when
>> struct page initialisation was deferred. One approach is to initialise memory
>> on demand but it interferes with page allocator paths. This patch creates
>> dedicated threads to initialise memory before basic setup. It then blocks
>> on a rw_semaphore until completion as a wait_queue and counter is overkill.
>> This may be slower to boot but it's simplier overall and also gets rid of a
>> section mangling which existed so kswapd could do the initialisation.
> Seems a reasonable compromise. It makes a bit of a mess of the patch
> sequencing.
>
> Have some tweaklets:
>
>
>
> From: Andrew Morton <akpm@...ux-foundation.org>
> Subject: mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix
>
> include rwsem.h, use DECLARE_RWSEM, fix comment, remove unneeded cast
>
> Cc: Daniel J Blueman <daniel@...ascale.com>
> Cc: Dave Hansen <dave.hansen@...el.com>
> Cc: Mel Gorman <mgorman@...e.de>
> Cc: Nathan Zimmer <nzimmer@....com>
> Cc: Scott Norton <scott.norton@...com>
> Cc: Waiman Long <waiman.long@...com
> Signed-off-by: Andrew Morton <akpm@...ux-foundation.org>
> ---
>
> mm/page_alloc.c | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff -puN mm/page_alloc.c~mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix mm/page_alloc.c
> --- a/mm/page_alloc.c~mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix
> +++ a/mm/page_alloc.c
> @@ -18,6 +18,7 @@
> #include <linux/mm.h>
> #include <linux/swap.h>
> #include <linux/interrupt.h>
> +#include <linux/rwsem.h>
> #include <linux/pagemap.h>
> #include <linux/jiffies.h>
> #include <linux/bootmem.h>
> @@ -1075,12 +1076,12 @@ static void __init deferred_free_range(s
> __free_pages_boot_core(page, pfn, 0);
> }
>
> -static struct rw_semaphore __initdata pgdat_init_rwsem;
> +static __initdata DECLARE_RWSEM(pgdat_init_rwsem);
>
> /* Initialise remaining memory on a node */
> static int __init deferred_init_memmap(void *data)
> {
> - pg_data_t *pgdat = (pg_data_t *)data;
> + pg_data_t *pgdat = data;
> int nid = pgdat->node_id;
> struct mminit_pfnnid_cache nid_init_state = { };
> unsigned long start = jiffies;
> @@ -1096,7 +1097,7 @@ static int __init deferred_init_memmap(v
> return 0;
> }
>
> - /* Bound memory initialisation to a local node if possible */
> + /* Bind memory initialisation thread to a local node if possible */
> if (!cpumask_empty(cpumask))
> set_cpus_allowed_ptr(current, cpumask);
>
> @@ -1200,7 +1201,6 @@ void __init page_alloc_init_late(void)
> {
> int nid;
>
> - init_rwsem(&pgdat_init_rwsem);
> for_each_node_state(nid, N_MEMORY) {
> down_read(&pgdat_init_rwsem);
> kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid);
> _
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists