Message-ID: <CAOAebxuHPieDWLyRmPauvuF7dkW=0GeigG=EfWQEb_cgrGgm0Q@mail.gmail.com>
Date:   Wed, 31 Jan 2018 13:24:55 -0500
From:   Pavel Tatashin <pasha.tatashin@...cle.com>
To:     "Koki.Sanagi@...fujitsu.com" <Koki.Sanagi@...fujitsu.com>
Cc:     Mel Gorman <mgorman@...hsingularity.net>,
        Michal Hocko <mhocko@...nel.org>,
        YASUAKI ISHIMATSU <yasu.isimatu@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Linux Memory Management List <linux-mm@...ck.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Steve Sistare <steven.sistare@...cle.com>,
        "msys.mizuma@...il.com" <msys.mizuma@...il.com>
Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if
 trace_buf_size is specified

Hi Koki,

Yes, the patch is here:
https://lkml.org/lkml/2018/1/12/600

It has not been reviewed yet.

Pavel

On Wed, Jan 31, 2018 at 12:28 PM, Koki.Sanagi@...fujitsu.com
<Koki.Sanagi@...fujitsu.com> wrote:
> Pavel,
>
> I assume you are working on the fix.
> Do you have any progress ?
>
> Koki
>
>>>-----Original Message-----
>>>From: Mel Gorman [mailto:mgorman@...hsingularity.net]
>>>Sent: Wednesday, December 06, 2017 5:50 AM
>>>To: Pavel Tatashin <pasha.tatashin@...cle.com>
>>>Cc: Michal Hocko <mhocko@...nel.org>; YASUAKI ISHIMATSU
>>><yasu.isimatu@...il.com>; Andrew Morton <akpm@...ux-foundation.org>;
>>>Linux Memory Management List <linux-mm@...ck.org>; linux-
>>>kernel@...r.kernel.org; Sanagi, Koki <Koki.Sanagi@...fujitsu.com>; Steve
>>>Sistare <steven.sistare@...cle.com>
>>>Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if
>>>trace_buf_size is specified
>>>
>>>On Wed, Nov 29, 2017 at 10:41:59PM -0500, Pavel Tatashin wrote:
>>>> Hi Mel,
>>>>
>>>> Thank you very much for your feedback, my replies below:
>>>>
>>>> > A lack of involvement from admins is indeed desirable. For example,
>>>> > while I might concede on using a disable-everything-switch, I would
>>>> > not be happy to introduce a switch that specified how much memory
>>>> > per node to initialise.
>>>> >
>>>> > For the forth approach, I really would be only thinking of a blunt
>>>> > "initialise everything instead of going OOM". I was wary of making
>>>> > things too complicated and I worried about some side-effects I'll cover later.
>>>>
>>>> I see, I misunderstood your suggestion. Switching to serial
>>>> initialization when OOM works, however, boot time becomes
>>>> unpredictable, with some configurations boot is fast with others it is
>>>> slow. All of that depends on whether predictions in
>>>> reset_deferred_meminit() were good or not which is not easy to debug
>>>> for users. Also, overtime predictions in reset_deferred_meminit() can
>>>> become very off, and I do not think that we want to continuously
>>>> adjust this function.
>>>>
>>>
>>>You could increase the probabilty of a report by doing a WARN_ON_ONCE if the
>>>serialised meminit is used.
>>>
>>>> >> With this approach we could always init a very small amount of
>>>> >> struct pages, and allow the rest to be initialized on demand as
>>>> >> boot requires until deferred struct pages are initialized. Since,
>>>> >> having deferred pages feature assumes that the machine is large,
>>>> >> there is no drawback of having some extra byte of dead code,
>>>> >> especially that all the checks can be permanently switched of via
>>>> >> static branches once deferred init is complete.
>>>> >>
>>>> >
>>>> > This is where I fear there may be dragons. If we minimse the number
>>>> > of struct pages and initialise serially as necessary, there is a
>>>> > danger that we'll allocate remote memory in cases where local memory
>>>> > would have done because a remote node had enough memory.
>>>>
>>>> True, but is not what we have now has the same issue as well? If one
>>>> node is gets out of memory we start using memory from another node,
>>>> before deferred pages are initialized?
>>>>
>>>
>>>It's possible but I'm not aware of it happening currently.
>>>
>>>>  To offset that risk, it would be
>>>> > necessary at boot-time to force allocations from local node where
>>>> > possible and initialise more memory as necessary. That starts
>>>> > getting complicated because we'd need to adjust gfp-flags in the
>>>> > fast path with init-and-retry logic in the slow path and that could
>>>> > be a constant penalty. We could offset that in the fast path by
>>>> > using static branches
>>>>
>>>> I will try to implement this, and see how complicated the patch will
>>>> be, if it gets too complicated for the problem I am trying to solve we
>>>> can return to one of your suggestions.
>>>>
>>>> I was thinking to do something like this:
>>>>
>>>> Start with every small amount of initialized pages in every node.
>>>> If allocation fails, initialize enough struct pages to cover this
>>>> particular allocation with struct pages rounded up to section size but
>>>> in every single node.
>>>>
>>>
>>>Ok, just make sure it's all in the slow paths of the allocator when the alternative
>>>is to fail the allocation.
>>>
>>>> > but it's getting more and
>>>> > more complex for what is a minor optimisation -- shorter boot times
>>>> > on large machines where userspace itself could take a *long* time to
>>>> > get up and running (think database reading in 1TB of data from disk as it
>>>warms up).
>>>>
>>>> On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time,
>>>> and this is on SPARC with 8K pages, on x86 it would be around of 8
>>>> minutes because of twice as many pages. This feature improves
>>>> availability for larger machines quite a bit. Overtime, systems are
>>>> growing, so I expect this feature to become a default configuration in
>>>> the next several years on server configs.
>>>>
>>>
>>>Ok, when developing the series originally, I had no machine even close to 32T of
>>>memory.
>>>
>>>--
>>>Mel Gorman
>>>SUSE Labs
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@...ck.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a hrefmailto:"dont@...ck.org"> email@...ck.org </a>

Powered by blists - more mailing lists