netdev - Re: [PATCH] net/skbuff: silence warnings under memory pressure

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <28040332-4639-375a-0832-b47759d1d2d8@I-love.SAKURA.ne.jp>
Date:   Sat, 7 Sep 2019 20:00:34 +0900
From:   Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
To:     Michal Hocko <mhocko@...nel.org>,
        Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
Cc:     Qian Cai <cai@....pw>, Eric Dumazet <eric.dumazet@...il.com>,
        davem@...emloft.net, netdev@...r.kernel.org, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, Petr Mladek <pmladek@...e.com>,
        Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
        Steven Rostedt <rostedt@...dmis.org>
Subject: Re: [PATCH] net/skbuff: silence warnings under memory pressure

On 2019/09/04 17:25, Michal Hocko wrote:
> On Wed 04-09-19 16:00:42, Sergey Senozhatsky wrote:
>> On (09/04/19 15:41), Sergey Senozhatsky wrote:
>>> But the thing is different in case of dump_stack() + show_mem() +
>>> some other output. Because now we ratelimit not a single printk() line,
>>> but hundreds of them. The ratelimit becomes - 10 * $$$ lines in 5 seconds
>>> (IOW, now we talk about thousands of lines).
>>
>> And on devices with slow serial consoles this can be somewhat close to
>> "no ratelimit". *Suppose* that warn_alloc() adds 700 lines each time.
>> Within 5 seconds we can call warn_alloc() 10 times, which will add 7000
>> lines to the logbuf. If printk() can evict only 6000 lines in 5 seconds
>> then we have a growing number of pending logbuf messages.
> 
> Yes, ratelimit is problematic when the ratelimited operation is slow. I
> guess that is a well known problem and we would need to rework both the
> api and the implementation to make it work in those cases as well.
> Essentially we need to make the ratelimit act as a gatekeeper to an
> operation section - something like a critical section except you can
> tolerate more code executions but not too many. So effectively
> 
> 	start_throttle(rate, number);
> 	/* here goes your operation */
> 	end_throttle();
> 
> one operation is not considered done until the whole section ends.
> Or something along those lines.

Regarding OOM killer which is serialized by oom_lock mutex, I proposed
"mm, oom: More aggressively ratelimit dump_header()." at
https://lkml.kernel.org/r/1550325895-9291-2-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
but it was ignored.

> 
> In this particular case we can increase the rate limit parameters of
> course but I think that longterm we need a better api.
> 

I proposed "printk: Introduce "store now but print later" prefix." at
https://lkml.kernel.org/r/1550896930-12324-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
but it was not accepted.

But I think that more better solution for warn_alloc() is to defer printing
 from allocating context (which may not be allowed to sleep) to WQ context.
I proposed "mm,oom: Defer dump_tasks() output." at
https://lkml.kernel.org/r/7de2310d-afbd-e616-e83a-d75103b986c6@i-love.sakura.ne.jp
and answered to Michal's concerns. I hope we can apply the same thing for
warn_alloc() and show_mem(). Then, we can minimize latency for both
"memory allocation failures" and "OOM killer invocations".