linux-kernel - Re: [PATCH] net: add big honking pfmemalloc OOM warning

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iK9DcDi68WQn0z2R65L1fQpa88nUnpe_di1OWGhhoqAQw@mail.gmail.com>
Date:   Wed, 10 Apr 2019 07:16:21 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     Juha-Matti Tilli <juha-matti.tilli@...eca.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        netdev <netdev@...r.kernel.org>,
        Rafael Aquini <aquini@...hat.com>,
        Murphy Zhou <xzhou@...hat.com>,
        Yongcheng Yang <yoyang@...hat.com>,
        Jianhong Yin <jiyin@...hat.com>
Subject: Re: [PATCH] net: add big honking pfmemalloc OOM warning

On Wed, Apr 10, 2019 at 3:20 AM Juha-Matti Tilli
<juha-matti.tilli@...eca.com> wrote:
>
> A system administrator is not notified (except via an obscure SNMP
> counter that most sysadmins don't know to look for) if packets are
> dropped due to out-of-memory condition when SKBs use pfmemalloc
> reserves. This can for example lead to NFS connections hanging on
> high-volume systems.
>
> Implement a ratelimited big honking out of memory warning that directs
> the sysadmin to bump up vm.min_free_kbytes in case this problem happens.
> Our experience shows that with default vm.min_free_kbytes (90112, or
> about 90 megabytes), NFS connections hang approximately once per day,
> whereas with 901120 (default multiplied by 10) NFS connections never
> hang.
>

If NFS sessions hang, then there is a bug to eventually root cause and fix.

Just telling the user : Increase the limit is the same thing than admitting :

Our limit system or TCP or NFS stacks are broken and unable to
recover, so lets disable the limit system and work around a more
serious bug.

Maybe the bug is in a NIC driver, please share more details before
adding yet another noisy signal in syslog

SNMP counters are per netns, and more useful in the modern computing
era,  where a host is shared by many different containers.



>
> Signed-off-by: Juha-Matti Tilli <juha-matti.tilli@...eca.com>
> Cc: Eric Dumazet <edumazet@...gle.com>
> Cc: Rafael Aquini <aquini@...hat.com>
> Cc: Murphy Zhou <xzhou@...hat.com>
> Cc: Yongcheng Yang <yoyang@...hat.com>
> Cc: Jianhong Yin <jiyin@...hat.com>
> ---
>  net/core/filter.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index fc92ebc4e200..7d8ef239b9af 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -99,6 +99,8 @@ int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap)
>          * helping free memory
>          */
>         if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC)) {
> +               printk_ratelimited(KERN_WARNING
> +                                  "dropped packet due to out-of-memory condition, please bump up vm.min_free_kbytes\n");
>                 NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP);
>                 return -ENOMEM;
>         }
> --
> 2.17.1
>