Message-ID: <CALvZod4jx1r4k6gMrqFq+PzDT3D37_DFGyyokgPSsZFXbMV0ZA@mail.gmail.com>
Date:   Sun, 15 Jul 2018 20:09:01 -0700
From:   Shakeel Butt <shakeelb@...gle.com>
To:     Yafang Shao <laoar.shao@...il.com>
Cc:     Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Vladimir Davydov <vdavydov.dev@...il.com>,
        Cgroups <cgroups@...r.kernel.org>, Linux MM <linux-mm@...ck.org>,
        LKML <linux-kernel@...r.kernel.org>, Roman Gushchin <guro@...com>
Subject: Re: [PATCH] mm: avoid bothering interrupted task when charge memcg in softirq

On Sun, Jul 15, 2018 at 6:50 PM Yafang Shao <laoar.shao@...il.com> wrote:
>
> On Sun, Jul 15, 2018 at 11:04 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
> > On Sun, Jul 15, 2018 at 1:02 AM Yafang Shao <laoar.shao@...il.com> wrote:
> >>
> >> On Sun, Jul 15, 2018 at 2:34 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
> >> > On Sat, Jul 14, 2018 at 10:26 PM Yafang Shao <laoar.shao@...il.com> wrote:
> >> >>
> >> >> On Sun, Jul 15, 2018 at 12:25 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
> >> >> > On Sat, Jul 14, 2018 at 7:10 PM Yafang Shao <laoar.shao@...il.com> wrote:
> >> >> >>
> >> >> >> On Sat, Jul 14, 2018 at 11:38 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
> >> >> >> > On Sat, Jul 14, 2018 at 1:32 AM Yafang Shao <laoar.shao@...il.com> wrote:
> >> >> >> >>
> >> >> >> >> try_charge maybe executed in packet receive path, which is in interrupt
> >> >> >> >> context.
> >> >> >> >> In this situation, the 'current' is the interrupted task, which may has
> >> >> >> >> no relation to the rx softirq, So it is nonsense to use 'current'.
> >> >> >> >>
> >> >> >> >
> >> >> >> > Have you actually seen this occurring?
> >> >> >>
> >> >> >> Hi Shakeel,
> >> >> >>
> >> >> >> I'm trying to produce this issue, but haven't find it occur yet.
> >> >> >>
> >> >> >> > I am not very familiar with the
> >> >> >> > network code but I can think of two ways try_charge() can be called
> >> >> >> > from network code. Either through kmem charging or through
> >> >> >> > mem_cgroup_charge_skmem() and both locations correctly handle
> >> >> >> > interrupt context.
> >> >> >> >
> >> >> >>
> >> >> >> Why do you say that mem_cgroup_charge_skmem() correctly hanle
> >> >> >> interrupt context ?
> >> >> >>
> >> >> >> Let me show you why mem_cgroup_charge_skmem isn't hanling interrupt
> >> >> >> context correctly.
> >> >> >>
> >> >> >> mem_cgroup_charge_skmem() is calling  try_charge() twice.
> >> >> >> The first one is with GFP_NOWAIT as the gfp_mask, and the second one
> >> >> >> is with  (GFP_NOWAIT |  __GFP_NOFAIL) as the gfp_mask.
> >> >> >>
> >> >> >> If page_counter_try_charge() failes at the first time, -ENOMEM is returned.
> >> >> >> Then mem_cgroup_charge_skmem() will call try_charge() once more with
> >> >> >> __GFP_NOFAIL set, and this time if If page_counter_try_charge() failes
> >> >> >> again the '
> >> >> >> force' label in  try_charge() will be executed and 0 is returned.
> >> >> >>
> >> >> >> No matter what, the 'current' will be used and touched, that is
> >> >> >> meaning mem_cgroup_charge_skmem() isn't hanling the interrupt context
> >> >> >> correctly.
> >> >> >>
> >> >> >
> >> >> > Hi Yafang,
> >> >> >
> >> >> > If you check mem_cgroup_charge_skmem(), the memcg passed is not
> >> >> > 'current' but is from the sock object i.e. sk->sk_memcg for which the
> >> >> > network buffer is allocated for.
> >> >> >
> >> >>
> >> >> That's correct, the memcg if from the sock object.
> >> >> But the point is, in this situation why 'current' is used in try_charge() ?
> >> >> As 'current' is not related with the memcg, which is just a interrupted task.
> >> >>
> >> >
> >> > Hmm so you mean the behavior of memcg charging in the interrupt
> >> > context depends on the state of the interrupted task.
> >>
> >> Yes.
> >>
> >> > As you have
> >> > noted, mem_cgroup_charge_skmem() tries charging again with
> >> > __GFP_NOFAIL and the charge succeeds. Basically the memcg charging by
> >> > mem_cgroup_charge_skmem() will always succeed irrespective of the
> >> > state of the interrupted task. However mem_cgroup_charge_skmem() can
> >> > return true if the interrupted task was exiting or a fatal signal is
> >> > pending or oom victim or reclaiming memory. Can you please explain why
> >> > this is bad?
> >> >
> >>
> >> Let me show you the possible issues cause by this behavoir.
> >> 1.  In mem_cgroup_oom(), some  members in 'current' is set.
> >>      That means an innocent task will be in  task_in_memcg_oom state.
> >>      But this task may be in a different memcg, I mean the memcg of
> >> the 'current' may be differenct with the sk->sk_memcg.
> >>      Then when this innocent 'current' do try_charge it will hit  "if
> >> (unlikely(task_in_memcg_oom(current)))" and  -ENOMEM is returned,
> >> While there're maybe some free memory (or some memory could be freed )
> >> in the memcg of the innocent 'task'.
> >>
> >
> > No memory will be freed as try_charge() is in interrupt context.
> >
>
> I mean when this interrupted 'current' is running, that's in process context.
> In process context it should call try_to_free_mem_cgroup_pages() to
> free some memory,
> but it will hit "if (unlikely(task_in_memcg_oom(current)))"  before as
> it is set in the interrupt context.
>
> That's an obviously issue. Do you understand ?
>

Not really. I couldn't find where current->memcg_in_oom can be set in
the interrupt context.

> >> 2.  If the interrupted task was exiting or a fatal signal is  pending
> >> or oom victim,
> >>      it will directly goto force and 0 is returned, and then
> >> mem_cgroup_charge_skmem() will return true.
> >>      But mem_cgroup_charge_skmem() maybe need to try the second time
> >> and return false.
> >>
> >> That are all unexpected behavoir.
> >>
> >
> > Yes, this is inconsistent behavior. Can you explain how this will
> > affect network traffic? Basically mem_cgroup_charge_skmem() was
> > supposed to return false but sometime based on the interrupted task,
> > mem_cgroup_charge_skmem() returns true. How is this behavior bad for
> > network traffic?
> >
>
> You could see the funtion  __sk_mem_raise_allocated().
> If mem_cgroup_charge_skmem() return false, it will goto
> suppress_allocation and uncharge skmem,
> while when mem_cgroup_charge_skmem() return true, it will charge skmem
> sucessfully.
>
> The consequence behavior is  sk_rmem_schedule may fail while it should sucess.
> And then it will call  tcp_prune_queue() and tcp collapse may take a long time.
>

Is that a good thing or bad? From what I understand with your change
if charge fails, sk_rmem_schedule will always fail. However without
your change the interrupted task's state might help sk_rmem_schedule
to pass. I am all for consistent behavior but I wanted to make sure if
that is what you are aiming for.

Anyways, from what I remember Facebook is using the cgroup-v2's tcpmem
accounting. Johannes or Roman can shed some light if they have
observed this issue in production and might have opinion on how to
solve it.

thanks,
Shakeel

Powered by blists - more mailing lists