Message-ID: <CALOAHbAovZyh5Lju9rR6wDAFFovXnW9EBeJ6is7W7O4ooULqYA@mail.gmail.com>
Date:   Mon, 16 Jul 2018 11:38:45 +0800
From:   Yafang Shao <laoar.shao@...il.com>
To:     Shakeel Butt <shakeelb@...gle.com>
Cc:     Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...nel.org>,
        Vladimir Davydov <vdavydov.dev@...il.com>,
        Cgroups <cgroups@...r.kernel.org>, Linux MM <linux-mm@...ck.org>,
        LKML <linux-kernel@...r.kernel.org>, Roman Gushchin <guro@...com>
Subject: Re: [PATCH] mm: avoid bothering interrupted task when charge memcg in softirq

On Mon, Jul 16, 2018 at 11:09 AM, Shakeel Butt <shakeelb@...gle.com> wrote:
> On Sun, Jul 15, 2018 at 6:50 PM Yafang Shao <laoar.shao@...il.com> wrote:
>>
>> On Sun, Jul 15, 2018 at 11:04 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
>> > On Sun, Jul 15, 2018 at 1:02 AM Yafang Shao <laoar.shao@...il.com> wrote:
>> >>
>> >> On Sun, Jul 15, 2018 at 2:34 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
>> >> > On Sat, Jul 14, 2018 at 10:26 PM Yafang Shao <laoar.shao@...il.com> wrote:
>> >> >>
>> >> >> On Sun, Jul 15, 2018 at 12:25 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
>> >> >> > On Sat, Jul 14, 2018 at 7:10 PM Yafang Shao <laoar.shao@...il.com> wrote:
>> >> >> >>
>> >> >> >> On Sat, Jul 14, 2018 at 11:38 PM, Shakeel Butt <shakeelb@...gle.com> wrote:
>> >> >> >> > On Sat, Jul 14, 2018 at 1:32 AM Yafang Shao <laoar.shao@...il.com> wrote:
>> >> >> >> >>
>> >> >> >> >> try_charge maybe executed in packet receive path, which is in interrupt
>> >> >> >> >> context.
>> >> >> >> >> In this situation, the 'current' is the interrupted task, which may has
>> >> >> >> >> no relation to the rx softirq, So it is nonsense to use 'current'.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > Have you actually seen this occurring?
>> >> >> >>
>> >> >> >> Hi Shakeel,
>> >> >> >>
>> >> >> >> I'm trying to produce this issue, but haven't find it occur yet.
>> >> >> >>
>> >> >> >> > I am not very familiar with the
>> >> >> >> > network code but I can think of two ways try_charge() can be called
>> >> >> >> > from network code. Either through kmem charging or through
>> >> >> >> > mem_cgroup_charge_skmem() and both locations correctly handle
>> >> >> >> > interrupt context.
>> >> >> >> >
>> >> >> >>
>> >> >> >> Why do you say that mem_cgroup_charge_skmem() correctly hanle
>> >> >> >> interrupt context ?
>> >> >> >>
>> >> >> >> Let me show you why mem_cgroup_charge_skmem isn't hanling interrupt
>> >> >> >> context correctly.
>> >> >> >>
>> >> >> >> mem_cgroup_charge_skmem() is calling  try_charge() twice.
>> >> >> >> The first one is with GFP_NOWAIT as the gfp_mask, and the second one
>> >> >> >> is with  (GFP_NOWAIT |  __GFP_NOFAIL) as the gfp_mask.
>> >> >> >>
>> >> >> >> If page_counter_try_charge() failes at the first time, -ENOMEM is returned.
>> >> >> >> Then mem_cgroup_charge_skmem() will call try_charge() once more with
>> >> >> >> __GFP_NOFAIL set, and this time if If page_counter_try_charge() failes
>> >> >> >> again the '
>> >> >> >> force' label in  try_charge() will be executed and 0 is returned.
>> >> >> >>
>> >> >> >> No matter what, the 'current' will be used and touched, that is
>> >> >> >> meaning mem_cgroup_charge_skmem() isn't hanling the interrupt context
>> >> >> >> correctly.
>> >> >> >>
>> >> >> >
>> >> >> > Hi Yafang,
>> >> >> >
>> >> >> > If you check mem_cgroup_charge_skmem(), the memcg passed is not
>> >> >> > 'current' but is from the sock object i.e. sk->sk_memcg for which the
>> >> >> > network buffer is allocated for.
>> >> >> >
>> >> >>
>> >> >> That's correct, the memcg if from the sock object.
>> >> >> But the point is, in this situation why 'current' is used in try_charge() ?
>> >> >> As 'current' is not related with the memcg, which is just a interrupted task.
>> >> >>
>> >> >
>> >> > Hmm so you mean the behavior of memcg charging in the interrupt
>> >> > context depends on the state of the interrupted task.
>> >>
>> >> Yes.
>> >>
>> >> > As you have
>> >> > noted, mem_cgroup_charge_skmem() tries charging again with
>> >> > __GFP_NOFAIL and the charge succeeds. Basically the memcg charging by
>> >> > mem_cgroup_charge_skmem() will always succeed irrespective of the
>> >> > state of the interrupted task. However mem_cgroup_charge_skmem() can
>> >> > return true if the interrupted task was exiting or a fatal signal is
>> >> > pending or oom victim or reclaiming memory. Can you please explain why
>> >> > this is bad?
>> >> >
>> >>
>> >> Let me show you the possible issues cause by this behavoir.
>> >> 1.  In mem_cgroup_oom(), some  members in 'current' is set.
>> >>      That means an innocent task will be in  task_in_memcg_oom state.
>> >>      But this task may be in a different memcg, I mean the memcg of
>> >> the 'current' may be differenct with the sk->sk_memcg.
>> >>      Then when this innocent 'current' do try_charge it will hit  "if
>> >> (unlikely(task_in_memcg_oom(current)))" and  -ENOMEM is returned,
>> >> While there're maybe some free memory (or some memory could be freed )
>> >> in the memcg of the innocent 'task'.
>> >>
>> >
>> > No memory will be freed as try_charge() is in interrupt context.
>> >
>>
>> I mean when this interrupted 'current' is running, that's in process context.
>> In process context it should call try_to_free_mem_cgroup_pages() to
>> free some memory,
>> but it will hit "if (unlikely(task_in_memcg_oom(current)))"  before as
>> it is set in the interrupt context.
>>
>> That's an obviously issue. Do you understand ?
>>
>
> Not really. I couldn't find where current->memcg_in_oom can be set in
> the interrupt context.
>

You are right. current->memcg_in_oom can't be set in the interrupt context.

>> >> 2.  If the interrupted task was exiting or a fatal signal is  pending
>> >> or oom victim,
>> >>      it will directly goto force and 0 is returned, and then
>> >> mem_cgroup_charge_skmem() will return true.
>> >>      But mem_cgroup_charge_skmem() maybe need to try the second time
>> >> and return false.
>> >>
>> >> That are all unexpected behavoir.
>> >>
>> >
>> > Yes, this is inconsistent behavior. Can you explain how this will
>> > affect network traffic? Basically mem_cgroup_charge_skmem() was
>> > supposed to return false but sometime based on the interrupted task,
>> > mem_cgroup_charge_skmem() returns true. How is this behavior bad for
>> > network traffic?
>> >
>>
>> You could see the funtion  __sk_mem_raise_allocated().
>> If mem_cgroup_charge_skmem() return false, it will goto
>> suppress_allocation and uncharge skmem,
>> while when mem_cgroup_charge_skmem() return true, it will charge skmem
>> sucessfully.
>>
>> The consequence behavior is  sk_rmem_schedule may fail while it should sucess.
>> And then it will call  tcp_prune_queue() and tcp collapse may take a long time.
>>
>
> Is that a good thing or bad?
> From what I understand with your change
> if charge fails, sk_rmem_schedule will always fail. However without
> your change the interrupted task's state might help sk_rmem_schedule
> to pass. I am all for consistent behavior but I wanted to make sure if
> that is what you are aiming for.
>

Yes, with this change it will always fail. Without this change it may
sucess depends on the  interrupted task's state.
My previous statement makes some mistake.

I have no clear idea it is bad or good. That's why I'm trying to
produce the issue now.
But I think that we should avoid this unexpected behavior due to state
of the random interrupted task.

> Anyways, from what I remember Facebook is using the cgroup-v2's tcpmem
> accounting. Johannes or Roman can shed some light if they have
> observed this issue in production and might have opinion on how to
> solve it.
>
> thanks,
> Shakeel

Powered by blists - more mailing lists