netdev - Re: [RFC PATCH] sched&net: avoid over-pulling tasks due to network interrupts

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAGsJ_4wofduvT2BJipJppJza_ZyL2pU3Ni-B3R+A3_Zqv2v_4g@mail.gmail.com>
Date:   Mon, 8 Nov 2021 07:08:09 +1300
From:   Barry Song <21cnbao@...il.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     David Miller <davem@...emloft.net>, kuba@...nel.org,
        Eric Dumazet <edumazet@...gle.com>, pabeni@...hat.com,
        fw@...len.de, Ingo Molnar <mingo@...hat.com>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        Thomas Gleixner <tglx@...utronix.de>, netdev@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>,
        Linuxarm <linuxarm@...wei.com>,
        Guodong Xu <guodong.xu@...aro.org>,
        yangyicong <yangyicong@...wei.com>, shenyang39@...wei.com,
        tangchengchang@...wei.com, Barry Song <song.bao.hua@...ilicon.com>,
        Libo Chen <libo.chen@...cle.com>,
        Tim Chen <tim.c.chen@...ux.intel.com>
Subject: Re: [RFC PATCH] sched&net: avoid over-pulling tasks due to network interrupts

On Sat, Nov 6, 2021 at 1:25 AM Peter Zijlstra <peterz@...radead.org> wrote:
>
> On Fri, Nov 05, 2021 at 06:51:36PM +0800, Barry Song wrote:
> > From: Barry Song <song.bao.hua@...ilicon.com>
> >
> > In LPC2021, both Libo Chen and Tim Chen have reported the overpull
> > of network interrupts[1]. For example, while running one database,
> > ethernet is located in numa0, numa1 might be almost idle due to
> > interrupts are pulling tasks to numa0 because of wake_up affine.
> > I have seen the same problem. One way to solve this problem is
> > moving to a normal wakeup in network rather than using a sync
> > wakeup which will be more aggressively pulling tasks in scheduler
> > core.
> >
> > On kunpeng920 with 4numa, ethernet is located at numa0, storage
> > disk is located at numa2. While using sysbench to connect this
> > mysql machine, I am seeing numa1 is idle though numa0,2 and 3
> > are quite busy.
> >
>
> > I am not saying this patch is exactly the right approach, But I'd
> > like to use this RFC to connect the people of net and scheduler,
> > and start the discussion in this wider range.
>
> Well the normal way would be to use multi-queue crud and/or receive
> packet steering to get the interrupt/wakeup back to the cpu that data
> came from.

The test case has been a multi-queue ethernet and irqs are balanced
to NUMA0 by irqbalanced or pinned to NUMA0 where the card is located
by the script like:
#!/bin/bash
irq_list=(`cat /proc/interrupts | grep network_name| awk -F: '{print $1}'`)
cpunum=0
for irq in ${irq_list[@]}
do
echo $cpunum > /proc/irq/$irq/smp_affinity_list
echo `cat /proc/irq/$irq/smp_affinity_list`
(( cpunum+=1 ))
done

I have heard some people are working around this issue  by pinning
multi-queue IRQs to multiple NUMAs which can spread interrupts and
avoid over-pulling tasks to one NUMA only, but lose ethernet locality?
Hi, @Tim, it seems in LPC2021 you mentioned you are using this
solution?

And some other people are pinning ethernet IRQs to a couple of
CPUs within the NUMA ethernet belongs to, and then isolate these
CPUs from tasks and use those CPUs for interrupts only. This can
avoid wake-up pulling at all.

I think we need some generic way to resolve this problem. Hi,
@Libo , what is your solution to work around this issue?

Thanks
Barry