netdev - Re: RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAHNKnsQ_rhMOCfkAfsXjDg2eAfYkZTJ6DOJLmDb3OSB4AotXzA@mail.gmail.com>
Date:   Mon, 19 Sep 2022 04:59:28 +0300
From:   Sergey Ryazanov <ryazanov.s.a@...il.com>
To:     Василий Умрихин 
        <umrihin@...evt.ru>
Cc:     netdev <netdev@...r.kernel.org>
Subject: Re: RFH, is it possible to set ndo_start_xmit() cpu affinity in
 ethernet driver?

Hello Vasiliy,

On Mon, Sep 12, 2022 at 6:45 PM Василий Умрихин <umrihin@...evt.ru> wrote:
> On the receiving side we have the opportunity to choose the CPU that will process the receive queue (RPS).
> On the sender side XPS selects the send queue for the given CPU, but there is no way to select the CPU on which ndo_start_xmit() will be launched.
> Taskset is able to bind user task, but in ndo_start_xmit() binding differs.
> In my case CPU0 reserved for polling kthread, because our NIC have no interrupts, therefore it is necessary. I need nothing else to run on this CPU.
>
> For example, setting CPU1 for RPS on both nodes:
>
> host1: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
> host2: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
>
> Then run iperf on two nodes:
>
> host1: taskset -c 1 iperf -s
> host2: taskset -c 1 iperf -c host1
>
> After adding pr_info("cpu%d\n", smp_processor_id()); in my ndo_start_xmit() method, see in dmesg:
>
> host1: dmesg | grep cpu0 | wc -l
> 0
> host2: dmesg | grep cpu0 | wc -l
> 6512
>
> Is it possible to choose the CPU on which ndo_start_xmit() will be launched on the sender side?

AFAIK, there is no mechanism to force the ndo_start_xmit() invocation
on a specific core.

If I realize your goal correctly, then you can implement such
functionality in the driver. Just use an intermediate queue that is
filled by the ndo_start_xmit() callback and processed by a thread
bound to a specific core. E.g.

netdev_tx_t foodriver_start_xmit(struct sk_buff *skb, struct net_device *dev)
{
    struct foodev_priv *p = netdev_priv(dev);

    skb_queue_tail(&p->xmit_queue, skb);

    return NETDEV_TX_OK;
}

int foodriver_xmit_thread(void *arg)
{
    struct foodev_priv *p = arg;

    while (!kthread_should_stop()) {
        struct sk_buff *skb = skb_dequeue(&p->xmit_queue);

        /* Do main xmit actions here */

    }
}

int foodriver_open(struct net_device *dev)
{
    struct foodev_priv *p = netdev_priv(dev);
    ...
    t = kthread_create(foodriver_xmit_thread, p, "xmit_thread");
    kthread_bind(t, prefered_cpu);
   ....
}

--
Sergey