linux-kernel - [PATCHSET] sched/delayacct: get task SOFTIRQ delay

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20240402112112.GA17370@didi-ThinkCentre-M930t-N000>
Date: Tue, 2 Apr 2024 19:21:12 +0800
From: Tio Zhang <tiozhang@...iglobal.com>
To: <mingo@...hat.com>, <peterz@...radead.org>, <juri.lelli@...hat.com>,
	<vincent.guittot@...aro.org>, <rostedt@...dmis.org>, <bsingharora@...il.com>,
	<corbet@....net>, <akpm@...ux-foundation.org>
CC: <linux-kernel@...r.kernel.org>, <linux-doc@...r.kernel.org>,
	<dietmar.eggemann@....com>, <bsegall@...gle.com>, <mgorman@...e.de>,
	<bristot@...hat.com>, <vschneid@...hat.com>, <tiozhang@...iglobal.com>,
	<zyhtheonly@...il.com>, <zyhtheonly@...h.net>, <fuyuanli@...iglobal.com>
Subject: [PATCHSET] sched/delayacct: get task SOFTIRQ delay

We can only get IRQ/SOFTIRQ delay in total now in Delay accounting, but
getting SOFTIRQ delay and IRQ delay separetely would help users reduce
such delays in a more convenient way.
For IRQ delay, we can tuning irq CPU affinity or using threaded-irq.
For SOFTIRQ delay, we can tuning rps/xps or using kernel threads for NAPI.
And this is an example stack a task is delayed mainly by SOFTIRQ(delay
by receiving packets when sending packets):
	...
	ip_rcv
	__netif_receive_skb_core
	__netif_receive_skb
	process_backlog
	net_rx_action
	do_softirq
	__local_bh_enable_ip
	ip_finish_output2
	ip_finish_output
	ip_output
	ip_local_out
	ip_send_skb
	udp_send_skb
	udp_sendmsg
	inet_sendmsg
	sock_sendmsg
	__sys_sendto
	do_syscall_64
	__libc_sendto
	...

So this patchset tries to make SOFTIRQ delay observeable in Delay
accounting and available in taskstats.
(also update tools/accounting/getdelays.c)

Also for backward compatibility, we dont want to change the meaning of
origin IRQ/SOFTIRQ delay, instead we can get real IRQ(interrupt) delay by
the origin IRQ/SOFTIRQ delay minus SOFTIRQ delay added by this patch.

With this patch, the example above results by getdelays.c:
	# ./getdelays -t 4600 -d
	print delayacct stats ON
	TGID	4600
	
	CPU             count     real total  virtual total    delay total  delay average
                 	3973    10700014780    10698803222   312345815813         78.617ms
	IO              count    delay total  delay average
                    	0              0              0.000ms
	SWAP            count    delay total  delay average
                    	0              0              0.000ms
	RECLAIM         count    delay total  delay average
                    	0              0              0.000ms
	THRASHING       count    delay total  delay average
                    	0              0              0.000ms
	COMPACT         count    delay total  delay average
                    	0              0              0.000ms
	WPCOPY          count    delay total  delay average
                   	40         266859             0.007ms
	IRQ             count    delay total  delay average
                	13450    17756373906          1.320ms
	SOFTIRQ         count    delay total  delay average
        	        13450    17639154300          1.311ms

We find out SOFTIRQ impact the delay most, then tune RPS to reduce this.