netdev - 答复: Performance decrease in ipt_do

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date: Tue, 23 May 2023 02:01:48 +0000
From: hanhuihui <hanhuihui5@...wei.com>
To: "netdev@...r.kernel.org" <netdev@...r.kernel.org>, "davem@...emloft.net"
	<davem@...emloft.net>, "dsahern@...nel.org" <dsahern@...nel.org>,
	"pablo@...filter.org" <pablo@...filter.org>
CC: Caowangbao <caowangbao@...wei.com>, liaichun <liaichun@...wei.com>, "Yanan
 (Euler)" <yanan@...wei.com>, "Fengtao (fengtao, Euler)"
	<fengtao40@...wei.com>, yujiaying <yujiaying@...wei.com>
Subject: 答复: Performance decrease in ipt_do_table

After d43b75fbc23f and 8c9c296adfae, the vrf performance did indeed decrease by 7%. Can you help me take a look at this. Or how to configure qdisc to maintain performance without degradation


-----邮件原件-----
发件人: pengyi (M) 
发送时间: 2023年5月23日 9:27
收件人: hanhuihui <hanhuihui5@...wei.com>
主题: 转发: Performance decrease in ipt_do_table



-----邮件原件-----
发件人: pengyi (M) 
发送时间: 2023年4月19日 17:26
收件人: Caowangbao <caowangbao@...wei.com>; liaichun <liaichun@...wei.com>; Yanan (Euler) <yanan@...wei.com>; Fengtao (fengtao, Euler) <fengtao40@...wei.com>; yujiaying <yujiaying@...wei.com>
主题: Re: Performance decrease in ipt_do_table

Thank you for the reply, I have another question about the fixed patch mentioned in commit d43b75fbc23f. The commit 8c9c296adfae(vrf: run conntrack only in context of lower/physdev for locally generated packets) added the function vrf_nf_set_untracked() and was called in vrf_ip_out_direct(), to marking the packets as untrack. I was wondering why we should clear the packets out of the conntrack if the packet would be sent out directly, is it beneficial to the network performance? 

Another issue is that if the packet was marked as "untrack", the ipt_do_table() would spend more time because we didn't record it in the conntrack and had to analysize it. I deleted the "vrf_nf_set_untracked(skb);"and compared the two stacks. We could find the tcp_mt() would be called after the conntrack_mt_v3 and the performance increased. Any thoughts about this commit would be very thankful!

Here attached the stacks before and after the commit:
# Not deleting vrf_nf_set_untracked(skb)
# CPU  DURATION                  FUNCTION CALLS                    
# |     |   |                     |   |   |   |
  0)               |  ipt_do_table [ip_tables]() {
  0)               |    conntrack_mt_v3 [xt_conntrack]() {
  0)   0.360 us    |      conntrack_mt.isra.0 [xt_conntrack]();
  0)   1.090 us    |    }
  0)   0.370 us    |    tcp_mt();
  0)   0.300 us    |    tcp_mt();
  0)   0.300 us    |    tcp_mt();
  0)   0.290 us    |    tcp_mt();
  0)   0.310 us    |    tcp_mt();
  0)   0.300 us    |    tcp_mt();

# Deleting vrf_nf_set_untracked(skb)
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
17)               |  ipt_do_table [ip_tables]() {
17)               |    conntrack_mt_v3 [xt_conntrack]() {
17)   0.820 us    |      conntrack_mt.isra.1 [xt_conntrack]();
17)   1.690 us    |    }
17)   0.340 us    |    __local_bh_enable_ip();
17)   5.460 us    |  }
17)               |  ipt_do_table [ip_tables]() {
17)               |    conntrack_mt_v3 [xt_conntrack]() {
17)   0.720 us    |      conntrack_mt.isra.1 [xt_conntrack]();
17)   1.640 us    |    }
17)   0.330 us    |    __local_bh_enable_ip();
17)   4.470 us    |  }

Best wishes
Yi Peng

-----邮件原件-----
发件人: Nicolas Dichtel <nicolas.dichtel@...nd.com> 
发送时间: 2023年4月19日 0:28
收件人: pengyi (M) <pengyi37@...wei.com>; netdev@...r.kernel.org; davem@...emloft.net; dsahern@...nel.org; pablo@...filter.org
抄送: Caowangbao <caowangbao@...wei.com>; liaichun <liaichun@...wei.com>; Yanan (Euler) <yanan@...wei.com>
主题: Re: Performance decrease in ipt_do_table

Le 18/04/2023 à 16:37, pengyi (M) a écrit :
> Hi, I found after the commit d43b75fbc23f [vrf: don't run conntrack on vrf with !dflt qdisc] in vrf.c , the performance of ipt_do_table decreased. In vrf.c, the locally generated packet was marked as "untracked" in vrf_ip_out, because it won't be processed by the router(also mentioned in commit. d43b75fbc23f). And it will also made the statebit was set to XT_CONNTRACK_STATE_UNTRACKED in conntrack_mt, which means we didn't had the information about in conntrack so we had to re-analysize the packet by calling tcp_mt().
> 
> However, after the commit d43b75fbc23f , all packets in vrf_ip_out was marked as "untracked", which made more times of calling tcp_mt(). Hence the time of processing all packets became longer and it affected the performance of ipt_do_table() in total. I wonder is it an expected behavior to see the performance decrease after the commit d43b75fbc23f ?
It was not the goal of this commit :)
The performance regression seems to be specific to your rule set.
As explained in the commit log, the goal was to have the same packet processing, whatever qdisc is configured on the vrf interface.
Before this patch, you should have the same performance level (as now) if a qdisc is configured.


Regards,
Nicolas