Message-Id: <20240709153728.4139640-1-omid.ehtemamhaghighi@menlosecurity.com>
Date: Tue,  9 Jul 2024 08:37:28 -0700
From: Omid Ehtemam-Haghighi <omid.ehtemamhaghighi@...losecurity.com>
To: netdev@...r.kernel.org
Cc: dsahern@...il.com,
	adrian.oliver@...losecurity.com,
	Omid Ehtemam-Haghighi <omid.ehtemamhaghighi@...losecurity.com>
Subject: [PATCH v2] net/ipv6: Fix soft lockups in fib6_select_path under high next hop churn

Soft lockups have been observed on a cluster of Linux-based edge routers
located in a highly dynamic environment. Using the `bird` service, these
routers continuously update BGP-advertised routes due to frequently
changing nexthop destinations, while also managing significant IPv6
traffic. The lockups occur during the traversal of the multipath
circular linked-list in the `fib6_select_path` function, particularly
while iterating through the siblings in the list. The issue typically
arises when the nodes of the linked list are unexpectedly deleted
concurrently on a different core—indicated by their 'next' and
'previous' elements pointing back to the node itself and their reference
count dropping to zero. This results in an infinite loop, leading to a
soft lockup that triggers a system panic via the watchdog timer.

To fix this issue, I applied RCU primitives in the problematic code
sections, which successfully resolved the issue within our testing
parameters and in the production environment where the issue was first
observed. Additionally, all references to fib6_siblings have been updated
to annotate or use the RCU APIs.

A test script that reproduces this issue is included with this patch. The
script periodically updates the routing table while generating a heavy load
of outgoing IPv6 traffic through multiple iperf3 clients. I have tested
this script on various machines, ranging from low to high performance, as
detailed in the comment section of the test script. It consistently induces
soft lockups within a minute.

Kernel log:

 0 [ffffbd13003e8d30] machine_kexec at ffffffff8ceaf3eb
 1 [ffffbd13003e8d90] __crash_kexec at ffffffff8d0120e3
 2 [ffffbd13003e8e58] panic at ffffffff8cef65d4
 3 [ffffbd13003e8ed8] watchdog_timer_fn at ffffffff8d05cb03
 4 [ffffbd13003e8f08] __hrtimer_run_queues at ffffffff8cfec62f
 5 [ffffbd13003e8f70] hrtimer_interrupt at ffffffff8cfed756
 6 [ffffbd13003e8fd0] __sysvec_apic_timer_interrupt at ffffffff8cea01af
 7 [ffffbd13003e8ff0] sysvec_apic_timer_interrupt at ffffffff8df1b83d
-- <IRQ stack> --
 8 [ffffbd13003d3708] asm_sysvec_apic_timer_interrupt at ffffffff8e000ecb
    [exception RIP: fib6_select_path+299]
    RIP: ffffffff8ddafe7b  RSP: ffffbd13003d37b8  RFLAGS: 00000287
    RAX: ffff975850b43600  RBX: ffff975850b40200  RCX: 0000000000000000
    RDX: 000000003fffffff  RSI: 0000000051d383e4  RDI: ffff975850b43618
    RBP: ffffbd13003d3800   R8: 0000000000000000   R9: ffff975850b40200
    R10: 0000000000000000  R11: 0000000000000000  R12: ffffbd13003d3830
    R13: ffff975850b436a8  R14: ffff975850b43600  R15: 0000000000000007
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 9 [ffffbd13003d3808] ip6_pol_route at ffffffff8ddb030c
10 [ffffbd13003d3888] ip6_pol_route_input at ffffffff8ddb068c
11 [ffffbd13003d3898] fib6_rule_lookup at ffffffff8ddf02b5
12 [ffffbd13003d3928] ip6_route_input at ffffffff8ddb0f47
13 [ffffbd13003d3a18] ip6_rcv_finish_core.constprop.0 at ffffffff8dd950d0
14 [ffffbd13003d3a30] ip6_list_rcv_finish.constprop.0 at ffffffff8dd96274
15 [ffffbd13003d3a98] ip6_sublist_rcv at ffffffff8dd96474
16 [ffffbd13003d3af8] ipv6_list_rcv at ffffffff8dd96615
17 [ffffbd13003d3b60] __netif_receive_skb_list_core at ffffffff8dc16fec
18 [ffffbd13003d3be0] netif_receive_skb_list_internal at ffffffff8dc176b3
19 [ffffbd13003d3c50] napi_gro_receive at ffffffff8dc565b9
20 [ffffbd13003d3c80] ice_receive_skb at ffffffffc087e4f5 [ice]
21 [ffffbd13003d3c90] ice_clean_rx_irq at ffffffffc0881b80 [ice]
22 [ffffbd13003d3d20] ice_napi_poll at ffffffffc088232f [ice]
23 [ffffbd13003d3d80] __napi_poll at ffffffff8dc18000
24 [ffffbd13003d3db8] net_rx_action at ffffffff8dc18581
25 [ffffbd13003d3e40] __do_softirq at ffffffff8df352e9
26 [ffffbd13003d3eb0] run_ksoftirqd at ffffffff8ceffe47
27 [ffffbd13003d3ec0] smpboot_thread_fn at ffffffff8cf36a30
28 [ffffbd13003d3ee8] kthread at ffffffff8cf2b39f
29 [ffffbd13003d3f28] ret_from_fork at ffffffff8ce5fa64
30 [ffffbd13003d3f50] ret_from_fork_asm at ffffffff8ce03cbb

Reported-by: Omid Ehtemam-Haghighi <omid.ehtemamhaghighi@...losecurity.com>
Tested-by: Omid Ehtemam-Haghighi <omid.ehtemamhaghighi@...losecurity.com>
Signed-off-by: Omid Ehtemam-Haghighi <omid.ehtemamhaghighi@...losecurity.com>
---
v2:
	* list_del_rcu() is applied exclusively to legacy multipath code
	* All occurrences of fib6_siblings have been modified to utilize RCU
	  APIs for annotation and usage.
	* Additionally, a test script for reproducing the reported
	  issue is included
---
 include/net/ip6_fib.h                         |   2 +-
 net/ipv6/ip6_fib.c                            |  24 +-
 net/ipv6/route.c                              |  40 ++--
 tools/testing/selftests/net/Makefile          |   1 +
 .../net/ipv6_route_update_soft_lockup.sh      | 217 ++++++++++++++++++
 5 files changed, 260 insertions(+), 24 deletions(-)
 create mode 100755 tools/testing/selftests/net/ipv6_route_update_soft_lockup.sh

diff --git a/include/net/ip6_fib.h b/include/net/ip6_fib.h
index 6cb867ce4878..167ef421bcb0 100644
--- a/include/net/ip6_fib.h
+++ b/include/net/ip6_fib.h
@@ -166,7 +166,7 @@ struct fib6_info {
 	 * to speed up lookup.
 	 */
 	union {
-		struct list_head	fib6_siblings;
+		struct list_head __rcu	fib6_siblings;
 		struct list_head	nh_list;
 	};
 	unsigned int			fib6_nsiblings;
diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
index 83e4f9855ae1..6202575b2c20 100644
--- a/net/ipv6/ip6_fib.c
+++ b/net/ipv6/ip6_fib.c
@@ -518,7 +518,7 @@ int fib6_tables_dump(struct net *net, struct notifier_block *nb,
 static int fib6_dump_node(struct fib6_walker *w)
 {
 	int res;
-	struct fib6_info *rt;
+	struct fib6_info *rt, *sibling, *last_sibling;
 
 	for_each_fib6_walker_rt(w) {
 		res = rt6_dump_route(rt, w->args, w->skip_in_node);
@@ -540,10 +540,16 @@ static int fib6_dump_node(struct fib6_walker *w)
 		 * last sibling of this route (no need to dump the
 		 * sibling routes again)
 		 */
-		if (rt->fib6_nsiblings)
-			rt = list_last_entry(&rt->fib6_siblings,
-					     struct fib6_info,
-					     fib6_siblings);
+		rcu_read_lock();
+		if (rt->fib6_nsiblings) {
+			last_sibling = rt;
+			list_for_each_entry_rcu(sibling, &rt->fib6_siblings,
+						fib6_siblings)
+				last_sibling = sibling;
+
+			rt = last_sibling;
+		}
+		rcu_read_unlock();
 	}
 	w->leaf = NULL;
 	return 0;
@@ -1190,8 +1196,8 @@ static int fib6_add_rt2node(struct fib6_node *fn, struct fib6_info *rt,
 		while (sibling) {
 			if (sibling->fib6_metric == rt->fib6_metric &&
 			    rt6_qualify_for_ecmp(sibling)) {
-				list_add_tail(&rt->fib6_siblings,
-					      &sibling->fib6_siblings);
+				list_add_tail_rcu(&rt->fib6_siblings,
+						  &sibling->fib6_siblings);
 				break;
 			}
 			sibling = rcu_dereference_protected(sibling->fib6_next,
@@ -1252,7 +1258,7 @@ static int fib6_add_rt2node(struct fib6_node *fn, struct fib6_info *rt,
 							 fib6_siblings)
 					sibling->fib6_nsiblings--;
 				rt->fib6_nsiblings = 0;
-				list_del_init(&rt->fib6_siblings);
+				list_del_rcu(&rt->fib6_siblings);
 				rt6_multipath_rebalance(next_sibling);
 				return err;
 			}
@@ -1970,7 +1976,7 @@ static void fib6_del_route(struct fib6_table *table, struct fib6_node *fn,
 					 &rt->fib6_siblings, fib6_siblings)
 			sibling->fib6_nsiblings--;
 		rt->fib6_nsiblings = 0;
-		list_del_init(&rt->fib6_siblings);
+		list_del_rcu(&rt->fib6_siblings);
 		rt6_multipath_rebalance(next_sibling);
 	}
 
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 8d72ca0b086d..4bca06dce176 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -415,7 +415,7 @@ void fib6_select_path(const struct net *net, struct fib6_result *res,
 		      struct flowi6 *fl6, int oif, bool have_oif_match,
 		      const struct sk_buff *skb, int strict)
 {
-	struct fib6_info *sibling, *next_sibling;
+	struct fib6_info *sibling;
 	struct fib6_info *match = res->f6i;
 
 	if (!match->nh && (!match->fib6_nsiblings || have_oif_match))
@@ -442,8 +442,9 @@ void fib6_select_path(const struct net *net, struct fib6_result *res,
 	if (fl6->mp_hash <= atomic_read(&match->fib6_nh->fib_nh_upper_bound))
 		goto out;
 
-	list_for_each_entry_safe(sibling, next_sibling, &match->fib6_siblings,
-				 fib6_siblings) {
+	rcu_read_lock();
+	list_for_each_entry_rcu(sibling, &match->fib6_siblings,
+				fib6_siblings) {
 		const struct fib6_nh *nh = sibling->fib6_nh;
 		int nh_upper_bound;
 
@@ -455,6 +456,7 @@ void fib6_select_path(const struct net *net, struct fib6_result *res,
 		match = sibling;
 		break;
 	}
+	rcu_read_unlock();
 
 out:
 	res->f6i = match;
@@ -4711,10 +4713,12 @@ static int rt6_multipath_total_weight(const struct fib6_info *rt)
 	if (!rt6_is_dead(rt))
 		total += rt->fib6_nh->fib_nh_weight;
 
-	list_for_each_entry(iter, &rt->fib6_siblings, fib6_siblings) {
+	rcu_read_lock();
+	list_for_each_entry_rcu(iter, &rt->fib6_siblings, fib6_siblings) {
 		if (!rt6_is_dead(iter))
 			total += iter->fib6_nh->fib_nh_weight;
 	}
+	rcu_read_unlock();
 
 	return total;
 }
@@ -5197,14 +5201,16 @@ static void ip6_route_mpath_notify(struct fib6_info *rt,
 	 * nexthop. Since sibling routes are always added at the end of
 	 * the list, find the first sibling of the last route appended
 	 */
+	rcu_read_lock();
 	if ((nlflags & NLM_F_APPEND) && rt_last && rt_last->fib6_nsiblings) {
-		rt = list_first_entry(&rt_last->fib6_siblings,
-				      struct fib6_info,
-				      fib6_siblings);
+		rt = list_first_or_null_rcu(&rt_last->fib6_siblings,
+					    struct fib6_info,
+					    fib6_siblings);
 	}
 
 	if (rt)
 		inet6_rt_notify(RTM_NEWROUTE, rt, info, nlflags);
+	rcu_read_unlock();
 }
 
 static bool ip6_route_mpath_should_notify(const struct fib6_info *rt)
@@ -5549,17 +5555,19 @@ static size_t rt6_nlmsg_size(struct fib6_info *f6i)
 		nexthop_for_each_fib6_nh(f6i->nh, rt6_nh_nlmsg_size,
 					 &nexthop_len);
 	} else {
-		struct fib6_info *sibling, *next_sibling;
+		struct fib6_info *sibling;
 		struct fib6_nh *nh = f6i->fib6_nh;
 
 		nexthop_len = 0;
 		if (f6i->fib6_nsiblings) {
 			rt6_nh_nlmsg_size(nh, &nexthop_len);
 
-			list_for_each_entry_safe(sibling, next_sibling,
-						 &f6i->fib6_siblings, fib6_siblings) {
+			rcu_read_lock();
+			list_for_each_entry_rcu(sibling, &f6i->fib6_siblings,
+						fib6_siblings) {
 				rt6_nh_nlmsg_size(sibling->fib6_nh, &nexthop_len);
 			}
+			rcu_read_unlock();
 		}
 		nexthop_len += lwtunnel_get_encap_size(nh->fib_nh_lws);
 	}
@@ -5818,13 +5826,17 @@ static bool fib6_info_uses_dev(const struct fib6_info *f6i,
 		return true;
 
 	if (f6i->fib6_nsiblings) {
-		struct fib6_info *sibling, *next_sibling;
+		struct fib6_info *sibling;
 
-		list_for_each_entry_safe(sibling, next_sibling,
-					 &f6i->fib6_siblings, fib6_siblings) {
-			if (sibling->fib6_nh->fib_nh_dev == dev)
+		rcu_read_lock();
+		list_for_each_entry_rcu(sibling,
+					&f6i->fib6_siblings, fib6_siblings) {
+			if (sibling->fib6_nh->fib_nh_dev == dev) {
+				rcu_read_unlock();
 				return true;
+			}
 		}
+		rcu_read_unlock();
 	}
 
 	return false;
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index d9393569d03a..35e3687397da 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -93,6 +93,7 @@ TEST_PROGS += fdb_flush.sh
 TEST_PROGS += fq_band_pktlimit.sh
 TEST_PROGS += vlan_hw_filter.sh
 TEST_PROGS += bpf_offload.py
+TEST_PROGS += ipv6_route_update_soft_lockup.sh
 
 TEST_FILES := settings
 TEST_FILES += in_netns.sh lib.sh net_helper.sh setup_loopback.sh setup_veth.sh
diff --git a/tools/testing/selftests/net/ipv6_route_update_soft_lockup.sh b/tools/testing/selftests/net/ipv6_route_update_soft_lockup.sh
new file mode 100755
index 000000000000..6e2a1e4dd0a6
--- /dev/null
+++ b/tools/testing/selftests/net/ipv6_route_update_soft_lockup.sh
@@ -0,0 +1,217 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Testing for potential kernel soft lockup during IPv6 routing table
+# refresh under heavy outgoing IPv6 traffic. If a kernel soft lockup
+# occurs, a kernel panic will be triggered to prevent associated issues.
+#
+#
+#                            Test Environment Layout
+#
+# ┌-----------------┐                                             ┌------------------┐
+# |     SOURCE_NS   |                                             |     SINK_NS      |
+# |    NAMESPACE    |                                             |    NAMESPACE     |
+# |(iperf3 clients) |                                             | (iperf3 servers) |
+# |                 |                                             |                  |
+# |                 |                                             |                  |
+# |     ┌-----------|                             nexthops        |---------┐        |
+# |     |veth_source|<------------------------------------------->|veth_sink|<-┐     |
+# |     └-----------| 2001:0DB8:1::0:1/96     2001:0DB8:1::1:1/96 |---------┘  |     |
+# |                 |         ^               2001:0DB8:1::1:2/96 |            |     |
+# |                 |         .                       .           |            | fwd |
+# |   ┌---------┐   |         .                       .           |            |     |
+# |   |   IPv6  |   |         .                       .           |            V     |
+# |   | routing |   |         .               2001:0DB8:1::1:90/96|         ┌-----┐  |
+# |   |  table  |   |         .                                   |         | lo  |  |
+# |   | nexthop |   |         .                                   └---------┴-----┴--┘
+# |   | update  |   |         ..................................> 2001:0DB8:2::1:1/128
+# |   └-------- ┘   |
+# └-----------------┘
+#
+# The test script sets up two network namespaces, SOURCE_NS and SINK_NS,
+# connected via a veth link. Within SOURCE_NS, it continuously updates the
+# IPv6 routing table by flushing and inserting IPV6_NEXTHOP_ADDR_COUNT nexthop
+# IPs destined for SINK_LOOPBACK_IP_ADDR in SINK_NS. This refresh occurs at a
+# rate of 1/ROUTING_TABLE_REFRESH_PERIOD per second for TEST_DURATION seconds.
+#
+# Simultaneously, multiple iperf3 clients within SOURCE_NS generate heavy
+# outgoing IPv6 traffic. Each client is assigned a unique port number starting
+# at 5000 and incrementing sequentially. Each client targets a unique iperf3
+# server running in SINK_NS, connected to the SINK_LOOPBACK_IFACE interface
+# using the same port number.
+#
+# The number of iperf3 servers and clients is set to half of the total
+# available cores on each machine.
+#
+# NOTE:  We have tested this script on machines with various CPU specifications,
+# ranging from lower to higher performance as listed below. The test script
+# effectively triggered a kernel soft lockup on machines running an unpatched
+# kernel in under a minute:
+#
+# - 1x Intel Xeon E-2278G 8-Core Processor @ 3.40GHz
+# - 1x Intel Xeon E-2378G Processor 8-Core @ 2.80GHz
+# - 1x AMD EPYC 7401P 24-Core Processor @ 2.00GHz
+# - 1x AMD EPYC 7402P 24-Core Processor @ 2.80GHz
+# - 2x Intel Xeon Gold 5120 14-Core Processor @ 2.20GHz
+# - 1x Ampere Altra Q80-30 80-Core Processor @ 3.00GHz
+# - 2x Intel Xeon Gold 5120 14-Core Processor @ 2.20GHz
+# - 2x Intel Xeon Silver 4214 24-Core Processor @ 2.20GHz
+# - 1x AMD EPYC 7502P 32-Core @ 2.50GHz
+# - 1x Intel Xeon Gold 6314U 32-Core Processor @ 2.30GHz
+# - 2x Intel Xeon Gold 6338 32-Core Processor @ 2.00GHz
+#
+# On less performant machines, you may need to increase the TEST_DURATION
+# parameter to enhance the likelihood of encountering a race condition leading
+# to a kernel soft lockup and avoid a false negative result.
+
+
+TEST_DURATION=120
+export ROUTING_TABLE_REFRESH_PERIOD=0.01
+
+export IPERF3_BITRATE="300m"
+
+
+export IPV6_NEXTHOP_ADDR_COUNT="144"
+export IPV6_NEXTHOP_ADDR_MASK="96"
+export IPV6_NEXTHOP_PREFIX="2001:0DB8:1"
+
+
+export SOURCE_NS="source_ns"
+export SOURCE_TEST_IFACE="veth_source"
+export SOURCE_TEST_IP_ADDR="2001:0DB8:1::0:1/96"
+
+export SINK_NS="sink_ns"
+export SINK_TEST_IFACE="veth_sink"
+# ${SINK_TEST_IFACE} is populated with the following range of IPv6 addresses:
+# 2001:0DB8:1::1:1  to 2001:0DB8:1::1:${IPV6_NEXTHOP_ADDR_COUNT}
+export SINK_LOOPBACK_IFACE="lo"
+export SINK_LOOPBACK_IP_MASK="128"
+export SINK_LOOPBACK_IP_ADDR="2001:0DB8:2::1:1"
+
+nexthop_ip_list=""
+termination_signal=""
+
+cleanup() {
+	echo "info: cleaning up namespaces and terminating all processes within them..."
+
+	kill -9 $(pgrep -f "iperf3")
+
+	# Give some time for processes to terminate
+	sleep 2
+
+	# Check if any iperf3 instances are still running
+	if pgrep -f "iperf3" > /dev/null; then
+		echo "FAIL: unable to terminate some iperf3 instances. Soft lockup is underway. A kernel panic is on the way!"
+		exit 1
+	else
+		if [ "$termination_signal" == "SIGINT" ]; then
+			echo "info: Termination due to ^C (SIGINT)"
+		elif [ "$termination_signal" == "SIGALRM" ]; then
+			echo "PASS: No kernel soft lockup occurred during this ${TEST_DURATION} second test"
+		fi
+	fi
+
+	ip netns delete ${SINK_NS}
+	ip netns delete ${SOURCE_NS}
+}
+
+setup_prepare() {
+	ip netns add ${SOURCE_NS}
+	ip netns add ${SINK_NS}
+
+	ip link add ${SOURCE_TEST_IFACE} type veth peer name ${SINK_TEST_IFACE}
+
+	ip link set ${SOURCE_TEST_IFACE} netns ${SOURCE_NS}
+	ip link set ${SINK_TEST_IFACE}   netns ${SINK_NS}
+
+	# Setting up the Source namespace
+	ip netns exec ${SOURCE_NS} ip addr add ${SOURCE_TEST_IP_ADDR} dev ${SOURCE_TEST_IFACE}
+	ip netns exec ${SOURCE_NS} ip link set dev ${SOURCE_TEST_IFACE} qlen 10000
+	ip netns exec ${SOURCE_NS} ip link set dev ${SOURCE_TEST_IFACE} up
+	ip netns exec ${SOURCE_NS} sysctl -qw net.ipv6.fib_multipath_hash_policy=1
+
+	# Setting up the Sink namespace
+	ip netns exec ${SINK_NS} ip addr add ${SINK_LOOPBACK_IP_ADDR}/${SINK_LOOPBACK_IP_MASK} dev ${SINK_LOOPBACK_IFACE}
+	ip netns exec ${SINK_NS} ip link set dev ${SINK_LOOPBACK_IFACE} up
+	ip netns exec ${SINK_NS} sysctl -qw net.ipv6.conf.${SINK_LOOPBACK_IFACE}.forwarding=1
+
+	ip netns exec ${SINK_NS} ip link set ${SINK_TEST_IFACE} up
+	ip netns exec ${SINK_NS} sysctl -qw net.ipv6.conf.${SINK_TEST_IFACE}.forwarding=1
+
+
+	# Populating Nexthop IPv6 addresses on the test interface of the SINK_NS namespace
+	echo "info: populating ${IPV6_NEXTHOP_ADDR_COUNT} IPv6 addresses on the ${SINK_TEST_IFACE} interface ..."
+	ip netns exec ${SINK_NS} bash -c '
+		for IP in $(seq 1 ${IPV6_NEXTHOP_ADDR_COUNT}); do
+			ip addr add ${IPV6_NEXTHOP_PREFIX}::$(printf "1:%x" "${IP}")/${IPV6_NEXTHOP_ADDR_MASK} dev ${SINK_TEST_IFACE};
+		done'
+
+	# Preparing list of nexthops
+	for IP in $(seq 1 ${IPV6_NEXTHOP_ADDR_COUNT}); do
+		nexthop_ip_list=$nexthop_ip_list" nexthop via ${IPV6_NEXTHOP_PREFIX}::$(printf "1:%x" $IP) dev ${SOURCE_TEST_IFACE} weight 1"
+	done
+
+	export nexthop_ip_list
+}
+
+
+test_soft_lockup_during_routing_table_refresh() {
+	# Start num_of_iperf_servers iperf3 servers in the SINK_NS namespace, each listening on ports
+	# starting at 5001 and incrementing sequentially.
+	echo "info: starting ${num_of_iperf_servers} iperf3 servers in the ${SINK_NS} namespace ..."
+	for i in $(seq 1 ${num_of_iperf_servers}); do
+		cmd="iperf3 --bind ${SINK_LOOPBACK_IP_ADDR} -s -p $(printf '5%03d' ${i}) > /dev/null 2>&1"
+		ip netns exec ${SINK_NS} bash -c "while true; do ${cmd}; done &"
+	done
+
+	# Continuously refresh the routing table in background in the Source namespase
+	ip netns exec ${SOURCE_NS} bash -c '
+		while $(ip netns list | grep -q ${SOURCE_NS}); do
+			ip -6 route add ${SINK_LOOPBACK_IP_ADDR}/${SINK_LOOPBACK_IP_MASK} ${nexthop_ip_list};
+			sleep ${ROUTING_TABLE_REFRESH_PERIOD};
+			ip -6 route delete ${SINK_LOOPBACK_IP_ADDR}/${SINK_LOOPBACK_IP_MASK};
+		done &'
+
+	# Start num_of_iperf_servers iperf3 clients in the SOURCE_NS namespace, each sending UDP packets on sequential ports starting at 5001
+	echo "info: starting ${num_of_iperf_servers} iperf3 clients in the ${SOURCE_NS} namespace ..."
+	for i in $(seq 1 ${num_of_iperf_servers}); do
+		cmd="iperf3 -c ${SINK_LOOPBACK_IP_ADDR} -p $(printf '5%03d' ${i}) --udp --length 64 --bitrate ${IPERF3_BITRATE} -t 0 > /dev/null 2>&1"
+		ip netns exec ${SOURCE_NS} bash -c "while true; do ${cmd}; done &"
+	done
+
+	echo "info: IPv6 routing table is being updated at the rate of $(echo "1/${ROUTING_TABLE_REFRESH_PERIOD}" | bc)/s for ${TEST_DURATION} seconds ..."
+	echo "info: A kernel soft lockup, if detected, results in a kernel panic!"
+
+	wait
+}
+
+# Determine the number of cores on the machine
+num_of_iperf_servers=$(( $(nproc)/2 ))
+
+# Check if we are running on a multi-core machine, exit otherwise
+if [ "${num_of_iperf_servers}" -eq 0 ]; then
+	echo "FAIL: This test is not valid on a single core machine!"
+	exit 1
+fi
+
+sysctl -qw kernel.softlockup_panic=1
+
+handle_sigint() {
+	termination_signal="SIGINT"
+	cleanup
+	exit 1
+}
+
+handle_sigalrm() {
+	termination_signal="SIGALRM"
+	cleanup
+	exit 0
+}
+
+trap handle_sigint SIGINT
+trap handle_sigalrm SIGALRM
+
+(sleep ${TEST_DURATION} && kill -s SIGALRM $$)&
+
+setup_prepare
+test_soft_lockup_during_routing_table_refresh
-- 
2.34.1

Powered by blists - more mailing lists