Date: Wed, 26 Jul 2023 15:38:16 +0200
From: Ian Kumlien <ian.kumlien@...il.com>
To: Pablo Neira Ayuso <pablo@...filter.org>
Cc: Jakub Kicinski <kuba@...nel.org>, 
	Linux Kernel Network Developers <netdev@...r.kernel.org>, netfilter-devel@...r.kernel.org
Subject: Re: Kernel oops with 6.4.4 - flow offloads - NULL pointer deref

On Wed, Jul 26, 2023 at 2:00 PM Ian Kumlien <ian.kumlien@...il.com> wrote:
>
> On Wed, Jul 26, 2023 at 10:03 AM Pablo Neira Ayuso <pablo@...filter.org> wrote:
> >
> > Hi,
> >
> > On Mon, Jul 24, 2023 at 02:24:15PM -0700, Jakub Kicinski wrote:
> > > Adding netfilter to CC.
> > >
> > > On Sun, 23 Jul 2023 16:44:50 +0200 Ian Kumlien wrote:
> > > > Running vanilla 6.4.4 with cherry picked:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.4.5&id=7a59f29961cf97b98b02acaadf5a0b1f8dde938c
> > > >
> > [...]
> > > > [108431.305700] RSP: 0018:ffffac250ade7e28 EFLAGS: 00010206
> > > > [108431.311107] RAX: 0000000000000081 RBX: ffff9ebc413b42f8 RCX:
> > > > 0000000000000001
> > > > [108431.318420] RDX: 00000001067200c0 RSI: ffff9ebeda71ce58 RDI:
> > > > ffff9ebeda71ce58
> > > > [108431.325735] RBP: ffff9ebc413b4250 R08: ffff9ebc413b4250 R09:
> > > > ffff9ebe3d7fad58
> > > > [108431.333068] R10: 0000000000000000 R11: 0000000000000003 R12:
> > > > ffff9ebfafab0000
> > > > [108431.340415] R13: 0000000000000000 R14: ffff9ebfafab0005 R15:
> > > > ffff9ebd79a0f780
> > > > [108431.347764] FS:  0000000000000000(0000) GS:ffff9ebfafa80000(0000)
> > > > knlGS:0000000000000000
> > > > [108431.356069] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [108431.362012] CR2: 0000000000000081 CR3: 000000045e99e000 CR4:
> > > > 00000000003526e0
> > > > [108431.369361] Call Trace:
> > > > [108431.371999]  <TASK>
> > > > [108431.374296] ? __die (arch/x86/kernel/dumpstack.c:421
> > > > arch/x86/kernel/dumpstack.c:434)
> > > > [108431.377553] ? page_fault_oops (arch/x86/mm/fault.c:707)
> > > > [108431.381850] ? load_balance (kernel/sched/fair.c:10926)
> > > > [108431.385884] ? exc_page_fault (arch/x86/mm/fault.c:1279
> > > > arch/x86/mm/fault.c:1486 arch/x86/mm/fault.c:1542)
> > > > [108431.390094] ? asm_exc_page_fault (./arch/x86/include/asm/idtentry.h:570)
> > > > [108431.394482] ? flow_offload_teardown
> > > > (./arch/x86/include/asm/bitops.h:75
> > > > ./include/asm-generic/bitops/instrumented-atomic.h:42
> > > > net/netfilter/nf_flow_table_core.c:362)
> > > > [108431.399036] nf_flow_offload_gc_step
> > > > (./arch/x86/include/asm/bitops.h:207
> > > > ./arch/x86/include/asm/bitops.h:239
> > > > ./include/asm-generic/bitops/instrumented-non-atomic.h:142
> > > > net/netfilter/nf_flow_table_core.c:436)
> >
> > This crash points here.
> >
> > static void nf_flow_offload_gc_step(struct nf_flowtable *flow_table,
> >                                     struct flow_offload *flow, void *data)
> > {
> >         if (nf_flow_has_expired(flow) ||
> >             nf_ct_is_dying(flow->ct) ||
> >             nf_flow_is_outdated(flow))
> >                 flow_offload_teardown(flow);
> >
> >         if (test_bit(NF_FLOW_TEARDOWN, &flow->flags)) { <--
> >
> > Is this always reproducible on your testbed?
>
> That's a bit unknown, I don't quite know what triggers it... I only
> know it's happened twice :/
> (That i've noticed - the fw runs with a watchdog and it's always been
> a "uhuh... uptime is less than expected" kind of thing)

I should add that i do:
for interface in eno1 eno2 eno3 eno4 ; do
for offload in ntuple hw-tc-offload rx-udp-gro-forwarding rx-gro-list ; do
ethtool -K $interface $offload on > /dev/null
done
done

And that some interfaces are directly attached to a bridge while
others are more normal

lspci |grep Ethernet
06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
X553 1GbE (rev 11)
06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
X553 1GbE (rev 11)
07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
X553 1GbE (rev 11)
07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
X553 1GbE (rev 11)

This is since i added NET_SCHED etc support back in to the kernel

tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eno1 root
qdisc fq 0: dev eno1 parent :c limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :b limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :a limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :9 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :8 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :7 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :6 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :5 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :4 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :3 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :2 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno1 parent :1 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc mq 0: dev eno2 root
qdisc fq 0: dev eno2 parent :c limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :b limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :a limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :9 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :8 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :7 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :6 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :5 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :4 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :3 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :2 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno2 parent :1 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc mq 0: dev eno3 root
qdisc fq 0: dev eno3 parent :c limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :b limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :a limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :9 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :8 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :7 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :6 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :5 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :4 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :3 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :2 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno3 parent :1 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc mq 0: dev eno4 root
qdisc fq 0: dev eno4 parent :c limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :b limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :a limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :9 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :8 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :7 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :6 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :5 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :4 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :3 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :2 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc fq 0: dev eno4 parent :1 limit 10000p flow_limit 100p buckets
1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b
low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon
10s horizon_drop
qdisc noqueue 0: dev external-net root refcnt 2
qdisc noqueue 0: dev local-lan root refcnt 2
qdisc noqueue 0: dev virtual-net root refcnt 2
qdisc noqueue 8001: dev vnet0 root refcnt 2
qdisc noqueue 8002: dev vnet1 root refcnt 2
qdisc noqueue 8003: dev vnet2 root refcnt 2
qdisc noqueue 8004: dev vnet3 root refcnt 2
qdisc noqueue 8005: dev vnet4 root refcnt 2
qdisc noqueue 8006: dev vnet5 root refcnt 2
qdisc noqueue 8007: dev vnet6 root refcnt 2
qdisc noqueue 8008: dev vnet7 root refcnt 2
qdisc noqueue 0: dev int root refcnt 2


> > Thanks.

Powered by blists - more mailing lists