netdev - 4.15.2 kernel panic, nat, ppp bug?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <ee73af5b3e036c3923ba40874d1ae872@nuclearcat.com>
Date:   Mon, 12 Feb 2018 12:02:16 +0200
From:   Denys Fedoryshchenko <nuclearcat@...learcat.com>
To:     Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: 4.15.2 kernel panic, nat, ppp bug?

Hello,

Got this and then server rebooted with panic (second message).
Workload: pppoe BRAS, lost of shapers, ppp interfaces

Please let me know if i need to provide more information

Feb 12 06:00:58 <srv> [13750.606169] WARNING: CPU: 6 PID: 0 at 
./include/net/dst.h:256 nf_xfrm_me_harder+0x52/0xd9 [nf_nat]
Feb 12 06:00:58 <srv> [13750.606747] Modules linked in: pppoe pppox 
ppp_generic slhc netconsole configfs coretemp nf_nat_pptp 
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre tun xt_TEE 
nf_dup_ipv4
  xt_REDIRECT nf_nat_redirect xt_nat xt_TCPMSS ipt_REJECT nf_reject_ipv4 
xt_set xt_string xt_connmark xt_DSCP xt_mark xt_tcpudp ip_set_hash_net 
ip_set_hash_ip ip_set nfnetlink iptable_mangle iptable_filter iptabl
e_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack 
ip_tables x_tables 8021q garp mrp stp llc ixgbe dca
Feb 12 06:00:58 <srv> [13750.608695] CPU: 6 PID: 0 Comm: swapper/6 Not 
tainted 4.15.2-build-0134 #5
Feb 12 06:00:58 <srv> [13750.609017] Hardware name: HP ProLiant DL320e 
Gen8 v2, BIOS P80 04/02/2015
Feb 12 06:00:58 <srv> [13750.609345] RIP: 
0010:nf_xfrm_me_harder+0x52/0xd9 [nf_nat]
Feb 12 06:00:58 <srv> [13750.609667] RSP: 0018:ffff880434183c88 EFLAGS: 
00010246
Feb 12 06:00:58 <srv> [13750.609985] RAX: 0000000000000000 RBX: 
ffff8803f997ce00 RCX: 0000000000000000
Feb 12 06:00:58 <srv> [13750.610306] RDX: 0000000000000001 RSI: 
ffff880406c09a00 RDI: ffff880434183cc8
Feb 12 06:00:58 <srv> [13750.610629] RBP: ffffffff822a81c0 R08: 
0000000000000005 R09: 0000000000000001
Feb 12 06:00:58 <srv> [13750.610949] R10: 00000000000000ce R11: 
ffff88043154c320 R12: 0000000000000001
Feb 12 06:00:58 <srv> [13750.611274] R13: ffff880434183d50 R14: 
0000000000e20008 R15: ffff88042e320078
Feb 12 06:00:58 <srv> [13750.611599] FS:  0000000000000000(0000) 
GS:ffff880434180000(0000) knlGS:0000000000000000
Feb 12 06:00:58 <srv> [13750.612181] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Feb 12 06:00:58 <srv> [13750.612500] CR2: 00007f12eed3a140 CR3: 
0000000446209003 CR4: 00000000001606e0
Feb 12 06:00:58 <srv> [13750.612823] Call Trace:
Feb 12 06:00:58 <srv> [13750.613138]  <IRQ>
Feb 12 06:00:58 <srv> [13750.613457]  nf_nat_ipv4_out+0xa5/0xb9 
[nf_nat_ipv4]
Feb 12 06:00:58 <srv> [13750.613780]  nf_hook_slow+0x31/0x93
Feb 12 06:00:58 <srv> [13750.614101]  ip_output+0x93/0xaf
Feb 12 06:00:58 <srv> [13750.614417]  ? 
ip_fragment.constprop.5+0x6e/0x6e
Feb 12 06:00:58 <srv> [13750.614739]  ip_forward+0x36d/0x378
Feb 12 06:00:58 <srv> [13750.615057]  ? ip_frag_mem+0x7/0x7
Feb 12 06:00:58 <srv> [13750.615376]  ip_rcv+0x2f0/0x325
Feb 12 06:00:58 <srv> [13750.615698]  ? 
ip_local_deliver_finish+0x1a8/0x1a8
Feb 12 06:00:58 <srv> [13750.616019]  
__netif_receive_skb_core+0x535/0x8b5
Feb 12 06:00:58 <srv> [13750.616340]  ? kmem_cache_free_bulk+0x21b/0x233
Feb 12 06:00:58 <srv> [13750.616661]  ? process_backlog+0x99/0x115
Feb 12 06:00:58 <srv> [13750.616981]  process_backlog+0x99/0x115
Feb 12 06:00:58 <srv> [13750.617300]  net_rx_action+0x11c/0x28a
Feb 12 06:00:58 <srv> [13750.617620]  __do_softirq+0xc8/0x1bf
Feb 12 06:00:58 <srv> [13750.617941]  irq_exit+0x49/0x88
Feb 12 06:00:58 <srv> [13750.618262]  
call_function_single_interrupt+0x92/0xa0
Feb 12 06:00:58 <srv> [13750.618587]  </IRQ>
Feb 12 06:00:58 <srv> [13750.618907] RIP: 0010:mwait_idle+0x4c/0x5e
Feb 12 06:00:58 <srv> [13750.619227] RSP: 0018:ffffc9000192bf08 EFLAGS: 
00000246 ORIG_RAX: ffffffffffffff04
Feb 12 06:00:58 <srv> [13750.619803] RAX: 0000000000000000 RBX: 
ffff88043296cc80 RCX: 0000000000000000
Feb 12 06:00:58 <srv> [13750.620120] RDX: 0000000000000000 RSI: 
0000000000000000 RDI: 0000000000000000
Feb 12 06:00:58 <srv> [13750.620436] RBP: 0000000000000000 R08: 
00525ccae333e271 R09: 0000000000023738
Feb 12 06:00:58 <srv> [13750.620750] R10: ffffc9000192be98 R11: 
00000000000236e0 R12: 0000000000000000
Feb 12 06:00:58 <srv> [13750.621065] R13: 0000000000000006 R14: 
ffff88043296cc80 R15: ffff88043296cc80
Feb 12 06:00:58 <srv> [13750.621394]  ? 
rcu_eqs_enter_common.constprop.54+0x57/0x5f
Feb 12 06:00:58 <srv> [13750.621714]  do_idle+0xa8/0x130
Feb 12 06:00:58 <srv> [13750.622032]  cpu_startup_entry+0x18/0x1a
Feb 12 06:00:58 <srv> [13750.622349]  secondary_startup_64+0xa5/0xb0
Feb 12 06:00:58 <srv> [13750.622667] Code: 48 83 e6 fe 48 83 7e 48 00 74 
07 48 8b b6 80 01 00 00 8b 86 80 00 00 00 85 c0 74 0f 8d 50 01 f0 0f b1 
96 80 00 00 00 74 04 eb ed <0f> ff 48 8b 4b 18 48 8d 54 24 08
  45 31 c0 48 89 ef e8 44 91 8d
Feb 12 06:00:58 <srv> [13750.623533] ---[ end trace 807c68f3da1711db 
]---
Feb 12 06:00:58 <srv> [13750.623863] dst_release: dst:00000000ad86ddff 
refcnt:-1


Feb 12 09:40:45 <srv> [26937.094365] WARNING: CPU: 5 PID: 0 at 
./include/net/dst.h:256 nf_xfrm_me_harder+0x52/0xd9 [nf_nat]
Feb 12 09:40:45 <srv> [26937.094958] Modules linked in: pppoe pppox 
ppp_generic slhc netconsole configfs coretemp nf_nat_pptp 
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre tun xt_TEE 
nf_dup_ipv4
  xt_REDIRECT nf_nat_redirect xt_nat xt_TCPMSS ipt_REJECT nf_reject_ipv4 
xt_set xt_string xt_connmark xt_DSCP xt_mark xt_tcpudp ip_set_hash_net 
ip_set_hash_ip ip_set nfnetlink iptable_mangle iptable_filter iptabl
e_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack 
ip_tables x_tables 8021q garp mrp stp llc ixgbe dca
Feb 12 09:40:45 <srv> [26937.096920] CPU: 5 PID: 0 Comm: swapper/5 
Tainted: G        W        4.15.2-build-0134 #5
Feb 12 09:40:45 <srv> [26937.097513] Hardware name: HP ProLiant DL320e 
Gen8 v2, BIOS P80 04/02/2015
Feb 12 09:40:45 <srv> [26937.097843] RIP: 
0010:nf_xfrm_me_harder+0x52/0xd9 [nf_nat]
Feb 12 09:40:45 <srv> [26937.098170] RSP: 0018:ffff880434143c88 EFLAGS: 
00010246
Feb 12 09:40:45 <srv> [26937.098495] RAX: 0000000000000000 RBX: 
ffff88040e5ce700 RCX: 0000000000000000
Feb 12 09:40:45 <srv> [26937.098823] RDX: 0000000000000001 RSI: 
ffff88040c95f900 RDI: ffff880434143cc8
Feb 12 09:40:45 <srv> [26937.099149] RBP: ffffffff822a81c0 R08: 
0000000000000005 R09: 0000000000000001
Feb 12 09:40:45 <srv> [26937.099477] R10: 00000000000000ce R11: 
ffff88043154c320 R12: 0000000000000001
Feb 12 09:40:45 <srv> [26937.099807] R13: ffff880434143d50 R14: 
0000000000e20008 R15: ffff88042e320078
Feb 12 09:40:45 <srv> [26937.100135] FS:  0000000000000000(0000) 
GS:ffff880434140000(0000) knlGS:0000000000000000
Feb 12 09:40:45 <srv> [26937.100728] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Feb 12 09:40:45 <srv> [26937.101054] CR2: 00005597aef7c188 CR3: 
0000000446209005 CR4: 00000000001606e0
Feb 12 09:40:45 <srv> [26937.101382] Call Trace:
Feb 12 09:40:45 <srv> [26937.101704]  <IRQ>
Feb 12 09:40:45 <srv> [26937.102030]  nf_nat_ipv4_out+0xa5/0xb9 
[nf_nat_ipv4]
Feb 12 09:40:45 <srv> [26937.102358]  nf_hook_slow+0x31/0x93
Feb 12 09:40:45 <srv> [26937.102686]  ip_output+0x93/0xaf
Feb 12 09:40:45 <srv> [26937.103010]  ? 
ip_fragment.constprop.5+0x6e/0x6e
Feb 12 09:40:45 <srv> [26937.103337]  ip_forward+0x36d/0x378
Feb 12 09:40:45 <srv> [26937.103661]  ? ip_frag_mem+0x7/0x7
Feb 12 09:40:45 <srv> [26937.103987]  ip_rcv+0x2f0/0x325
Feb 12 09:40:45 <srv> [26937.104317]  ? 
ip_local_deliver_finish+0x1a8/0x1a8
Feb 12 09:40:45 <srv> [26937.104645]  
__netif_receive_skb_core+0x535/0x8b5
Feb 12 09:40:45 <srv> [26937.104974]  ? process_backlog+0x99/0x115
Feb 12 09:40:45 <srv> [26937.105302]  process_backlog+0x99/0x115
Feb 12 09:40:45 <srv> [26937.105628]  net_rx_action+0x11c/0x28a
Feb 12 09:40:45 <srv> [26937.105955]  __do_softirq+0xc8/0x1bf
Feb 12 09:40:45 <srv> [26937.106280]  irq_exit+0x49/0x88
Feb 12 09:40:45 <srv> [26937.106604]  
call_function_single_interrupt+0x92/0xa0
Feb 12 09:40:45 <srv> [26937.106934]  </IRQ>
Feb 12 09:40:45 <srv> [26937.107261] RIP: 0010:mwait_idle+0x4c/0x5e
Feb 12 09:40:45 <srv> [26937.107585] RSP: 0018:ffffc90001923f08 EFLAGS: 
00000246 ORIG_RAX: ffffffffffffff04
Feb 12 09:40:45 <srv> [26937.108177] RAX: 0000000000000000 RBX: 
ffff88043296bfc0 RCX: 0000000000000000
Feb 12 09:40:45 <srv> [26937.108500] RDX: 0000000000000000 RSI: 
0000000000000000 RDI: 0000000000000000
Feb 12 09:40:45 <srv> [26937.108830] RBP: 0000000000000000 R08: 
0052844734cde3b2 R09: 000000000001ee80
Feb 12 09:40:45 <srv> [26937.109160] R10: ffffc90001923e98 R11: 
0000000000000046 R12: 0000000000000000
Feb 12 09:40:45 <srv> [26937.109489] R13: 0000000000000005 R14: 
ffff88043296bfc0 R15: ffff88043296bfc0
Feb 12 09:40:45 <srv> [26937.109815]  ? 
rcu_eqs_enter_common.constprop.54+0x57/0x5f
Feb 12 09:40:45 <srv> [26937.110136]  do_idle+0xa8/0x130
Feb 12 09:40:45 <srv> [26937.110449]  cpu_startup_entry+0x18/0x1a
Feb 12 09:40:45 <srv> [26937.110769]  secondary_startup_64+0xa5/0xb0
Feb 12 09:40:45 <srv> [26937.111090] Code: 48 83 e6 fe 48 83 7e 48 00 74 
07 48 8b b6 80 01 00 00 8b 86 80 00 00 00 85 c0 74 0f 8d 50 01 f0 0f b1 
96 80 00 00 00 74 04 eb ed <0f> ff 48 8b 4b 18 48 8d 54 24 08
  45 31 c0 48 89 ef e8 44 91 8d
Feb 12 09:40:45 <srv> [26937.111976] ---[ end trace 807c68f3da1711dc 
]---
Feb 12 09:40:45 <srv> [26937.112325] dst_release: dst:00000000c89976c9 
refcnt:-1


Paniced:

Feb 12 10:03:53 <srv> [28325.320917] general protection fault: 0000 [#1] 
SMP
Feb 12 10:03:53 <srv> [28325.321244] Modules linked in: pppoe pppox 
ppp_generic slhc netconsole configfs coretemp nf_nat_pptp 
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre tun xt_TEE 
nf_dup_ipv4
  xt_REDIRECT nf_nat_redirect xt_nat xt_TCPMSS ipt_REJECT nf_reject_ipv4 
xt_set xt_string xt_connmark xt_DSCP xt_mark xt_tcpudp ip_set_hash_net 
ip_set_hash_ip ip_set nfnetlink iptable_mangle iptable_filter iptabl
e_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack 
ip_tables x_tables 8021q garp mrp stp llc ixgbe dca
Feb 12 10:03:53 <srv> [28325.329584] CPU: 0 PID: 12546 Comm: accel-pppd 
Tainted: G        W        4.15.2-build-0134 #5
Feb 12 10:03:53 <srv> [28325.330170] Hardware name: HP ProLiant DL320e 
Gen8 v2, BIOS P80 04/02/2015
Feb 12 10:03:53 <srv> [28325.330502] RIP: 0010:ppp_push+0x73/0x4ec 
[ppp_generic]
Feb 12 10:03:53 <srv> [28325.330824] RSP: 0018:ffffc90001fe7d50 EFLAGS: 
00010286
Feb 12 10:03:53 <srv> [28325.331149] RAX: 045da3d970030000 RBX: 
ffff8803fe8087b8 RCX: 0000000000000101
Feb 12 10:03:53 <srv> [28325.331475] RDX: 0000000000000000 RSI: 
ffff88041b68cb00 RDI: ffff8803e9a44dc3
Feb 12 10:03:53 <srv> [28325.331800] RBP: ffff88041b68cb00 R08: 
0000000000000001 R09: 00000000ffffffff
Feb 12 10:03:53 <srv> [28325.332129] R10: 00007f04d0620768 R11: 
0000000000000293 R12: 0000000000008021
Feb 12 10:03:53 <srv> [28325.332455] R13: ffff8804159af880 R14: 
0000000000008021 R15: ffff8804159af800
Feb 12 10:03:53 <srv> [28325.332783] FS:  00007f04db733700(0000) 
GS:ffff880434000000(0000) knlGS:0000000000000000
Feb 12 10:03:53 <srv> [28325.333372] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Feb 12 10:03:53 <srv> [28325.333697] CR2: 00007fb7b9650140 CR3: 
000000042a318002 CR4: 00000000001606f0
Feb 12 10:03:53 <srv> [28325.334023] Call Trace:
Feb 12 10:03:53 <srv> [28325.334347]  ? 
__kmalloc_node_track_caller+0xb5/0xd6
Feb 12 10:03:53 <srv> [28325.334674]  __ppp_xmit_process+0x35/0x4c6 
[ppp_generic]
Feb 12 10:03:53 <srv> [28325.334997]  ppp_xmit_process+0x35/0x88 
[ppp_generic]
Feb 12 10:03:53 <srv> [28325.335318]  ppp_write+0xb1/0xbb [ppp_generic]
Feb 12 10:03:53 <srv> [28325.335636]  __vfs_write+0x1c/0x118
Feb 12 10:03:53 <srv> [28325.335955]  ? SyS_epoll_ctl+0x399/0x871
Feb 12 10:03:53 <srv> [28325.336272]  vfs_write+0xc6/0x169
Feb 12 10:03:53 <srv> [28325.336588]  SyS_write+0x48/0x81
Feb 12 10:03:53 <srv> [28325.336904]  do_syscall_64+0x5f/0xea
Feb 12 10:03:53 <srv> [28325.337229]  
entry_SYSCALL_64_after_hwframe+0x21/0x86
Feb 12 10:03:53 <srv> [28325.337555] RIP: 0033:0x7f04daf1ab2d
Feb 12 10:03:53 <srv> [28325.337878] RSP: 002b:00007f04db732b00 EFLAGS: 
00000293 ORIG_RAX: 0000000000000001
Feb 12 10:03:53 <srv> [28325.338467] RAX: ffffffffffffffda RBX: 
00007f04d9a661e3 RCX: 00007f04daf1ab2d
Feb 12 10:03:53 <srv> [28325.338795] RDX: 000000000000000c RSI: 
00007f04d1ad0560 RDI: 0000000000002b07
Feb 12 10:03:53 <srv> [28325.339123] RBP: 00007f04db732b30 R08: 
0000000000000000 R09: 000055cfb7641e40
Feb 12 10:03:53 <srv> [28325.339451] R10: 00007f04d0620768 R11: 
0000000000000293 R12: 00007f04d1312c98
Feb 12 10:03:53 <srv> [28325.339776] R13: 00007ffe1a7bb06f R14: 
0000000000000000 R15: 00007f04db733700
Feb 12 10:03:53 <srv> [28325.340105] Code: 94 00 00 00 49 89 ff 0f ba e0 
0a 72 43 48 8b 5f 68 48 8d 7b e8 e8 48 4f 84 e1 48 8b 7b b8 48 85 ff 74 
10 48 8b 47 08 48 8b 34 24 <ff> 10 85 c0 75 0b eb 14 48 8b 3c
  24 e8 c3 6c 76 e1 49 c7 87 c8
Feb 12 10:03:53 <srv> [28325.340983] RIP: ppp_push+0x73/0x4ec 
[ppp_generic] RSP: ffffc90001fe7d50
Feb 12 10:03:53 <srv> [28325.341350] ---[ end trace 807c68f3da1711dd 
]---
Feb 12 10:03:53 <srv> [28325.345063] Kernel panic - not syncing: Fatal 
exception in interrupt
Feb 12 10:03:53 <srv> [28325.345407] Kernel Offset: disabled
Feb 12 10:03:53 <srv> [28325.349038] Rebooting in 5 seconds..