| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1287028842.11178.68.camel@localhost>
Date: Thu, 14 Oct 2010 05:00:42 +0100
From: Ben Hutchings <ben@...adent.org.uk>
To: netdev@...r.kernel.org
Cc: Beatrice Barbe <beatrice.barbe@...il.com>, 599816@...s.debian.org
Subject: Nested GRE locking bug
Beatrice Barbe reported a reproducible crash after creating large
numbers of nested GRE tunnels and then pinging with the source address
forced. I was able to reproduce this using net-2.6. I'm attaching the
kernel config I used and a script to reproduce this based on the script
she provided. The magic number of tunnels to create is apparently 37.
With lockdep enabled, I get the following output:
=============================================
[ INFO: possible recursive locking detected ]
2.6.36-rc7-00040-gb0057c5 #5
---------------------------------------------
ping/2199 is trying to acquire lock:
(_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454
but task is already holding lock:
(_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454
other info that might help us debug this:
4 locks held by ping/2199:
#0: (sk_lock-AF_INET){+.+.+.}, at: [<c1168c46>] raw_sendmsg+0x590/0x64c
#1: (rcu_read_lock_bh){.+....}, at: [<c11395ea>] dev_queue_xmit+0x0/0x454
#2: (_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454
#3: (rcu_read_lock_bh){.+....}, at: [<c11395ea>] dev_queue_xmit+0x0/0x454
stack backtrace:
Pid: 2199, comm: ping Not tainted 2.6.36-rc7-00040-gb0057c5 #5
Call Trace:
[<c1187b3c>] ? printk+0xf/0x13
[<c103a942>] __lock_acquire+0xbda/0x1311
[<c103a32b>] ? __lock_acquire+0x5c3/0x1311
[<c103b0d2>] lock_acquire+0x59/0x77
[<c1139968>] ? dev_queue_xmit+0x37e/0x454
[<c11898b4>] _raw_spin_lock+0x1b/0x2a
[<c1139968>] ? dev_queue_xmit+0x37e/0x454
[<c1139968>] dev_queue_xmit+0x37e/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c1151382>] ? ip_append_data+0x536/0x7dc
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c1151851>] ? ip_generic_getfrag+0x0/0x8a
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c1150dff>] ip_push_pending_frames+0x260/0x2ad
[<c1168c85>] raw_sendmsg+0x5cf/0x64c
[<c11708ad>] inet_sendmsg+0x46/0x4f
[<c112cea9>] sock_sendmsg+0xa4/0xba
[<c105897d>] ? might_fault+0x35/0x6f
[<c105897d>] ? might_fault+0x35/0x6f
[<c1134614>] ? verify_iovec+0x3e/0x6a
[<c112d2af>] sys_sendmsg+0x149/0x196
[<c104b079>] ? unlock_page+0x3f/0x42
[<c103b176>] ? lock_release_non_nested+0x86/0x221
[<c105897d>] ? might_fault+0x35/0x6f
[<c105897d>] ? might_fault+0x35/0x6f
[<c112e287>] sys_socketcall+0x146/0x18b
[<c10cb5c8>] ? trace_hardirqs_on_thunk+0xc/0x10
[<c1189f5d>] syscall_call+0x7/0xb
------------[ cut here ]------------
WARNING: at kernel/softirq.c:143 local_bh_enable_ip+0x39/0xa5()
Hardware name: Bochs
Pid: 2199, comm: ping Not tainted 2.6.36-rc7-00040-gb0057c5 #5
Call Trace:
[<c101a092>] warn_slowpath_common+0x60/0x75
[<c101e534>] ? local_bh_enable_ip+0x39/0xa5
[<c114b993>] ? rt_intern_hash+0x4da/0x4f9
[<c101a0b6>] warn_slowpath_null+0xf/0x13
[<c101e534>] local_bh_enable_ip+0x39/0xa5
[<c1189d4e>] _raw_spin_unlock_bh+0x25/0x28
[<c114b993>] rt_intern_hash+0x4da/0x4f9
[<c114c1b8>] __ip_route_output_key+0x806/0x860
[<c114c220>] ip_route_output_flow+0xe/0x3e
[<c114c25c>] ip_route_output_key+0xc/0xe
[<c11793d6>] ipgre_tunnel_xmit+0x1ac/0x757
[<c1139968>] ? dev_queue_xmit+0x37e/0x454
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
[<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
[<c11394fc>] dev_hard_start_xmit+0x33a/0x428
[<c1139987>] dev_queue_xmit+0x39d/0x454
[<c1151382>] ? ip_append_data+0x536/0x7dc
[<c115292e>] ip_finish_output+0x29d/0x2c7
[<c1151851>] ? ip_generic_getfrag+0x0/0x8a
[<c11529e2>] ip_output+0x8a/0x8f
[<c1150b9c>] ip_local_out+0x50/0x53
[<c1150dff>] ip_push_pending_frames+0x260/0x2ad
[<c1168c85>] raw_sendmsg+0x5cf/0x64c
[<c11708ad>] inet_sendmsg+0x46/0x4f
[<c112cea9>] sock_sendmsg+0xa4/0xba
[<c105897d>] ? might_fault+0x35/0x6f
[<c105897d>] ? might_fault+0x35/0x6f
[<c1134614>] ? verify_iovec+0x3e/0x6a
[<c112d2af>] sys_sendmsg+0x149/0x196
[<c104b079>] ? unlock_page+0x3f/0x42
[<c103b176>] ? lock_release_non_nested+0x86/0x221
[<c105897d>] ? might_fault+0x35/0x6f
[<c105897d>] ? might_fault+0x35/0x6f
[<c112e287>] sys_socketcall+0x146/0x18b
[<c10cb5c8>] ? trace_hardirqs_on_thunk+0xc/0x10
[<c1189f5d>] syscall_call+0x7/0xb
<IRQ>
Ben.
--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
View attachment ".config" of type "text/x-mpsub" (28654 bytes)
Download attachment "tunnels.sh" of type "application/x-shellscript" (1116 bytes)
Download attachment "signature.asc" of type "application/pgp-signature" (829 bytes)
Powered by blists - more mailing lists