linux-kernel - Re: [next-20130204] hard LOCKUP on cpu 5, 3, 4, 2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <1360732466.3634.3.camel@thor.lan>
Date:	Wed, 13 Feb 2013 00:14:26 -0500
From:	Peter Hurley <peter@...leysoftware.com>
To:	Alexander Viro <viro@...iv.linux.org.uk>,
	Alasdair Kergon <agk@...hat.com>, Theodore Ts'o <tytso@....edu>
Cc:	linux-fsdevel@...r.kernel.org, linux-ext4@...r.kernel.org,
	dm-devel@...hat.com, linux-kernel@...r.kernel.org,
	Ingo Molnar <mingo@...hat.com>, x86@...nel.org
Subject: Re: [next-20130204] hard LOCKUP on cpu 5, 3, 4, 2

On Tue, 2013-02-12 at 23:45 -0500, Peter Hurley wrote:
> Got this message log (excerpted below) while building a kernel on
> next-20130204.
> 
> The build was taking place on an ext4 fs which is device-mapped on
> stripped fakeraid.

Happened directly on boot on a lockdep next-20130204 kernel. Log
attached.

> [I have no idea what gvfs was doing.]
> 
> Besides the obvious hard lockups in the fs, the IPIs failed to trigger
> stack dumps for any of the other CPUs [thus +cc x86]. I'm building the
> NMI selftest in now.
> 
> [ 2635.021602] ------------[ cut here ]------------
> [ 2635.021602] WARNING: at /home/peter/src/kernels/next/kernel/watchdog.c:245 watchdog_overflow_callback+0x9f/0xd0()
> [ 2635.021602] Hardware name: Precision WorkStation T5400  
> [ 2635.021602] Watchdog detected hard LOCKUP on cpu 5
> [ 2635.021602] Modules linked in: ext2 ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep rfcomm lp snd_ctxfi snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq joydev snd_timer coretemp btusb kvm_intel snd_seq_device snd psmouse soundcore ppdev bluetooth parport_pc kvm i5400_edac snd_page_alloc edac_core mac_hid i5k_amb dm_multipath parport gpio_ich scsi_dh lpc_ich shpchp serio_raw dcdbas microcode netconsole configfs hid_logitech_dj dm_mirror dm_region_hash dm_log hid_generic usbhid hid usb_storage tg3 nouveau e1000e ptp ttm pps_core drm_kms_helper drm i2c_algo_bit[ 2656.019995] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [ 2656.020000] 	5: (0 ticks this GP) idle=cfd/140000000000001/0 last_accelerate: e84a/fccd, nonlazy_posted: 0, ..
> [ 2656.020000] 	(detected by 1, t=5252 jiffies, g=99488, c=99487, q=215)
> [ 2656.020000] sending NMI to all CPUs:
> [ 2656.020036] NMI backtrace for cpu 2
> [ 2656.020053] CPU 2 
> [ 2656.020060] Pid: 3116, comm: gvfs-afc-volume Tainted: G        W    3.8.0-next-20130204+pcipatch-xeon #20130204+pcipatch Dell Inc. Precision WorkStation T5400  /0RW203
> [ 2656.020065] RIP: 0010:[<ffffffff8104554d>]  [<ffffffff8104554d>] __ticket_spin_lock+0x1d/0x30
> [ 2656.020076] RSP: 0018:ffff8802a0989c28  EFLAGS: 00000297
> [ 2656.020080] RAX: 0000000000000042 RBX: 0000000000010e10 RCX: ffff8802a0989db8
> [ 2656.020083] RDX: 0000000000000043 RSI: 0000000000000000 RDI: ffff8802afc90e10
> [ 2656.020085] RBP: ffff8802a0989c28 R08: 0000000000000000 R09: 8d8d9adfc58cdaff
> [ 2656.020088] R10: 8080808080808080 R11: 0000000000000000 R12: ffff8800b88cb020
> [ 2656.020090] R13: 0000000000000041 R14: 00000000ffffff9c R15: ffff8802a0989db8
> [ 2656.020093] FS:  00007faef9388700(0000) GS:ffff8802afc80000(0000) knlGS:0000000000000000
> [ 2656.020096] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2656.020099] CR2: 00002aaaab01d8c0 CR3: 0000000285a32000 CR4: 00000000000007e0
> [ 2656.020102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2656.020104] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2656.020107] Process gvfs-afc-volume (pid: 3116, threadinfo ffff8802a0988000, task ffff8802a0f64530)
> [ 2656.020109] Stack:
> [ 2656.020112]  ffff8802a0989c48 ffffffff810895f5 ffff8802a0989c58 ffff8802a0989db8
> [ 2656.020120]  ffff8802a0989ca8 ffffffff811a17d5 ffff8802a0989c88 0000000000008b97
> [ 2656.020128]  ffff8802a0989cb8 ffffffff81098929 ffff8802a0989c88 ffff8802a0989db8
> [ 2656.020136] Call Trace:
> [ 2656.020144]  [<ffffffff810895f5>] lg_local_lock+0x35/0x40
> [ 2656.020151]  [<ffffffff811a17d5>] path_init+0x205/0x430
> [ 2656.020156]  [<ffffffff81098929>] ? update_curr+0x99/0x1a0
> [ 2656.020160]  [<ffffffff811a1a33>] path_lookupat+0x33/0x760
> [ 2656.020166]  [<ffffffff8101dd29>] ? sched_clock+0x9/0x10
> [ 2656.020170]  [<ffffffff811a2194>] filename_lookup+0x34/0xc0
> [ 2656.020175]  [<ffffffff811a505e>] user_path_at_empty+0x8e/0x110
> [ 2656.020181]  [<ffffffff816f14b2>] ? _raw_spin_lock_irqsave+0x42/0x60
> [ 2656.020186]  [<ffffffff81084c50>] ? lock_hrtimer_base.isra.21+0x30/0x60
> [ 2656.020190]  [<ffffffff81085055>] ? hrtimer_try_to_cancel+0x55/0x120
> [ 2656.020194]  [<ffffffff811a50f1>] user_path_at+0x11/0x20
> [ 2656.020198]  [<ffffffff8119a502>] vfs_fstatat+0x52/0xb0
> [ 2656.020203]  [<ffffffff8119a59b>] vfs_stat+0x1b/0x20
> [ 2656.020206]  [<ffffffff8119a7da>] sys_newstat+0x1a/0x40
> [ 2656.020210]  [<ffffffff810856b4>] ? hrtimer_start_range_ns+0x14/0x20
> [ 2656.020214]  [<ffffffff81085f16>] ? sys_nanosleep+0x66/0x80
> [ 2656.020219]  [<ffffffff816f9059>] system_call_fastpath+0x16/0x1b
> [ 2656.020222] Code: ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 b8 00 01 00 00 48
> 
> <snipped - full log attached>
> 


View attachment "hard_lockup.next-20130204+lockdep.log" of type "text/x-log" (32207 bytes)