| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Pine.LNX.4.64.0710160944470.21430@bizon.gios.gov.pl>
Date: Tue, 16 Oct 2007 09:54:09 +0200 (CEST)
From: Krzysztof Oledzki <olel@....pl>
To: Nick Piggin <nickpiggin@...oo.com.au>
cc: Peter Zijlstra <peterz@...radead.org>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Strange system hangs
On Sat, 29 Sep 2007, Nick Piggin wrote:
> On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
>> Hello,
>>
>> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
>> and stops accepting remote connections, so it is no longer possible to
>> connect to most important services: smtp (postfix), www (squid) or even
>> ssh. Such connection is accepted but then it hangs.
>>
>> What is strange, that previously established ssh session is usable. It is
>> possible to work on such system until you do something stupid like "less
>> /var/log/all.log". Using strace I found that process blocks on:
>
> Is this a regression? If so, what's the most recent kernel that didn't show
> the problem?
>
> The symptoms could be consistent with some place doing a
> balance_dirty_pages while holding a lock that is required for IO, but I can't
> see a smoking gun (you've got contention on i_mutex, but that should be
> OK).
OK. It did it again :(, so I send additional debug information. Today I
discovered that "echo i > /proc/sysrq-trigger" makes my system being
usable.
> Can you see if there is any memory under writeback that isn't being
> completed (sysrq+M),
SysRq : Show Memory
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 11 Cold: hi: 62, btch: 15 usd: 8
CPU 1: Hot: hi: 186, btch: 31 usd: 170 Cold: hi: 62, btch: 15 usd: 23
Active:281859 inactive:140551 dirty:46161 writeback:0 unstable:0
free:39081 slab:50571 mapped:3261 pagetables:356 bounce:0
DMA free:8704kB min:512kB low:640kB high:768kB active:48kB inactive:0kB present:16256kB pages_scanned:12 all_unreclaimable? no
lowmem_reserve[]: 0 2015
Normal free:147620kB min:65016kB low:81268kB high:97524kB active:1127388kB inactive:562204kB present:2064260kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0
DMA: 4*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8704kB
Normal: 15781*4kB 522*8kB 264*16kB 56*32kB 9*64kB 8*128kB 6*256kB 1*512kB 1*1024kB 2*2048kB 16*4096kB = 147620kB
Swap cache: add 5022, delete 4928, find 2281/2804, race 0+0
Free swap = 2927208kB
Total swap = 2927608kB
Free swap: 2927208kB
524224 pages of RAM
0 pages of HIGHMEM
6345 reserved pages
249411 pages shared
94 pages swap cached
46161 pages dirty
0 pages writeback
3261 pages mapped
50571 pages slab
356 pages pagetables
> also a list the locks held after the hang might be
> helpful (compile in lockdep and sysrq+D)
SysRq : Show Locks Held
Showing all locks held in the system:
1 lock held by syslogd/2197:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by freshclam/2898:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by crond/3061:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by snmpd/3546:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by squid/3606:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by zabbix_agentd/3728:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by agetty/3791:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3793:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3794:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3795:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3796:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3797:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3798:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3799:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
2 locks held by bash/3887:
#0: (sysrq_key_table_lock){....}, at: [<782a6ea9>] __handle_sysrq+0x17/0x109
#1: (tasklist_lock){..-?}, at: [<7813d883>] debug_show_all_locks+0x3b/0x12d
1 lock held by amavisd/7227:
#0: (&mm->mmap_sem){----}, at: [<781196ad>] do_page_fault+0x152/0x535
1 lock held by sadc/8202:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by emerge/8204:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by mkpir/8297:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by screen/8365:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
=============================================
> Is anything currently running? (sysrq+P
Empty.
> and even a full sysrq+T task list
> could be useful).
SysRq : Show State
free sibling
task PC stack pid father child younger older
init S 7A1C1B4C 0 1 0 (NOTLB)
7a1c1b60 00000016 00000002 7a1c1b4c 7a1c1b48 00000000 7840d6e3 7a1d2000
0000000a 00000001 f7fbf490 f7fbe030 9c45a103 0001ff61 00005b8a f7fbf59c
7a022980 f71bdd00 219904e6 00000003 00000000 00000000 7a1c1b70 2199186c
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7837cb10>] dev_queue_xmit+0x10b/0x213
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78172a90>] __d_lookup+0xbb/0x10b
[<78172a3f>] __d_lookup+0x6a/0x10b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78172ac3>] __d_lookup+0xee/0x10b
[<7816a323>] do_lookup+0x4f/0x143
[<78171ebc>] dput+0x16/0xe4
[<7816be16>] __link_path_walk+0xa1a/0xb5e
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7816c003>] link_path_walk+0xa9/0xb3
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
kthreadd S F7FBEA60 0 2 0 (L-TLB)
7a1c3fd0 00000016 7851aa94 f7fbea60 00000001 786b7ec8 f7fbef90 f7fbef90
0000000a 00000000 f7fbea60 edcd0030 e200f0d9 0001df92 00000d92 f7fbeb6c
7a01a980 f7cafa40 00001890 00000246 7851aa80 00001890 7851aa60 00001890
Call Trace:
[<78134f4d>] kthreadd+0x61/0xfe
[<78134eec>] kthreadd+0x0/0xfe
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
migration/0 S 7A022980 0 3 2 (L-TLB)
7a1c9fa4 00000002 7811e4eb 7a022980 f7c7d550 7a1c9fa4 7811f199 7a1c7a00
00000001 00000000 7a1c74d0 f7e84b60 78531ca2 0001ff5b 00001431 7a1c75dc
7a01a980 e913a340 7a0229e8 00000046 7a01a980 7a01a980 7a01b2f4 7a01a980
Call Trace:
[<7811e4eb>] resched_task+0x3b/0x59
[<7811f199>] move_tasks+0x1f5/0x2bd
[<781221cf>] migration_thread+0x14a/0x1cd
[<78122085>] migration_thread+0x0/0x1cd
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksoftirqd/0 S 7A1CBFB4 0 4 2 (L-TLB)
7a1cbfc8 00000002 00000002 7a1cbfb4 7a1cbfb0 00000000 edcd1490 edcd1490
00000009 00000000 7a1c6aa0 785143e0 f1a2fd1e 0001e9af 0000065e 7a1c6bac
7a01a980 c053d340 202bebb7 00000003 00000000 00000000 7a018100 00000000
Call Trace:
[<78129a29>] ksoftirqd+0x55/0x134
[<781299d4>] ksoftirqd+0x0/0x134
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
watchdog/0 S 7A01A980 0 5 2 (L-TLB)
7a1cdfcc 00000006 7840d6e3 7a01a980 7a1cdfcc 7813e99c 00000002 00000046
00000001 00000000 7a1c6070 f7fbf490 35f147b4 0000000a 00002c3f 7a1c617c
7a01a980 78514180 7a1c6530 ffffffff 78596a0c 01a85000 00000000 78146910
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78146910>] watchdog+0x0/0x58
[<78146958>] watchdog+0x48/0x58
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
migration/1 S 7A1D1F90 0 6 2 (L-TLB)
7a1d1fa4 00000002 00000002 7a1d1f90 7a1d1f8c 00000000 7a1cfa40 7a1cfa40
00000001 00000001 7a1cf510 f7fbe030 04d4c284 0001ff60 00001b5e 7a1cf61c
7a022980 e8d34580 2198ea1b 00000003 00000000 00000000 7a0232f4 7a022980
Call Trace:
[<781221cf>] migration_thread+0x14a/0x1cd
[<78122085>] migration_thread+0x0/0x1cd
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksoftirqd/1 S 7A1D5FB4 0 7 2 (L-TLB)
7a1d5fc8 00000002 00000002 7a1d5fb4 7a1d5fb0 00000000 00000046 00000000
0000000a 00000001 7a1ceae0 f7fbe030 07b31f2a 0001ff3a 0004c80b 7a1cebec
7a022980 e913a340 21966ac7 00000003 00000000 00000000 7a020100 00000001
Call Trace:
[<78129a29>] ksoftirqd+0x55/0x134
[<781299d4>] ksoftirqd+0x0/0x134
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
watchdog/1 S 7A022980 0 8 2 (L-TLB)
7a1d7fcc 00000006 7840d6e3 7a022980 7a1d7fcc 7813e99c 00000002 00000046
00000001 00000001 7a1ce0b0 7a1ceae0 372914ac 0000000a 00003a96 7a1ce1bc
7a022980 78514180 7a1ce570 ffffffff 78596a0c 01a8d000 00000001 78146910
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78146910>] watchdog+0x0/0x58
[<78146958>] watchdog+0x48/0x58
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
events/0 S F7FBDFA0 0 9 2 (L-TLB)
f7fbdfb4 00000086 00000002 f7fbdfa0 f7fbdf9c 00000000 f7ea1a80 00000000
0000000a 00000000 f7ea1550 785143e0 959db3cd 0001ff61 0000158b f7ea165c
7a01a980 c053d840 21990475 00000003 00000000 00000000 f7f940c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
events/1 S F7FB9FA0 0 10 2 (L-TLB)
f7fb9fb4 00000086 00000002 f7fb9fa0 f7fb9f9c 00000000 f7ea1050 00000000
0000000a 00000001 f7ea0b20 f7fbe030 992e5f8c 0001ff61 000013bc f7ea0c2c
7a022980 e913a340 219904b2 00000003 00000000 00000000 f7c4df40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
khelper S 00000000 0 11 2 (L-TLB)
f7fbbfb4 00000086 00000046 00000000 00000000 f7c4defc f7ea0620 00000000
00000009 00000001 f7ea00f0 f7cc8130 6fcbad27 0001df91 000011c4 f7ea01fc
7a022980 f6b15840 00000046 f7c4dee8 00000286 f7c4dee8 f7c4dec0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kblockd/0 S F7E3BFA0 0 64 2 (L-TLB)
f7e3bfb4 00000086 00000002 f7e3bfa0 f7e3bf9c 00000000 f7e85090 00000000
0000000a 00000000 f7e84b60 785143e0 6ff1ad71 0001ff61 00001779 f7e84c6c
7a01a980 e913a340 219901fd 00000003 00000000 00000000 f7e951c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kblockd/1 S F7E3DFA0 0 65 2 (L-TLB)
f7e3dfb4 00000086 00000002 f7e3dfa0 f7e3df9c 00000000 00000000 00000000
0000000a 00000001 f7e85590 f7fbe030 593eee3f 0001ff61 00001932 f7e8569c
7a022980 d3e4acc0 2199007e 00000002 00000000 00000000 f7e95140 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kacpid S 00000000 0 66 2 (L-TLB)
f7e3ffb4 00000086 00000046 00000000 00000000 f7c76cfc f7c7c620 00000000
00000001 00000000 f7c7c0f0 f7fbf490 3bbce431 0000000a 0000072f f7c7c1fc
7a01a980 78514180 00000046 f7c76ce8 00000286 f7c76ce8 f7c76cc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kacpi_notify S 00000000 0 67 2 (L-TLB)
f7e41fb4 00000086 00000046 00000000 00000000 f7c76bfc f7c7d050 00000000
00000001 00000000 f7c7cb20 f7fbf490 3bbd4590 0000000a 00000720 f7c7cc2c
7a01a980 78514180 00000046 f7c76be8 00000286 f7c76be8 f7c76bc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata/0 S 00000000 0 147 2 (L-TLB)
f7e21fb4 00000086 00000046 00000000 00000000 f7c4bd7c f7e23a40 00000000
0000000a 00000000 f7e23510 f7cd0030 a23c198a 0000000b 0002d0fb f7e2361c
7a01a980 78514180 00000046 f7c4bd68 00000286 f7c4bd68 f7c4bd40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata/1 S 00000000 0 148 2 (L-TLB)
f7e19fb4 00000086 00000046 00000000 00000000 f7c4bcfc f7e2a620 00000000
0000000a 00000001 f7e2a0f0 f7fbf490 7e41e2a8 0000000b 00005ba5 f7e2a1fc
7a022980 78514180 00000046 f7c4bce8 00000286 f7c4bce8 f7c4bcc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata_aux S F7E43FA0 0 149 2 (L-TLB)
f7e43fb4 00000086 00000002 f7e43fa0 f7e43f9c 00000000 f7e1a5a0 00000000
00000001 00000000 f7e1a070 785143e0 3dbcad9d 0000000a 00000e19 f7e1a17c
7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bc40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksuspend_usbd S F7E29FA0 0 150 2 (L-TLB)
f7e29fb4 00000086 00000002 f7e29fa0 f7e29f9c 00000000 f7e84660 00000000
00000001 00000000 f7e84130 785143e0 3dbd4cb7 0000000a 00001042 f7e8423c
7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bb40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
khubd S 00000000 0 153 2 (L-TLB)
f7e05f54 00000086 00000046 00000000 00000000 78538a34 f7e225e0 00000000
0000000a 00000000 f7e220b0 f7fbf490 d1ac4454 0000000b 00005717 f7e221bc
7a01a980 78514180 00000046 78538a20 00000286 78538a20 00000000 f7d36200
Call Trace:
[<78316b2e>] hub_thread+0x98f/0x9f9
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840ae59>] __sched_text_start+0x701/0x810
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7831619f>] hub_thread+0x0/0x9f9
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kseriod S 00000000 0 155 2 (L-TLB)
7a1fdf9c 00000002 00000046 00000000 00000000 7853bcd4 f7e0a660 00000000
0000000a 00000000 f7e0a130 f7fbf490 0dfac2e3 0000000c 00003730 f7e0a23c
7a01a980 78514180 00000046 7853bcc0 00000216 7853bcc0 f7d50580 7832efe5
Call Trace:
[<7832efe5>] serio_thread+0x0/0x2bb
[<7832f255>] serio_thread+0x270/0x2bb
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7832efe5>] serio_thread+0x0/0x2bb
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kswapd0 S F7E59F44 0 187 2 (L-TLB)
f7e59f58 00000086 00000002 f7e59f44 f7e59f40 00000000 7a1fb010 00000000
00000009 00000001 7a1faae0 f7fbe030 1a81a860 0001e6a0 00e7fbcc 7a1fabec
7a022980 abc7dd40 1ff863fd 00000003 00000000 00000000 00000001 ffffffff
Call Trace:
[<78150a2e>] kswapd+0xb0/0x40f
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7815097e>] kswapd+0x0/0x40f
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
aio/0 S 00000000 0 188 2 (L-TLB)
f7e5dfb4 00000086 00000046 00000000 00000000 f7ed547c 7a1fba40 00000000
00000001 00000000 7a1fb510 f7fbea60 42056d94 0000000a 00000dba 7a1fb61c
7a01a980 78514180 00000046 f7ed5468 00000286 f7ed5468 f7ed5440 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
aio/1 S 00000000 0 189 2 (L-TLB)
f7e5ffb4 00000086 00000046 00000000 00000000 f7ed53fc 7a1f25a0 00000000
00000001 00000001 7a1f2070 f7fbf490 4205dc87 0000000a 000009cd 7a1f217c
7a022980 78514180 00000046 f7ed53e8 00000286 f7ed53e8 f7ed53c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
cryptd S F7E7FFA8 0 190 2 (L-TLB)
f7e7ffbc 00000082 00000002 f7e7ffa8 f7e7ffa4 00000000 787ce0d8 f7fb5590
00000001 00000000 f7fb5590 785143e0 42116cfc 0000000a 000076c7 f7fb569c
7a01a980 78514180 fffb6d71 00000003 00000000 00000000 00000000 00000000
Call Trace:
[<7820a221>] cryptd_thread+0x95/0xa5
[<7820a18c>] cryptd_thread+0x0/0xa5
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_0 S 7A1EA030 0 320 2 (L-TLB)
f7ca1f80 00000092 f7e0f058 7a1ea030 7840d6e3 f7e0f000 f7cef000 7813e99c
0000000a 00000000 7a1ea030 f7fbf490 7ebe3537 0000000b 00005ddb 7a1ea13c
7a01a980 78514180 00000292 f7cef000 f7e0f000 00000000 00000282 f7e0f000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_1 S F7CA5F6C 0 322 2 (L-TLB)
f7ca5f80 00000092 00000002 f7ca5f6c f7ca5f68 00000000 00000000 7813e99c
00000003 00000000 f7cf94d0 785143e0 7e3733ad 0000000b 0000c8a9 f7cf95dc
7a01a980 78514180 fffb8236 00000003 00000000 00000000 00000282 f7cbd800
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_2 S F7E0B590 0 339 2 (L-TLB)
f7ca7f80 00000092 7a1d8058 f7e0b590 7840d6e3 7a1d8000 00000000 7813e99c
00000006 00000000 f7e0b590 f7fbf490 9835b106 0000000b 004284a4 f7e0b69c
7a01a980 78514180 00000292 00000282 7a1d8000 00000000 00000282 7a1d8000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_3 S F7CD0030 0 341 2 (L-TLB)
f7ca3f80 00000092 f7cbd058 f7cd0030 7840d6e3 f7cbd000 00000000 7813e99c
00000008 00000000 f7cd0030 f7fbf490 a28957bc 0000000b 003e72b9 f7cd013c
7a01a980 78514180 00000292 00000282 f7cbd000 00000000 00000282 f7cbd000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kpsmoused S 00000000 0 407 2 (L-TLB)
f7d75fb4 00000086 00000046 00000000 00000000 f7d1767c f7d005e0 00000000
00000009 00000000 f7d000b0 f7fbf490 d3b0a6af 0000000b 0000087a f7d001bc
7a01a980 78514180 00000046 f7d17668 00000286 f7d17668 f7d17640 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kcryptd/0 S 00000000 0 412 2 (L-TLB)
f7d77fb4 00000086 00000046 00000000 00000000 f7d170fc f7e33090 00000000
00000009 00000000 f7e32b60 f7fbf490 e6cc0d2f 0000000b 000007f4 f7e32c6c
7a01a980 78514180 00000046 f7d170e8 00000286 f7d170e8 f7d170c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kcryptd/1 S F7D71FA0 0 413 2 (L-TLB)
f7d71fb4 00000086 00000002 f7d71fa0 f7d71f9c 00000000 f7d01010 00000000
00000008 00000001 f7d00ae0 f7fbe030 e6cc9219 0000000b 000010ad f7d00bec
7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5cf40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kmpathd/0 S 00000000 0 414 2 (L-TLB)
f7d73fb4 00000086 00000046 00000000 00000000 f7d5cd7c f7cf85a0 00000000
00000009 00000000 f7cf8070 f7fbf490 e6cd348c 0000000b 00000786 f7cf817c
7a01a980 78514180 00000046 f7d5cd68 00000286 f7d5cd68 f7d5cd40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kmpathd/1 S F7D6DFA0 0 415 2 (L-TLB)
f7d6dfb4 00000086 00000002 f7d6dfa0 f7d6df9c 00000000 f7cf8fd0 00000000
00000008 00000001 f7cf8aa0 f7fbe030 e6cda902 0000000b 00000f2f f7cf8bac
7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5ccc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksnapd S 00000000 0 416 2 (L-TLB)
f7d31fb4 00000086 00000046 00000000 00000000 f7d5c9fc f7d01a40 00000000
00000008 00000000 f7d01510 f7fbf490 e72c4b66 0000000b 0000070f f7d0161c
7a01a980 78514180 00000046 f7d5c9e8 00000286 f7d5c9e8 f7d5c9c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kedac S F7D7BF80 0 419 2 (L-TLB)
f7d7bf94 00000086 00000002 f7d7bf80 f7d7bf7c 00000000 7840d6e3 7a1d2000
0000000a 00000001 f7e0ab60 f7fbe030 acabac43 0001ff61 00000d5e f7e0ac6c
7a022980 e913a340 219905f9 00000003 00000000 00000000 f7d7bfa4 219909e0
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<7812cada>] process_timeout+0x0/0x5
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<78363ea0>] edac_kernel_thread+0xc7/0xf3
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md4_raid5 S F7E2AB20 0 434 2 (L-TLB)
7a255f84 00000006 7a21a920 f7e2ab20 7840d727 00000000 00000000 7a224d5c
0000000a 00000000 f7e2ab20 f6928030 63aadcb9 0001ff61 00002f44 f7e2ac2c
7a01a980 e913a340 00000000 00000000 f7e2ab20 f7e2b050 7fffffff 7fffffff
Call Trace:
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md15_raid10 S 7A1F3A00 0 437 2 (L-TLB)
7a253f84 00000006 7a1f34d0 7a1f3a00 785cde94 7a1f34d0 7840d6e3 785cde80
0000000a 00000000 7a1f34d0 f6928a60 39bb729c 0001ff61 000011b8 7a1f35dc
7a01a980 f58bf040 785cde80 7812ccc1 00000000 00000216 7a253f94 219911f1
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812cada>] process_timeout+0x0/0x5
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md0_raid1 S F7C7DA80 0 439 2 (L-TLB)
f719ff84 00000086 f7c7d550 f7c7da80 785cde94 f7c7d550 7840d6e3 785cde80
0000000a 00000000 f7c7d550 f6928030 706bbe1a 0001ff61 00002cc3 f7c7d65c
7a01a980 e913a340 785cde80 7812ccc1 00000000 00000296 f719ff94 2199158a
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812cada>] process_timeout+0x0/0x5
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 441 2 (L-TLB)
f717dfa8 00000082 00000046 00000000 00000000 f7199d3c f7c74608 f7c745e0
0000000a 00000000 f7c740b0 f6928030 575613ff 0001ff61 000092f3 f7c741bc
7a01a980 f6e31800 00000046 00000246 f7199c14 f7199c00 f7199c14 f7199c00
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S F6DD5F94 0 584 2 (L-TLB)
f6dd5fa8 00000082 00000002 f6dd5f94 f6dd5f90 00000000 f7fb4688 f7fb4660
0000000a 00000001 f7fb4130 f7fbe030 545ba352 0001ff61 0000700b f7fb423c
7a022980 e632d2c0 2199002b 00000003 00000000 00000000 f68c8814 f68c8800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S F588DF94 0 585 2 (L-TLB)
f588dfa8 00000082 00000002 f588df94 f588df90 00000000 f7cd19e8 f7cd19c0
0000000a 00000000 f7cd1490 785143e0 f0b5365d 0001ff24 00001614 f7cd159c
7a01a980 f7cafa40 219507da 00000003 00000000 00000000 f58c1814 f58c1800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 586 2 (L-TLB)
f6fdbfa8 00000082 00000046 00000000 00000000 f58c113c 7a1dcfb8 7a1dcf90
00000009 00000000 7a1dca60 f7cb80b0 53c6eaba 0000000d 00001d05 7a1dcb6c
7a01a980 f68e9ac0 00000046 00000246 f58c1014 f58c1000 f58c1014 f58c1000
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 587 2 (L-TLB)
f587bfa8 00000082 00000046 00000000 00000000 f64c993c f7c6da28 f7c6da00
0000000a 00000001 f7c6d4d0 f7ea0b20 f44b1328 0001e632 0000149c f7c6d5dc
7a022980 e632d2c0 00000046 00000246 f64c9814 f64c9800 f64c9814 f64c9800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 588 2 (L-TLB)
f58c9fa8 00000082 00000046 00000000 00000000 f64c913c 7a1dc588 7a1dc560
00000008 00000000 7a1dc030 f7cb80b0 5d289ec0 0000000d 00001bb0 7a1dc13c
7a01a980 f68e9ac0 00000046 00000246 f64c9014 f64c9000 f64c9014 f64c9000
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
syslogd D 00000000 0 2197 1 (NOTLB)
f6267c74 00000082 00000000 00000000 7813e7a9 f7cc8130 7840d6e3 785cde80
00000007 00000000 f7cc8130 f7c74ae0 abd727e3 0001ff61 00002fdc f7cc823c
7a01a980 f6b15840 785cde80 7812ccc1 00000000 00000292 f6267c84 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<7816402f>] do_sync_readv_writev+0xc1/0xfe
[<78139494>] getnstimeofday+0x30/0xbc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78163ee4>] rw_copy_check_uvector+0x50/0xaa
[<781646b4>] do_readv_writev+0x99/0x164
[<78199b90>] ext3_file_write+0x0/0x92
[<781647bc>] vfs_writev+0x3d/0x48
[<78164b9e>] sys_writev+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
klogd S F5F37DB8 0 2211 1 (NOTLB)
f5f37dcc 00000086 00000002 f5f37db8 f5f37db4 00000000 00000000 f6f77f54
00000007 00000000 f7c6caa0 785143e0 579dd14b 0001ff41 000c175e f7c6cbac
7a01a980 f62ada40 2196e5cc 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78370f09>] sock_aio_write+0xcb/0xd7
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781649e7>] vfs_read+0xcf/0x10a
[<781648aa>] vfs_write+0x9e/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
acpid S F5C35BE8 0 2265 1 (NOTLB)
f5c35bfc 00000082 00000002 f5c35be8 f5c35be4 00000000 78528e74 f7cc1550
00000001 00000000 f7cc1550 785143e0 87711dce 0000000f 0000efdb f7cc165c
7a01a980 f6b150c0 fffbc622 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783cbaf6>] unix_poll+0x1a/0x9a
[<7816e77e>] do_sys_poll+0x258/0x327
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840af41>] __sched_text_start+0x7e9/0x810
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
[<7840d33a>] _read_lock+0x33/0x3e
[<7840d699>] _read_unlock+0x25/0x3b
[<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<78171ebc>] dput+0x16/0xe4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<781383ba>] up_read+0x14/0x26
[<781197d3>] do_page_fault+0x278/0x535
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816e881>] sys_poll+0x34/0x37
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd S F31C9B4C 0 2823 1 (NOTLB)
f31c9b60 00000096 00000002 f31c9b4c f31c9b48 00000000 7840d6e3 7a1d2000
00000007 00000001 f7e32130 f7fbe030 65ed4f6c 0001ff60 000151f2 f7e3223c
7a022980 f64f1800 2198f07d 00000003 00000000 00000000 f31c9b70 2199178d
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
[<7816f1be>] __pollwait+0x0/0xac
[<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
[<7838eaf9>] tcp_packet+0xa0f/0xa3e
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<783bb722>] inet_sock_destruct+0x179/0x1bb
[<783b147b>] tcp_v4_rcv+0x51a/0x787
[<783994b1>] ip_local_deliver+0x1bd/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<78139494>] getnstimeofday+0x30/0xbc
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7812e2a3>] group_send_sig_info+0x12/0x56
[<7812e378>] kill_pid_info+0x61/0x7b
[<7812f183>] sys_kill+0x108/0x125
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
clamd S 00000000 0 2889 1 (NOTLB)
f32d1e58 00000086 00000002 00000000 00010822 00000000 00000000 ee9a69ac
00000001 00000001 f7cb80b0 db4c8030 6f522ff7 0001e655 00010e05 f7cb81bc
7a022980 f68e9ac0 00000000 00000000 f7cb80b0 f7cb85e0 7fffffff f5e2d380
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783787ab>] skb_recv_datagram+0x13f/0x1c4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783ccdf1>] unix_accept+0x51/0xde
[<7837262e>] sys_accept+0xb2/0x187
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<781671ca>] sys_fstat64+0x1e/0x23
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
freshclam D 00000000 0 2898 1 (NOTLB)
f2f43cc4 00000082 00000000 00000000 7813e7a9 f6928030 7840d6e3 785cde80
00000008 00000000 f6928030 f7cc8130 abd6f807 0001ff61 00003332 f692813c
7a01a980 f58bfa40 785cde80 7812ccc1 00000000 00000296 f2f43cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
named S F0989D0C 0 2959 1 (NOTLB)
f0989d20 00200092 00000002 f0989d0c f0989d08 00000000 00000000 f6f77f54
0000000a 00000000 f7e13490 785143e0 0e816649 0001e659 00013c17 f7e1359c
7a01a980 f58bf2c0 1ff3b86f 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<78171ebc>] dput+0x16/0xe4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78371b64>] sys_sendto+0x11b/0x13b
[<78169ed9>] path_release+0xa/0x20
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<78371bbb>] sys_send+0x37/0x3b
[<78372830>] sys_socketcall+0x12d/0x242
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
crond D 00000000 0 3061 1 (NOTLB)
f1367cc4 00000082 00000000 00000000 7813e7a9 f7e02b20 7840d6e3 7a1d2000
00000008 00000001 f7e02b20 edcd0a60 abd708dc 0001ff61 0000367c f7e02c2c
7a022980 f58bfcc0 7a1d2000 7812ccc1 00000000 00000296 f1367cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
mdadm S F1ADDB4C 0 3110 1 (NOTLB)
f1addb60 00000096 00000002 f1addb4c f1addb48 00000000 7840d6e3 785cde80
00000007 00000000 f6928a60 785143e0 39bb937f 0001ff61 000020e3 f6928b6c
7a01a980 f58bf040 2198fe6a 00000003 00000000 00000000 f1addb70 2199e8c9
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840af41>] __sched_text_start+0x7e9/0x810
[<7840ae59>] __sched_text_start+0x701/0x810
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7811e3ba>] __wake_up+0x32/0x43
[<7840b09e>] preempt_schedule+0x46/0x58
[<7834dc68>] md_wakeup_thread+0x26/0x28
[<78354e25>] md_ioctl+0x1261/0x12e6
[<7840d623>] _spin_unlock+0x25/0x3b
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7840bd93>] __mutex_unlock_slowpath+0x105/0x127
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78352e61>] md_open+0x51/0x58
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<781335c2>] call_rcu+0x64/0x69
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
portmap S F1571BE8 0 3161 1 (NOTLB)
f1571bfc 00000082 00000002 f1571be8 f1571be4 00000000 00000000 00000000
00000007 00000001 7a1dd490 f7fbe030 e696d8aa 00000013 000281e5 7a1dd59c
7a022980 f6e31080 fffc0fae 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<783b435a>] udp_poll+0x10/0xdb
[<7816e77e>] do_sys_poll+0x258/0x327
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783b57a5>] udp_sendmsg+0x4a7/0x5b9
[<783ba7cb>] inet_sendmsg+0x3b/0x45
[<78371838>] sock_sendmsg+0xbc/0xd4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<78377ed6>] verify_iovec+0x3e/0x70
[<783719e4>] sys_sendmsg+0x194/0x1f9
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7814cb18>] free_hot_cold_page+0x13b/0x167
[<7814cb24>] free_hot_cold_page+0x147/0x167
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814cb58>] __pagevec_free+0x14/0x1a
[<7814ecc5>] release_pages+0x126/0x12e
[<78159060>] anon_vma_unlink+0x13/0x53
[<7840d623>] _spin_unlock+0x25/0x3b
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816e881>] sys_poll+0x34/0x37
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
rpciod/0 S 00000000 0 3216 2 (L-TLB)
ec88dfb4 00000086 00000046 00000000 00000000 f6343c7c 7a1eb9c0 00000000
0000000a 00000000 7a1eb490 89333550 78dbc731 0001e655 0000401b 7a1eb59c
7a01a980 c053d840 00000046 f6343c68 00000286 f6343c68 f6343c40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
rpciod/1 S 00000000 0 3217 2 (L-TLB)
ec8e7fb4 00000086 00000046 00000000 00000000 f5a6867c f69299c0 00000000
0000000a 00000001 f6929490 a79ec0b0 2e2b55c5 00000834 0000ce3a f692959c
7a022980 a71d7340 00000046 f5a68668 00000286 f5a68668 f5a68640 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
lockd S 00000001 0 3218 2 (L-TLB)
ec865f18 00000082 7813534c 00000001 784d6071 f7e02628 00000000 00000000
00000001 00000000 f7e020f0 f7ca8a60 e69b3708 00000013 0000a1e7 f7e021fc
7a01a980 f68e90c0 f7e02620 f7e020f0 781299b4 f0fecd7c 7fffffff f0fec000
Call Trace:
[<7813534c>] add_wait_queue+0x12/0x30
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<783ffbab>] svc_recv+0x249/0x3ca
[<7812027d>] default_wake_function+0x0/0xc
[<781e6643>] lockd+0xe8/0x1fd
[<78121e77>] schedule_tail+0x0/0xb6
[<78103e68>] restore_nocheck+0x12/0x15
[<781e655b>] lockd+0x0/0x1fd
[<781e655b>] lockd+0x0/0x1fd
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ntpd S EC959B4C 0 3269 1 (NOTLB)
ec959b60 00200096 00000002 ec959b4c ec959b48 00000000 f7cb05a0 00200046
0000000a 00000000 f7cb0070 785143e0 a27c1eef 0001ff61 0000aea1 f7cb017c
7a01a980 f7cafa40 2199054d 00000003 00000000 00000000 7fffffff ec959f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783b435a>] udp_poll+0x10/0xdb
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783760ad>] skb_dequeue+0x39/0x3f
[<783786ed>] skb_recv_datagram+0x81/0x1c4
[<783b5b8f>] udp_recvmsg+0x57/0x1cd
[<78372cc9>] sock_common_recvmsg+0x3e/0x54
[<78371763>] sock_recvmsg+0xcf/0xe8
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7837221d>] sys_recvmsg+0x11d/0x1cf
[<78371b64>] sys_sendto+0x11b/0x13b
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<78103986>] do_notify_resume+0x511/0x5f3
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sqlgrey S 00000000 0 3323 1 (NOTLB)
ec79fb60 00200096 786b7ed0 00000000 7837cbf7 00200046 00000000 00000000
00000004 00000001 f7cb0aa0 f7fbf490 b29583a6 0001f67f 00068968 f7cb0bac
7a022980 f68e90c0 7813e99c 00000000 00200046 ec975398 7fffffff ec79ff9c
Call Trace:
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<783b133f>] tcp_v4_rcv+0x3de/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7839949e>] ip_local_deliver+0x1aa/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c73c>] process_backlog+0xef/0xfa
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c9d8>] net_rx_action+0x162/0x18f
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
smartd S EA68DD0C 0 3487 1 (NOTLB)
ea68dd20 00000092 00000002 ea68dd0c ea68dd08 00000000 00000000 f6f77f54
00000009 00000000 ec907490 785143e0 ad75944f 0001e66c 00034dc2 ec90759c
7a01a980 f6e11040 1ff502a1 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813facb>] lock_release_non_nested+0xec/0x14d
[<783cdc63>] unix_dgram_connect+0x8d/0x157
[<783cdd00>] unix_dgram_connect+0x12a/0x157
[<783cdc63>] unix_dgram_connect+0x8d/0x157
[<78371b64>] sys_sendto+0x11b/0x13b
[<7840d623>] _spin_unlock+0x25/0x3b
[<783cdd00>] unix_dgram_connect+0x12a/0x157
[<7837135e>] sys_connect+0x72/0x9c
[<78371bbb>] sys_send+0x37/0x3b
[<78372830>] sys_socketcall+0x12d/0x242
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
snmpd D 00000000 0 3546 1 (NOTLB)
ea333cc4 00200082 00000000 00000000 7813e7a9 f7c74ae0 7840d6e3 785cde80
00000007 00000000 f7c74ae0 89333550 abd756a0 0001ff61 00002ebd f7c74bec
7a01a980 f6e31800 785cde80 7812ccc1 00000000 00200296 ea333cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squid S E9A41F20 0 3604 1 (NOTLB)
e9a41f34 00200082 00000002 e9a41f20 e9a41f1c 00000000 e9e6da40 e9e6da40
00000005 00000000 e9e6d510 785143e0 74bc2237 00000016 00014e53 e9e6d61c
7a01a980 f6e31d00 fffc3aae 00000003 00000000 00000000 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squid D 00000000 0 3606 3604 (NOTLB)
e9a65cc4 00000082 00000000 00000000 7813e7a9 f7cb9510 7840d6e3 7a1d2000
00000007 00000001 f7cb9510 edcd1490 abd7ba4d 0001ff61 000021db f7cb961c
7a022980 f6ce25c0 7a1d2000 7812ccc1 00000000 00000296 e9a65cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E9239B4C 0 3663 1 (NOTLB)
e9239b60 00200096 00000002 e9239b4c e9239b48 00000000 00000000 00000000
00000006 00000000 e9b7e0f0 785143e0 61b30795 0001ff2d 000245bd e9b7e1fc
7a01a980 f62ad540 219595ea 00000003 00000000 00000000 7fffffff e9239f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816e4d2>] free_poll_entry+0xe/0x16
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 7A1F2AA0 0 3722 1 (NOTLB)
e9369f34 00200082 7854ec14 7a1f2aa0 00000001 786b7e90 7a1f2fd0 7a1f2fd0
00000001 00000001 7a1f2aa0 f7fb4b60 0ec74ab6 00000017 0000b6ef 7a1f2bac
7a022980 e9314540 00000000 00200246 7854ec00 00000001 00000e8c 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<781197d3>] do_page_fault+0x278/0x535
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S E93D1F2C 0 3724 3722 (NOTLB)
e93d1f40 00200092 00000002 e93d1f2c e93d1f28 00000000 7813e99c 7a02391c
00000008 00000001 ec8de0f0 f7fbe030 aa5c0061 0001ff61 0004acea ec8de1fc
7a022980 e9314cc0 219905d2 00000003 00000000 00000000 e93d1f58 00000001
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840c5af>] do_nanosleep+0x48/0x73
[<78137d84>] hrtimer_nanosleep+0x39/0xdc
[<78137879>] hrtimer_wakeup+0x0/0x18
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78137e70>] sys_nanosleep+0x49/0x59
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 00000070 0 3725 3722 (NOTLB)
e8ccbe44 00200086 00000001 00000070 7a01b3e0 00000002 00000000 ead5e140
00000001 00000000 e90b8b60 e9e6cae0 0f7701b8 00000017 000051d5 e90b8c6c
7a01a980 e913a0c0 fffc44d7 00000003 00000000 00000070 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S E8CCDE30 0 3726 3722 (NOTLB)
e8ccde44 00200086 00000002 e8ccde30 e8ccde2c 00000000 00000000 ead5e140
00000001 00000000 e9e6cae0 785143e0 0fa415f1 00000017 0001181a e9e6cbec
7a01a980 e93147c0 fffc44d7 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 00000000 0 3727 3722 (NOTLB)
e8cede44 00200086 00000000 00000000 7813e7a9 00000000 00000000 ead5e140
00000001 00000001 f7cb8ae0 e9e6c0b0 0fbbed36 00000017 0003651f f7cb8bec
7a022980 f62ad040 f7cb9010 f7cb8ae0 781299b4 e8cedea0 7fffffff 7fffffff
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd D E8CEFCB0 0 3728 3722 (NOTLB)
e8cefcc4 00200082 00000002 e8cefcb0 e8cefcac 00000000 7840d6e3 7a1d2000
00000008 00000001 e9e6c0b0 f7fbe030 abd7fd2c 0001ff61 00002173 e9e6c1bc
7a022980 e913a340 219905eb 00000003 00000000 00000000 e8cefcd4 2199064e
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7814ce3d>] get_page_from_freelist+0x293/0x35e
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<78153e4d>] __handle_mm_fault+0x451/0x83b
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781197d3>] do_page_fault+0x278/0x535
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<7811955b>] do_page_fault+0x0/0x535
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S E8849EC8 0 3791 1 (NOTLB)
e8849edc 00000086 00000002 e8849ec8 e8849ec4 00000000 00000000 00000000
00000007 00000001 f7fb4b60 f7fbe030 6d0455f6 00000017 0018bcb8 f7fb4c6c
7a022980 f6b15d40 fffc4afe 00000003 00000000 00000000 7fffffff f539f000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3793 1 (NOTLB)
e8d19edc 00000086 00000000 00000000 f53a5c8c ea120a60 00000000 00000000
00000007 00000001 ea120a60 f7e12030 70e57c7e 00000017 0010cf04 ea120b6c
7a022980 e9314040 ea120f98 ea120a60 00000000 00000000 7fffffff f53a5800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3794 1 (NOTLB)
e8d61edc 00000086 00000000 00000000 f56dcc8c f7e12030 00000000 00000000
00000007 00000001 f7e12030 ec8deb20 70f48c60 00000017 000f0fe2 f7e1213c
7a022980 f62adcc0 f7e12568 f7e12030 00000000 00000000 7fffffff f56dc800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3795 1 (NOTLB)
e8d23edc 00000086 00000000 00000000 f52fcc8c ec8deb20 00000000 00000000
00000007 00000001 ec8deb20 ec8ff590 71036a29 00000017 000eddc9 ec8dec2c
7a022980 e913a840 ec8df058 ec8deb20 00000000 00000000 7fffffff f52fc800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3796 1 (NOTLB)
e8903edc 00000086 00000000 00000000 f583748c ec8ff590 00000000 00000000
00000007 00000001 ec8ff590 e8eea130 71118e5d 00000017 000e2434 ec8ff69c
7a022980 f6f5b080 ec8ffac8 ec8ff590 00000000 00000000 7fffffff f5837000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S E8921EC8 0 3797 1 (NOTLB)
e8921edc 00000086 00000002 e8921ec8 e8921ec4 00000000 00000000 00000000
00000007 00000001 e8eea130 f7fbe030 713eaf24 00000017 00023594 e8eea23c
7a022980 e913a5c0 fffc4b41 00000003 00000000 00000000 7fffffff f7242000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3798 1 (NOTLB)
e8923edc 00000086 00000000 00000000 f5d5048c ec906030 00000000 00000000
00000007 00000001 ec906030 ea024070 712da585 00000017 000ebb02 ec90613c
7a022980 e913aac0 ec906568 ec906030 00000000 00000000 7fffffff f5d50000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3799 1 (NOTLB)
e8925edc 00000086 00000000 00000000 f6f2348c ea024070 00000000 00000000
00000007 00000001 ea024070 e8eea130 713c7990 00000017 000ed40b ea02417c
7a022980 f6e31580 ea0245a8 ea024070 00000000 00000000 7fffffff f6f23000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E3429DC4 0 3835 3663 (NOTLB)
e3429dd8 00000082 00000002 e3429dc4 e3429dc0 00000000 00000000 e12799ac
00000007 00000000 7a1eaa60 785143e0 1e0f5b36 00000025 00001291 7a1eab6c
7a01a980 e8d34d00 fffd3160 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E6C59B4C 0 3838 3835 (NOTLB)
e6c59b60 00000096 00000002 e6c59b4c e6c59b48 00000000 e90b8688 e90b8668
0000000a 00000000 e90b8130 785143e0 28faff08 0001ff23 0001a008 e90b823c
7a01a980 f71bd580 2194e9e6 00000003 00000000 00000000 7fffffff e6c59f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cb0e>] normal_poll+0x0/0x11f
[<78298d67>] tty_poll+0x56/0x63
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7839dedc>] ip_output+0x270/0x2ac
[<7839c7d5>] ip_finish_output+0x0/0x212
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7839b270>] dst_output+0x0/0x7
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<78370f09>] sock_aio_write+0xcb/0xd7
[<78164133>] do_sync_write+0xc7/0x10a
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S EA120030 0 3839 3838 (NOTLB)
e3c81f34 00000082 7854ec14 ea120030 00000001 786b7e90 ea120560 ea120560
00000009 00000000 ea120030 e7428030 3ac85a37 00000028 00019fc2 ea12013c
7a01a980 f71bd800 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S E7428030 0 3866 3839 (NOTLB)
e5f65f34 00000082 7854ec14 e7428030 00000001 786b7e90 e7428560 e7428560
00000008 00000000 e7428030 f7c6c070 8cd13ab9 0001ff22 0001ae9d e742813c
7a01a980 f71bda80 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S 00000127 0 3878 3663 (NOTLB)
e3d2bdd8 00000082 00000000 00000127 00000000 00000000 00000000 e4c4c0ac
00000007 00000001 e21fa0b0 e21fb510 5abe5861 0000002b 00001417 e21fa1bc
7a022980 f71bd080 e21fa5e0 e21fa0b0 00000000 e21fa0b0 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S 00000000 0 3880 3878 (NOTLB)
e406fb60 00000096 00000046 00000000 00000002 00000001 e21fba68 e21fba48
0000000a 00000000 e21fb510 e4632130 adc501bc 0001ff61 00010239 e21fb61c
7a01a980 e93142c0 e91624c0 e1acb80c 7813e99c e91624c0 7fffffff e406ff9c
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cb0e>] normal_poll+0x0/0x11f
[<78298d67>] tty_poll+0x56/0x63
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7839dedc>] ip_output+0x270/0x2ac
[<7839c7d5>] ip_finish_output+0x0/0x212
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7839b270>] dst_output+0x0/0x7
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<78120273>] try_to_wake_up+0x393/0x39d
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<78370f09>] sock_aio_write+0xcb/0xd7
[<78164133>] do_sync_write+0xc7/0x10a
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S E401D550 0 3881 3880 (NOTLB)
e401ff34 00000082 7854ec14 e401d550 00000001 786b7e90 e401da80 e401da80
00000009 00000000 e401d550 e8eeab60 53749886 0000002c 0001b062 e401d65c
7a01a980 e4743d40 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash R running 0 3887 3881 (NOTLB)
unlinkd S 00000218 0 1139 3606 (NOTLB)
c1d17e58 00000086 00000002 00000218 7a0233e0 00000002 00000000 89332b20
00000007 00000001 89332b20 f7cd1490 861dbffc 0001e62a 00045f0d 89332c2c
7a022980 f6e11a40 1ff0a94d 00000003 00000000 00000218 f5a60c00 f7cb5b78
Call Trace:
[<7816943d>] pipe_wait+0x53/0x73
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78169b7d>] pipe_read+0x2af/0x321
[<78174b7c>] file_update_time+0x22/0x6a
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<78164176>] do_sync_read+0x0/0x10a
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
pdflush S ACC9FF9C 0 944 2 (L-TLB)
acc9ffb0 00000092 00000002 acc9ff9c acc9ff98 00000000 9c505590 9c505590
00000007 00000001 7a1fa0b0 f7fbe030 6ce9b6b3 0001e655 0000065b 7a1fa1bc
7a022980 f68e9340 1ff37b50 00000003 00000000 00000000 acc9ffc8 7814e33a
Call Trace:
[<7814e33a>] pdflush+0x0/0x1a6
[<7814e3e8>] pdflush+0xae/0x1a6
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
amavisd S 00000000 0 6000 2823 (NOTLB)
929dfe44 00000086 00000000 00000000 7813e7a9 00000000 00000000 f532f940
0000000a 00000001 e8eeb590 f7e03550 a54ef8a7 0001e62a 000028bb e8eeb69c
7a022980 d3e4a040 e8eebac0 e8eeb590 781299b4 fffffe00 7fffffff 7fffffff
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<781332be>] __rcu_process_callbacks+0xe3/0x172
[<781332a7>] __rcu_process_callbacks+0xcc/0x172
[<7840d623>] _spin_unlock+0x25/0x3b
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 95039DC4 0 6285 3606 (NOTLB)
95039dd8 00000082 00000002 95039dc4 95039dc0 00000000 00000000 c31076ac
00000005 00000001 7f498aa0 f7fbe030 84d480c7 0001e4d0 00001b01 7f498bac
7a022980 f6e117c0 1fd9ea3c 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 00000127 0 6286 3606 (NOTLB)
bc6c7dd8 00000082 00000000 00000127 00000000 00000000 00000000 d18f63ac
00000007 00000001 e7429490 f7e2b550 15274906 0001df92 0004a752 e742959c
7a022980 f68e9d40 e74299c0 e7429490 00000000 e7429490 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 00000127 0 6287 3606 (NOTLB)
e31d5dd8 00000082 00000000 00000127 00000000 00000000 00000000 9cb76cac
00000007 00000001 f7e2b550 7f498aa0 152ad54d 0001df92 00038c47 f7e2b65c
7a022980 c053dac0 f7e2ba80 f7e2b550 00000000 f7e2b550 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
pdflush S F6A75F9C 0 6288 2 (L-TLB)
f6a75fb0 00000092 00000002 f6a75f9c f6a75f98 00000000 00000292 edcd0560
0000000a 00000001 edcd0030 f7fbe030 bc332535 0001ff60 00002604 edcd013c
7a022980 f6e31800 2198f62b 00000003 00000000 00000000 f6a75fc8 7814e33a
Call Trace:
[<7814e33a>] pdflush+0x0/0x1a6
[<7814e3e8>] pdflush+0xae/0x1a6
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
amavisd S 85DD5E30 0 6350 2823 (NOTLB)
85dd5e44 00000086 00000002 85dd5e30 85dd5e2c 00000000 00000000 f532f940
00000007 00000001 a163eaa0 f7fbe030 bcc842a9 0001e62a 000024c3 a163ebac
7a022980 f62ad2c0 1ff0acdf 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd S F7E03550 0 6922 2823 (NOTLB)
d7bdfb60 00000096 786b7ed0 f7e03550 00000001 00000046 00000000 00000000
0000000a 00000001 f7e03550 edcd0a60 6a543cc5 0001e655 0008e72a f7e0365c
7a022980 f6e11cc0 7813e99c 00000000 00000046 f5829698 7fffffff d7bdff9c
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<783b133f>] tcp_v4_rcv+0x3de/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7839949e>] ip_local_deliver+0x1aa/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c73c>] process_backlog+0xef/0xfa
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c9d8>] net_rx_action+0x162/0x18f
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7838108c>] neigh_resolve_output+0x1f2/0x224
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<7812938d>] _local_bh_enable+0xb1/0xbe
[<781197d3>] do_page_fault+0x278/0x535
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd D 00000000 0 7227 2823 (NOTLB)
d1953e2c 00000082 00000000 00000000 7813e7a9 edcd0a60 7840d6e3 7a1d2000
00000007 00000001 edcd0a60 f7cb14d0 abd73b81 0001ff61 000032a5 edcd0b6c
7a022980 e632d2c0 7a1d2000 7812ccc1 00000000 00000292 d1953e3c 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<78152ded>] do_wp_page+0x3c4/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<78138305>] down_read_trylock+0x47/0x4e
[<7811976d>] do_page_fault+0x212/0x535
[<7811955b>] do_page_fault+0x0/0x535
[<7840d96a>] error_code+0x72/0x78
=======================
amavisd S 00000002 0 7836 2823 (NOTLB)
98947b60 00000096 00000000 00000002 7840ae59 00000000 00000000 00000000
0000000a 00000000 ec8feb60 89333550 bca917ba 0001e62a 00001d50 ec8fec6c
7a01a980 e8d34a80 1ff0ab63 00000003 00000000 00000000 7fffffff 98947f9c
Call Trace:
[<7840ae59>] __sched_text_start+0x701/0x810
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sadc D 00000000 0 8202 1 (NOTLB)
ccff9cc4 00000082 00000000 00000000 7813e7a9 edcd1490 7840d6e3 7a1d2000
00000007 00000001 edcd1490 e9e6c0b0 abd7dbb9 0001ff61 0000216c edcd159c
7a022980 c053d340 7a1d2000 7812ccc1 00000000 00000296 ccff9cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7814cb18>] free_hot_cold_page+0x13b/0x167
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
emerge D B7BF9CB0 0 8204 3061 (NOTLB)
b7bf9cc4 00200082 00000002 b7bf9cb0 b7bf9cac 00000000 7840d6e3 785cde80
00000007 00000000 89333550 785143e0 abd783fe 0001ff61 00002d5e 8933365c
7a01a980 c053d840 219905ed 00000003 00000000 00000000 b7bf9cd4 2199064e
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7814ce3d>] get_page_from_freelist+0x293/0x35e
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<78153e4d>] __handle_mm_fault+0x451/0x83b
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781197d3>] do_page_fault+0x278/0x535
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<7811955b>] do_page_fault+0x0/0x535
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
mkpir D 00000000 0 8297 3061 (NOTLB)
9c507cc4 00000082 00000000 00000000 7813e7a9 9c505590 7840d6e3 7a1d2000
00000007 00000001 9c505590 f7cb9510 abd79872 0001ff61 00002bf0 9c50569c
7a022980 f68e9340 7a1d2000 7812ccc1 00000000 00000296 9c507cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<781a8658>] journal_stop+0x1ce/0x1da
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78148be7>] file_read_actor+0x0/0xde
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
screen S 8DBAFF98 0 8364 3866 (NOTLB)
8dbaffac 00000086 00000002 8dbaff98 8dbaff94 00000000 00000000 7812850a
00000007 00000000 f7c6c070 785143e0 455dd02b 0001ff61 00005ab9 f7c6c17c
7a01a980 e913ad40 2198ff2e 00000003 00000000 00000000 ffffffff 00000000
Call Trace:
[<7812850a>] do_setitimer+0x16e/0x30b
[<7812f315>] sys_pause+0x11/0x17
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
screen D 00000000 0 8365 8364 (NOTLB)
eca0fcc4 00000082 00000000 00000000 7813e7a9 f7cb14d0 7840d6e3 7a1d2000
00000007 00000001 f7cb14d0 9c505590 abd76c82 0001ff61 00003101 f7cb15dc
7a022980 d3e4acc0 7a1d2000 7812ccc1 00000000 00000296 eca0fcd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7840c019>] __mutex_lock_slowpath+0x25f/0x267
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
> Are any IO errors occurring at all?
No.
Best regards,
Krzysztof Olędzki
Powered by blists - more mailing lists