Feb 26 09:41:19 [kernel] [1726920.709038] INFO: task kswapd0:590 blocked for more than 120 seconds. Feb 26 09:41:19 [kernel] [1726920.709042] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 26 09:41:19 [kernel] [1726920.709045] kswapd0 D 0000000000000000 1800 590 2 0x00000000 Feb 26 09:41:19 [kernel] [1726920.709052] ffff8801b626b9b0 0000000000000046 00000000001d3300 0000000000000000 Feb 26 09:41:19 [kernel] [1726920.709058] ffff8801b626b940 00000000001d2880 ffff8801b6262680 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709064] ffff8801b626bfd8 ffff8801b626a000 00000000001d2880 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709069] Call Trace: Feb 26 09:41:19 [kernel] [1726920.709079] [] ? mutex_lock_nested+0x205/0x330 Feb 26 09:41:19 [kernel] [1726920.709083] [] schedule+0x3a/0x50 Feb 26 09:41:19 [kernel] [1726920.709087] [] mutex_lock_nested+0x15d/0x330 Feb 26 09:41:19 [kernel] [1726920.709092] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709096] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709100] [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709106] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709111] [] ? sched_clock_cpu+0xa8/0x120 Feb 26 09:41:19 [kernel] [1726920.709115] [] ? trace_hardirqs_off+0xd/0x10 Feb 26 09:41:19 [kernel] [1726920.709119] [] ? local_clock+0x4f/0x60 Feb 26 09:41:19 [kernel] [1726920.709122] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709127] [] ? xfs_ail_push_all+0x80/0x90 Feb 26 09:41:19 [kernel] [1726920.709132] [] ? _raw_spin_unlock+0x26/0x30 Feb 26 09:41:19 [kernel] [1726920.709136] [] xfs_reclaim_inodes_nr+0x2e/0x40 Feb 26 09:41:19 [kernel] [1726920.709139] [] xfs_fs_free_cached_objects+0x10/0x20 Feb 26 09:41:19 [kernel] [1726920.709144] [] prune_super+0x101/0x1b0 Feb 26 09:41:19 [kernel] [1726920.709149] [] shrink_slab+0x165/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709153] [] kswapd+0x70a/0xa60 Feb 26 09:41:19 [kernel] [1726920.709158] [] ? wake_up_bit+0x40/0x40 Feb 26 09:41:19 [kernel] [1726920.709161] [] ? try_to_free_pages+0x110/0x110 Feb 26 09:41:19 [kernel] [1726920.709165] [] kthread+0xa6/0xb0 Feb 26 09:41:19 [kernel] [1726920.709173] [] ? retint_restore_args+0xe/0xe Feb 26 09:41:19 [kernel] [1726920.709177] [] ? __init_kthread_worker+0x70/0x70 Feb 26 09:41:19 [kernel] [1726920.709180] [] ? gs_change+0xb/0xb Feb 26 09:41:19 [kernel] [1726920.709183] 3 locks held by kswapd0/590: Feb 26 09:41:19 [kernel] [1726920.709185] #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x34/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709192] #1: (&type->s_umount_key#31){.+.+.+}, at: [] grab_super_passive+0x4f/0xc0 Feb 26 09:41:19 [kernel] [1726920.709199] #2: (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709211] INFO: task gkrellmd:2341 blocked for more than 120 seconds. Feb 26 09:41:19 [kernel] [1726920.709214] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 26 09:41:19 [kernel] [1726920.709216] gkrellmd D 0000000000000000 1552 2341 1 0x00000000 Feb 26 09:41:19 [kernel] [1726920.709221] ffff8801b4b5d6f8 0000000000000046 ffffffff8189813a 0000000000000000 Feb 26 09:41:19 [kernel] [1726920.709226] ffff880100000000 00000000001d2880 ffff8801b5ac9340 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709232] ffff8801b4b5dfd8 ffff8801b4b5c000 00000000001d2880 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709237] Call Trace: Feb 26 09:41:19 [kernel] [1726920.709240] [] ? __schedule+0x70a/0x920 Feb 26 09:41:19 [kernel] [1726920.709244] [] ? mutex_lock_nested+0x205/0x330 Feb 26 09:41:19 [kernel] [1726920.709247] [] schedule+0x3a/0x50 Feb 26 09:41:19 [kernel] [1726920.709251] [] mutex_lock_nested+0x15d/0x330 Feb 26 09:41:19 [kernel] [1726920.709254] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709258] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709262] [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709265] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709269] [] ? sched_clock_cpu+0xa8/0x120 Feb 26 09:41:19 [kernel] [1726920.709273] [] ? trace_hardirqs_off+0xd/0x10 Feb 26 09:41:19 [kernel] [1726920.709304] [] ? local_clock+0x4f/0x60 Feb 26 09:41:19 [kernel] [1726920.709312] [] ? xfs_ail_push_all+0x80/0x90 Feb 26 09:41:19 [kernel] [1726920.709316] [] ? _raw_spin_unlock+0x26/0x30 Feb 26 09:41:19 [kernel] [1726920.709319] [] xfs_reclaim_inodes_nr+0x2e/0x40 Feb 26 09:41:19 [kernel] [1726920.709323] [] xfs_fs_free_cached_objects+0x10/0x20 Feb 26 09:41:19 [kernel] [1726920.709327] [] prune_super+0x101/0x1b0 Feb 26 09:41:19 [kernel] [1726920.709330] [] shrink_slab+0x165/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709334] [] do_try_to_free_pages+0x267/0x460 Feb 26 09:41:19 [kernel] [1726920.709337] [] try_to_free_pages+0x96/0x110 Feb 26 09:41:19 [kernel] [1726920.709342] [] __alloc_pages_nodemask+0x4cd/0x820 Feb 26 09:41:19 [kernel] [1726920.709347] [] ? sched_clock_cpu+0xa8/0x120 Feb 26 09:41:19 [kernel] [1726920.709352] [] alloc_pages_current+0xa1/0x110 Feb 26 09:41:19 [kernel] [1726920.709356] [] new_slab+0x265/0x300 Feb 26 09:41:19 [kernel] [1726920.709359] [] ? mutex_lock_nested+0x28c/0x330 Feb 26 09:41:19 [kernel] [1726920.709363] [] __slab_alloc+0x2ca/0x540 Feb 26 09:41:19 [kernel] [1726920.709367] [] ? getname_flags+0x36/0x270 Feb 26 09:41:19 [kernel] [1726920.709371] [] ? local_clock+0x4f/0x60 Feb 26 09:41:19 [kernel] [1726920.709374] [] ? getname_flags+0x36/0x270 Feb 26 09:41:19 [kernel] [1726920.709378] [] kmem_cache_alloc+0xdb/0x120 Feb 26 09:41:19 [kernel] [1726920.709381] [] getname_flags+0x36/0x270 Feb 26 09:41:19 [kernel] [1726920.709384] [] getname+0xd/0x10 Feb 26 09:41:19 [kernel] [1726920.709388] [] do_sys_open+0xc8/0x1d0 Feb 26 09:41:19 [kernel] [1726920.709391] [] sys_open+0x1b/0x20 Feb 26 09:41:19 [kernel] [1726920.709395] [] system_call_fastpath+0x16/0x1b Feb 26 09:41:19 [kernel] [1726920.709397] 3 locks held by gkrellmd/2341: Feb 26 09:41:19 [kernel] [1726920.709399] #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x34/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709405] #1: (&type->s_umount_key#31){.+.+.+}, at: [] grab_super_passive+0x4f/0xc0 Feb 26 09:41:19 [kernel] [1726920.709412] #2: (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709423] INFO: task nfsd:2821 blocked for more than 120 seconds. Feb 26 09:41:19 [kernel] [1726920.709425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 26 09:41:19 [kernel] [1726920.709428] nfsd D 0000000000000000 1160 2821 2 0x00000000 Feb 26 09:41:19 [kernel] [1726920.709432] ffff8801ae9577a0 0000000000000046 ffffffff8189813a 0000000000000000 Feb 26 09:41:19 [kernel] [1726920.709438] ffff880100000000 00000000001d2880 ffff8801b7ae4d00 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709443] ffff8801ae957fd8 ffff8801ae956000 00000000001d2880 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709448] Call Trace: Feb 26 09:41:19 [kernel] [1726920.709452] [] ? __schedule+0x70a/0x920 Feb 26 09:41:19 [kernel] [1726920.709455] [] ? mutex_lock_nested+0x205/0x330 Feb 26 09:41:19 [kernel] [1726920.709459] [] schedule+0x3a/0x50 Feb 26 09:41:19 [kernel] [1726920.709462] [] mutex_lock_nested+0x15d/0x330 Feb 26 09:41:19 [kernel] [1726920.709466] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709469] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709473] [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709477] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709481] [] ? sched_clock_cpu+0xa8/0x120 Feb 26 09:41:19 [kernel] [1726920.709484] [] ? trace_hardirqs_off+0xd/0x10 Feb 26 09:41:19 [kernel] [1726920.709488] [] ? local_clock+0x4f/0x60 Feb 26 09:41:19 [kernel] [1726920.709491] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709495] [] ? xfs_ail_push_all+0x80/0x90 Feb 26 09:41:19 [kernel] [1726920.709499] [] ? _raw_spin_unlock+0x26/0x30 Feb 26 09:41:19 [kernel] [1726920.709502] [] xfs_reclaim_inodes_nr+0x2e/0x40 Feb 26 09:41:19 [kernel] [1726920.709506] [] xfs_fs_free_cached_objects+0x10/0x20 Feb 26 09:41:19 [kernel] [1726920.709509] [] prune_super+0x101/0x1b0 Feb 26 09:41:19 [kernel] [1726920.709513] [] shrink_slab+0x165/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709517] [] do_try_to_free_pages+0x267/0x460 Feb 26 09:41:19 [kernel] [1726920.709520] [] try_to_free_pages+0x96/0x110 Feb 26 09:41:19 [kernel] [1726920.709524] [] __alloc_pages_nodemask+0x4cd/0x820 Feb 26 09:41:19 [kernel] [1726920.709534] [] svc_recv+0xec/0x900 Feb 26 09:41:19 [kernel] [1726920.709539] [] ? try_to_wake_up+0x2d0/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709543] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709547] [] nfsd+0x9d/0x150 Feb 26 09:41:19 [kernel] [1726920.709550] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709553] [] kthread+0xa6/0xb0 Feb 26 09:41:19 [kernel] [1726920.709557] [] kernel_thread_helper+0x4/0x10 Feb 26 09:41:19 [kernel] [1726920.709561] [] ? retint_restore_args+0xe/0xe Feb 26 09:41:19 [kernel] [1726920.709565] [] ? __init_kthread_worker+0x70/0x70 Feb 26 09:41:19 [kernel] [1726920.709568] [] ? gs_change+0xb/0xb Feb 26 09:41:19 [kernel] [1726920.709570] 3 locks held by nfsd/2821: Feb 26 09:41:19 [kernel] [1726920.709572] #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x34/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709578] #1: (&type->s_umount_key#31){.+.+.+}, at: [] grab_super_passive+0x4f/0xc0 Feb 26 09:41:19 [kernel] [1726920.709585] #2: (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709592] INFO: task nfsd:2822 blocked for more than 120 seconds. Feb 26 09:41:19 [kernel] [1726920.709594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 26 09:41:19 [kernel] [1726920.709596] nfsd D 0000000000000000 1224 2822 2 0x00000000 Feb 26 09:41:19 [kernel] [1726920.709601] ffff8801aea517a0 0000000000000046 00000000001d3300 0000000000000002 Feb 26 09:41:19 [kernel] [1726920.709607] ffff8801aea51730 00000000001d2880 ffff8801b5acb9c0 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709612] ffff8801aea51fd8 ffff8801aea50000 00000000001d2880 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709617] Call Trace: Feb 26 09:41:19 [kernel] [1726920.709621] [] ? mutex_lock_nested+0x205/0x330 Feb 26 09:41:19 [kernel] [1726920.709624] [] schedule+0x3a/0x50 Feb 26 09:41:19 [kernel] [1726920.709627] [] mutex_lock_nested+0x15d/0x330 Feb 26 09:41:19 [kernel] [1726920.709631] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709634] [] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709638] [] xfs_reclaim_inodes_ag+0x2ee/0x3a0 Feb 26 09:41:19 [kernel] [1726920.709642] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709646] [] ? sched_clock_cpu+0xa8/0x120 Feb 26 09:41:19 [kernel] [1726920.709649] [] ? trace_hardirqs_off+0xd/0x10 Feb 26 09:41:19 [kernel] [1726920.709653] [] ? local_clock+0x4f/0x60 Feb 26 09:41:19 [kernel] [1726920.709656] [] ? lock_release_holdtime+0x3d/0x1a0 Feb 26 09:41:19 [kernel] [1726920.709660] [] ? xfs_ail_push_all+0x80/0x90 Feb 26 09:41:19 [kernel] [1726920.709664] [] ? _raw_spin_unlock+0x26/0x30 Feb 26 09:41:19 [kernel] [1726920.709667] [] xfs_reclaim_inodes_nr+0x2e/0x40 Feb 26 09:41:19 [kernel] [1726920.709671] [] xfs_fs_free_cached_objects+0x10/0x20 Feb 26 09:41:19 [kernel] [1726920.709674] [] prune_super+0x101/0x1b0 Feb 26 09:41:19 [kernel] [1726920.709677] [] shrink_slab+0x165/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709681] [] do_try_to_free_pages+0x267/0x460 Feb 26 09:41:19 [kernel] [1726920.709685] [] try_to_free_pages+0x96/0x110 Feb 26 09:41:19 [kernel] [1726920.709689] [] __alloc_pages_nodemask+0x4cd/0x820 Feb 26 09:41:19 [kernel] [1726920.709694] [] ? svc_tcp_recvfrom+0x50d/0x760 Feb 26 09:41:19 [kernel] [1726920.709698] [] alloc_pages_current+0xa1/0x110 Feb 26 09:41:19 [kernel] [1726920.709702] [] svc_recv+0xec/0x900 Feb 26 09:41:19 [kernel] [1726920.709705] [] ? try_to_wake_up+0x2d0/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709708] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709712] [] nfsd+0x9d/0x150 Feb 26 09:41:19 [kernel] [1726920.709715] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709718] [] kthread+0xa6/0xb0 Feb 26 09:41:19 [kernel] [1726920.709722] [] kernel_thread_helper+0x4/0x10 Feb 26 09:41:19 [kernel] [1726920.709726] [] ? retint_restore_args+0xe/0xe Feb 26 09:41:19 [kernel] [1726920.709729] [] ? __init_kthread_worker+0x70/0x70 Feb 26 09:41:19 [kernel] [1726920.709732] [] ? gs_change+0xb/0xb Feb 26 09:41:19 [kernel] [1726920.709735] 3 locks held by nfsd/2822: Feb 26 09:41:19 [kernel] [1726920.709737] #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x34/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709757] INFO: task nfsd:2824 blocked for more than 120 seconds. Feb 26 09:41:19 [kernel] [1726920.709759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 26 09:41:19 [kernel] [1726920.709761] nfsd D 0000000000000000 1208 2824 2 0x00000000 Feb 26 09:41:19 [kernel] [1726920.709766] ffff8801ae065d60 0000000000000046 00000000001d3300 0000000000000000 Feb 26 09:41:19 [kernel] [1726920.709771] ffff8801ae065cf0 00000000001d2880 ffff8801b54aa680 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709777] ffff8801ae065fd8 ffff8801ae064000 00000000001d2880 00000000001d2880 Feb 26 09:41:19 [kernel] [1726920.709782] Call Trace: Feb 26 09:41:19 [kernel] [1726920.709785] [] ? mutex_lock_nested+0x205/0x330 Feb 26 09:41:19 [kernel] [1726920.709789] [] schedule+0x3a/0x50 Feb 26 09:41:19 [kernel] [1726920.709792] [] mutex_lock_nested+0x15d/0x330 Feb 26 09:41:19 [kernel] [1726920.709796] [] ? svc_send+0x59/0xf0 Feb 26 09:41:19 [kernel] [1726920.709799] [] ? svc_send+0x59/0xf0 Feb 26 09:41:19 [kernel] [1726920.709803] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709806] [] svc_send+0x59/0xf0 Feb 26 09:41:19 [kernel] [1726920.709809] [] ? try_to_wake_up+0x2d0/0x2d0 Feb 26 09:41:19 [kernel] [1726920.709813] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709816] [] svc_process+0x120/0x150 Feb 26 09:41:19 [kernel] [1726920.709819] [] nfsd+0xb5/0x150 Feb 26 09:41:19 [kernel] [1726920.709823] [] ? nfsd_shutdown+0x30/0x30 Feb 26 09:41:19 [kernel] [1726920.709826] [] kthread+0xa6/0xb0 Feb 26 09:41:19 [kernel] [1726920.709830] [] kernel_thread_helper+0x4/0x10 Feb 26 09:41:19 [kernel] [1726920.709833] [] ? retint_restore_args+0xe/0xe Feb 26 09:41:19 [kernel] [1726920.709837] [] ? __init_kthread_worker+0x70/0x70 Feb 26 09:41:19 [kernel] [1726920.709841] [] ? gs_change+0xb/0xb Feb 26 09:41:19 [kernel] [1726920.709843] 1 lock held by nfsd/2824: