| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20231018094157.gyfdkgemoissjups@awork3.anarazel.de>
Date: Wed, 18 Oct 2023 02:41:57 -0700
From: Andres Freund <andres@...razel.de>
To: Theodore Ts'o <tytso@....edu>
Cc: Thorsten Leemhuis <regressions@...mhuis.info>,
Shreeya Patel <shreeya.patel@...labora.com>,
linux-ext4@...r.kernel.org,
Ricardo Cañuelo <ricardo.canuelo@...labora.com>,
gustavo.padovan@...labora.com, zsm@...gle.com, garrick@...gle.com,
Linux regressions mailing list <regressions@...ts.linux.dev>
Subject: Re: task hung in ext4_fallocate #2
HI,
On 2023-10-17 19:50:09 -0700, Andres Freund wrote:
> I switched it to CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y and am rebuilding
> with lockdep enabled. Curious to see how long it'll take to trigger the
> problem with it enabled...
After booting into the lockdep enabled kernel I was immediately greeted with a
lockdep splat - in nouveau. Not realizing that would prevent subsequent
lockdep reports, I continued trying to reproduce the issue. Took 2-3h.
The next boot quickly triggered another unrelated lockdep report, possibly
triggered by a usb hub disconnecting.
Luckily the boot after that the problem very quickly reproduced. Lockdep did
not report a bug, but the list of held locks after the "hung task" warning
seems interesting:
[ 367.987710] INFO: task iou-wrk-3385:3753 blocked for more than 122 seconds.
[ 367.987882] Not tainted 6.6.0-rc6-andres-00001-g01edcfe38260 #79
[ 367.987896] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 367.987907] task:iou-wrk-3385 state:D stack:0 pid:3753 ppid:3320 flags:0x00004000
[ 367.987931] Call Trace:
[ 367.987942] <TASK>
[ 367.987964] __schedule+0x4c3/0x17f0
[ 367.988000] ? schedule+0xa6/0xe0
[ 367.988030] schedule+0x5f/0xe0
[ 367.988049] schedule_preempt_disabled+0x15/0x20
[ 367.988068] rwsem_down_read_slowpath+0x2b9/0x590
[ 367.988110] down_read+0x64/0x150
[ 367.988128] ext4_file_write_iter+0x435/0xa90
[ 367.988150] ? find_held_lock+0x2b/0x80
[ 367.988170] ? io_write+0x366/0x4d0
[ 367.988184] ? lock_release+0xba/0x260
[ 367.988199] ? lock_is_held_type+0x84/0xf0
[ 367.988225] io_write+0x12b/0x4d0
[ 367.988264] ? lock_acquire+0xb3/0x2a0
[ 367.988280] ? io_worker_handle_work+0x10c/0x560
[ 367.988303] io_issue_sqe+0x5a/0x340
[ 367.988323] io_wq_submit_work+0x86/0x240
[ 367.988342] io_worker_handle_work+0x156/0x560
[ 367.988371] io_wq_worker+0xf6/0x370
[ 367.988392] ? find_held_lock+0x2b/0x80
[ 367.988409] ? ret_from_fork+0x17/0x50
[ 367.988427] ? lock_release+0xba/0x260
[ 367.988441] ? io_worker_handle_work+0x560/0x560
[ 367.988459] ? io_worker_handle_work+0x560/0x560
[ 367.988477] ret_from_fork+0x2d/0x50
[ 367.988493] ? io_worker_handle_work+0x560/0x560
[ 367.988509] ret_from_fork_asm+0x11/0x20
[ 367.988551] </TASK>
[ 367.988562] INFO: task iou-wrk-3385:3855 blocked for more than 122 seconds.
[ 367.988575] Not tainted 6.6.0-rc6-andres-00001-g01edcfe38260 #79
[ 367.988586] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 367.988596] task:iou-wrk-3385 state:D stack:0 pid:3855 ppid:3320 flags:0x00004000
[ 367.988615] Call Trace:
[ 367.988625] <TASK>
[ 367.988641] __schedule+0x4c3/0x17f0
[ 367.988670] ? schedule+0xa6/0xe0
[ 367.988699] schedule+0x5f/0xe0
[ 367.988718] schedule_preempt_disabled+0x15/0x20
[ 367.988736] rwsem_down_read_slowpath+0x2b9/0x590
[ 367.988777] down_read+0x64/0x150
[ 367.988795] ext4_file_write_iter+0x435/0xa90
[ 367.988813] ? find_held_lock+0x2b/0x80
[ 367.988831] ? io_write+0x366/0x4d0
[ 367.988844] ? lock_release+0xba/0x260
[ 367.988858] ? lock_is_held_type+0x84/0xf0
[ 367.988883] io_write+0x12b/0x4d0
[ 367.988922] ? lock_acquire+0xb3/0x2a0
[ 367.988937] ? io_worker_handle_work+0x10c/0x560
[ 367.988959] io_issue_sqe+0x5a/0x340
[ 367.988978] io_wq_submit_work+0x86/0x240
[ 367.988998] io_worker_handle_work+0x156/0x560
[ 367.989026] io_wq_worker+0xf6/0x370
[ 367.989047] ? find_held_lock+0x2b/0x80
[ 367.989064] ? ret_from_fork+0x17/0x50
[ 367.989080] ? lock_release+0xba/0x260
[ 367.989094] ? io_worker_handle_work+0x560/0x560
[ 367.989112] ? io_worker_handle_work+0x560/0x560
[ 367.989130] ret_from_fork+0x2d/0x50
[ 367.989146] ? io_worker_handle_work+0x560/0x560
[ 367.989162] ret_from_fork_asm+0x11/0x20
[ 367.989202] </TASK>
[ 367.989251] INFO: task postgres:3409 blocked for more than 122 seconds.
[ 367.989263] Not tainted 6.6.0-rc6-andres-00001-g01edcfe38260 #79
[ 367.989273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 367.989283] task:postgres state:D stack:0 pid:3409 ppid:3320 flags:0x00020002
[ 367.989301] Call Trace:
[ 367.989310] <TASK>
[ 367.989327] __schedule+0x4c3/0x17f0
[ 367.989345] ? prepare_to_wait+0x19/0x90
[ 367.989383] schedule+0x5f/0xe0
[ 367.989403] inode_dio_wait+0xd5/0x100
[ 367.989424] ? sugov_start+0x120/0x120
[ 367.989445] ext4_fallocate+0x149/0x1140
[ 367.989461] ? lock_acquire+0xb3/0x2a0
[ 367.989476] ? __x64_sys_fallocate+0x42/0x70
[ 367.989513] vfs_fallocate+0x135/0x450
[ 367.989534] __x64_sys_fallocate+0x42/0x70
[ 367.989554] do_syscall_64+0x38/0x80
[ 367.989573] entry_SYSCALL_64_after_hwframe+0x46/0xb0
[ 367.989589] RIP: 0033:0x7f9bbc458f82
[ 367.989605] RSP: 002b:00007fff59b4b6a8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
[ 367.989624] RAX: ffffffffffffffda RBX: 00000000000d0000 RCX: 00007f9bbc458f82
[ 367.989637] RDX: 00000000139ec000 RSI: 0000000000000000 RDI: 0000000000000046
[ 367.989649] RBP: 00000000139ec000 R08: 00000000139ec000 R09: 0000558ea7726ed0
[ 367.989661] R10: 00000000000d0000 R11: 0000000000000246 R12: 0000000000000b98
[ 367.989672] R13: 000000000a000013 R14: 0000000000000035 R15: 0000558ea719c070
[ 367.989710] </TASK>
[ 367.989723]
Showing all locks held in the system:
[ 367.989749] 1 lock held by khungtaskd/130:
[ 367.989762] #0: ffffffff8412d020 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x32/0x1b0
[ 367.989864] 1 lock held by iou-wrk-3322/3768:
[ 367.989875] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.989918] 1 lock held by iou-wrk-3322/4142:
[ 367.989929] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.989968] 1 lock held by iou-wrk-3322/4143:
[ 367.989978] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990015] 1 lock held by iou-wrk-3322/4144:
[ 367.990025] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990062] 1 lock held by iou-wrk-3322/4145:
[ 367.990072] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990109] 1 lock held by iou-wrk-3322/4146:
[ 367.990119] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990157] 1 lock held by iou-wrk-3322/4147:
[ 367.990167] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990204] 1 lock held by iou-wrk-3322/4148:
[ 367.990214] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990252] 1 lock held by iou-wrk-3385/3753:
[ 367.990263] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990300] 1 lock held by iou-wrk-3385/3855:
[ 367.990310] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990347] 1 lock held by iou-wrk-3386/3953:
[ 367.990357] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990394] 1 lock held by iou-wrk-3386/4016:
[ 367.990404] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990478] 1 lock held by iou-wrk-3386/4018:
[ 367.990489] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990527] 1 lock held by iou-wrk-3386/4026:
[ 367.990537] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990574] 1 lock held by iou-wrk-3386/4027:
[ 367.990584] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990622] 1 lock held by iou-wrk-3386/4030:
[ 367.990632] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990669] 1 lock held by iou-wrk-3386/4031:
[ 367.990679] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990716] 1 lock held by iou-wrk-3386/4034:
[ 367.990726] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990763] 1 lock held by iou-wrk-3386/4036:
[ 367.990773] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990810] 1 lock held by iou-wrk-3386/4037:
[ 367.990820] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990857] 1 lock held by iou-wrk-3386/4039:
[ 367.990868] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990905] 1 lock held by iou-wrk-3386/4041:
[ 367.990915] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990952] 1 lock held by iou-wrk-3386/4043:
[ 367.990962] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.990999] 1 lock held by iou-wrk-3386/4045:
[ 367.991009] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991046] 1 lock held by iou-wrk-3386/4047:
[ 367.991056] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991094] 1 lock held by iou-wrk-3386/4049:
[ 367.991104] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991141] 1 lock held by iou-wrk-3386/4052:
[ 367.991151] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991188] 1 lock held by iou-wrk-3386/4054:
[ 367.991198] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991235] 1 lock held by iou-wrk-3386/4056:
[ 367.991245] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991283] 1 lock held by iou-wrk-3386/4058:
[ 367.991293] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991330] 1 lock held by iou-wrk-3386/4060:
[ 367.991340] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991377] 1 lock held by iou-wrk-3386/4061:
[ 367.991387] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991424] 1 lock held by iou-wrk-3386/4063:
[ 367.991434] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991471] 1 lock held by iou-wrk-3386/4066:
[ 367.991481] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991518] 1 lock held by iou-wrk-3386/4069:
[ 367.991528] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991566] 1 lock held by iou-wrk-3386/4071:
[ 367.991576] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991613] 1 lock held by iou-wrk-3386/4073:
[ 367.991623] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991660] 1 lock held by iou-wrk-3386/4075:
[ 367.991670] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991707] 1 lock held by iou-wrk-3386/4076:
[ 367.991717] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991754] 1 lock held by iou-wrk-3386/4077:
[ 367.991764] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991801] 1 lock held by iou-wrk-3386/4078:
[ 367.991811] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991848] 1 lock held by iou-wrk-3386/4079:
[ 367.991859] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991896] 1 lock held by iou-wrk-3387/3990:
[ 367.991906] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991943] 1 lock held by iou-wrk-3387/3998:
[ 367.991953] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.991990] 1 lock held by iou-wrk-3387/4083:
[ 367.992001] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992038] 1 lock held by iou-wrk-3387/4085:
[ 367.992048] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992085] 1 lock held by iou-wrk-3387/4087:
[ 367.992095] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992132] 1 lock held by iou-wrk-3387/4089:
[ 367.992142] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992179] 1 lock held by iou-wrk-3387/4091:
[ 367.992189] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992226] 1 lock held by iou-wrk-3387/4093:
[ 367.992236] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992273] 1 lock held by iou-wrk-3387/4095:
[ 367.992283] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992321] 1 lock held by iou-wrk-3387/4097:
[ 367.992331] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992368] 1 lock held by iou-wrk-3387/4099:
[ 367.992378] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992415] 1 lock held by iou-wrk-3387/4101:
[ 367.992425] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992462] 1 lock held by iou-wrk-3387/4103:
[ 367.992472] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992509] 1 lock held by iou-wrk-3387/4105:
[ 367.992519] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992556] 1 lock held by iou-wrk-3387/4107:
[ 367.992566] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992603] 1 lock held by iou-wrk-3387/4109:
[ 367.992613] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992650] 1 lock held by iou-wrk-3387/4111:
[ 367.992660] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992697] 1 lock held by iou-wrk-3387/4113:
[ 367.992707] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992745] 1 lock held by iou-wrk-3387/4115:
[ 367.992755] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992792] 1 lock held by iou-wrk-3387/4117:
[ 367.992817] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992823] 1 lock held by iou-wrk-3387/4119:
[ 367.992825] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992832] 1 lock held by iou-wrk-3387/4122:
[ 367.992834] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992840] 1 lock held by iou-wrk-3387/4124:
[ 367.992842] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992849] 1 lock held by iou-wrk-3387/4126:
[ 367.992851] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992858] 1 lock held by iou-wrk-3387/4128:
[ 367.992859] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992866] 1 lock held by iou-wrk-3387/4130:
[ 367.992868] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992875] 1 lock held by iou-wrk-3387/4132:
[ 367.992876] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992883] 1 lock held by iou-wrk-3387/4134:
[ 367.992885] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992892] 1 lock held by iou-wrk-3387/4136:
[ 367.992893] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992900] 1 lock held by iou-wrk-3387/4138:
[ 367.992902] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992909] 1 lock held by iou-wrk-3387/4139:
[ 367.992910] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992917] 1 lock held by iou-wrk-3387/4141:
[ 367.992919] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992926] 1 lock held by iou-wrk-3388/3968:
[ 367.992927] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992934] 1 lock held by iou-wrk-3388/4013:
[ 367.992936] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992943] 1 lock held by iou-wrk-3388/4015:
[ 367.992944] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992951] 1 lock held by iou-wrk-3388/4021:
[ 367.992953] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992960] 1 lock held by iou-wrk-3388/4150:
[ 367.992962] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992968] 1 lock held by iou-wrk-3388/4151:
[ 367.992970] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992977] 1 lock held by iou-wrk-3388/4152:
[ 367.992979] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992985] 1 lock held by iou-wrk-3388/4153:
[ 367.992987] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.992994] 1 lock held by iou-wrk-3388/4154:
[ 367.992996] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993002] 1 lock held by iou-wrk-3388/4155:
[ 367.993004] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993011] 1 lock held by iou-wrk-3388/4156:
[ 367.993013] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993019] 1 lock held by iou-wrk-3388/4157:
[ 367.993021] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993028] 1 lock held by iou-wrk-3388/4158:
[ 367.993030] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993036] 1 lock held by iou-wrk-3388/4159:
[ 367.993038] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993045] 1 lock held by iou-wrk-3388/4160:
[ 367.993047] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993054] 1 lock held by iou-wrk-3388/4161:
[ 367.993055] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993062] 1 lock held by iou-wrk-3388/4162:
[ 367.993064] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993071] 1 lock held by iou-wrk-3388/4163:
[ 367.993073] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993079] 1 lock held by iou-wrk-3388/4164:
[ 367.993081] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993088] 1 lock held by iou-wrk-3389/3965:
[ 367.993089] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993096] 1 lock held by iou-wrk-3389/4005:
[ 367.993098] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993105] 1 lock held by iou-wrk-3389/4023:
[ 367.993106] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993113] 1 lock held by iou-wrk-3389/4024:
[ 367.993115] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993122] 1 lock held by iou-wrk-3389/4025:
[ 367.993124] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993130] 1 lock held by iou-wrk-3389/4028:
[ 367.993132] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993139] 1 lock held by iou-wrk-3389/4029:
[ 367.993140] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993147] 1 lock held by iou-wrk-3389/4032:
[ 367.993149] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993156] 1 lock held by iou-wrk-3389/4033:
[ 367.993157] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993164] 1 lock held by iou-wrk-3389/4035:
[ 367.993166] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993172] 1 lock held by iou-wrk-3389/4038:
[ 367.993174] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993181] 1 lock held by iou-wrk-3389/4040:
[ 367.993183] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993189] 1 lock held by iou-wrk-3389/4042:
[ 367.993191] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993198] 1 lock held by iou-wrk-3389/4044:
[ 367.993200] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993206] 1 lock held by iou-wrk-3389/4046:
[ 367.993208] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993215] 1 lock held by iou-wrk-3389/4048:
[ 367.993217] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993223] 1 lock held by iou-wrk-3389/4050:
[ 367.993225] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993232] 1 lock held by iou-wrk-3389/4051:
[ 367.993234] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993240] 1 lock held by iou-wrk-3389/4053:
[ 367.993242] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993249] 1 lock held by iou-wrk-3389/4055:
[ 367.993251] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993257] 1 lock held by iou-wrk-3389/4057:
[ 367.993259] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993266] 1 lock held by iou-wrk-3389/4059:
[ 367.993268] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993274] 1 lock held by iou-wrk-3389/4062:
[ 367.993276] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993283] 1 lock held by iou-wrk-3389/4064:
[ 367.993284] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993291] 1 lock held by iou-wrk-3389/4065:
[ 367.993293] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993300] 1 lock held by iou-wrk-3389/4067:
[ 367.993301] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993308] 1 lock held by iou-wrk-3389/4068:
[ 367.993310] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993317] 1 lock held by iou-wrk-3389/4070:
[ 367.993318] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993325] 1 lock held by iou-wrk-3389/4072:
[ 367.993327] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993334] 1 lock held by iou-wrk-3389/4074:
[ 367.993335] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993342] 1 lock held by iou-wrk-3389/4080:
[ 367.993344] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993351] 1 lock held by iou-wrk-3389/4081:
[ 367.993352] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993359] 1 lock held by iou-wrk-3390/3856:
[ 367.993361] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993368] 1 lock held by iou-wrk-3390/4014:
[ 367.993370] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993376] 1 lock held by iou-wrk-3390/4017:
[ 367.993378] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993385] 1 lock held by iou-wrk-3390/4165:
[ 367.993387] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993393] 1 lock held by iou-wrk-3390/4166:
[ 367.993395] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993402] 1 lock held by iou-wrk-3390/4167:
[ 367.993404] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993410] 1 lock held by iou-wrk-3390/4168:
[ 367.993412] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993419] 1 lock held by iou-wrk-3390/4169:
[ 367.993420] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993427] 1 lock held by iou-wrk-3390/4170:
[ 367.993429] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993436] 1 lock held by iou-wrk-3390/4171:
[ 367.993437] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993444] 1 lock held by iou-wrk-3390/4172:
[ 367.993446] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993453] 1 lock held by iou-wrk-3390/4173:
[ 367.993454] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993461] 1 lock held by iou-wrk-3390/4174:
[ 367.993463] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993470] 1 lock held by iou-wrk-3390/4175:
[ 367.993471] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993478] 1 lock held by iou-wrk-3390/4176:
[ 367.993480] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993486] 1 lock held by iou-wrk-3390/4177:
[ 367.993488] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993495] 1 lock held by iou-wrk-3390/4178:
[ 367.993497] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993504] 1 lock held by iou-wrk-3392/4000:
[ 367.993506] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993512] 1 lock held by iou-wrk-3392/4019:
[ 367.993514] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993521] 1 lock held by iou-wrk-3392/4082:
[ 367.993523] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993529] 1 lock held by iou-wrk-3392/4084:
[ 367.993531] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993538] 1 lock held by iou-wrk-3392/4086:
[ 367.993540] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993546] 1 lock held by iou-wrk-3392/4088:
[ 367.993548] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993555] 1 lock held by iou-wrk-3392/4090:
[ 367.993557] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993563] 1 lock held by iou-wrk-3392/4092:
[ 367.993565] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993572] 1 lock held by iou-wrk-3392/4094:
[ 367.993574] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993580] 1 lock held by iou-wrk-3392/4096:
[ 367.993582] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993589] 1 lock held by iou-wrk-3392/4098:
[ 367.993591] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993597] 1 lock held by iou-wrk-3392/4100:
[ 367.993599] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993606] 1 lock held by iou-wrk-3392/4102:
[ 367.993608] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993614] 1 lock held by iou-wrk-3392/4104:
[ 367.993616] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993623] 1 lock held by iou-wrk-3392/4106:
[ 367.993625] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993631] 1 lock held by iou-wrk-3392/4108:
[ 367.993633] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993640] 1 lock held by iou-wrk-3392/4110:
[ 367.993642] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993648] 1 lock held by iou-wrk-3392/4112:
[ 367.993650] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993657] 1 lock held by iou-wrk-3392/4114:
[ 367.993659] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993665] 1 lock held by iou-wrk-3392/4116:
[ 367.993667] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993674] 1 lock held by iou-wrk-3392/4118:
[ 367.993676] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993682] 1 lock held by iou-wrk-3392/4120:
[ 367.993684] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993691] 1 lock held by iou-wrk-3392/4121:
[ 367.993693] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993699] 1 lock held by iou-wrk-3392/4123:
[ 367.993701] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993708] 1 lock held by iou-wrk-3392/4125:
[ 367.993710] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993716] 1 lock held by iou-wrk-3392/4127:
[ 367.993718] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993725] 1 lock held by iou-wrk-3392/4129:
[ 367.993727] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993733] 1 lock held by iou-wrk-3392/4131:
[ 367.993735] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993742] 1 lock held by iou-wrk-3392/4133:
[ 367.993743] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993750] 1 lock held by iou-wrk-3392/4135:
[ 367.993752] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993759] 1 lock held by iou-wrk-3392/4137:
[ 367.993760] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993767] 1 lock held by iou-wrk-3392/4140:
[ 367.993769] #0: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_file_write_iter+0x435/0xa90
[ 367.993780] 2 locks held by postgres/3409:
[ 367.993801] #0: ffff888144c3e3f8 (sb_writers#7){.+.+}-{0:0}, at: __x64_sys_fallocate+0x42/0x70
[ 367.993809] #1: ffff888466a00d70 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: ext4_fallocate+0x110/0x1140
[ 367.993818] 2 locks held by less/3875:
[ 367.993820] #0: ffff888142f518a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x50
[ 367.993828] #1: ffffc900022ea2f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x51b/0x650
[ 367.993838] =============================================
Translating the various lock acquisition locations:
addr2line -p -i -e /home/andres/src/kernel/vmlinux ext4_file_write_iter+0x435/0xa90
/home/andres/src/kernel/./include/linux/fs.h:1073
(inlined by) /home/andres/src/kernel/fs/ext4/file.c:57
(inlined by) /home/andres/src/kernel/fs/ext4/file.c:564
(inlined by) /home/andres/src/kernel/fs/ext4/file.c:715
addr2line -p -i -e /home/andres/src/kernel/vmlinux __x64_sys_fallocate+0x42/0x70
/home/andres/src/kernel/./include/linux/file.h:45
(inlined by) /home/andres/src/kernel/fs/open.c:348
(inlined by) /home/andres/src/kernel/fs/open.c:355
(inlined by) /home/andres/src/kernel/fs/open.c:353
(inlined by) /home/andres/src/kernel/fs/open.c:353
addr2line -p -i -e /home/andres/src/kernel/vmlinux ext4_fallocate+0x110/0x1140
/home/andres/src/kernel/./arch/x86/include/asm/bitops.h:207
(inlined by) /home/andres/src/kernel/./arch/x86/include/asm/bitops.h:239
(inlined by) /home/andres/src/kernel/./include/asm-generic/bitops/instrumented-non-atomic.h:142
(inlined by) /home/andres/src/kernel/fs/ext4/ext4.h:1922
(inlined by) /home/andres/src/kernel/fs/ext4/extents.c:4735
I think the addresses that lockdep computes must be off a bit - it sure looks
like instead of the address that ext4_fallocate+0x110 maps to, the acquisition
instead was the preceding line, i.e. the inode_lock().
fs/ext4/extents.c:
4727 max_blocks = EXT4_MAX_BLOCKS(len, offset, blkbits);
0xffffffff815d9de0 <ext4_fallocate+256>: and %rdx,%rax
0xffffffff815d9de3 <ext4_fallocate+259>: sar %cl,%rax
0xffffffff815d9de6 <ext4_fallocate+262>: sub %esi,%eax
0xffffffff815d9de8 <ext4_fallocate+264>: mov %eax,-0x60(%rbp)
./include/linux/fs.h:
802 down_write(&inode->i_rwsem);
0xffffffff815d9deb <ext4_fallocate+267>: call 0xffffffff8276c0c0 <down_write>
./arch/x86/include/asm/bitops.h:
206 return ((1UL << (nr & (BITS_PER_LONG-1))) &
0xffffffff815d9df0 <ext4_fallocate+272>: mov -0x260(%rbx),%rax
fs/ext4/extents.c:
4735 if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS))) {
Looking at the code, I assume the reason lockdep doesn't fire is that it
doesn't understand wait edges via __inode_dio_wait().
Greetings,
Andres Freund
Powered by blists - more mailing lists