| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20170724083515.GA652@quack2.suse.cz>
Date: Mon, 24 Jul 2017 10:35:15 +0200
From: Jan Kara <jack@...e.cz>
To: Jerry Lee <leisurelysw24@...il.com>
Cc: Jan Kara <jack@...e.cz>, linux-ext4@...r.kernel.org,
Theodore Ts'o <tytso@....edu>, jack@...e.com
Subject: Re: lockdep detects possible deadlock scenario
On Fri 21-07-17 09:41:54, Jerry Lee wrote:
> On 20 July 2017 at 20:16, Jan Kara <jack@...e.cz> wrote:
> > Hi!
> >
> > On Thu 20-07-17 19:29:28, Jerry Lee wrote:
> >> I hit the following lockdep trace on linux-4.2.8 and I could steadily
> >> re-produce it on some of my machine. Although the trace shows up, the
> >> file system works quite well without seeing any operations being stuck
> >> on it. Does it mean that the trace is just a false alarm? Thanks.
> >>
> >> BTW, I've saw some similar traces previously in the mailing list and
> >> found that the patch, "ext4: add lockdep annotations for i_data_sem
> >> (daf647d2dd58)", which is already included in my kernel.
> >
> > I don't think that patch is included in the kernel reporting this trace -
> > from the trace ei->i_data_sem obtained on quota file (the first stack
> > trace) did not use the special I_DATA_SEM_QUOTA locking class which commit
> > daf647d2dd58 introduced and it should have... In either case this report
> > is a false positive.
> >
> > Honza
>
> You are right that the patch is not included in the linux-4.2.8 kernel
> on the mainstream. I'm sorry that I didn't clearly describe my setup
> in previous post. Before I sent the mail, I found the patch and
> back-ported it to my kernel to get rid of possible false positive.
> But, with the patch, I still got the trace. Does it mean that I miss
> some other patches when directly back-porting the patch on my kernel?
Well, I'm not sure. Was it the same trace? There's followup fix for commit
daf647d2dd58 - commit 964edf66bf9ab "ext4: clear lockdep subtype for quota
files on quota off" so you may miss that one.
Honza
> Thanks for your quick reply.
>
> >
> >>
> >> ======================================================
> >> <4>[ 205.633705] [ INFO: possible circular locking dependency detected ]
> >> <4>[ 205.639962] 4.2.8 #3 Tainted: G W O
> >> <4>[ 205.644395] -------------------------------------------------------
> >> <4>[ 205.650650] rm/19302 is trying to acquire lock:
> >> <4>[ 205.655174] (&s->s_dquot.dqio_mutex){+.+...}, at:
> >> [<ffffffff81250678>] dquot_commit+0x28/0xc0
> >> <4>[ 205.663835]
> >> <4>[ 205.663835] but task is already holding lock:
> >> <4>[ 205.669659] (&ei->i_data_sem){++++..}, at: [<ffffffff812b2329>]
> >> ext4_truncate+0x379/0x680
> >> <4>[ 205.677960]
> >> <4>[ 205.677960] which lock already depends on the new lock.
> >> <4>[ 205.677960]
> >> <4>[ 205.686119]
> >> <4>[ 205.686119] the existing dependency chain (in reverse order) is:
> >> <4>[ 205.693586]
> >> <4>[ 205.693586] -> #1 (&ei->i_data_sem){++++..}:
> >> <4>[ 205.698071] [<ffffffff810cde05>] lock_acquire+0xd5/0x280
> >> <4>[ 205.703995] [<ffffffff81c82727>] down_read+0x47/0x60
> >> <4>[ 205.709573] [<ffffffff812acefb>] ext4_map_blocks+0x48b/0x5f0
> >> <4>[ 205.715845] [<ffffffff812ad653>] ext4_getblk+0x43/0x190
> >> <4>[ 205.721682] [<ffffffff812ad7ae>] ext4_bread+0xe/0xa0
> >> <4>[ 205.727260] [<ffffffff812c225d>] ext4_quota_read+0xcd/0x110
> >> <4>[ 205.733442] [<ffffffff81254687>] read_blk+0x47/0x50
> >> <4>[ 205.738933] [<ffffffff81255452>] find_tree_dqentry+0x42/0x230
> >> <4>[ 205.745290] [<ffffffff812555b4>] find_tree_dqentry+0x1a4/0x230
> >> <4>[ 205.751730] [<ffffffff812555b4>] find_tree_dqentry+0x1a4/0x230
> >> <4>[ 205.758172] [<ffffffff812555b4>] find_tree_dqentry+0x1a4/0x230
> >> <4>[ 205.764612] [<ffffffff81255773>] qtree_read_dquot+0x133/0x260
> >> <4>[ 205.770968] [<ffffffff81253d89>] v2_read_dquot+0x29/0x30
> >> <4>[ 205.776892] [<ffffffff8124f7f6>] dquot_acquire+0xe6/0x130
> >> <4>[ 205.782902] [<ffffffff812c1c9a>] ext4_acquire_dquot+0x6a/0xb0
> >> <4>[ 205.789258] [<ffffffff81251970>] dqget+0x3c0/0x420
> >> <4>[ 205.794662] [<ffffffff81251afd>] __dquot_initialize+0x12d/0x230
> >> <4>[ 205.801187] [<ffffffff81251c0e>] dquot_initialize+0xe/0x10
> >> <4>[ 205.807283] [<ffffffff812d215b>] ext4_fill_super+0x2d9b/0x3150
> >> <4>[ 205.813729] [<ffffffff811e0f00>] mount_bdev+0x180/0x1b0
> >> <4>[ 205.819566] [<ffffffff812c18c0>] ext4_mount+0x10/0x20
> >> <4>[ 205.825230] [<ffffffff811e17a4>] mount_fs+0x14/0xa0
> >> <4>[ 205.830719] [<ffffffff81203736>] vfs_kern_mount+0x66/0x150
> >> <4>[ 205.836815] [<ffffffff812066a5>] do_mount+0x1e5/0xd00
> >> <4>[ 205.842476] [<ffffffff812074b6>] SyS_mount+0x86/0xc0
> >> <4>[ 205.848048] [<ffffffff81c84bd7>]
> >> entry_SYSCALL_64_fastpath+0x12/0x6f
> >> <4>[ 205.855010]
> >> <4>[ 205.855010] -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
> >> <4>[ 205.860103] [<ffffffff810ccccc>] __lock_acquire+0x1fdc/0x23a0
> >> <4>[ 205.866464] [<ffffffff810cde05>] lock_acquire+0xd5/0x280
> >> <4>[ 205.872393] [<ffffffff81c80990>] mutex_lock_nested+0x60/0x370
> >> <4>[ 205.878753] [<ffffffff81250678>] dquot_commit+0x28/0xc0
> >> <4>[ 205.884594] [<ffffffff812c1d4e>] ext4_write_dquot+0x6e/0xa0
> >> <4>[ 205.890783] [<ffffffff812c1dbe>] ext4_mark_dquot_dirty+0x3e/0x60
> >> <4>[ 205.897408] [<ffffffff81250857>] __dquot_free_space+0x147/0x310
> >> <4>[ 205.903943] [<ffffffff812ebe0d>] ext4_free_blocks+0x77d/0x1010
> >> <4>[ 205.910390] [<ffffffff812dbb66>] ext4_ext_remove_space+0x8f6/0x16a0
> >> <4>[ 205.917271] [<ffffffff812de49f>] ext4_ext_truncate+0xaf/0xe0
> >> <4>[ 205.923547] [<ffffffff812b23f0>] ext4_truncate+0x440/0x680
> >> <4>[ 205.929651] [<ffffffff812b2a9f>] ext4_evict_inode+0x46f/0x730
> >> <4>[ 205.936017] [<ffffffff811fe473>] evict+0xb3/0x180
> >> <4>[ 205.941337] [<ffffffff811fecc7>] iput+0x187/0x350
> >> <4>[ 205.946662] [<ffffffff811f0243>] do_unlinkat+0x163/0x340
> >> <4>[ 205.952588] [<ffffffff811f0481>] SyS_unlink+0x11/0x20
> >> <4>[ 205.958258] [<ffffffff81c84bd7>]
> >> entry_SYSCALL_64_fastpath+0x12/0x6f
> >> <4>[ 205.965226]
> >> <4>[ 205.965226] other info that might help us debug this:
> >> <4>[ 205.965226]
> >> <4>[ 205.973217] Possible unsafe locking scenario:
> >> <4>[ 205.973217]
> >> <4>[ 205.979127] CPU0 CPU1
> >> <4>[ 205.983652] ---- ----
> >> <4>[ 205.988174] lock(&ei->i_data_sem);
> >> <4>[ 205.991772] lock(&s->s_dquot.dqio_mutex);
> >> <4>[ 205.998487] lock(&ei->i_data_sem);
> >> <4>[ 206.004596] lock(&s->s_dquot.dqio_mutex);
> >> <4>[ 206.008795]
> >> <4>[ 206.008795] *** DEADLOCK ***
> >> <4>[ 206.008795]
> >> <4>[ 206.014708] 5 locks held by rm/19302:
> >> <4>[ 206.018364] #0: (sb_writers#10){.+.+.+}, at:
> >> [<ffffffff81204bbf>] mnt_want_write+0x1f/0x50
> >> <4>[ 206.026870] #1: (sb_internal){.+.+..}, at:
> >> [<ffffffff812b27a9>] ext4_evict_inode+0x179/0x730
> >> <4>[ 206.035538] #2: (jbd2_handle){+.+...}, at:
> >> [<ffffffff81306cc1>] start_this_handle+0x191/0x630
> >> <4>[ 206.044298] #3: (&ei->i_data_sem){++++..}, at:
> >> [<ffffffff812b2329>] ext4_truncate+0x379/0x680
> >> <4>[ 206.053052] #4: (dquot_srcu){......}, at: [<ffffffff8125076a>]
> >> __dquot_free_space+0x5a/0x310
> >> <4>[ 206.061719]
> >> <4>[ 206.061719] stack backtrace:
> >> <4>[ 206.066072] CPU: 0 PID: 19302 Comm: rm Tainted: G W O 4.2.8 #3
> >> <4>[ 206.072848] Hardware name: To be filled by O.E.M. To be filled
> >> by O.E.M./MAHOBAY, BIOS QC30AR23 08/14/2014
> >> <4>[ 206.082486] ffffffff82effc40 ffff88004cebf7d8 ffffffff81c767eb
> >> 0000000000000007
> >> <4>[ 206.089919] ffffffff82effc40 ffff88004cebf828 ffffffff81c739cd
> >> ffff880044724e08
> >> <4>[ 206.097364] ffff88004cebf898 ffff88004cebf828 0000000000000005
> >> ffff880044724640
> >> <4>[ 206.104807] Call Trace:
> >> <4>[ 206.107257] [<ffffffff81c767eb>] dump_stack+0x4c/0x65
> >> <4>[ 206.112393] [<ffffffff81c739cd>] print_circular_bug+0x202/0x213
> >> <4>[ 206.118394] [<ffffffff810ccccc>] __lock_acquire+0x1fdc/0x23a0
> >> <4>[ 206.124222] [<ffffffff810cde05>] lock_acquire+0xd5/0x280
> >> <4>[ 206.129616] [<ffffffff81250678>] ? dquot_commit+0x28/0xc0
> >> <4>[ 206.135099] [<ffffffff81c80990>] mutex_lock_nested+0x60/0x370
> >> <4>[ 206.140931] [<ffffffff81250678>] ? dquot_commit+0x28/0xc0
> >> <4>[ 206.146414] [<ffffffff812c1d3a>] ? ext4_write_dquot+0x5a/0xa0
> >> <4>[ 206.152245] [<ffffffff8130784a>] ? jbd2__journal_start+0x1a/0x20
> >> <4>[ 206.158766] [<ffffffff81250678>] dquot_commit+0x28/0xc0
> >> <4>[ 206.164074] [<ffffffff812c1d4e>] ext4_write_dquot+0x6e/0xa0
> >> <4>[ 206.169731] [<ffffffff812c1dbe>] ext4_mark_dquot_dirty+0x3e/0x60
> >> <4>[ 206.175821] [<ffffffff81250857>] __dquot_free_space+0x147/0x310
> >> <4>[ 206.181825] [<ffffffff8125076a>] ? __dquot_free_space+0x5a/0x310
> >> <4>[ 206.187917] [<ffffffff812ebc62>] ? ext4_free_blocks+0x5d2/0x1010
> >> <4>[ 206.194005] [<ffffffff812ebe0d>] ext4_free_blocks+0x77d/0x1010
> >> <4>[ 206.199920] [<ffffffff810ca7e1>] ? mark_held_locks+0x71/0x90
> >> <4>[ 206.205662] [<ffffffff811cadc6>] ? __kmalloc+0xa6/0x5d0
> >> <4>[ 206.210972] [<ffffffff810c8f4d>] ? __lock_is_held+0x4d/0x70
> >> <4>[ 206.216627] [<ffffffff812db2ca>] ? ext4_ext_remove_space+0x5a/0x16a0
> >> <4>[ 206.223061] [<ffffffff812dbb66>] ext4_ext_remove_space+0x8f6/0x16a0
> >> <4>[ 206.229412] [<ffffffff812de49f>] ext4_ext_truncate+0xaf/0xe0
> >> <4>[ 206.235157] [<ffffffff812b23f0>] ext4_truncate+0x440/0x680
> >> <4>[ 206.240723] [<ffffffff812b2a9f>] ext4_evict_inode+0x46f/0x730
> >> <4>[ 206.246551] [<ffffffff811fe473>] evict+0xb3/0x180
> >> <4>[ 206.251339] [<ffffffff811fecc7>] iput+0x187/0x350
> >> <4>[ 206.256129] [<ffffffff811f0243>] do_unlinkat+0x163/0x340
> >> <4>[ 206.261525] [<ffffffff812041c0>] ? mnt_get_count+0x60/0x60
> >> <4>[ 206.267092] [<ffffffff81002044>] ? lockdep_sys_exit_thunk+0x12/0x14
> >> <4>[ 206.273441] [<ffffffff811f0481>] SyS_unlink+0x11/0x20
> >> <4>[ 206.278578] [<ffffffff81c84bd7>] entry_SYSCALL_64_fastpath+0x12/0x6f
> >>
> > --
> > Jan Kara <jack@...e.com>
> > SUSE Labs, CR
--
Jan Kara <jack@...e.com>
SUSE Labs, CR
Powered by blists - more mailing lists