linux-ext4 - Re: jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20210712090459.GA27936@quack2.suse.cz>
Date:   Mon, 12 Jul 2021 11:04:59 +0200
From:   Jan Kara <jack@...e.cz>
To:     Ivan Zahariev <famzah@...zah.net>
Cc:     Theodore Ts'o <tytso@....edu>, linux-ext4@...r.kernel.org
Subject: Re: jbd2: fix deadlock while checkpoint thread waits commit thread
 to finish (backport to 4.14)

On Thu 08-07-21 06:45:35, Ivan Zahariev wrote:
> Out of thousand machines, one would trigger the problem about every 1 to 10
> days. Some machines trigger the problem much often than others. So I can say
> that we have a way to verify quickly if applying the patch will fix this for
> us.
> 
> The most important question is: Is it safe to apply the patch on production
> machines with kernel 4.14?
> 
> We can't risk data loss. And I lack the expertise to asses what risks this
> small patch brings.

The fix should work correctly even for older kernels. I'm not aware of any
changes in this area in the past that could conflict...

								Honza

> On 8.7.2021 г. 2:52, Theodore Ts'o wrote:
> > On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
> > > Hello,
> > > 
> > > We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
> > > which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
> > > 
> > > This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
> > > applies cleanly. The two files "fs/jbd2/checkpoint.c" and
> > > "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
> > > to kernel 5.4 where we have this bugfix already applied.
> > > 
> > > Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
> > > that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
> > > system in 4.14 and 5.4 fundamentally different and this would lead to data
> > > loss or other problems?
> > The commit was over two years ago, so my memory is not going to be
> > perfect.  However, Jan had made a comment suggesting the approach in
> > this commit because it should be easier to backport into older stble
> > kernels[1].
> > 
> >     "Since proper locking change is going to be a bit more involved, can you
> >      perhaps fix this deadlock by just dropping j_checkpoint_mutex in
> >      log_do_checkpoint() when we are going to wait for transaction commit. I've
> >      checked and that should be fine and that is going to be much easier change
> >      to backport into stable kernels..."
> > 
> > [1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
> > 
> > So I suspect it was just that I failed to remember to add a "Cc:
> > stable@...nel.org" and so it was never automatically backported into
> > 4.14 or 4.19.
> > 
> > Do you have a reliable reproduction which is triggering the deadlock
> > on your kernels?  If so, have you tried applying the patch and does it
> > make the problem go away for you?
> > 
> > Cheers,
> > 
> > 						- Ted
-- 
Jan Kara <jack@...e.com>
SUSE Labs, CR