linux-kernel - Re: [PATCH 1/1] quota: flush quota_release

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <ZzjdggicyuGqaVs8@li-bb2b2a4c-3307-11b2-a85c-8fa5c3a69313.ibm.com>
Date: Sat, 16 Nov 2024 23:29:30 +0530
From: Ojaswin Mujoo <ojaswin@...ux.ibm.com>
To: Ritesh Harjani <ritesh.list@...il.com>
Cc: linux-ext4@...r.kernel.org, Jan Kara <jack@...e.com>,
        linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org,
        Disha Goel <disgoel@...ux.ibm.com>, Baokun Li <libaokun1@...wei.com>
Subject: Re: [PATCH 1/1] quota: flush quota_release_work upon quota writeback

On Sat, Nov 16, 2024 at 02:20:26AM +0530, Ritesh Harjani wrote:
> Ojaswin Mujoo <ojaswin@...ux.ibm.com> writes:
> 
> > One of the paths quota writeback is called from is:
> >
> > freeze_super()
> >   sync_filesystem()
> >     ext4_sync_fs()
> >       dquot_writeback_dquots()
> >
> > Since we currently don't always flush the quota_release_work queue in
> > this path, we can end up with the following race:
> >
> >  1. dquot are added to releasing_dquots list during regular operations.
> >  2. FS freeze starts, however, this does not flush the quota_release_work queue.
> >  3. Freeze completes.
> >  4. Kernel eventually tries to flush the workqueue while FS is frozen which
> >     hits a WARN_ON since transaction gets started during frozen state:
> >
> >   ext4_journal_check_start+0x28/0x110 [ext4] (unreliable)
> >   __ext4_journal_start_sb+0x64/0x1c0 [ext4]
> >   ext4_release_dquot+0x90/0x1d0 [ext4]
> >   quota_release_workfn+0x43c/0x4d0
> >
> > Which is the following line:
> >
> >   WARN_ON(sb->s_writers.frozen == SB_FREEZE_COMPLETE);
> >
> > Which ultimately results in generic/390 failing due to dmesg
> > noise. This was detected on powerpc machine 15 cores.
> >
> > To avoid this, make sure to flush the workqueue during
> > dquot_writeback_dquots() so we dont have any pending workitems after
> > freeze.
> 
> Not just that, sync_filesystem can also be called from other places and
> quota_release_workfn() could write out and and release the dquot
> structures if such are found during processing of releasing_dquots list. 
> IIUC, this was earlier done in the same dqput() context but had races
> with dquot_mark_dquot_dirty(). Hence the final dqput() will now add the
> dquot structures to releasing_dquots list and will schedule a delayed
> workfn which will process the releasing_dquots list. 
Hi Ritesh,

Ohh right, thanks for the context. I see this was done here:

  dabc8b207566 quota: fix dqput() to follow the guarantees dquot_srcu
  should provide

Which went in v6.5. Let me cc Baokun as well.
> 
> And so after the final dqput and before the release_workfn gets
> scheduled, if dquot gets marked as dirty or dquot_transfer gets called -
> then I am suspecting that it could lead to a dirty or an active dquot.
> 
> Hence, flushing the delayed quota_release_work at the end of
> dquot_writeback_dquots() looks like the right thing to do IMO.
> 
> But I can give another look as this part of the code is not that well
> known to me. 
> 
> >
> > Reported-by: Disha Goel <disgoel@...ux.ibm.com>
> > Signed-off-by: Ojaswin Mujoo <ojaswin@...ux.ibm.com>
> > ---
> 
> Maybe a fixes tag as well?

Right, I'll add that in the next revision. I believe it would be:

Fixes: dabc8b207566 ("quota: fix dqput() to follow the guarantees dquot_srcu should provide")

Regards,
ojaswin

> 
> >  fs/quota/dquot.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c
> > index 3dd8d6f27725..2782cfc8c302 100644
> > --- a/fs/quota/dquot.c
> > +++ b/fs/quota/dquot.c
> > @@ -729,6 +729,8 @@ int dquot_writeback_dquots(struct super_block *sb, int type)
> >  			sb->dq_op->write_info(sb, cnt);
> >  	dqstats_inc(DQST_SYNCS);
> >  
> > +	flush_delayed_work(&quota_release_work);
> > +
> >  	return ret;
> >  }
> >  EXPORT_SYMBOL(dquot_writeback_dquots);
> > -- 
> > 2.43.5