| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Mon, 11 Nov 2019 12:14:41 +0530
From: Sahitya Tummala <stummala@...eaurora.org>
To: Chao Yu <yuchao0@...wei.com>
Cc: Jaegeuk Kim <jaegeuk@...nel.org>,
linux-f2fs-devel@...ts.sourceforge.net,
linux-kernel@...r.kernel.org
Subject: Re: [PATCH] f2fs: Fix deadlock under storage almost full/dirty
condition
Hi Chao,
On Mon, Nov 11, 2019 at 02:28:47PM +0800, Chao Yu wrote:
> Hi Sahitya,
>
> On 2019/11/11 11:40, Sahitya Tummala wrote:
> > Hi Chao,
> >
> > On Mon, Nov 11, 2019 at 10:51:10AM +0800, Chao Yu wrote:
> >> On 2019/11/8 19:03, Sahitya Tummala wrote:
> >>> There could be a potential deadlock when the storage capacity
> >>> is almost full and theren't enough free segments available, due
> >>> to which FG_GC is needed in the atomic commit ioctl as shown in
> >>> the below callstack -
> >>>
> >>> schedule_timeout
> >>> io_schedule_timeout
> >>> congestion_wait
> >>> f2fs_drop_inmem_pages_all
> >>> f2fs_gc
> >>> f2fs_balance_fs
> >>> __write_node_page
> >>> f2fs_fsync_node_pages
> >>> f2fs_do_sync_file
> >>> f2fs_ioctl
> >>>
> >>> If this inode doesn't have i_gc_failures[GC_FAILURE_ATOMIC] set,
> >>> then it waits forever in f2fs_drop_inmem_pages_all(), for this
> >>> atomic inode to be dropped. And the rest of the system is stuck
> >>> waiting for sbi->gc_mutex lock, which is acquired by f2fs_balance_fs()
> >>> in the stack above.
> >>
> >> I think the root cause of this issue is there is potential infinite loop in
> >> f2fs_drop_inmem_pages_all() for the case of gc_failure is true, because once the
> >> first inode in inode_list[ATOMIC_FILE] list didn't suffer gc failure, we will
> >> skip dropping its in-memory cache and calling iput(), and traverse the list
> >> again, most possibly there is the same inode in the head of that list.
> >>
> >
> > I thought we are expecting for those atomic updates (without any gc failures) to be
> > committed by doing congestion_wait() and thus retrying again. Hence, I just
>
> Nope, we only need to drop inode which encounter gc failures, and keep the rest
> inodes.
>
> > fixed only if we are ending up waiting for commit to happen in the atomic
> > commit path itself, which will be a deadlock.
>
> Look into call stack you provide, I don't think it's correct to drop such inode,
> as its dirty pages should be committed before f2fs_fsync_node_pages(), so
> calling f2fs_drop_inmem_pages won't release any inmem pages, and won't help
> looped GC caused by skipping due to inmem pages.
>
> And then I figure out below fix...
>
Thanks for the explanation.
The fix below looks good to me.
Thanks,
Sahitya.
> >
> >> Could you please check below fix:
> >>
> >> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> >> index 7bf7b0194944..8a3a35b42a37 100644
> >> --- a/fs/f2fs/f2fs.h
> >> +++ b/fs/f2fs/f2fs.h
> >> @@ -1395,6 +1395,7 @@ struct f2fs_sb_info {
> >> unsigned int gc_mode; /* current GC state */
> >> unsigned int next_victim_seg[2]; /* next segment in victim section */
> >> /* for skip statistic */
> >> + unsigned int atomic_files; /* # of opened atomic file */
> >> unsigned long long skipped_atomic_files[2]; /* FG_GC and BG_GC */
> >> unsigned long long skipped_gc_rwsem; /* FG_GC only */
> >>
> >> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> >> index ecd063239642..79f4b348951a 100644
> >> --- a/fs/f2fs/file.c
> >> +++ b/fs/f2fs/file.c
> >> @@ -2047,6 +2047,7 @@ static int f2fs_ioc_start_atomic_write(struct file *filp)
> >> spin_lock(&sbi->inode_lock[ATOMIC_FILE]);
> >> if (list_empty(&fi->inmem_ilist))
> >> list_add_tail(&fi->inmem_ilist, &sbi->inode_list[ATOMIC_FILE]);
> >> + sbi->atomic_files++;
> >> spin_unlock(&sbi->inode_lock[ATOMIC_FILE]);
> >>
> >> /* add inode in inmem_list first and set atomic_file */
> >> diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
> >> index 8b977bbd6822..6aa0bb693697 100644
> >> --- a/fs/f2fs/segment.c
> >> +++ b/fs/f2fs/segment.c
> >> @@ -288,6 +288,8 @@ void f2fs_drop_inmem_pages_all(struct f2fs_sb_info *sbi,
> >> bool gc_failure)
> >> struct list_head *head = &sbi->inode_list[ATOMIC_FILE];
> >> struct inode *inode;
> >> struct f2fs_inode_info *fi;
> >> + unsigned int count = sbi->atomic_files;
> >
> > If the sbi->atomic_files decrements just after this, then the below exit condition
> > may not work. In that case, looped will never be >= count.
> >
> >> + unsigned int looped = 0;
> >> next:
> >> spin_lock(&sbi->inode_lock[ATOMIC_FILE]);
> >> if (list_empty(head)) {
> >> @@ -296,22 +298,29 @@ void f2fs_drop_inmem_pages_all(struct f2fs_sb_info *sbi,
> >> bool gc_failure)
> >> }
> >> fi = list_first_entry(head, struct f2fs_inode_info, inmem_ilist);
> >> inode = igrab(&fi->vfs_inode);
> >> + if (inode)
> >> + list_move_tail(&fi->inmem_ilist, head);
> >> spin_unlock(&sbi->inode_lock[ATOMIC_FILE]);
> >>
> >> if (inode) {
> >> if (gc_failure) {
> >> - if (fi->i_gc_failures[GC_FAILURE_ATOMIC])
> >> - goto drop;
> >> - goto skip;
> >> + if (!fi->i_gc_failures[GC_FAILURE_ATOMIC])
> >> + goto skip;
> >> }
> >> -drop:
> >> set_inode_flag(inode, FI_ATOMIC_REVOKE_REQUEST);
> >> f2fs_drop_inmem_pages(inode);
> >> +skip:
> >> iput(inode);
> >
> > Does this result into f2fs_evict_inode() in this context for this inode?
>
> Yup, we need to call igrab/iput in pair in f2fs_drop_inmem_pages_all() anyway.
>
> Previously, we may have .i_count leak...
>
> Thanks,
>
> >
> > thanks,
> >
> >> }
> >> -skip:
> >> +
> >> congestion_wait(BLK_RW_ASYNC, HZ/50);
> >> cond_resched();
> >> +
> >> + if (gc_failure) {
> >> + if (++looped >= count)
> >> + return;
> >> + }
> >> +
> >> goto next;
> >> }
> >>
> >> @@ -334,6 +343,7 @@ void f2fs_drop_inmem_pages(struct inode *inode)
> >> spin_lock(&sbi->inode_lock[ATOMIC_FILE]);
> >> if (!list_empty(&fi->inmem_ilist))
> >> list_del_init(&fi->inmem_ilist);
> >> + sbi->atomic_files--;
> >> spin_unlock(&sbi->inode_lock[ATOMIC_FILE]);
> >> }
> >>
> >> Thanks,
> >
--
--
Sent by a consultant of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Powered by blists - more mailing lists