linux-kernel - Re: [PATCH] iomap: Fix the write_count in iomap_add_to

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200824154841.GB295033@bfoster>
Date:   Mon, 24 Aug 2020 11:48:41 -0400
From:   Brian Foster <bfoster@...hat.com>
To:     Christoph Hellwig <hch@...radead.org>
Cc:     Dave Chinner <david@...morbit.com>,
        Ritesh Harjani <riteshh@...ux.ibm.com>,
        Anju T Sudhakar <anju@...ux.vnet.ibm.com>,
        darrick.wong@...cle.com, linux-xfs@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        willy@...radead.org
Subject: Re: [PATCH] iomap: Fix the write_count in iomap_add_to_ioend().

On Mon, Aug 24, 2020 at 04:04:17PM +0100, Christoph Hellwig wrote:
> On Mon, Aug 24, 2020 at 10:28:23AM -0400, Brian Foster wrote:
> > Do I understand the current code (__bio_try_merge_page() ->
> > page_is_mergeable()) correctly in that we're checking for physical page
> > contiguity and not necessarily requiring a new bio_vec per physical
> > page?
> 
> 
> Yes.
> 

Ok. I also realize now that this occurs on a kernel without commit
07173c3ec276 ("block: enable multipage bvecs"). That is probably a
contributing factor, but it's not clear to me whether it's feasible to
backport whatever supporting infrastructure is required for that
mechanism to work (I suspect not).

> > With regard to Dave's earlier point around seeing excessively sized bio
> > chains.. If I set up a large memory box with high dirty mem ratios and
> > do contiguous buffered overwrites over a 32GB range followed by fsync, I
> > can see upwards of 1GB per bio and thus chains on the order of 32+ bios
> > for the entire write. If I play games with how the buffered overwrite is
> > submitted (i.e., in reverse) however, then I can occasionally reproduce
> > a ~32GB chain of ~32k bios, which I think is what leads to problems in
> > I/O completion on some systems. Granted, I don't reproduce soft lockup
> > issues on my system with that behavior, so perhaps there's more to that
> > particular issue.
> > 
> > Regardless, it seems reasonable to me to at least have a conservative
> > limit on the length of an ioend bio chain. Would anybody object to
> > iomap_ioend growing a chain counter and perhaps forcing into a new ioend
> > if we chain something like more than 1k bios at once?
> 
> So what exactly is the problem of processing a long chain in the
> workqueue vs multiple small chains?  Maybe we need a cond_resched()
> here and there, but I don't see how we'd substantially change behavior.
> 

The immediate problem is a watchdog lockup detection in bio completion:

  NMI watchdog: Watchdog detected hard LOCKUP on cpu 25

This effectively lands at the following segment of iomap_finish_ioend():

		...
               /* walk each page on bio, ending page IO on them */
                bio_for_each_segment_all(bv, bio, iter_all)
                        iomap_finish_page_writeback(inode, bv->bv_page, error);

I suppose we could add a cond_resched(), but is that safe directly
inside of a ->bi_end_io() handler? Another option could be to dump large
chains into the completion workqueue, but we may still need to track the
length to do that. Thoughts?

Brian