| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAKYAXd8-sZo0XcdHuyOQ1qT_s3kJXyphXsjSS7e1-sJ1QaAOgg@mail.gmail.com> Date: Thu, 3 Jan 2013 13:35:39 +0900 From: Namjae Jeon <linkinjeon@...il.com> To: Jan Kara <jack@...e.cz> Cc: Wanpeng Li <liwanp@...ux.vnet.ibm.com>, fengguang.wu@...el.com, linux-fsdevel@...r.kernel.org, linux-mm@...ck.org, linux-kernel@...r.kernel.org, Namjae Jeon <namjae.jeon@...sung.com>, Vivek Trivedi <t.vivek@...sung.com>, Dave Chinner <dchinner@...hat.com> Subject: Re: [PATCH] writeback: fix writeback cache thrashing 2013/1/2, Jan Kara <jack@...e.cz>: > On Tue 01-01-13 08:51:04, Wanpeng Li wrote: >> On Mon, Dec 31, 2012 at 12:30:54PM +0100, Jan Kara wrote: >> >On Sun 30-12-12 14:59:50, Namjae Jeon wrote: >> >> From: Namjae Jeon <namjae.jeon@...sung.com> >> >> >> >> Consider Process A: huge I/O on sda >> >> doing heavy write operation - dirty memory becomes more >> >> than dirty_background_ratio >> >> on HDD - flusher thread flush-8:0 >> >> >> >> Consider Process B: small I/O on sdb >> >> doing while [1]; read 1024K + rewrite 1024K + sleep 2sec >> >> on Flash device - flusher thread flush-8:16 >> >> >> >> As Process A is a heavy dirtier, dirty memory becomes more >> >> than dirty_background_thresh. Due to this, below check becomes >> >> true(checking global_page_state in over_bground_thresh) >> >> for all bdi devices(even for very small dirtied bdi - sdb): >> >> >> >> In this case, even small cached data on 'sdb' is forced to flush >> >> and writeback cache thrashing happens. >> >> >> >> When we added debug prints inside above 'if' condition and ran >> >> above Process A(heavy dirtier on bdi with flush-8:0) and >> >> Process B(1024K frequent read/rewrite on bdi with flush-8:16) >> >> we got below prints: >> >> >> >> [Test setup: ARM dual core CPU, 512 MB RAM] >> >> >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84720 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94720 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 384 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 960 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92160 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 768 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 320 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92032 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91968 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 1024 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 576 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84352 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 512 KB >> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92608 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92544 KB >> >> >> >> As mentioned in above log, when global dirty memory > global >> >> background_thresh >> >> small cached data is also forced to flush by flush-8:16. >> >> >> >> If removing global background_thresh checking code, we can reduce >> >> cache >> >> thrashing of frequently used small data. >> > It's not completely clear to me: >> > Why is this a problem? Wearing of the flash? Power consumption? I'd >> > like >> >to understand this before changing the code... Hi Jan. Yes, it can reduce wearing and fragmentation of flash. And also from one scenario - we think it might reduce power consumption also. >> > >> >> And It will be great if we can reserve a portion of writeback cache >> >> using >> >> min_ratio. >> >> >> >> After applying patch: >> >> $ echo 5 > /sys/block/sdb/bdi/min_ratio >> >> $ cat /sys/block/sdb/bdi/min_ratio >> >> 5 >> >> >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84160 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 96960 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94080 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91520 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 89600 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 72960 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB >> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90688 KB >> >> >> >> As mentioned in the above logs, once cache is reserved for Process B, >> >> and patch is applied there is less writeback cache thrashing on sdb >> >> by frequent forced writeback by flush-8:16 in over_bground_thresh. >> >> >> >> After all, small cached data will be flushed by periodic writeback >> >> once every dirty_writeback_interval. >> > OK, in principle something like this makes sence to me. But if there >> > are >> >more BDIs which are roughly equally used, it could happen none of them >> > are >> >over threshold due to percpu counter & rounding errors. So I'd rather >> >change the conditions to something like: >> > reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE); >> > bdi_bground_thresh = bdi_dirty_limit(bdi, background_thresh); >> > >> > if (reclaimable > bdi_bground_thresh) >> > return true; >> > /* >> > * If global background limit is exceeded, kick the writeback on >> > * BDI if there's a reasonable amount of data to write (at least >> > * 1/2 of BDI's background dirty limit). >> > */ >> > if (global_page_state(NR_FILE_DIRTY) + >> > global_page_state(NR_UNSTABLE_NFS) > background_thresh && >> > reclaimable * 2 > bdi_bground_thresh) >> > return true; >> > >> >> Hi Jan, >> >> If there are enough BDIs and percpu counter of each bdi roughly equally >> used less than 1/2 of BDI's background dirty limit, still nothing will >> be flushed even if over global background_thresh. > Yes, although then the percpu counter error would have to be quite big. > Anyway, we can change the last condition to: > if (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh && > reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh) > > That should be safe and for machines with resonable number of CPUs it > should save the wakeup as well. I agree and will send v2 patch as your suggestion. Thanks Jan. > > Honza > >> >> Suggested-by: Wanpeng Li <liwanp@...ux.vnet.ibm.com> >> >> Signed-off-by: Namjae Jeon <namjae.jeon@...sung.com> >> >> Signed-off-by: Vivek Trivedi <t.vivek@...sung.com> >> >> Cc: Fengguang Wu <fengguang.wu@...el.com> >> >> Cc: Jan Kara <jack@...e.cz> >> >> Cc: Dave Chinner <dchinner@...hat.com> >> >> --- >> >> fs/fs-writeback.c | 4 ---- >> >> 1 file changed, 4 deletions(-) >> >> >> >> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c >> >> index 310972b..070b773 100644 >> >> --- a/fs/fs-writeback.c >> >> +++ b/fs/fs-writeback.c >> >> @@ -756,10 +756,6 @@ static bool over_bground_thresh(struct >> >> backing_dev_info *bdi) >> >> >> >> global_dirty_limits(&background_thresh, &dirty_thresh); >> >> >> >> - if (global_page_state(NR_FILE_DIRTY) + >> >> - global_page_state(NR_UNSTABLE_NFS) > background_thresh) >> >> - return true; >> >> - >> >> if (bdi_stat(bdi, BDI_RECLAIMABLE) > >> >> bdi_dirty_limit(bdi, background_thresh)) >> >> return true; >> >> -- >> >> 1.7.9.5 >> >> >> >-- >> >Jan Kara <jack@...e.cz> >> >SUSE Labs, CR >> > >> >-- >> >To unsubscribe, send a message with 'unsubscribe linux-mm' in >> >the body to majordomo@...ck.org. For more info on Linux MM, >> >see: http://www.linux-mm.org/ . >> >Don't email: <a href=mailto:"dont@...ck.org"> email@...ck.org </a> >> > -- > Jan Kara <jack@...e.cz> > SUSE Labs, CR > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists