linux-ext4 - Re: [PATCH] mm, oom: report compaction/migration stats for higher order requests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Mon, 22 Aug 2016 09:02:50 +0200
From:   Michal Hocko <mhocko@...nel.org>
To:     arekm@...en.pl
Cc:     Vlastimil Babka <vbabka@...e.cz>, linux-ext4@...r.kernel.org,
        linux-mm@...ck.org, Jan Kara <jack@...e.cz>,
        Joonsoo Kim <iamjoonsoo.kim@....com>
Subject: Re: [PATCH] mm, oom: report compaction/migration stats for higher
 order requests

On Sun 21-08-16 23:19:50, Arkadiusz Miskiewicz wrote:
> On Friday 19 of August 2016, Vlastimil Babka wrote:
> > On 08/18/2016 08:49 PM, Arkadiusz Miskiewicz wrote:
> > > On Wednesday 17 of August 2016, Michal Hocko wrote:
> > >> On Wed 17-08-16 10:34:54, Arkadiusz Miśkiewicz wrote:
> > >> [...]
> > >> 
> > >>> With "[PATCH] mm, oom: report compaction/migration stats for higher
> > >>> order requests" patch:
> > >>> https://ixion.pld-linux.org/~arekm/p2/ext4/log-20160817.txt
> > >>> 
> > >>> Didn't count much - all counters are 0
> > >>> compaction_stall:0 compaction_fail:0 compact_migrate_scanned:0
> > >>> compact_free_scanned:0 compact_isolated:0 pgmigrate_success:0
> > >>> pgmigrate_fail:0
> > >> 
> > >> Dohh, COMPACTION counters are events and those are different than other
> > >> counters we have. They only have per-cpu representation and so we would
> > >> have to do
> > >> +       for_each_online_cpu(cpu) {
> > >> +               struct vm_event_state *this = &per_cpu(vm_event_states,
> > >> cpu); +               ret += this->event[item];
> > >> +       }
> > >> 
> > >> which is really nasty because, strictly speaking, we would have to do
> > >> {get,put}_online_cpus around that loop and that uses locking and we do
> > >> not want to possibly block in this path just because something is in the
> > >> middle of the hotplug. So let's scratch that patch for now and sorry I
> > >> haven't realized that earlier.
> > >> 
> > >>> two processes were killed by OOM (rm and cp), the rest of rm/cp didn't
> > >>> finish
> > >>> 
> > >>> and I'm interrupting it to try that next patch:
> > >>>> Could you try to test with
> > >>>> patch from
> > >>>> http://lkml.kernel.org/r/20160816031222.GC16913@js1304-P5Q-DELUXE
> > >>>> please? Ideally on top of linux-next. You can add both the compaction
> > >>>> counters patch in the oom report and high order atomic reserves patch
> > >>>> on top.
> > >>> 
> > >>> Uhm, was going to use it on top of 4.7.[01] first.
> > >> 
> > >> OK
> > > 
> > > So with 
> > > http://lkml.kernel.org/r/20160816031222.GC16913@js1304-P5Q-DELUXE OOM no
> > > longer happens (all 10x rm/cp processes finished).
> > 
> > Is it on top of 4.7 then? 
> 
> Yes, it was on top of 4.7.0.
> 
> > That's a bit different from the other reporter
> > who needed both linux-next and this patch to avoid OOM.
> > In any case the proper solution should restrict this disabled heuristic
> > to highest compaction priority, which needs the patches from linux-next
> > anyway.
> > 
> > So can you please also try linux-next with the patch from
> > http://marc.info/?l=linux-mm&m=147158805719821 ?
> 
> https://ixion.pld-linux.org/~arekm/p2/ext4/log-20160819.txt
> https://ixion.pld-linux.org/~arekm/p2/ext4/log-trace_pipe-20160819.txt.gz
> 
> rm/cp -al x10 succeeded without any OOM
> 
> so the question is - which solution is "the one" for stable/4.7.x ?

I will send an email later today with other people reporting pre-mature
OOMs later today and will make sure you are on the CC list as well.

Thanks for the testing!
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html