Message-ID: <56018695.5030700@gmail.com>
Date:	Tue, 22 Sep 2015 12:49:25 -0400
From:	Austin S Hemmelgarn <ahferroin7@...il.com>
To:	Minchan Kim <minchan@...nel.org>,
	Vitaly Wool <vitalywool@...il.com>
Cc:	Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>,
	Dan Streetman <ddstreet@...e.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
	LKML <linux-kernel@...r.kernel.org>, linux-mm@...ck.org
Subject: Re: [PATCH 0/2] prepare zbud to be used by zram as underlying
 allocator

On 2015-09-22 11:36, Minchan Kim wrote:
> Hi Vitaly,
>
> On Mon, Sep 21, 2015 at 11:11:00PM +0200, Vitaly Wool wrote:
>> Hello Minchan,
>>
>>> Sorry, because you wrote up "zram" in the title.
>>> As I said earlier, we need several numbers to investigate.
>>>
>>> First of all, what is culprit of your latency?
>>> It seems you are thinking about compaction. so compaction what?
>>> Frequent scanning? lock collision? or frequent sleeping in compaction
>>> code somewhere? And then why does zbud solve it? If we use zbud for zram,
>>> we lose memory efficiency so there is something to justify it.
>>
>> The data I've got so far strongly suggests that in some use cases (see
>> below) with zsmalloc
>> * there are more allocstalls
>> * memory compaction is triggered more frequently
>> * allocstalls happen more often
>> * page migrations are way more frequent, too.
>>
>> Please also keep in mind that I do not advise you or anyone to use
>> zbud instead of zsmalloc. The point I'm trying to make is that zbud
>> fits my particular case better and I want to be able to choose it in
>> the kernel without hacking it with my private patches.
>
> I understand your goal well. ;-) But, please understand my goal which
> is to find fundamental reason why zbud removes latency.
>
> You gave some compaction-related stats but it is just one of result,
> not the cause. I guess you could find another stats as well as compaction
> stats which affect your workload. Once you find them all, please
> investigate what is major factor for your latency among them.
> Then, we should think over what is best solution for it and if zbud is
> best to remove the cause, yes, why not. I can merge it into zram.
>
> IOW, I should maintain zram so I need to know when,where,how to use zbud
> with zram is useful so that I can guide it to zram users and you should
> *justify* the overhead to me. Overhead means I should maintain two allocators
> for zram from now on. It means when I want to add some feature for zsmalloc,
> I should take care of zbud and I should watch zbud patches, too which could
> be very painful and starting point of diverge for zram.
>
> Compared to zsmalloc, zsmalloc packs lots of compressed objects into
> a page while zbud just stores two objects so if there are different
> life time objects in a page, zsmalloc may make higher fragmentation
> but zbud is not a good choice for memory efficiency either so my concern
> starts from here.
>
> For solving such problem, we added compaction into recent zram to
> reduce waste memory space so it should solve internal fragment problem.
> Other problem we don't solve now is external fragmentation which
> is related to compaction stats you are seeing now.
> Although you are seeing mitigation with zbud, it would be still problem
> if you begin to use more memory for zbud. One of example, a few years
> ago, some guys tried to support zbud page migration.
>
> If external fragmentation is really problem in here, we should proivde
> a feature VM can migrate zsmalloc page and it was alomost done as I told
> you previous thread and I think it is really way to go.
>
> Even, we are trying to add zlib which is better compress ratio algorithm
> to reduce working memory size so without the feature, the problem would be
> more severe.
>
> So, I am thinking now we should enhance it rather than hiding a problem
> by merging zbud.
You know,from my perspective, most of the above argument 'against' zbud 
being supported really provides no actual reason to not support zbud 
without reading between the lines (which as far as I can tell, is 'I 
just don't want to support it'), it just points out all the differences 
between zsmalloc and zbud.  They are different API's, they have 
different semantics, they are intended for different workloads and use 
cases, and this will always be the case.  To be entirely honest, the 
more complicated you make zsmalloc in an attempt to obviate the need to 
support zbud, the more attractive zbud looks as far as I'm concerned.
>
>> FWIW, given that I am not an author of either, I don't see why anyone
>> would consider me biased. :-)
>>
>> As of the memory efficiency, you seem to be quite comfortable with
>> storing uncompressed pages when they compress to more than 3/4 of a
>> page. I observed ~13% reported ratio increase (3.8x to 4.3x) when I
>> increased max_zpage_size to PAGE_SIZE / 32 * 31. Doesn't look like a
>> fight for every byte to me.
>
> Thanks for the report. It could be another patch. :)
>
>>
>>> The reason I am asking is I have investigated similar problems
>>> in android and other plaforms and the reason of latency was not zsmalloc
>>> but agressive high-order allocations from subsystems, watermark check
>>> race, deferring of compaction, LMK not working and too much swapout so
>>> it causes to reclaim lots of page cache pages which was main culprit
>>> in my cases. When I checks with perf, compaction stall count is increased,
>>> the time spent in there is not huge so it was not main factor of latency.
>>
>> The main use case where the difference is seen is switching between
>> users on an Android device. It does cause a lot of reclaim, too, as
>> you say, but this is in the nature of zbud that reclaim happens in a
>> more deterministic way and worst-case looks substantially nicer. That
>
> Interesting!
> Why is reclaim more deterministic with zbud?
> That's really one of part what I want with data.
I'm pretty sure this is due to the fact that zbud stores at most 2 
compressed pages in a given page, combined with fewer conditionals in 
the reclaim path.
>
>
>> said, the standard deviation calculated over 20 iterations of a
>> change-user-multiple-times-test is 2x less for zbud than the one of
>> zsmalloc.
>
> One thing I can guess is a page could be freed easily if just two objects
> in a page are freed by munmap or kill. IOW, we could remove pinned page
> easily so we could get higher-order page easily.
>
> However, it would be different once zbud's memory usgae is higher
> as I mentioned. As well, we lose memory efficieny significantly for zram. :(
Not everyone who uses zram cares hugely about memory efficiency, I for 
one would rather have a more deterministic compression ratio (which zbud 
achieves) than slightly better memory efficiencyg.
>
> IMO, more fundamentatal solution is to support VM-aware compaction of
> zsmalloc/zbud rather than hiding a problem with zbud.



Download attachment "smime.p7s" of type "application/pkcs7-signature" (3019 bytes)

Powered by blists - more mailing lists