linux-kernel - Re: [PATCH -V3 0/3] memory tiering: hot page selection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <872bdaee-21a0-005b-b66c-893eb331e39a@linux.alibaba.com>
Date:   Mon, 20 Jun 2022 11:19:23 +0800
From:   Baolin Wang <baolin.wang@...ux.alibaba.com>
To:     Huang Ying <ying.huang@...el.com>,
        Andrew Morton <akpm@...ux-foundation.org>
Cc:     linux-mm@...ck.org, linux-kernel@...r.kernel.org,
        Johannes Weiner <hannes@...xchg.org>,
        Michal Hocko <mhocko@...e.com>,
        Rik van Riel <riel@...riel.com>,
        Mel Gorman <mgorman@...hsingularity.net>,
        Peter Zijlstra <peterz@...radead.org>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        Yang Shi <shy828301@...il.com>, Zi Yan <ziy@...dia.com>,
        Wei Xu <weixugc@...gle.com>, osalvador <osalvador@...e.de>,
        Shakeel Butt <shakeelb@...gle.com>,
        Zhong Jiang <zhongjiang-ali@...ux.alibaba.com>
Subject: Re: [PATCH -V3 0/3] memory tiering: hot page selection



On 6/14/2022 4:16 PM, Huang Ying wrote:
> To optimize page placement in a memory tiering system with NUMA
> balancing, the hot pages in the slow memory nodes need to be
> identified.  Essentially, the original NUMA balancing implementation
> selects the mostly recently accessed (MRU) pages to promote.  But this
> isn't a perfect algorithm to identify the hot pages.  Because the
> pages with quite low access frequency may be accessed eventually given
> the NUMA balancing page table scanning period could be quite long
> (e.g. 60 seconds).  So in this patchset, we implement a new hot page
> identification algorithm based on the latency between NUMA balancing
> page table scanning and hint page fault.  Which is a kind of mostly
> frequently accessed (MFU) algorithm.
> 
> In NUMA balancing memory tiering mode, if there are hot pages in slow
> memory node and cold pages in fast memory node, we need to
> promote/demote hot/cold pages between the fast and cold memory nodes.
> 
> A choice is to promote/demote as fast as possible.  But the CPU cycles
> and memory bandwidth consumed by the high promoting/demoting
> throughput will hurt the latency of some workload because of accessing
> inflating and slow memory bandwidth contention.
> 
> A way to resolve this issue is to restrict the max promoting/demoting
> throughput.  It will take longer to finish the promoting/demoting.
> But the workload latency will be better.  This is implemented in this
> patchset as the page promotion rate limit mechanism.
> 
> The promotion hot threshold is workload and system configuration
> dependent.  So in this patchset, a method to adjust the hot threshold
> automatically is implemented.  The basic idea is to control the number
> of the candidate promotion pages to match the promotion rate limit.
> 
> We used the pmbench memory accessing benchmark tested the patchset on
> a 2-socket server system with DRAM and PMEM installed.  The test
> results are as follows,
> 
> 		pmbench score		promote rate
> 		 (accesses/s)			MB/s
> 		-------------		------------
> base		  146887704.1		       725.6
> hot selection     165695601.2		       544.0
> rate limit	  162814569.8		       165.2
> auto adjustment	  170495294.0                  136.9
> 
>  From the results above,
> 
> With hot page selection patch [1/3], the pmbench score increases about
> 12.8%, and promote rate (overhead) decreases about 25.0%, compared with
> base kernel.
> 
> With rate limit patch [2/3], pmbench score decreases about 1.7%, and
> promote rate decreases about 69.6%, compared with hot page selection
> patch.
> 
> With threshold auto adjustment patch [3/3], pmbench score increases
> about 4.7%, and promote rate decrease about 17.1%, compared with rate
> limit patch.

I did a simple testing with mysql on my machine which contains 1 DRAM 
node (30G) and 1 PMEM node (126G).

sysbench /usr/share/sysbench/oltp_read_write.lua \
......
--tables=200 \
--table-size=1000000 \
--report-interval=10 \
--threads=16 \
--time=120

The tps can be improved about 5% from below data, and I think this is a 
good start to optimize the promotion. So for this series, please feel 
free to add:

Reviewed-by: Baolin Wang <baolin.wang@...ux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@...ux.alibaba.com>

Without this patchset:
  transactions:                        2080188 (3466.48 per sec.)

With this patch set:
  transactions:                        2174296 (3623.40 per sec.)