[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <8fea74ec-8feb-1709-14f2-cecb63fdc9ed@amd.com>
Date: Fri, 24 Feb 2023 09:06:49 +0530
From: Bharata B Rao <bharata@....com>
To: "Huang, Ying" <ying.huang@...el.com>
Cc: linux-kernel@...r.kernel.org, linux-mm@...ck.org, mgorman@...e.de,
peterz@...radead.org, mingo@...hat.com, bp@...en8.de,
dave.hansen@...ux.intel.com, x86@...nel.org,
akpm@...ux-foundation.org, luto@...nel.org, tglx@...utronix.de,
yue.li@...verge.com, Ravikumar.Bangoria@....com
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
On 17-Feb-23 11:33 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@....com> writes:
>
>> On 14-Feb-23 10:25 AM, Bharata B Rao wrote:
>>> On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>>>>> I have a microbenchmark where two sets of threads bound to two
>>>>> NUMA nodes access the two different halves of memory which is
>>>>> initially allocated on the 1st node.
>>>>>
>>>>> On a two node Zen4 system, with 64 threads in each set accessing
>>>>> 8G of memory each from the initial allocation of 16G, I see that
>>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>>>>> to complete a fixed number of memory accesses. This could well
>>>>> be the best case and real workloads/benchmarks may not get this much
>>>>> uplift, but it does show the potential gain to be had.
>>>>
>>>> Can you find a way to show the overhead of the original implementation
>>>> and your method? Then we can compare between them? Because you think
>>>> the improvement comes from the reduced overhead.
>>>
>>> Sure, will measure the overhead.
>>
>> I used ftrace function_graph tracer to measure the amount of time (in us)
>> spent in fault handling and task_work handling in both the methods when
>> the above mentioned benchmark was running.
>>
>> Default IBS
>> Fault handling 29879668.71 1226770.84
>> Task work handling 24878.894 10635593.82
>> Sched switch handling 78159.846
>>
>> Total 29904547.6 11940524.51
>
> Thanks! You have shown the large overhead difference between the
> original method and your method. Can you show the number of the pages
> migrated too? I think the overhead / page can be a good overhead
> indicator too.
>
> Can it be translated to the performance improvement? Per my
> understanding, the total overhead is small compared with total run time.
I captured some of the numbers that you wanted for two different runs.
The first case shows the data for a short run (less number of memory access
iterations) and the second one is for a long run (more number of iterations)
Short-run
=========
Time taken or overhead (us) for fault, task_work and sched_switch
handling
Default IBS
Fault handling 29017953.99 1196828.67
Task work handling 10354.40 10356778.53
Sched switch handling 56572.21
Total overhead 29028308.39 11610179.41
Benchmark score(us) 194050290 53963650
numa_pages_migrated 2097256 662755
Overhead / page 13.84 17.51
Pages migrated per sec 72248.64 57083.95
Default
-------
Total Min Max Avg
do_numa_page 29017953.99 0.1 307.63 15.97
task_numa_work 10354.40 2.86 4573.60 175.50
Total 29028308.39
IBS
---
Total Min Max Avg
ibs_overflow_handler 1196828.67 0.15 100.28 1.26
task_ibs_access_work 10356778.53 0.21 10504.14 28.42
hw_access_sched_in 56572.21 0.15 16.94 1.45
Total 11610179.41
Long-run
========
Time taken or overhead (us) for fault, task_work and sched_switch
handling
Default IBS
Fault handling 27437756.73 901406.37
Task work handling 1741.66 4902935.32
Sched switch handling 100590.33
Total overhead 27439498.38 5904932.02
Benchmark score(us) 306786210.0 153422489.0
numa_pages_migrated 2097218 1746099
Overhead / page 13.08 3.38
Pages migrated per sec 6836.08 11380.98
Default
-------
Total Min Max Avg
do_numa_page 27437756.73 0.08 363.475 15.03
task_numa_work 1741.66 3.294 1200.71 42.48
Total 27439498.38
IBS
---
Total Min Max Avg
ibs_overflow_handler 901406.37 0.15 95.51 1.06
task_ibs_access_work 4902935.32 0.22 11013.68 9.64
hw_access_sched_in 100590.33 0.14 91.97 1.52
Total 5904932.02
Regards,
Bharata.
Powered by blists - more mailing lists