Message-Id: <20211218071041.24077-1-bot@edi.works>
Date:   Fri, 17 Dec 2021 23:10:41 -0800
From:   bot@....works
To:     yuzhao@...gle.com
Cc:     linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        page-reclaim@...gle.com, corbet@....net,
        michael@...haellarabel.com, sofia.trinh@....works
Subject: Re: [PATCH v5 00/10] Multigenerational LRU Framework

Kernel / PostgreSQL benchmark with MGLRU

TLDR
====
With the MGLRU, PostgreSQL achieved 95% CI [1.75, 6.42]% more
transactions per minute (TPM) under the high-concurrency conditions,
when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more
TPM, respectively, under the medium- and the high-concurrency
conditions, when swap was on. There were no statistically significant
changes in TPM for the rest of the test matrix.

Background
==========
Memory overcommit can increase utilization and, if carried out
properly, can also increase throughput. The challenges are to improve
working set estimation and to optimize page reclaim. The risks are
performance degradation and OOM kills. Short of overcoming the
challenges, the only way to reduce the risks is to underutilize
memory.

PostgreSQL is one of the most popular open-source RDBMSs. HammerDB is
the leading open-source benchmarking software derived from the TPC
specifications. OLTP is the most important use case for RDBMSs.

Matrix
======
Kernels: version [+ patchset]
* Baseline: 5.15
* Patched: 5.15 + MGLRU

Swap configurations:
* Off
* On (vm.swappiness=1)

Concurrency conditions: average # of users per CPU
* Low: ~4
* Medium: ~8
* High: ~12

Total configurations: 12
Data points per configuration: 10
Total run duration (minutes) per data point: ~50

Procedure
=========
The latest MGLRU patchset for the 5.15 kernel is available at
git fetch https://linux-mm.googlesource.com/page-reclaim \
    refs/changes/30/1430/2

The baseline and the patched 5.15 kernel images are available at
https://drive.google.com/drive/folders/1eMkQleAFGkP2vzM_JyRA21oKE0ESHBqp

<install and configure OS>
hammerdbcli auto prep_tpcc.tcl
systemctl stop postgresql
e2image <backup /mnt/data>

<for each kernel>
    grub-set-default <baseline, patched>
    <for each swap configuration>
        <swapoff, swapon>
        <update /etc/postgresql/13/main/postgresql.conf>
        <for each concurrency condition>
            <update run_tpcc.tcl>
            <for each data point>
                systemctl stop postgresql
                e2image <restore /mnt/data>
                reboot
                hammerdbcli auto run_tpcc.tcl
                <collect TPM>

Hardware
========
Memory (GB): 256
CPU (total #): 48
NVMe SSD (GB): 2048

OS
==
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.10
DISTRIB_CODENAME=impish
DISTRIB_DESCRIPTION="Ubuntu 21.10"

$ cat /proc/swaps
Filename          Type          Size          Used     Priority
/swap.img         partition     67108860      0        -2

$ cat /proc/sys/vm/overcommit_memory
1

$ cat /proc/sys/vm/swappiness
1

$ cat /sys/fs/cgroup/user.slice/memory.min
4294967296

PostgreSQL
==========
$ pg_config --version
PostgreSQL 13.5 (Ubuntu 13.5-0ubuntu0.21.10.1)

$ cat /etc/postgresql/13/main/postgresql.conf
<existing parameters>

data_directory = '/mnt/data'
max_connections = 1000
shared_buffers = <swapoff: 120GB, swapon: 150GB>
temp_buffers = 1GB
work_mem = 1GB
maintenance_work_mem = 1GB
logical_decoding_work_mem = 1GB
bgwriter_delay = 1000ms
bgwriter_lru_maxpages = 100000
bgwriter_lru_multiplier = 1.0
bgwriter_flush_after = 0
effective_io_concurrency = 1000
wal_compression = on
wal_writer_delay = 1000ms
wal_writer_flush_after = 128MB
max_wal_size = 100GB
min_wal_size = 10GB
checkpoint_flush_after = 0
effective_cache_size = 150GB

<existing parameters>

HammerDB
========
$ hammerdbcli -h
HammerDB CLI v4.3
Copyright (C) 2003-2021 Steve Shaw
Type "help" for a list of commands
Usage: hammerdbcli [ auto [ script_to_autoload.tcl  ] ]

$ cat prep_tpcc.tcl
dbset db pg
diset connection pg_host /var/run/postgresql
diset tpcc pg_count_ware 2400
diset tpcc pg_num_vu 48
buildschema
waittocomplete
quit

$ cat run_tpcc.tcl
dbset db pg
diset connection pg_host /var/run/postgresql
diset tpcc pg_total_iterations 20000000
diset tpcc pg_driver timed
diset tpcc pg_rampup 30
diset tpcc pg_duration 10
diset tpcc pg_allwarehouse true
vuset logtotemp 1
vuset unique 1
loadscript
vuset vu <200, 400, 600>
vucreate
vurun
runtimer 3000
vudestroy

Results
=======
Comparing the patched with the baseline kernel, PostgreSQL achieved
95% CI [1.75, 6.42]% more TPM under the high-concurrency conditions,
when swap was off; 95% CIs [12.82, 18.69]% and [22.70, 46.86]% more
TPM, respectively, under the medium- and the high-concurrency
conditions, when swap was on. There were no statistically significant
changes in TPM for the rest of the test matrix.

+--------------------+--------------------+--------------------+
| Mean TPM [95% CI]  | Swap off           | Swap on            |
+--------------------+--------------------+--------------------+
| Low concurrency    | 466430 / 467521    | 475060 / 475047    |
|                    | [-6931, 9112]      | [-7431, 7405]      |
+--------------------+--------------------+--------------------+
| Medium concurrency | 453871 / 459592    | 388245 / 449409    |
|                    | [-774, 12216]      | [49755, 72572]     |
+--------------------+--------------------+--------------------+
| High concurrency   | 443014 / 461112    | 157106 / 211752    |
|                    | [7771, 28423]      | [35664, 73627]     |
+--------------------+--------------------+--------------------+
Table 1. Comparison between the baseline and the patched kernels

Comparing swap on with swap off, PostgreSQL achieved 95% CIs [0.46,
3.24]%, [-16.91, -12.01]% and [-68.64, -60.43]% more TPM,
respectively, under the low-, the medium- and the high-concurrency
conditions, when using the baseline kernel; 95% CIs [-3.76, -0.67]%
and [-56.70, -51.46]% more TPM, respectively, under the medium- and
the high-concurrency conditions, when using the patched kernel. There
were no statistically significant changes in TPM for the rest of the
test matrix.

+--------------------+--------------------+--------------------+
| Mean TPM [95% CI]  | Baseline kernel    | Patched kernel     |
+--------------------+--------------------+--------------------+
| Low concurrency    | 466430 / 475060    | 467521 / 475047    |
|                    | [2160, 15100]      | [-1204, 16256]     |
+--------------------+--------------------+--------------------+
| Medium concurrency | 453871 / 388245    | 459592 / 449409    |
|                    | [-76757, -54494]   | [-17292, -3073]    |
+--------------------+--------------------+--------------------+
| High concurrency   | 443014 / 157106    | 461112 / 211752    |
|                    | [-304097, -267718] | [-261442, -237275] |
+--------------------+--------------------+--------------------+
Table 2. Comparison between swap off and swap on

Metrics collected during each run are available at
https://github.com/ediworks/KernelPerf/tree/master/mglru/postgres/5.15

Appendix
========
$ cat raw_data_postgres.r
v <- c(
    # baseline swapoff 200vu
    462379, 462998, 463363, 464949, 465605, 466977, 467290, 468658, 469682, 472404,
    # baseline swapoff 400vu
    446111, 446305, 447339, 448043, 450604, 452160, 453846, 461309, 465101, 467893,
    # baseline swapoff 600vu
    434061, 435645, 435974, 436026, 436581, 439138, 442121, 445990, 454687, 469926,
    # baseline swapon 200vu
    466546, 467298, 467882, 469185, 472114, 473868, 475217, 481319, 483246, 493931,
    # baseline swapon 400vu
    367605, 371855, 373991, 380763, 388456, 389768, 395270, 403536, 404457, 406749,
    # baseline swapon 600vu
    123036, 127174, 131863, 150724, 155572, 158938, 170892, 179302, 183783, 189785,
    # patched swapoff 200vu
    456088, 457197, 457341, 458069, 459630, 472291, 474782, 475727, 478015, 486071,
    # patched swapoff 400vu
    452681, 453758, 455800, 457675, 458812, 459304, 460897, 461252, 465269, 470475,
    # patched swapoff 600vu
    448009, 452465, 453655, 454333, 456111, 456304, 465371, 471431, 475092, 478351,
    # patched swapon 200vu
    465540, 468681, 471682, 473134, 473148, 474015, 475734, 476691, 481974, 489873,
    # patched swapon 400vu
    436300, 440202, 441955, 445214, 445817, 452176, 452379, 456931, 457724, 465393,
    # patched swapon 600vu
    195315, 197186, 199332, 199667, 209630, 211162, 214787, 222783, 230000, 237667
)

a <- array(v, dim = c(10, 3, 2, 2))

# baseline vs patched
for (swap in 1:2) {
    for (vu in 1:3) {
        r <- t.test(a[, vu, swap, 1], a[, vu, swap, 2])
        print(r)

        p <- r$conf.int * 100 / r$estimate[1]
        if ((p[1] > 0 && p[2] < 0) || (p[1] < 0 && p[2] > 0)) {
            s <- sprintf("swap%d vu%d: no significance", swap, vu)
        } else {
            s <- sprintf("swap%d vu%d: [%.2f, %.2f]%%", swap, vu, -p[2], -p[1])
        }
        print(s)
    }
}

# swapoff vs swapon
for (kern in 1:2) {
    for (vu in 1:3) {
        r <- t.test(a[, vu, 1, kern], a[, vu, 2, kern])
        print(r)

        p <- r$conf.int * 100 / r$estimate[1]
        if ((p[1] > 0 && p[2] < 0) || (p[1] < 0 && p[2] > 0)) {
            s <- sprintf("kern%d vu%d: no significance", kern, vu)
        } else {
            s <- sprintf("kern%d vu%d: [%.2f, %.2f]%%", kern, vu, -p[2], -p[1])
        }
        print(s)
    }
}

$ R -q -s -f raw_data_postgres.r

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = -0.3009, df = 10.521, p-value = 0.7694
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -9112.559  6931.359
sample estimates:
mean of x mean of y
 466430.5  467521.1

[1] "swap1 vu1: no significance"

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = -1.8711, df = 15.599, p-value = 0.08021
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -12216.64    774.24
sample estimates:
mean of x mean of y
 453871.1  459592.3

[1] "swap1 vu2: no significance"

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = -3.6832, df = 17.919, p-value = 0.001712
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -28423.515  -7771.085
sample estimates:
mean of x mean of y
 443014.9  461112.2

[1] "swap1 vu3: [1.75, 6.42]%"

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = 0.0038109, df = 17.001, p-value = 0.997
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -7405.094  7431.894
sample estimates:
mean of x mean of y
 475060.6  475047.2

[1] "swap2 vu1: no significance"

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = -11.413, df = 15.222, p-value = 7.301e-09
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -72572.5 -49755.7
sample estimates:
mean of x mean of y
 388245.0  449409.1

[1] "swap2 vu2: [12.82, 18.69]%"

        Welch Two Sample t-test

data:  a[, vu, swap, 1] and a[, vu, swap, 2]
t = -6.1414, df = 14.853, p-value = 1.97e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -73627.83 -35664.17
sample estimates:
mean of x mean of y
 157106.9  211752.9

[1] "swap2 vu3: [22.70, 46.86]%"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = -2.9241, df = 11.372, p-value = 0.0134
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -15100.107  -2160.093
sample estimates:
mean of x mean of y
 466430.5  475060.6

[1] "kern1 vu1: [0.46, 3.24]%"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = 12.629, df = 14.192, p-value = 4.129e-09
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 54494.92 76757.28
sample estimates:
mean of x mean of y
 453871.1  388245.0

[1] "kern1 vu2: [-16.91, -12.01]%"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = 34.005, df = 12.822, p-value = 5.981e-14
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 267718.5 304097.5
sample estimates:
mean of x mean of y
 443014.9  157106.9

[1] "kern1 vu3: [-68.64, -60.43]%"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = -1.8367, df = 15.057, p-value = 0.08607
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -16256.986   1204.786
sample estimates:
mean of x mean of y
 467521.1  475047.2

[1] "kern2 vu1: no significance"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = 3.061, df = 14.554, p-value = 0.008153
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
  3073.49 17292.91
sample estimates:
mean of x mean of y
 459592.3  449409.1

[1] "kern2 vu2: [-3.76, -0.67]%"

        Welch Two Sample t-test

data:  a[, vu, 1, kern] and a[, vu, 2, kern]
t = 43.656, df = 16.424, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 237275.9 261442.7
sample estimates:
mean of x mean of y
 461112.2  211752.9

[1] "kern2 vu3: [-56.70, -51.46]%"

Powered by blists - more mailing lists