| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 29 Oct 2019 10:15:53 +0100
From: Dario Faggioli <dfaggioli@...e.com>
To: Aaron Lu <aaron.lu@...ux.alibaba.com>,
Aubrey Li <aubrey.intel@...il.com>
Cc: Tim Chen <tim.c.chen@...ux.intel.com>,
Julien Desfossez <jdesfossez@...italocean.com>,
"Li, Aubrey" <aubrey.li@...ux.intel.com>,
Subhra Mazumdar <subhra.mazumdar@...cle.com>,
Vineeth Remanan Pillai <vpillai@...italocean.com>,
Nishanth Aravamudan <naravamudan@...italocean.com>,
Peter Zijlstra <peterz@...radead.org>,
Ingo Molnar <mingo@...nel.org>,
Thomas Gleixner <tglx@...utronix.de>,
Paul Turner <pjt@...gle.com>,
Linus Torvalds <torvalds@...ux-foundation.org>,
Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
Frédéric Weisbecker <fweisbec@...il.com>,
Kees Cook <keescook@...omium.org>,
Greg Kerr <kerrnel@...gle.com>, Phil Auld <pauld@...hat.com>,
Valentin Schneider <valentin.schneider@....com>,
Mel Gorman <mgorman@...hsingularity.net>,
Pawan Gupta <pawan.kumar.gupta@...ux.intel.com>,
Paolo Bonzini <pbonzini@...hat.com>
Subject: Re: [RFC PATCH v3 00/16] Core scheduling v3
On Tue, 2019-10-29 at 10:11 +0100, Dario Faggioli wrote:
> On Sun, 2019-09-15 at 22:14 +0800, Aaron Lu wrote:
> > I'm using the following branch as base which is v5.1.5 based:
> > https://github.com/digitalocean/linux-coresched coresched-v3-
> > v5.1.5-
> > test
> >
> > And I have pushed Tim's branch to:
> > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-tim
> >
> > Mine:
> > https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-
> > core_vruntime
> >
> Hello,
>
> As anticipated, I've been trying to follow the development of this
> feature and, in the meantime, I have done some benchmarks.
>
> I actually have a lot of data (and am planning for more), so I am
> sending a few emails, each one with a subset of the numbers in it,
> instead than just one which would be beyond giant! :-)
>
HACKBENCH-PROCESS-PIPES
=======================
http://xenbits.xen.org/people/dariof/benchmarks/results/linux/core-sched/mmtests/boxes/wayrath/coresched-email-2_hackbench.txt
v v BM BM BM BM BM BM
BM-HT BM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.4372 ( 0.00%) 0.6714 * -53.57%* 0.4242 ( 2.97%) 0.6654 * -52.20%* 1.1748 *-168.71%* 1.1666 *-166.83%* 1.1496 *-162.95%* 1.3696 *-213.27%*
Amean 3 1.1466 ( 0.00%) 2.0284 * -76.91%* 1.1700 ( -2.04%) 2.0320 * -77.22%* 6.8484 *-497.28%* 6.7128 *-485.45%* 6.8068 *-493.65%* 7.9702 *-595.12%*
Amean 5 2.0140 ( 0.00%) 2.7834 * -38.20%* 2.0116 ( 0.12%) 2.6688 * -32.51%* 11.9494 *-493.32%* 11.9450 *-493.10%* 12.0904 *-500.32%* 14.4190 *-615.94%*
Amean 7 2.5396 ( 0.00%) 3.3064 * -30.19%* 2.6002 ( -2.39%) 2.9074 * -14.48%* 16.0142 *-530.58%* 16.3418 *-543.48%* 17.0896 *-572.92%* 20.2050 *-695.60%*
Amean 12 3.2930 ( 0.00%) 2.9830 ( 9.41%) 3.4226 ( -3.94%) 2.8522 ( 13.39%) 28.1482 *-754.79%* 27.3916 *-731.81%* 28.5938 *-768.32%* 34.6184 *-951.27%*
Amean 18 3.9452 ( 0.00%) 4.0806 ( -3.43%) 3.7950 ( 3.81%) 3.8938 ( 1.30%) 41.7120 *-957.28%* 42.1062 *-967.28%* 44.2136 *-1020.69%* 51.0200 *-1193.22%*
Amean 24 4.1618 ( 0.00%) 4.8466 * -16.45%* 4.2258 ( -1.54%) 5.0720 * -21.87%* 56.8598 *-1266.23%* 57.3568 *-1278.17%* 61.2660 *-1372.10%* 68.2538 *-1540.01%*
Amean 30 4.7790 ( 0.00%) 5.9726 * -24.98%* 4.8756 ( -2.02%) 5.9434 * -24.36%* 72.1602 *-1409.94%* 75.8036 *-1486.18%* 80.0066 *-1574.13%* 81.3768 *-1602.80%*
Amean 32 5.1680 ( 0.00%) 6.6000 * -27.71%* 5.2004 ( -0.63%) 6.5490 * -26.72%* 78.1974 *-1413.11%* 82.9812 *-1505.67%* 87.7340 *-1597.64%* 85.6876 *-1558.04%*
Stddev 1 0.0173 ( 0.00%) 0.0624 (-259.99%) 0.0129 ( 25.54%) 0.0451 (-160.36%) 0.0727 (-319.12%) 0.0954 (-450.05%) 0.0545 (-214.16%) 0.0960 (-453.74%)
Stddev 3 0.0471 ( 0.00%) 0.3035 (-544.20%) 0.0547 ( -16.12%) 0.2745 (-482.52%) 0.2029 (-330.60%) 0.1102 (-133.97%) 0.2391 (-407.38%) 0.1864 (-295.62%)
Stddev 5 0.1492 ( 0.00%) 0.4178 (-180.05%) 0.1419 ( 4.88%) 0.2569 ( -72.22%) 0.3878 (-159.96%) 0.2584 ( -73.22%) 0.4259 (-185.47%) 0.3340 (-123.89%)
Stddev 7 0.2077 ( 0.00%) 0.2941 ( -41.59%) 0.2281 ( -9.85%) 0.2049 ( 1.33%) 0.3922 ( -88.86%) 0.8178 (-293.78%) 0.4064 ( -95.67%) 0.6127 (-195.02%)
Stddev 12 0.5560 ( 0.00%) 0.2038 ( 63.34%) 0.2113 ( 62.00%) 0.2490 ( 55.21%) 1.0797 ( -94.21%) 0.7564 ( -36.06%) 1.0225 ( -83.91%) 1.5233 (-174.01%)
Stddev 18 0.3556 ( 0.00%) 0.3110 ( 12.55%) 0.3054 ( 14.11%) 0.1265 ( 64.43%) 1.0258 (-188.47%) 1.4386 (-304.53%) 1.1818 (-232.33%) 2.6710 (-651.08%)
Stddev 24 0.1844 ( 0.00%) 0.1614 ( 12.46%) 0.1135 ( 38.46%) 0.3679 ( -99.54%) 2.0997 (-1038.84%) 1.1493 (-523.36%) 0.5214 (-182.79%) 2.9229 (-1485.35%)
Stddev 30 0.1420 ( 0.00%) 0.0875 ( 38.37%) 0.1799 ( -26.66%) 0.1076 ( 24.26%) 4.5079 (-3073.51%) 1.5704 (-1005.58%) 1.4054 (-889.42%) 4.5743 (-3120.26%)
Stddev 32 0.2184 ( 0.00%) 0.2427 ( -11.11%) 0.3143 ( -43.92%) 0.3517 ( -61.01%) 3.7345 (-1609.81%) 1.3564 (-521.00%) 2.1822 (-899.11%) 4.0896 (-1772.40%)
v v VM VM VM VM VM VM
VM-HT VM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.6762 ( 0.00%) 1.6824 *-148.80%* 0.6630 ( 1.95%) 1.5706 *-132.27%* 0.6600 ( 2.40%) 0.6514 ( 3.67%) 0.6780 ( -0.27%) 0.6964 * -2.99%*
Amean 3 1.7130 ( 0.00%) 2.2806 * -33.13%* 1.8074 * -5.51%* 2.1882 * -27.74%* 1.7992 ( -5.03%) 1.7896 * -4.47%* 1.7760 ( -3.68%) 1.8264 * -6.62%*
Amean 5 2.8688 ( 0.00%) 2.8048 ( 2.23%) 2.5880 * 9.79%* 2.8474 ( 0.75%) 2.5486 * 11.16%* 2.7896 ( 2.76%) 2.4402 * 14.94%* 2.6020 * 9.30%*
Amean 7 3.3432 ( 0.00%) 3.3434 ( -0.01%) 3.4564 ( -3.39%) 3.4704 ( -3.80%) 3.2424 ( 3.02%) 3.5620 ( -6.54%) 3.0352 ( 9.21%) 3.0874 ( 7.65%)
Amean 12 5.8936 ( 0.00%) 4.9968 * 15.22%* 5.1560 * 12.52%* 5.0670 * 14.03%* 4.2722 * 27.51%* 5.6570 ( 4.01%) 4.8270 * 18.10%* 4.2514 * 27.86%*
Amean 18 6.3938 ( 0.00%) 7.0542 ( -10.33%) 6.4900 ( -1.50%) 6.9682 ( -8.98%) 5.6478 ( 11.67%) 7.4324 ( -16.24%) 6.3160 ( 1.22%) 6.5124 ( -1.85%)
Amean 24 7.6278 ( 0.00%) 9.5096 * -24.67%* 7.0062 * 8.15%* 9.0278 * -18.35%* 7.5650 ( 0.82%) 8.2604 ( -8.29%) 8.5372 ( -11.92%) 7.7008 ( -0.96%)
Amean 30 10.5534 ( 0.00%) 10.9456 ( -3.72%) 11.4470 ( -8.47%) 11.1330 ( -5.49%) 8.8486 ( 16.15%) 10.8508 ( -2.82%) 12.6182 ( -19.57%) 10.5836 ( -0.29%)
Amean 32 11.6024 ( 0.00%) 12.6052 ( -8.64%) 10.8236 ( 6.71%) 11.2156 ( 3.33%) 9.0654 * 21.87%* 11.5074 ( 0.82%) 10.3592 ( 10.72%) 11.7174 ( -0.99%)
Stddev 1 0.0143 ( 0.00%) 0.3700 (-2480.08%) 0.0261 ( -82.29%) 0.1206 (-741.16%) 0.0444 (-209.74%) 0.0270 ( -88.42%) 0.0238 ( -65.73%) 0.0180 ( -25.17%)
Stddev 3 0.0473 ( 0.00%) 0.1384 (-192.63%) 0.0739 ( -56.38%) 0.0714 ( -51.01%) 0.0931 ( -96.94%) 0.0517 ( -9.28%) 0.0960 (-103.02%) 0.0553 ( -17.04%)
Stddev 5 0.1236 ( 0.00%) 0.2251 ( -82.10%) 0.1607 ( -29.98%) 0.1464 ( -18.41%) 0.1319 ( -6.69%) 0.1842 ( -48.96%) 0.0647 ( 47.66%) 0.1977 ( -59.92%)
Stddev 7 0.3597 ( 0.00%) 0.1105 ( 69.27%) 0.2288 ( 36.38%) 0.1335 ( 62.90%) 0.3131 ( 12.94%) 0.2958 ( 17.76%) 0.1581 ( 56.05%) 0.2361 ( 34.37%)
Stddev 12 0.4677 ( 0.00%) 0.0846 ( 81.91%) 0.6319 ( -35.11%) 0.1526 ( 67.37%) 0.3591 ( 23.23%) 0.6898 ( -47.48%) 0.4853 ( -3.76%) 0.2963 ( 36.64%)
Stddev 18 1.2289 ( 0.00%) 0.1849 ( 84.95%) 1.1160 ( 9.18%) 0.2497 ( 79.68%) 0.9843 ( 19.90%) 0.9542 ( 22.35%) 0.6621 ( 46.12%) 0.7668 ( 37.60%)
Stddev 24 0.5202 ( 0.00%) 0.9344 ( -79.60%) 0.1940 ( 62.71%) 0.4706 ( 9.53%) 0.8362 ( -60.73%) 1.0819 (-107.96%) 1.0229 ( -96.63%) 0.8881 ( -70.72%)
Stddev 30 2.1557 ( 0.00%) 0.7984 ( 62.96%) 1.2499 ( 42.02%) 0.9804 ( 54.52%) 0.4846 ( 77.52%) 1.2901 ( 40.15%) 1.5532 ( 27.95%) 1.2932 ( 40.01%)
Stddev 32 2.2255 ( 0.00%) 1.1321 ( 49.13%) 2.2380 ( -0.56%) 0.2127 ( 90.44%) 0.3654 ( 83.58%) 1.5727 ( 29.33%) 2.3291 ( -4.66%) 2.0936 ( 5.93%)
v v VM VM VM VM VM VM
VM-v4-HT VM-v4-noHT v4-HT v4-noHT v4-csc-HT v4-csc_stallfix-HT v4-csc_tim-HT v4-csc_vruntime-HT
Amean 1 1.2194 ( 0.00%) 1.1974 ( 1.80%) 1.0308 * 15.47%* 1.1054 * 9.35%* 1.0522 * 13.71%* 1.2290 ( -0.79%) 1.1392 * 6.58%* 1.0524 * 13.70%*
Amean 3 2.2568 ( 0.00%) 2.0588 ( 8.77%) 2.0708 ( 8.24%) 2.2352 ( 0.96%) 2.1808 ( 3.37%) 2.2820 ( -1.12%) 2.1682 ( 3.93%) 2.2598 ( -0.13%)
Amean 5 2.9848 ( 0.00%) 2.9912 ( -0.21%) 2.4938 * 16.45%* 2.4634 * 17.47%* 2.6890 ( 9.91%) 2.8908 ( 3.15%) 2.8636 ( 4.06%) 2.4158 * 19.06%*
Amean 7 3.4500 ( 0.00%) 3.2538 ( 5.69%) 3.3646 ( 2.48%) 3.2666 ( 5.32%) 3.0800 ( 10.72%) 4.2206 ( -22.34%) 3.1016 ( 10.10%) 3.2186 ( 6.71%)
Amean 12 6.0414 ( 0.00%) 5.0624 ( 16.20%) 5.1276 ( 15.13%) 5.1066 ( 15.47%) 4.7728 * 21.00%* 5.5068 ( 8.85%) 4.7544 * 21.30%* 5.8920 ( 2.47%)
Amean 16 7.5510 ( 0.00%) 7.6888 ( -1.82%) 6.9732 ( 7.65%) 5.9098 * 21.73%* 6.5542 * 13.20%* 6.9492 ( 7.97%) 6.4372 * 14.75%* 6.1968 * 17.93%*
Stddev 1 0.0786 ( 0.00%) 0.1166 ( -48.34%) 0.1762 (-124.09%) 0.0712 ( 9.45%) 0.1541 ( -95.99%) 0.0814 ( -3.55%) 0.0452 ( 42.57%) 0.1817 (-131.05%)
Stddev 3 0.2220 ( 0.00%) 0.1887 ( 15.03%) 0.2174 ( 2.07%) 0.1368 ( 38.37%) 0.1928 ( 13.17%) 0.4342 ( -95.57%) 0.2353 ( -5.97%) 0.1753 ( 21.06%)
Stddev 5 0.4586 ( 0.00%) 0.4689 ( -2.23%) 0.3202 ( 30.19%) 0.2666 ( 41.88%) 0.3108 ( 32.24%) 0.2876 ( 37.29%) 0.4067 ( 11.33%) 0.3010 ( 34.37%)
Stddev 7 0.3769 ( 0.00%) 0.5242 ( -39.09%) 0.2523 ( 33.06%) 0.2498 ( 33.71%) 0.4527 ( -20.13%) 1.5089 (-300.36%) 0.3998 ( -6.09%) 0.4018 ( -6.62%)
Stddev 12 1.0604 ( 0.00%) 0.5194 ( 51.02%) 0.5646 ( 46.75%) 0.5255 ( 50.45%) 0.4872 ( 54.06%) 0.3447 ( 67.49%) 0.5492 ( 48.21%) 0.5487 ( 48.26%)
Stddev 16 0.6245 ( 0.00%) 0.5220 ( 16.40%) 0.6914 ( -10.71%) 0.6984 ( -11.83%) 0.3543 ( 43.27%) 0.5137 ( 17.74%) 0.5083 ( 18.61%) 1.0744 ( -72.04%)
v v VMx2 VMx2 VMx2 VMx2 VMx2 VMx2
VMx2-HT VMx2-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.6780 ( 0.00%) 2.1522 *-217.43%* 1.6790 *-147.64%* 2.2440 *-230.97%* 15.6090 *-2202.21%* 9.1920 *-1255.75%* 2.0732 *-205.78%* 2.2390 *-230.24%*
Amean 3 1.8452 ( 0.00%) 3.2456 * -75.89%* 2.5214 * -36.65%* 3.3944 * -83.96%* 13.7258 *-643.87%* 9.7654 *-429.23%* 2.6668 * -44.53%* 2.9044 * -57.40%*
Amean 5 2.8564 ( 0.00%) 4.2548 * -48.96%* 3.2500 * -13.78%* 4.3958 * -53.89%* 11.6808 *-308.93%* 8.4724 *-196.61%* 3.4000 * -19.03%* 3.5298 * -23.58%*
Amean 7 3.3712 ( 0.00%) 5.1056 * -51.45%* 4.1636 * -23.50%* 4.9874 * -47.94%* 14.0516 (-316.81%) 9.1454 *-171.28%* 4.1396 * -22.79%* 4.3570 * -29.24%*
Amean 12 4.8134 ( 0.00%) 7.4516 * -54.81%* 6.1254 * -27.26%* 7.3488 * -52.67%* 13.7680 *-186.03%* 13.7400 *-185.45%* 6.0424 * -25.53%* 6.3404 * -31.72%*
Amean 18 6.1980 ( 0.00%) 10.4126 * -68.00%* 8.5942 * -38.66%* 9.7554 * -57.40%* 10.7234 * -73.01%* 13.5214 *-118.16%* 8.6628 * -39.77%* 8.5910 * -38.61%*
Amean 24 8.2116 ( 0.00%) 12.7570 * -55.35%* 10.9386 * -33.21%* 12.9032 * -57.13%* 17.8492 *-117.37%* 14.4432 * -75.89%* 10.7488 * -30.90%* 11.1930 * -36.31%*
Amean 30 9.3264 ( 0.00%) 15.3704 * -64.81%* 13.6630 * -46.50%* 15.4806 * -65.99%* 16.9272 * -81.50%* 18.4184 * -97.49%* 13.6174 * -46.01%* 13.5026 * -44.78%*
Amean 32 10.1954 ( 0.00%) 16.5494 * -62.32%* 14.7450 * -44.62%* 16.4650 * -61.49%* 13.9146 * -36.48%* 17.5790 * -72.42%* 14.6072 * -43.27%* 14.0346 * -37.66%*
Stddev 1 0.0129 ( 0.00%) 0.2665 (-1959.36%) 0.2262 (-1647.54%) 0.0703 (-443.26%) 9.5450 (-73651.40%) 6.3226 (-48752.57%) 0.1736 (-1241.41%) 0.3070 (-2272.09%)
Stddev 3 0.1156 ( 0.00%) 0.3653 (-215.89%) 0.1388 ( -20.00%) 0.2067 ( -78.77%) 3.3781 (-2821.39%) 3.2241 (-2688.20%) 0.0970 ( 16.10%) 0.1247 ( -7.87%)
Stddev 5 0.0817 ( 0.00%) 0.2968 (-263.19%) 0.0938 ( -14.79%) 0.1367 ( -67.25%) 3.1273 (-3726.47%) 3.9175 (-4693.34%) 0.1155 ( -41.31%) 0.1864 (-128.10%)
Stddev 7 0.1190 ( 0.00%) 0.1337 ( -12.31%) 0.3136 (-163.48%) 0.2034 ( -70.91%) 17.1856 (-14336.96%) 2.8676 (-2308.94%) 0.0794 ( 33.32%) 0.1362 ( -14.42%)
Stddev 12 0.5237 ( 0.00%) 0.2929 ( 44.08%) 0.1386 ( 73.53%) 0.2330 ( 55.51%) 7.1869 (-1272.26%) 8.0085 (-1429.13%) 0.2535 ( 51.59%) 0.1441 ( 72.49%)
Stddev 18 0.4791 ( 0.00%) 0.1528 ( 68.10%) 0.4254 ( 11.21%) 0.2953 ( 38.37%) 2.8492 (-494.71%) 4.3767 (-813.54%) 0.3205 ( 33.10%) 0.3094 ( 35.43%)
Stddev 24 1.1591 ( 0.00%) 0.2616 ( 77.43%) 0.2446 ( 78.90%) 0.4363 ( 62.36%) 5.3380 (-360.53%) 3.2433 (-179.81%) 0.2704 ( 76.67%) 0.4402 ( 62.02%)
Stddev 30 0.6561 ( 0.00%) 0.4008 ( 38.91%) 0.2670 ( 59.31%) 0.1451 ( 77.88%) 4.5765 (-597.54%) 4.2380 (-545.95%) 0.4768 ( 27.33%) 0.2693 ( 58.96%)
Stddev 32 1.2197 ( 0.00%) 0.4459 ( 63.44%) 0.5496 ( 54.94%) 0.4179 ( 65.74%) 1.3553 ( -11.12%) 4.3537 (-256.96%) 0.5602 ( 54.07%) 0.5312 ( 56.45%)
So, situation looks pretty *terrible* on baremetal. Well, something to
investigate, I guess.
In VMs, on the other hand, things don't look too bad. Although, it's a
bit weird that the benchmark seems to be sensitive to HT when run on
baremetal, but not so much when run in VMs. Among the various core
scheduling implementations/patches, plain v3 seems to have issues with
this workload, while Tim's and vruntime are better.
It is confirmed that, in virtualization scenario, under
overcommittment, core scheduling is more effective than disabling
HyperThreading, at least either with Tim's or with vruntime patches.
Overhead wise, it is an hard call. Numbers vary a lot, and I think each
group of measurements needs to be looked at carefully to come up with a
sensible analysis.
Overhead is there, that's for sure, in the baremetal case. Also, it
looks more severe when HT is disabled (e.g., comapre v-BM-noHT with BM-
noHT).
In virt cases, it's really an hard call. E.g., when a VM with 4 vCPUs
is used, core scheduling seems to be able to make the system
significantly faster! :-O
--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)
Powered by blists - more mailing lists