June 2024 - Graviton3-test-report

Linux Kernel v6.10-rc5 vs v6.9 Performance Testing Results

by konstantin.belov＠linaro.org

Hello, Performance evaluation of Linux Kernel v6.10-rc5 is finished. Comparison is done to baseline Linux Kernel version: v6.9. The following configuration was used for all targets: - Instance CPU: graviton3 - Number of cores: 2 - RAM size, GB: 7.6 - OS: Debian Linux 12 (bookworm) Regressions by test Benchmark/Test/Metric, Units | Baseline Mean | Current Mean | Diff, % | SD | Samples | Confidence interval ------------------------------------------------------------------------------------------------------------------------------------------------------ aim9/brk_test/System Memory Allocations, OPS | 2024031.020 | 2008833.335 | 0.841 | 15981.304 | 120 | 2025343.934 / 2031098.615 fio/writer/writer-write, OPS | 97798.885 | 67325.199 | 25.764 | 72300.888 | 60 | 85492.996 / 122116.078 stressng/context/Context Switches, switches/s - context-1 | 2413.745 | 2366.265 | 1.974 | 5.545 | 21 | 2411.869 / 2416.544 - context-2 | 4841.600 | 4757.130 | 1.716 | 8.553 | 21 | 4836.322 / 4843.231 - context-4 | 4752.505 | 4712.375 | 0.646 | 56.213 | 21 | 4715.545 / 4762.271 - context-8 | 4753.285 | 4713.330 | 0.725 | 42.188 | 21 | 4727.178 / 4762.383 - context-16 | 4753.605 | 4711.875 | 0.861 | 26.756 | 21 | 4740.353 / 4762.158 - context-32 | 4756.705 | 4721.310 | 0.628 | 42.255 | 21 | 4729.786 / 4764.480 - context-64 | 4759.445 | 4728.605 | 0.505 | 46.395 | 21 | 4732.590 / 4770.716 - context-128 | 4814.425 | 4759.170 | 0.936 | 43.430 | 21 | 4784.785 / 4820.130 stressng/fork/Forks, OPS - fork-4 | 6922.335 | 6846.845 | 0.595 | 181.515 | 105 | 6905.004 / 6973.439 - fork-8 | 6751.675 | 6645.580 | 1.540 | 175.721 | 105 | 6733.361 / 6800.310 - fork-16 | 6639.680 | 6536.265 | 1.157 | 230.738 | 105 | 6605.904 / 6691.617 - fork-32 | 6469.605 | 6351.585 | 1.843 | 199.299 | 105 | 6435.213 / 6512.344 - fork-64 | 6226.470 | 6050.965 | 3.055 | 229.732 | 105 | 6202.757 / 6290.235 - fork-128 | 5938.430 | 5824.840 | 2.447 | 260.748 | 105 | 5922.094 / 6020.485 stressng/get/Read Throughput, MB/s - get-1 | 2828.900 | 2793.915 | 1.113 | 41.018 | 70 | 2814.521 / 2833.488 - get-4 | 2878.250 | 2848.490 | 1.126 | 83.756 | 70 | 2853.543 / 2892.408 stressng/madvise/Access calls, OPS - madvise-1 | 39.925 | 38.445 | 3.726 | 0.901 | 105 | 39.717 / 40.055 - madvise-2 | 77.400 | 74.895 | 3.051 | 2.653 | 105 | 76.821 / 77.835 - madvise-4 | 87.550 | 85.125 | 2.941 | 1.662 | 105 | 87.288 / 87.916 - madvise-8 | 88.070 | 85.405 | 3.131 | 1.859 | 105 | 87.739 / 88.446 - madvise-16 | 87.235 | 84.830 | 3.080 | 1.978 | 105 | 86.901 / 87.640 - madvise-32 | 86.300 | 83.885 | 3.077 | 1.805 | 105 | 85.999 / 86.684 - madvise-64 | 85.615 | 83.190 | 3.066 | 1.923 | 105 | 85.282 / 86.023 - madvise-128 | 84.730 | 81.780 | 3.566 | 1.741 | 105 | 84.442 / 85.099 stressng/vm-splice/Transfer Rate, MB/s - vm-splice-1 | 393962.125 | 380884.680 | 3.338 | 1375.412 | 70 | 393557.679 / 394193.506 - vm-splice-2 | 985842.100 | 955996.310 | 3.014 | 3171.138 | 70 | 984987.661 / 986448.773 - vm-splice-4 | 1112662.825 | 1085601.655 | 2.410 | 4974.978 | 70 | 1111703.539 / 1114047.566 - vm-splice-8 | 1196704.405 | 1167308.690 | 2.471 | 5363.710 | 70 | 1195240.562 / 1197725.298 - vm-splice-16 | 1241430.610 | 1206691.725 | 2.798 | 5841.721 | 70 | 1239884.083 / 1242606.569 - vm-splice-32 | 1243408.735 | 1205210.870 | 3.071 | 5133.764 | 70 | 1242267.819 / 1244628.719 - vm-splice-64 | 1241989.475 | 1210209.025 | 2.556 | 5385.394 | 70 | 1240609.336 / 1243102.559 - vm-splice-128 | 1248649.105 | 1212717.895 | 2.864 | 6197.063 | 70 | 1247271.017 / 1250112.895 sysbench/mutex/Events, OPS | 9991.017 | 9599.048 | 4.210 | 1446.210 | 56 | 9434.313 / 10169.427 wis/futex1-threads/Futex, OPS - futex1-threads-2 | 7331384.000 | 7284264.500 | 0.590 | 19642.783 | 40 | 7320680.786 / 7332620.586 - futex1-threads-5 | 7345486.500 | 7292154.000 | 0.730 | 18849.538 | 40 | 7338685.872 / 7350282.645 - futex1-threads-8 | 7345137.750 | 7298011.000 | 0.514 | 37983.921 | 40 | 7324579.433 / 7347620.029 - futex1-threads-12 | 7355575.500 | 7303781.500 | 0.719 | 11125.573 | 40 | 7351143.157 / 7358021.168 - futex1-threads-21 | 7361320.500 | 7305515.000 | 0.751 | 9466.763 | 40 | 7356681.171 / 7362429.214 - futex1-threads-30 | 7359961.000 | 7300230.500 | 0.807 | 7729.312 | 40 | 7357069.488 / 7361786.599 - futex1-threads-48 | 7351592.500 | 7309618.000 | 0.591 | 6461.848 | 40 | 7351385.098 / 7355354.709 - futex1-threads-79 | 7358242.000 | 7308607.500 | 0.692 | 11973.766 | 40 | 7353975.894 / 7361353.356 - futex1-threads-110 | 7354987.000 | 7304531.000 | 0.673 | 7643.182 | 40 | 7351888.976 / 7356472.944 - futex1-threads-128 | 7488506.000 | 7428399.500 | 0.746 | 27867.865 | 40 | 7475077.688 / 7492044.614 wis/futex3-threads/Futex, OPS - futex3-threads-30 | 9408300.500 | 9380256.000 | 0.639 | 16189.516 | 40 | 9402458.322 / 9412406.551 - futex3-threads-79 | 9419871.250 | 9379065.000 | 0.643 | 9673.762 | 40 | 9417409.600 / 9423269.750 wis/pipe1-threads/Pipe, OPS - pipe1-threads-2 | 2953808.000 | 2836051.000 | 4.096 | 32265.170 | 40 | 2947591.564 / 2967304.694 - pipe1-threads-5 | 2902003.750 | 2808807.500 | 3.244 | 35961.675 | 40 | 2891343.652 / 2913314.310 - pipe1-threads-8 | 2902555.250 | 2797466.500 | 3.955 | 38328.801 | 40 | 2890867.899 / 2914501.971 - pipe1-threads-12 | 2893482.250 | 2798925.000 | 3.338 | 37131.352 | 40 | 2884542.474 / 2907327.818 - pipe1-threads-21 | 2898269.500 | 2781471.000 | 4.220 | 38567.680 | 40 | 2888091.931 / 2911775.462 - pipe1-threads-30 | 2893691.750 | 2772565.000 | 4.414 | 37063.065 | 40 | 2886426.264 / 2908849.549 - pipe1-threads-48 | 2897777.250 | 2785544.000 | 3.849 | 36639.153 | 40 | 2887350.688 / 2909847.167 - pipe1-threads-79 | 2899279.250 | 2768002.500 | 4.559 | 36845.516 | 40 | 2888371.391 / 2910716.188 - pipe1-threads-110 | 2884872.250 | 2770956.000 | 4.240 | 51646.569 | 40 | 2866295.525 / 2897785.042 - pipe1-threads-128 | 2910726.750 | 2831037.000 | 2.793 | 42884.068 | 40 | 2897187.420 / 2923784.775 wis/pthread_mutex1-threads/Mutex, OPS - pthread_mutex1-threads-5 | 14964128.250 | 14225412.000 | 3.902 | 967263.602 | 40 | 14413067.341 / 15008825.215 - pthread_mutex1-threads-21 | 15001573.500 | 13764075.500 | 6.863 | 695259.838 | 40 | 14691176.210 / 15112893.191 wis/sched_yield-threads/Context Switches, OPS - sched_yield-threads-5 | 2803251.000 | 2739170.000 | 1.637 | 44282.527 | 40 | 2789015.746 / 2816083.465 - sched_yield-threads-8 | 2676167.750 | 2621844.500 | 2.074 | 24688.255 | 40 | 2670476.766 / 2685268.877 - sched_yield-threads-12 | 2537788.750 | 2510135.500 | 1.227 | 45024.342 | 40 | 2521879.751 / 2549589.639 - sched_yield-threads-21 | 2471273.250 | 2442379.500 | 0.778 | 42739.709 | 40 | 2447357.476 / 2473133.006 - sched_yield-threads-30 | 2379894.500 | 2311476.000 | 2.301 | 25939.719 | 40 | 2366828.438 / 2382368.296 - sched_yield-threads-79 | 2202153.500 | 2149404.500 | 2.836 | 20013.488 | 40 | 2195495.042 / 2207819.754 - sched_yield-threads-110 | 2143881.500 | 2059775.500 | 4.010 | 21334.701 | 40 | 2137079.910 / 2150291.032 - sched_yield-threads-128 | 2102815.500 | 2026696.000 | 3.346 | 27519.539 | 40 | 2097120.591 / 2114175.826 wis/signal1-threads/Signals, OPS - signal1-threads-2 | 1032419.750 | 967942.500 | 5.658 | 39861.529 | 40 | 1017668.801 / 1042203.932 - signal1-threads-5 | 1040060.750 | 1017754.500 | 2.443 | 29931.169 | 40 | 1030378.573 / 1048564.717 - signal1-threads-12 | 1033586.750 | 984592.500 | 4.575 | 33248.929 | 40 | 1022808.441 / 1043256.098 - signal1-threads-21 | 1037118.250 | 979339.000 | 3.928 | 20657.264 | 40 | 1032019.797 / 1044798.986 - signal1-threads-30 | 1025704.500 | 983937.000 | 4.030 | 31987.654 | 40 | 1015466.709 / 1034958.376 - signal1-threads-48 | 1027878.250 | 975944.000 | 5.507 | 29338.353 | 40 | 1024320.523 / 1042120.100 - signal1-threads-79 | 1030575.250 | 992948.250 | 3.856 | 38482.082 | 40 | 1020043.830 / 1043527.170 - signal1-threads-110 | 1034733.750 | 1005838.500 | 2.806 | 19589.115 | 35 | 1028380.922 / 1041184.536 - signal1-threads-128 | 1037267.000 | 988638.000 | 4.543 | 24313.801 | 35 | 1027944.596 / 1043670.576 * index after test name indicate number of threads or processes used for the test. Detailed test results and raw data can be found here: https://artifacts.codelinaro.org/artifactory/linaro-373-reports/aws-gravito… Best regards.

8 months, 2 weeks

1
0
0 0

Re: Linux Kernel v6.10-rc3 vs v6.9 Performance Testing Results

by Konstantin Belov

Hello James, I can say from statistical point of view the following: 1. madvise results: All madvise test scenarios show a performance degradation 4.4% - 6.6% compared to the baseline (v6.9). It's clear from the numbers and chart (see aws-graviton3-2024-06-17/stressng/stressng.html). Given the consistency of the regression across different 'madvise' settings (number of threads 1-128), this tells of a potentially major issue introduced in Linux Kernel v6.10-rc3 that impacts the 'madvise' system call. This observation also can be strengthened with individual threads cases (madvise-1, .., -128) observation across previous versions starting from 6.8 (aws-graviton3-2024-06-17stressng/madvise-1/comparison.html, for example). Moreover, from git log it's clear that some active work was done in windows v6.8-v6.9, v6.9-v6.10-rc1, etc. This could be a reason for such degradations. With implemented bisection procedures it will be possible to tell which commits have made what contribution to such degradations. 2. fork results: The 'fork' tests with 32, 64 and 128 processes show performance regressions ranging 3.7% - 4.85%. While not as severe as the madvise results, these still indicate a measurable degradation in the fork system call performance. The madvise results are more significant for a the following reasons: - The regressions are larger in magnitude (up to 6.6% vs 4.85% for fork) - The regressions are consistent across all tested 'madvise' settings and versions, while 'fork' only shows notable regressions at higher counts. So in summary, based on this data, the 'madvise' performance issues introduced in Linux Kernel v6.10-rc3 appear to be the more significant and require further investigation and optimization efforts. Best regards, Konstantin. On Fri, 14 Jun 2024 at 22:16, James Greenhalgh <James.Greenhalgh(a)arm.com> wrote: > Should I read this as identifying new issues with madvise and fork > performance, with the madvise result likely to be more significant? > > > > Thanks for the effort to make the report public – it is now much easier to > access and click through to see the detailed stats. > > > > Thanks, > > James > > > > *From: *konstantin.belov(a)linaro.org <konstantin.belov(a)linaro.org> > *Date: *Friday, June 14, 2024 at 15:03 > *To: *graviton3-test-report(a)lists.linaro.org < > graviton3-test-report(a)lists.linaro.org> > *Subject: *[Graviton3-test-report] Linux Kernel v6.10-rc3 vs v6.9 > Performance Testing Results > > > Hello, > Performance evaluation of Linux Kernel v6.10-rc3 is finished. > Comparison is done to baseline Linux Kernel version: v6.9. > The following configuration was used for all targets: > - Instance CPU: graviton3 > - Number of cores: 2 > - RAM size, GB: 7.6 > - OS: Debian Linux 12 (bookworm) > > > Regressions by test > > Benchmark/Test/Metric, Units | Baseline Median | > Current Median | Diff, % | SD | Samples | Confidence interval > > ------------------------------------------------------------------------------------------------------------------------------------------------------ > aim9/brk_test/System Memory Allocations, OPS | 2024133.335 | > 2013262.470 | 0.537 | 16053.459 | 60 | 2024467.853 / > 2032591.875 > aim9/exec_test/Program Loads, OPS | 2019.665 | > 2005.835 | 0.685 | 17.517 | 60 | 2015.828 / 2024.693 > aim9/page_test/System Allocations & Pages, OPS | 570520.000 | > 550091.667 | 3.581 | 8356.271 | 60 | 568085.990 / > 572314.769 > stressng/context/Context Switches, switches/s > - context-1 | 2413.745 | > 2376.445 | 1.545 | 5.545 | 21 | 2411.760 / 2416.503 > - context-2 | 4841.600 | > 4769.580 | 1.488 | 8.553 | 21 | 4836.387 / 4843.704 > - context-4 | 4752.505 | > 4712.600 | 0.840 | 56.213 | 21 | 4716.312 / 4764.397 > - context-8 | 4753.285 | > 4724.605 | 0.603 | 42.188 | 21 | 4728.812 / 4764.899 > - context-16 | 4753.605 | > 4726.140 | 0.578 | 26.756 | 21 | 4740.173 / 4763.060 > - context-128 | 4814.425 | > 4774.555 | 0.828 | 43.430 | 21 | 4786.041 / 4823.191 > stressng/fork/Forks, OPS > - fork-32 | 6659.030 | > 6412.310 | 3.705 | 174.515 | 21 | 6588.293 / 6737.573 > - fork-64 | 6516.635 | > 6200.470 | 4.852 | 266.061 | 21 | 6394.023 / 6621.612 > - fork-128 | 6128.850 | > 5843.435 | 4.657 | 353.797 | 21 | 6005.178 / 6307.815 > stressng/get/Read Throughput, MB/s > - get-1 | 2835.125 | > 2785.405 | 1.754 | 25.632 | 21 | 2823.417 / 2845.342 > - get-2 | 3295.495 | > 3250.850 | 1.355 | 50.308 | 21 | 3271.035 / 3314.069 > - get-16 | 3097.905 | > 3062.850 | 1.132 | 36.459 | 21 | 3081.725 / 3112.912 > stressng/madvise/Access calls, OPS > - madvise-1 | 39.845 | > 37.240 | 6.538 | 0.864 | 21 | 39.425 / 40.164 > - madvise-2 | 76.950 | > 72.480 | 5.809 | 2.384 | 21 | 76.016 / 78.055 > - madvise-4 | 86.900 | > 83.080 | 4.396 | 1.283 | 21 | 86.317 / 87.414 > - madvise-8 | 87.615 | > 82.920 | 5.359 | 1.634 | 21 | 86.943 / 88.341 > - madvise-16 | 87.230 | > 82.345 | 5.600 | 1.879 | 21 | 86.472 / 88.079 > - madvise-32 | 85.895 | > 80.780 | 5.955 | 1.543 | 21 | 85.294 / 86.614 > - madvise-64 | 85.955 | > 80.625 | 6.201 | 2.257 | 21 | 84.615 / 86.546 > - madvise-128 | 85.005 | > 79.375 | 6.623 | 1.706 | 21 | 84.143 / 85.603 > stressng/vm-splice/Transfer Rate, MB/s > - vm-splice-1 | 393825.950 | > 384976.075 | 2.247 | 1326.151 | 21 | 393146.946 / > 394281.333 > - vm-splice-2 | 985022.850 | > 970009.675 | 1.524 | 3305.843 | 21 | 983522.945 / > 986350.758 > - vm-splice-4 | 1111278.105 | > 1098156.625 | 1.181 | 5058.131 | 21 | 1108947.236 / > 1113273.953 > - vm-splice-8 | 1195749.630 | > 1183949.325 | 0.987 | 5994.316 | 21 | 1192658.405 / > 1197785.934 > - vm-splice-16 | 1238939.755 | > 1226909.580 | 0.971 | 7273.147 | 21 | 1235968.212 / > 1242189.651 > - vm-splice-32 | 1241984.390 | > 1227662.560 | 1.153 | 5226.860 | 21 | 1239895.031 / > 1244366.079 > - vm-splice-64 | 1241623.855 | > 1227060.990 | 1.173 | 5099.827 | 21 | 1239863.809 / > 1244226.193 > - vm-splice-128 | 1248609.720 | > 1233938.390 | 1.175 | 7083.372 | 21 | 1245315.449 / > 1251374.554 > > * index after test name indicate number of threads or processes used for > the test. > > Detailed test results and raw data can be found here: > https://artifacts.codelinaro.org/artifactory/linaro-373-reports/aws-gravito… > > Best regards. > _______________________________________________ > Graviton3-test-report mailing list -- > graviton3-test-report(a)lists.linaro.org > To unsubscribe send an email to > graviton3-test-report-leave(a)lists.linaro.org > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. >

8 months, 3 weeks

1
0
0 0

Linux Kernel v6.10-rc3 vs v6.9 Performance Testing Results

by konstantin.belov＠linaro.org

Hello, Performance evaluation of Linux Kernel v6.10-rc3 is finished. Comparison is done to baseline Linux Kernel version: v6.9. The following configuration was used for all targets: - Instance CPU: graviton3 - Number of cores: 2 - RAM size, GB: 7.6 - OS: Debian Linux 12 (bookworm) Regressions by test Benchmark/Test/Metric, Units | Baseline Median | Current Median | Diff, % | SD | Samples | Confidence interval ------------------------------------------------------------------------------------------------------------------------------------------------------ aim9/brk_test/System Memory Allocations, OPS | 2024133.335 | 2013262.470 | 0.537 | 16053.459 | 60 | 2024467.853 / 2032591.875 aim9/exec_test/Program Loads, OPS | 2019.665 | 2005.835 | 0.685 | 17.517 | 60 | 2015.828 / 2024.693 aim9/page_test/System Allocations & Pages, OPS | 570520.000 | 550091.667 | 3.581 | 8356.271 | 60 | 568085.990 / 572314.769 stressng/context/Context Switches, switches/s - context-1 | 2413.745 | 2376.445 | 1.545 | 5.545 | 21 | 2411.760 / 2416.503 - context-2 | 4841.600 | 4769.580 | 1.488 | 8.553 | 21 | 4836.387 / 4843.704 - context-4 | 4752.505 | 4712.600 | 0.840 | 56.213 | 21 | 4716.312 / 4764.397 - context-8 | 4753.285 | 4724.605 | 0.603 | 42.188 | 21 | 4728.812 / 4764.899 - context-16 | 4753.605 | 4726.140 | 0.578 | 26.756 | 21 | 4740.173 / 4763.060 - context-128 | 4814.425 | 4774.555 | 0.828 | 43.430 | 21 | 4786.041 / 4823.191 stressng/fork/Forks, OPS - fork-32 | 6659.030 | 6412.310 | 3.705 | 174.515 | 21 | 6588.293 / 6737.573 - fork-64 | 6516.635 | 6200.470 | 4.852 | 266.061 | 21 | 6394.023 / 6621.612 - fork-128 | 6128.850 | 5843.435 | 4.657 | 353.797 | 21 | 6005.178 / 6307.815 stressng/get/Read Throughput, MB/s - get-1 | 2835.125 | 2785.405 | 1.754 | 25.632 | 21 | 2823.417 / 2845.342 - get-2 | 3295.495 | 3250.850 | 1.355 | 50.308 | 21 | 3271.035 / 3314.069 - get-16 | 3097.905 | 3062.850 | 1.132 | 36.459 | 21 | 3081.725 / 3112.912 stressng/madvise/Access calls, OPS - madvise-1 | 39.845 | 37.240 | 6.538 | 0.864 | 21 | 39.425 / 40.164 - madvise-2 | 76.950 | 72.480 | 5.809 | 2.384 | 21 | 76.016 / 78.055 - madvise-4 | 86.900 | 83.080 | 4.396 | 1.283 | 21 | 86.317 / 87.414 - madvise-8 | 87.615 | 82.920 | 5.359 | 1.634 | 21 | 86.943 / 88.341 - madvise-16 | 87.230 | 82.345 | 5.600 | 1.879 | 21 | 86.472 / 88.079 - madvise-32 | 85.895 | 80.780 | 5.955 | 1.543 | 21 | 85.294 / 86.614 - madvise-64 | 85.955 | 80.625 | 6.201 | 2.257 | 21 | 84.615 / 86.546 - madvise-128 | 85.005 | 79.375 | 6.623 | 1.706 | 21 | 84.143 / 85.603 stressng/vm-splice/Transfer Rate, MB/s - vm-splice-1 | 393825.950 | 384976.075 | 2.247 | 1326.151 | 21 | 393146.946 / 394281.333 - vm-splice-2 | 985022.850 | 970009.675 | 1.524 | 3305.843 | 21 | 983522.945 / 986350.758 - vm-splice-4 | 1111278.105 | 1098156.625 | 1.181 | 5058.131 | 21 | 1108947.236 / 1113273.953 - vm-splice-8 | 1195749.630 | 1183949.325 | 0.987 | 5994.316 | 21 | 1192658.405 / 1197785.934 - vm-splice-16 | 1238939.755 | 1226909.580 | 0.971 | 7273.147 | 21 | 1235968.212 / 1242189.651 - vm-splice-32 | 1241984.390 | 1227662.560 | 1.153 | 5226.860 | 21 | 1239895.031 / 1244366.079 - vm-splice-64 | 1241623.855 | 1227060.990 | 1.173 | 5099.827 | 21 | 1239863.809 / 1244226.193 - vm-splice-128 | 1248609.720 | 1233938.390 | 1.175 | 7083.372 | 21 | 1245315.449 / 1251374.554 * index after test name indicate number of threads or processes used for the test. Detailed test results and raw data can be found here: https://artifacts.codelinaro.org/artifactory/linaro-373-reports/aws-gravito… Best regards.

9 months

1
0
0 0

2025

2024

Graviton3-test-report June 2024 ----- 2025 ----- March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024

Graviton3-test-report June 2024