This is an automated email from the git hooks/post-receive script.

bernie.ogden pushed a change to branch benchmarking in repository toolchain/abe.

from 20f031f Remove tcwg tag from kvm example job adds 2e7d7da Handle error being an empty string adds 678878e Initialise buildtar to empty adds 08b0beb Drop pointless .git from log dir adds 43534a8 Accept any topdir in prebuilt tarball adds b933449 Be precise about the Juno CPUs adds c420331 Remove checks that cannot work adds 6702e29 Make .services file optional adds 0d603bb Copy return code along with results adds b0b12a8 Support uinstance jobs adds c3f9f46 Sanity check location when running trusted adds 28be528 Check that runbenchmark.sh has been given a thing to run adds a9cc551 Fix fakebench.conf adds db3b3f8 Remove bench dir for being confusing adds 9ab9122 Use ip_addr in error report, if available adds 0b27cca Merge Benchmark.job into host-session adds d47984c Allow benchmark.sh to take prebuilt tarball adds 8cfe45e Simplify benchmark.sh invocations adds a23e1a4 Support post-run-cmd adds e85387a Support gathering of results to bundle stream adds 2aa54ee Add some standard settings to target-session adds f28db30 Remove redundant exit from remote_exec_async adds 739d1d8 Attach raw files when gathering fakebench adds a03e8aa Clean up path handling adds a630e1e gather/fakebench.sh: All output to console adds 0e149e9 Gather script + tests for CPU2006 adds 508b69b fakebench.sh: Attach to run, not a test case adds bc98d46 CPU2006.sh: Attach raw output to run adds 1399e74 Rework CPU2006.sh to use 'raw' output adds 69ef488 test_CPU2006.sh: Give doit a sensible name adds cb8610b Rename *CPU2006.sh to *CPU200x.sh adds aa5096f test_CPU200x: Paramaterise bits that vary for 2k adds 708a90e CPU200x*: Support CPU2000 results. adds ae6a127 Lead regexp with ^ adds d6c911b Remove unused variables adds 6e3dfe6 raw is global, stop passing it around everywhere. adds fffbc69 Handle different spellings of benchmark names adds 02be24e Move boilerplate to generate_benchmark and rename adds d4d3c45 Test cases where only int or fp was run adds 64d90ff Add (commented-out) debugging printf adds 222796a Add a set -o pipefail adds 92f02ad Add exit, error handlers adds 5764caf Separate benchset definitions from individual runs adds 098bccc Add tests for individual benchmark runs adds 0096b3d Test random mixes of subbenchmarks adds 1917007 Fix off-by-1 in iteration counts adds e966d7c Test non-ref input sizes adds aa48cd9 Update error message adds 8dd3514 Add tests for cases where the script should fail adds d0ae3f2 Correct naming style of test input adds 711a076 Test more realistic values for input sizes adds fa4a4f8 Print the leading 0 if integer part is 0 adds e90eb06 Make locals in generate_subbenchmark actually local adds 9f37afc Multiply ratios for CPU2000 adds 866ffdd Use spec.cpu200x.invalid rather than ...valid adds f09e56a Test case where some subbenchmarks fail adds 38826e2 'Semi-trusted' concept, for testing purposes adds 0521d9e Remove redundant TODO adds 1753346 Add script to derive all job parameters adds c6d7bae Import template.yaml from lava-jobs adds a803692 Benchmark.sh: Allow optional parms to be undefined adds d019643 Benchmark.sh: Braces-and-spaces adds 2b45d79 Benchmark.sh: Get bundle server from external LAVA_SERVER def adds 8ee6618 Benchmark.sh: Make LAVA_USER available to params adds 55ca654 Benchmark.sh,template.yaml: s/ABE_/TESTDEF_/ adds a3227e4 Benchmark.sh,template.yaml: Accept public key adds 1212d25 Benchmark.sh,template.yaml: Map from target type to image adds dd218c6 Benchmark.sh: Remove redundant comments adds d5e5ddf Benchmark.sh: Set the right default branch adds c30abe9 Benchmark.sh: Set BUNDLE_STREAM_NAME only if unset adds a28dc75 Benchmark.sh: Handle enquoting in parameters adds eeb06d8 Benchmark.sh: Make bundle stream default paranoid adds 439b40a Benchmark.sh,template.yaml: s/BUNDLE_STREAM_NAME/BUNDLE_STREAM/ adds 731aee7 Remove old jobdefs adds ebf80ac template.yaml: Metadata must be a string adds 927250b dispatch-benchmark.py: Script to dispatch benchmarks adds 71ac192 dispatch-benchmark.py: Move dryrun exit to more natural location adds adabef4 dispatch-benchmark.py: Message on dry-run exit adds fbb7343 dispatch-benchmark.py: Validate input adds 8c5e81e dispatch-benchmark.py: Option to set bundle stream adds 751555c dispatch-benchmark.py: More detail on keyring fail adds b33631d Support external sysroot adds afcadd5 Apply target context when looking for binaries adds b26847e Always read configs with target context adds a745b27 Add some (commented-out) debugging output adds e4f867d CoremarkPro support adds 2755ec9 dispatch-benchmark.py: Support Coremark Pro adds d10f47a Gather script for Coremark Pro adds 0c69511 CoremarkPro.sh: Unique names for time and rate adds c0c13a0 CoremarkPro.sh: Attach raw data before processing adds 4d1f345 test_CoremarkPro.sh: Don't output errors when testing fail cases adds c986230 CoremarkPro.sh: Whitespace: Old comment,blank line adds e68d8b1 CoremarkPro.sh: Name size measurements sanely adds 9116bdc CoremarkPro.sh: [] notation for iterations only adds 2500638 CoremarkPro.sh: Eliminate spaces in TEST_CASE_ID adds 27664f1 CoremarkPro.sh: Cosmetic changes to test IDs adds e9edf0e CoremarkPro.sh: Count everything that can be multiple adds 58950cb CoremarkPro.sh: Make function-local var local adds 370e2a5 CoremarkPro.sh: Pass name to report_measured adds 85a75e0 CoremarkPro.sh: Remove commented-out code adds 4de6b50 CoremarkPro.sh: Require exactly one log adds 74d5121 CoremarkPro.sh: Report sizes first adds 5ee32c8 CoremarkPro.conf: Fix native paths adds 059a517 Ensure that unset params get value None adds f98f406 Clean up toolchain usage adds 9b4ad5b benchmark.sh: Rename GCC to something more generic adds 1ac4b63 Do not report names in tcpdump adds 8c94f48 Match tcpdump output even if user gave hostnames adds fd1af66 Match tcpdump output even if user gave username adds c9ba85f runbenchmark.sh: Fix indentation adds b0ceb31 Check post-run command exit code adds 2d06d8f Simplify post-run-cmd-not-set NOP adds fe4c5a1 target-session: Tidy up kill/wait code adds d8d1d46 Add comments about post-run/gather.sh adds 9b5081e Convert post-run command into multiline string adds 8bddeb4 Convert post-run cmd to function adds 419540e Wait for post-run command to complete adds 5ce2268 Return gather script exit code within post-run cmd adds 50fbd0d gather: set -x in all gather scripts adds f5f69e8 benchmark.sh preserves -x for runbenchmark.sh adds 14843bb gather scripts: Provide mime type for attachments adds 544ae0a host-session: Consistent use of {} with args adds 4df9bed Remove redundant ABE_BRANCH parameter adds 9d84d6e host-session invokes benchmark scripts with -x adds a693134 host-session: shift CLI args when parsing adds 3ff63d5 Decouple triple from toolchain adds 3f646d8 Do not infer device validation from triple adds 652d9de Rationalize *-session YAML adds 4a9b70d Defer to user-specified TARGET_SESSION adds f30c654 Make ssh messages depend on instance adds 2900fbb Some more detail around data collection message adds d6db174 Remove redundant 'tag' option adds 1f7649e Update CLI-related messages adds 2646b63 Remove redundant boot_timeout parameter adds cad3b0f Make TIMEOUT overrideable adds 50b0954 Increase default timeout from 0.5 to 1.5 hours. adds 80b22bb controlledrun.sh output log set by board conf adds cf6de56 controlledrun.sh logs to /dev/console if running in LAVA adds b554388 Always set safe_output when in a trusted LAVA adds 841332c tee stderr to log, when there is a log. adds 4b22180 Output 'Running ${cmd}' to the log adds 00c3424 Preserve stdout/stderr separation in cmd output adds d5566cc Benchmark.sh: Support deploy_linaro_kernel adds 07e3b8d Benchmark.sh: Support mustang host/target adds b002ace Close all tmpdir files before deletion adds 7547d6a Permit host machine to be of any type adds 597a910 Add missing tcpdump dependency adds 56f064b Template for use in trusted LAVA uinstance adds 3cefd21 Tighten CoremarkPro gather script, add many tests adds 77d0fbb s/bench-session-debian/host-session/ adds b3cd81c Move output_param to top of Benchmark.sh adds 06fcb70 Apply NAME=VALUE CLI overrides before Benchmark.sh adds 05949b5 Replace output_param with output_{value,slot} adds 93c354b Support injection of metadata into test runs adds 1fcc296 Centralize default-setting and input validation adds 9ac5985 Make JOB_NAME user-settable adds ed71b46 Only look up token in keychain for non-dry-runs adds 0e084be Confirm that one of TOOLCHAIN and PREBUILT is set adds 2454830 Create/support CPU200{0,6}.sh symlinks adds 55db400 dispatch-benchmark.py: Force explicit --triple adds b0fd2d0 Benchmark.sh: Let user override BUNDLE_SERVER adds 3e63f3d Benchmark.sh: Validate/fix up {LAVA,BUNDLE}_SERVER adds 2253030 Benchmark.sh: Ensure BUNDLE_STREAM has trailing / adds ce30010 CPU200x: Fix unit for geomean of runtime adds f489162 Benchmark.sh: Catch 2 triple/toolchain mismatches adds 498e5c8 Benchmark.sh: Isomorphic tidying of validate() adds 09c13e9 CPU200x.sh: Work with arbitrary name adds f245188 Handle possibility that gather script is a symlink adds 1d7ef33 host-session: Add missing 'test' adds 4224e83 Get Coremark-Pro from new Coremark-Pro repository adds 07d4a9a Set 'build' in Coremark-Pro default_makeflags adds b1beb64 Benchmark.sh: Set timeout depending on benchmark. adds f781052 Benchmark.sh: Add comment to validation func adds 4538068 Remove EEMBC.conf adds 09a7b69 Benchmark.sh: Only set timeout if user didn't adds 5b0e185 Use LAVA_JOB_NAME, not JOB_NAME, in LAVA wrapper adds 75c5411 Pass COMPILER_FLAGS through default_makeflags adds 2bf6b4a Add a TODO about setting COMPILER_FLAGS on CLI

No new revisions were added by this update.

Summary of changes: bench/create-tables.sql | 185 ----------- bench/host2sql.sh | 130 -------- bench/make-benchdata.sh | 121 ------- bench/make-datafiles.sh | 56 ---- bench/plot-bench.sh | 82 ----- bench/run2sql.sh | 75 ----- bench/tab2sql.sh | 44 --- config/CPU2000.conf | 4 +- config/CPU2006.conf | 4 +- config/Coremark-Pro.conf | 31 ++ config/EEMBC.conf | 13 - config/bench/boards/arndale.conf | 1 - config/bench/boards/arndale.services | 87 ----- config/bench/boards/juno-a53.conf | 3 +- config/bench/boards/juno-a53.services | 0 config/bench/boards/juno-a57.conf | 3 +- config/bench/boards/juno-a57.services | 0 config/bench/boards/kvm.conf | 1 - config/bench/boards/kvm.services | 0 config/bench/boards/localhost.services | 0 config/bench/boards/mustang.conf | 1 - config/bench/boards/mustang.services | 0 config/bench/boards/panda-es.conf | 1 - config/bench/boards/panda-es.services | 81 ----- config/bench/gather/CPU2000.sh | 1 + config/bench/gather/CPU2006.sh | 1 + config/bench/gather/CPU200x.sh | 223 +++++++++++++ config/bench/gather/Coremark-Pro.sh | 264 +++++++++++++++ config/bench/gather/fakebench.sh | 79 +++++ config/bench/gather/test_CPU200x.sh | 356 ++++++++++++++++++++ config/bench/gather/test_Coremark-Pro.sh | 331 +++++++++++++++++++ .../cfp2006_aarch32-juno_a53_x1-juno_a57_x1.json | 126 ------- .../cfp2006_aarch64-juno_a53_x1-juno_a57_x1.json | 126 ------- config/bench/jobdefs/cfp2006_pre-mustang.json | 92 ------ config/bench/jobdefs/cint2006-kvm_x1.json | 89 ----- .../cint2006_aarch32_juno_a53_x1-juno_a57_x1.json | 126 ------- .../cint2006_aarch64-juno_a53_x1-juno_a57_x1.json | 126 ------- config/bench/jobdefs/cint2006_pre-mustang.json | 92 ------ config/bench/jobdefs/template-uinstance.yaml | 109 +++++++ config/bench/jobdefs/template.yaml | 122 +++++++ config/bench/lava/host-session | 361 +++++++++++++++++---- ...ost-session.yaml => host-session-multilib.yaml} | 9 +- ...-session.yaml => host-session-no-multilib.yaml} | 11 +- config/bench/lava/host-session.yaml | 29 +- config/bench/lava/target-session | 31 +- config/bench/lava/target-session-mustang.yaml | 14 - ...-session-kvm.yaml => target-session-tools.yaml} | 6 +- config/bench/lava/target-session.yaml | 7 - config/fakebench.conf | 14 +- config/sources.conf | 1 + scripts/Benchmark.job | 113 ------- scripts/Benchmark.sh | 276 ++++++++++++++++ scripts/benchmark.sh | 162 +++++---- scripts/controlledrun.sh | 20 +- scripts/dispatch-benchmark.py | 213 ++++++++++++ scripts/runbenchmark.sh | 89 +++-- 56 files changed, 2538 insertions(+), 2004 deletions(-) delete mode 100644 bench/create-tables.sql delete mode 100755 bench/host2sql.sh delete mode 100755 bench/make-benchdata.sh delete mode 100644 bench/make-datafiles.sh delete mode 100755 bench/plot-bench.sh delete mode 100755 bench/run2sql.sh delete mode 100755 bench/tab2sql.sh create mode 100644 config/Coremark-Pro.conf delete mode 100644 config/EEMBC.conf delete mode 100644 config/bench/boards/arndale.services delete mode 100644 config/bench/boards/juno-a53.services delete mode 100644 config/bench/boards/juno-a57.services delete mode 100644 config/bench/boards/kvm.services delete mode 100644 config/bench/boards/localhost.services delete mode 100644 config/bench/boards/mustang.services delete mode 100644 config/bench/boards/panda-es.services create mode 120000 config/bench/gather/CPU2000.sh create mode 120000 config/bench/gather/CPU2006.sh create mode 100755 config/bench/gather/CPU200x.sh create mode 100755 config/bench/gather/Coremark-Pro.sh create mode 100755 config/bench/gather/fakebench.sh create mode 100755 config/bench/gather/test_CPU200x.sh create mode 100755 config/bench/gather/test_Coremark-Pro.sh delete mode 100644 config/bench/jobdefs/cfp2006_aarch32-juno_a53_x1-juno_a57_x1.json delete mode 100644 config/bench/jobdefs/cfp2006_aarch64-juno_a53_x1-juno_a57_x1.json delete mode 100644 config/bench/jobdefs/cfp2006_pre-mustang.json delete mode 100644 config/bench/jobdefs/cint2006-kvm_x1.json delete mode 100644 config/bench/jobdefs/cint2006_aarch32_juno_a53_x1-juno_a57_x1.json delete mode 100644 config/bench/jobdefs/cint2006_aarch64-juno_a53_x1-juno_a57_x1.json delete mode 100644 config/bench/jobdefs/cint2006_pre-mustang.json create mode 100644 config/bench/jobdefs/template-uinstance.yaml create mode 100644 config/bench/jobdefs/template.yaml copy config/bench/lava/{host-session.yaml => host-session-multilib.yaml} (78%) copy config/bench/lava/{host-session.yaml => host-session-no-multilib.yaml} (77%) delete mode 100644 config/bench/lava/target-session-mustang.yaml rename config/bench/lava/{target-session-kvm.yaml => target-session-tools.yaml} (81%) delete mode 100755 scripts/Benchmark.job create mode 100755 scripts/Benchmark.sh create mode 100755 scripts/dispatch-benchmark.py