Oleksandr Terentiev oterenti@cisco.com writes:
Hi Abibal,
In our project we need to analyze a total number of passed and failed tests for each packet. To distinguish packets we use lava-test-set feature. In order to implement that I modified ptest.py and send-to-lava.sh scripts. Could you please look at the patch and express your opinion? Maybe this code can be added to git.linaro.org/qa/test-definitions.git ?
Best regards, Alex
automated/linux/ptest: Analyze each test in package tests
Currently ptest.py analyze only exit code of each package test to decide if it passed or not. However, ptest-runner can return success code even though some tests failed. So we need to parse test output and analyze it.
It also quite useful to see exactly which tests failed. So results are recorded for each particular test, and lava-test-set feature is used to distinguish packages.
Signed-off-by: Oleksandr Terentiev oterenti@cisco.com
I gave this a quick test and found a minor problem below...
diff --git a/automated/linux/ptest/ptest.py b/automated/linux/ptest/ptest.py index 13feb4d..a28d7f0 100755 --- a/automated/linux/ptest/ptest.py +++ b/automated/linux/ptest/ptest.py @@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests, exclude):
return filter_ptests
+def parse_line(line): + test_status_list = { + 'pass': re.compile("^PASS:(.+)"), + 'fail': re.compile("^FAIL:(.+)"), + 'skip': re.compile("^SKIP:(.+)") + }
+ for test_status, status_regex in test_status_list.items(): + test_name = status_regex.search(line) + if test_name: + return [test_name.group(1), test_status]
-def check_ptest(ptest_dir, ptest_name, output_log): - status = 'pass' + return None
- try: - output = subprocess.check_call('ptest-runner -d %s %s' % - (ptest_dir, ptest_name), shell=True, - stderr=subprocess.STDOUT) - except subprocess.CalledProcessError: - status = 'fail' +def parse_ptest(log_file): + result = []
- with open(output_log, 'a+') as f: - f.write("%s %s\n" % (ptest_name, status)) + with open(log_file, 'r') as f: + for line in f: + result_tuple = parse_line(line) + if not result_tuple: + continue + print(result_tuple) + result.append(result_tuple) + continue
+ return result
+def run_command(command, log_file): + process = subprocess.Popen(command, + shell=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + with open(log_file, 'w') as f: + while True: + output = process.stdout.readline() + if output == '' and process.poll() is not None: + break + if output: + print output.strip()
This causes a syntax error in python. Don't you need to wrap this in () ?
Kevin