Hi Abibal,

In our project we need to analyze a total number of passed and failed tests for each packet. To distinguish packets we use lava-test-set feature. In order to implement that I modified ptest.py and send-to-lava.sh scripts. Could you please look at the patch and express your opinion? Maybe this code can be added to git.linaro.org/qa/test-definitions.git ?

Best regards, Alex

automated/linux/ptest: Analyze each test in package tests

Currently ptest.py analyze only exit code of each package test to decide if it passed or not. However, ptest-runner can return success code even though some tests failed. So we need to parse test output and analyze it.

It also quite useful to see exactly which tests failed. So results are recorded for each particular test, and lava-test-set feature is used to distinguish packages.

Signed-off-by: Oleksandr Terentiev oterenti@cisco.com

diff --git a/automated/linux/ptest/ptest.py b/automated/linux/ptest/ptest.py index 13feb4d..a28d7f0 100755 --- a/automated/linux/ptest/ptest.py +++ b/automated/linux/ptest/ptest.py @@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests, exclude):

return filter_ptests

+def parse_line(line): +    test_status_list = { +        'pass': re.compile("^PASS:(.+)"), +        'fail': re.compile("^FAIL:(.+)"), +        'skip': re.compile("^SKIP:(.+)") +    } + +    for test_status, status_regex in test_status_list.items(): +            test_name = status_regex.search(line) +            if test_name: +                return [test_name.group(1), test_status]

-def check_ptest(ptest_dir, ptest_name, output_log): -    status = 'pass' +    return None

-    try: -        output = subprocess.check_call('ptest-runner -d %s %s' % -                                       (ptest_dir, ptest_name), shell=True, -                                       stderr=subprocess.STDOUT) -    except subprocess.CalledProcessError: -        status = 'fail' +def parse_ptest(log_file): +    result = []

-    with open(output_log, 'a+') as f: -        f.write("%s %s\n" % (ptest_name, status)) +    with open(log_file, 'r') as f: +        for line in f: +            result_tuple = parse_line(line) +            if not result_tuple: +                continue +            print(result_tuple) +            result.append(result_tuple) +            continue

+    return result + +def run_command(command, log_file): +    process = subprocess.Popen(command, +                               shell=True, +                               stdout=subprocess.PIPE, +                               stderr=subprocess.STDOUT) +    with open(log_file, 'w') as f: +        while True: +            output = process.stdout.readline() +            if output == '' and process.poll() is not None: +                break +            if output: +                print output.strip() +                f.write("%s\n" % output.strip()) +    rc = process.poll() +    return rc + +def check_ptest(ptest_dir, ptest_name, output_log): +    log_name = os.path.join(os.getcwd(), '%s.log' % ptest_name) +    status = run_command('ptest-runner -d %s %s' % (ptest_dir, ptest_name), log_name) + +    with open(output_log, 'a+') as f: +        f.write("lava-test-set start %s\n" % ptest_name) +        f.write("%s %s\n" % (ptest_name, "pass" if status == 0 else "fail")) +        for test, test_status in parse_ptest(log_name): +            f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test), test_status)) +        f.write("lava-test-set stop %s\n" % ptest_name)

def main():      parser = argparse.ArgumentParser(description="LAVA/OE ptest script", diff --git a/automated/utils/send-to-lava.sh b/automated/utils/send-to-lava.sh index bf2a477..db4442c 100755 --- a/automated/utils/send-to-lava.sh +++ b/automated/utils/send-to-lava.sh @@ -4,6 +4,8 @@ RESULT_FILE="$1"

which lava-test-case > /dev/null 2>&1  lava_test_case="$?" +which lava-test-set > /dev/null 2>&1 +lava_test_set="$?"

if [ -f "${RESULT_FILE}" ]; then      while read -r line; do @@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then              else                 echo "<TEST_CASE_ID=${test} RESULT=${result} MEASUREMENT=${measurement} UNITS=${units}>"              fi +        elif echo "${line}" | egrep -iq "^lava-test-set.*"; then +            test_set_status="$(echo "${line}" | awk '{print $2}')" +            test_set_name="$(echo "${line}" | awk '{print $3}')" +            if [ "${lava_test_set}" -eq 0 ]; then +                lava-test-set "${test_set_status}" "${test_set_name}" +            else +                if [ "${test_set_status}" == "start" ]; then +                    echo "<LAVA_SIGNAL_TESTSET START ${test_set_name}>" +                else +                    echo "<LAVA_SIGNAL_TESTSET STOP>" +                fi +            fi          fi      done < "${RESULT_FILE}"  else

On 01.10.18 17:09, Anibal Limon wrote:

Hi,

I was on vacation, that's the reason for the slow response, comments below,

On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev <oterenti@cisco.com mailto:oterenti@cisco.com> wrote:

Hi,

I would like to discuss the following question.
As it was said now we have to analyze pass/fail of every ptest.
From my point of view there are couple options.

The first we can parse output and mark ptest as failed if there is
even only one failed test found.

Right I will choice this approach changes needs to be done in the ptest lava script [1] to fail when any of the package tests failed like [2].

The second we can analyze each test within some packet and record
corresponding results.
I see a few issues here. First of all there will be a large number
of test results as each ptest can run lots of tests.

Right but that need to be handled in every OE ptest script, I mean if you want to fail if certain test inside a ptest fails needs to be done in OE.

Another thing is that we need somehow separate test results
between particular packets.

Currently we use QA reports to see only the package test result, now If you want to be able look at details you need to see the ptest log.

As an option we can use lava-test-set feature for that. So each
test within ptest will be marked as test case and packet name we
will see as test set.

What do you think about that?

May be the lava-test-set is an option.

I would go to do the 1st option and then start to review/implement the idea of use lava-test-feature. Regards, Anibal

[1] https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/pt... [2] http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cas...

Regards,
Alex

On 23.08.18 16:10, Anibal Limon wrote:

...

On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev
<oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

    Thank you Anibal for the fast response


    On 22.08.18 19:50, Anibal Limon wrote:

...

    On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev
    <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

        Hi,

        I launched util-linux ptest using
        automated/linux/ptest/ptest.yaml from
        https://git.linaro.org/qa/test-definitions.git and
        received the
        following results:
        https://pastebin.com/nj9PYQzE

        As you can see some tests failed. However, case
        util-linux marked as
        passed. It looks like ptest.py only analyze return code
        of ptest-runner
        -d <ptest_dir> <ptest_name> command. And since
        ptest-runner finishes
        correctly exit code is 0. Therefore all tests are always
        marked as
        passed, and users never know when some of the tests fail.

        Maybe it worth to analyze each test?


    Talking about each ptest the result comes from the ptest
    script in the OE recipe [1], for convention if the OE ptest
    returns 0 means pass, so
    needs to be fixed in the OE ptest [2].

    I’ve read https://wiki.yoctoproject.org/wiki/Ptest carefully
    a few times more. There are prescriptions about output
    format. But I didn’t find any mention about return code
    processing or a reference to the convention you mentioned in
    the answer.

    I looked through some OE run-ptest scripts. I suspect they
    don’t verify if some of their tests failed, and exit with 0
    even if all their tests failed.

    http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest



Right, looks that the OEQA test case was update since i worked on
it [1], so now it takes into account the pass/fail of every ptest.
So the ptest.py needs to implement the same behavior.

Regards,
Anibal

[1]
http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80

...

    Regarding the LAVA ptest.py script, I mark the run as
    succeed if there is no critical error in the ptest-runner
    and we have a QA-reports tool to analyse pass/fails
    in detail for every ptest executed [3].

        I heard about QA-reports tool but I’ve never used it
        before, so maybe I missed something.
        From
        https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/suite/linux-ptest/tests/
        I see all ptests passed. Still, in log
        https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/log
        I found 54 failed tests and wasn’t able to find a report
        which indicates those failures.

        Is there such a report? It would be really useful to know
        that some tests failed.

        Thanks

...

    [1]
    http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
    [2] https://wiki.yoctoproject.org/wiki/Ptest
    [3]
    https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/

    Regards,
    Anibal


        Best regards,
        Alex

Thanks for replay. Is there any public LAVA instance where I can run the example test? Maybe on staging.validation.linaro.org https://staging.validation.linaro.org/ ? Can I be registered there and how? Or maybe there is another approach to run that.

On 27.11.18 19:41, Anibal Limon wrote:

On Tue, 27 Nov 2018 at 08:29, Oleksandr Terentiev <oterenti@cisco.com mailto:oterenti@cisco.com> wrote:

Hi Abibal,

In our project we need to analyze a total number of passed and
failed tests for each packet. To distinguish packets we use
lava-test-set feature.
In order to implement that I modified ptest.py and send-to-lava.sh
scripts. Could you please look at the patch and express your opinion?
Maybe this code can be added to
git.linaro.org/qa/test-definitions.git
<http://git.linaro.org/qa/test-definitions.git> ?

Hi Oleksandr,

The code looks good, can you have an example of the LAVA test run to see the actual results?

Regards, Anibal

Best regards,
Alex

automated/linux/ptest: Analyze each test in package tests

Currently ptest.py analyze only exit code of each package test
to decide if it passed or not. However, ptest-runner can return
success code even though some tests failed. So we need to parse
test output and analyze it.

It also quite useful to see exactly which tests failed. So results
are
recorded for each particular test, and lava-test-set feature is used
to distinguish packages.

Signed-off-by: Oleksandr Terentiev <oterenti@cisco.com>
<mailto:oterenti@cisco.com>

diff --git a/automated/linux/ptest/ptest.py
b/automated/linux/ptest/ptest.py
index 13feb4d..a28d7f0 100755
--- a/automated/linux/ptest/ptest.py
+++ b/automated/linux/ptest/ptest.py
@@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests,
exclude):

     return filter_ptests

+def parse_line(line):
+    test_status_list = {
+        'pass': re.compile("^PASS:(.+)"),
+        'fail': re.compile("^FAIL:(.+)"),
+        'skip': re.compile("^SKIP:(.+)")
+    }
+
+    for test_status, status_regex in test_status_list.items():
+            test_name = status_regex.search(line)
+            if test_name:
+                return [test_name.group(1), test_status]

-def check_ptest(ptest_dir, ptest_name, output_log):
-    status = 'pass'
+    return None

-    try:
-        output = subprocess.check_call('ptest-runner -d %s %s' %
-                                       (ptest_dir, ptest_name),
shell=True,
- stderr=subprocess.STDOUT)
-    except subprocess.CalledProcessError:
-        status = 'fail'
+def parse_ptest(log_file):
+    result = []

-    with open(output_log, 'a+') as f:
-        f.write("%s %s\n" % (ptest_name, status))
+    with open(log_file, 'r') as f:
+        for line in f:
+            result_tuple = parse_line(line)
+            if not result_tuple:
+                continue
+            print(result_tuple)
+            result.append(result_tuple)
+            continue

+    return result
+
+def run_command(command, log_file):
+    process = subprocess.Popen(command,
+                               shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT)
+    with open(log_file, 'w') as f:
+        while True:
+            output = process.stdout.readline()
+            if output == '' and process.poll() is not None:
+                break
+            if output:
+                print output.strip()
+                f.write("%s\n" % output.strip())
+    rc = process.poll()
+    return rc
+
+def check_ptest(ptest_dir, ptest_name, output_log):
+    log_name = os.path.join(os.getcwd(), '%s.log' % ptest_name)
+    status = run_command('ptest-runner -d %s %s' % (ptest_dir,
ptest_name), log_name)
+
+    with open(output_log, 'a+') as f:
+        f.write("lava-test-set start %s\n" % ptest_name)
+        f.write("%s %s\n" % (ptest_name, "pass" if status == 0
else "fail"))
+        for test, test_status in parse_ptest(log_name):
+            f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test),
test_status))
+        f.write("lava-test-set stop %s\n" % ptest_name)

 def main():
     parser = argparse.ArgumentParser(description="LAVA/OE ptest
script",
diff --git a/automated/utils/send-to-lava.sh
b/automated/utils/send-to-lava.sh
index bf2a477..db4442c 100755
--- a/automated/utils/send-to-lava.sh
+++ b/automated/utils/send-to-lava.sh
@@ -4,6 +4,8 @@ RESULT_FILE="$1"

 which lava-test-case > /dev/null 2>&1
 lava_test_case="$?"
+which lava-test-set > /dev/null 2>&1
+lava_test_set="$?"

 if [ -f "${RESULT_FILE}" ]; then
     while read -r line; do
@@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then
             else
                echo "<TEST_CASE_ID=${test} RESULT=${result}
MEASUREMENT=${measurement} UNITS=${units}>"
             fi
+        elif echo "${line}" | egrep -iq "^lava-test-set.*"; then
+            test_set_status="$(echo "${line}" | awk '{print $2}')"
+            test_set_name="$(echo "${line}" | awk '{print $3}')"
+            if [ "${lava_test_set}" -eq 0 ]; then
+                lava-test-set "${test_set_status}"
"${test_set_name}"
+            else
+                if [ "${test_set_status}" == "start" ]; then
+                    echo "<LAVA_SIGNAL_TESTSET START
${test_set_name}>"
+                else
+                    echo "<LAVA_SIGNAL_TESTSET STOP>"
+                fi
+            fi
         fi
     done < "${RESULT_FILE}"
 else



On 01.10.18 17:09, Anibal Limon wrote:

...

Hi,

I was on vacation, that's the reason for the slow response,
comments below,

On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev
<oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

    Hi,

    I would like to discuss the following question.
    As it was said now we have to analyze pass/fail of
    every ptest. From my point of view there are couple options.

    The first we can parse output and mark ptest as failed if
    there is even only one failed test found.


Right I will choice this approach changes needs to be done in the
ptest lava script [1] to fail when any of the package tests
failed like [2].



    The second we can analyze each test within some packet and
    record corresponding results.
    I see a few issues here. First of all there will be a large
    number of test results as each ptest can run lots of tests.


Right but that need to be handled in every OE ptest script, I
mean if you want to fail if certain test inside a ptest fails
needs to be done in OE.


    Another thing is that we need somehow separate test results
    between particular packets.


Currently we use QA reports to see only the package test result,
now If you want to be able look at details you need to see the
ptest log.

    As an option we can use lava-test-set feature for that. So
    each test within ptest will be marked as test case and packet
    name we will see as test set.

    What do you think about that?


May be the lava-test-set is an option.

I would go to do the 1st option and then start to
review/implement the idea of use lava-test-feature.
Regards,
Anibal

[1]
https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/ptest.py
[2]
http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n87

    Regards,
    Alex

    On 23.08.18 16:10, Anibal Limon wrote:

...

    On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev
    <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

        Thank you Anibal for the fast response


        On 22.08.18 19:50, Anibal Limon wrote:

...

        On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev
        <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

            Hi,

            I launched util-linux ptest using
            automated/linux/ptest/ptest.yaml from
            https://git.linaro.org/qa/test-definitions.git and
            received the
            following results:
            https://pastebin.com/nj9PYQzE

            As you can see some tests failed. However, case
            util-linux marked as
            passed. It looks like ptest.py only analyze return
            code of ptest-runner
            -d <ptest_dir> <ptest_name> command. And since
            ptest-runner finishes
            correctly exit code is 0. Therefore all tests are
            always marked as
            passed, and users never know when some of the tests
            fail.

            Maybe it worth to analyze each test?


        Talking about each ptest the result comes from the
        ptest script in the OE recipe [1], for convention if
        the OE ptest returns 0 means pass, so
        needs to be fixed in the OE ptest [2].

        I’ve read https://wiki.yoctoproject.org/wiki/Ptest
        carefully a few times more. There are prescriptions
        about output format. But I didn’t find any mention about
        return code processing or a reference to the convention
        you mentioned in the answer.

        I looked through some OE run-ptest scripts. I suspect
        they don’t verify if some of their tests failed, and
        exit with 0 even if all their tests failed.

        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest



    Right, looks that the OEQA test case was update since i
    worked on it [1], so now it takes into account the pass/fail
    of every ptest.
    So the ptest.py needs to implement the same behavior.

    Regards,
    Anibal

    [1]
    http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80

...

        Regarding the LAVA ptest.py script, I mark the run as
        succeed if there is no critical error in the
        ptest-runner and we have a QA-reports tool to analyse
        pass/fails
        in detail for every ptest executed [3].

            I heard about QA-reports tool but I’ve never used it
            before, so maybe I missed something.
            From
            https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/suite/linux-ptest/tests/
            I see all ptests passed. Still, in log
            https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/log
            I found 54 failed tests and wasn’t able to find a
            report which indicates those failures.

            Is there such a report? It would be really useful to
            know that some tests failed.

            Thanks

...

        [1]
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
        [2] https://wiki.yoctoproject.org/wiki/Ptest
        [3]
        https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/

        Regards,
        Anibal


            Best regards,
            Alex

On Tue, 27 Nov 2018 20:22:42 +0200 Oleksandr Terentiev oterenti@cisco.com wrote:

Thanks for replay. Is there any public LAVA instance where I can run the example test? Maybe on staging.validation.linaro.org https://staging.validation.linaro.org/ ? Can I be registered there and how?

Staging (and other instances hosted by Linaro) share the same authentication support as validation.linaro.org, which is described in the LAVA documentation.

https://staging.validation.linaro.org/static/docs/v2/first_steps.html#linaro...

If you do not have a Linaro LDAP account, you can register at https://register.linaro.org/.

You could use validation.linaro.org for your test.

Or maybe there is another approach to run that.

Once the script is available in a public git repository instead of only as a patch, it can be executed from any suitable LAVA test job. So push the script and associated support to public git somewhere and post the URL here. Someone on this list can put that into a test job and run that. If you have a test job definition ready to go, put that alongside the script and test definition.

Any public git repo will be fine, it doesn't have to be the original one on git.linaro.org.

On 27.11.18 19:41, Anibal Limon wrote:

...

On Tue, 27 Nov 2018 at 08:29, Oleksandr Terentiev <oterenti@cisco.com mailto:oterenti@cisco.com> wrote:

Hi Abibal,

In our project we need to analyze a total number of passed and
failed tests for each packet. To distinguish packets we use
lava-test-set feature.
In order to implement that I modified ptest.py and

send-to-lava.sh scripts. Could you please look at the patch and express your opinion? Maybe this code can be added to git.linaro.org/qa/test-definitions.git http://git.linaro.org/qa/test-definitions.git ?

Hi Oleksandr,

The code looks good, can you have an example of the LAVA test run to see the actual results?

Regards, Anibal

Best regards,
Alex

automated/linux/ptest: Analyze each test in package tests

Currently ptest.py analyze only exit code of each package test
to decide if it passed or not. However, ptest-runner can return
success code even though some tests failed. So we need to parse
test output and analyze it.

It also quite useful to see exactly which tests failed. So

results are recorded for each particular test, and lava-test-set feature is used to distinguish packages.

Signed-off-by: Oleksandr Terentiev <oterenti@cisco.com>
<mailto:oterenti@cisco.com>

diff --git a/automated/linux/ptest/ptest.py
b/automated/linux/ptest/ptest.py
index 13feb4d..a28d7f0 100755
--- a/automated/linux/ptest/ptest.py
+++ b/automated/linux/ptest/ptest.py
@@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests,
exclude):

     return filter_ptests

+def parse_line(line):
+    test_status_list = {
+        'pass': re.compile("^PASS:(.+)"),
+        'fail': re.compile("^FAIL:(.+)"),
+        'skip': re.compile("^SKIP:(.+)")
+    }
+
+    for test_status, status_regex in test_status_list.items():
+            test_name = status_regex.search(line)
+            if test_name:
+                return [test_name.group(1), test_status]

-def check_ptest(ptest_dir, ptest_name, output_log):
-    status = 'pass'
+    return None

-    try:
-        output = subprocess.check_call('ptest-runner -d %s %s'

% -                                       (ptest_dir, ptest_name), shell=True, - stderr=subprocess.STDOUT) -    except subprocess.CalledProcessError: -        status = 'fail' +def parse_ptest(log_file): +    result = []

-    with open(output_log, 'a+') as f:
-        f.write("%s %s\n" % (ptest_name, status))
+    with open(log_file, 'r') as f:
+        for line in f:
+            result_tuple = parse_line(line)
+            if not result_tuple:
+                continue
+            print(result_tuple)
+            result.append(result_tuple)
+            continue

+    return result
+
+def run_command(command, log_file):
+    process = subprocess.Popen(command,
+                               shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT)
+    with open(log_file, 'w') as f:
+        while True:
+            output = process.stdout.readline()
+            if output == '' and process.poll() is not None:
+                break
+            if output:
+                print output.strip()
+                f.write("%s\n" % output.strip())
+    rc = process.poll()
+    return rc
+
+def check_ptest(ptest_dir, ptest_name, output_log):
+    log_name = os.path.join(os.getcwd(), '%s.log' % ptest_name)
+    status = run_command('ptest-runner -d %s %s' % (ptest_dir,
ptest_name), log_name)
+
+    with open(output_log, 'a+') as f:
+        f.write("lava-test-set start %s\n" % ptest_name)
+        f.write("%s %s\n" % (ptest_name, "pass" if status == 0
else "fail"))
+        for test, test_status in parse_ptest(log_name):
+            f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test),
test_status))
+        f.write("lava-test-set stop %s\n" % ptest_name)

 def main():
     parser = argparse.ArgumentParser(description="LAVA/OE ptest
script",
diff --git a/automated/utils/send-to-lava.sh
b/automated/utils/send-to-lava.sh
index bf2a477..db4442c 100755
--- a/automated/utils/send-to-lava.sh
+++ b/automated/utils/send-to-lava.sh
@@ -4,6 +4,8 @@ RESULT_FILE="$1"

 which lava-test-case > /dev/null 2>&1
 lava_test_case="$?"
+which lava-test-set > /dev/null 2>&1
+lava_test_set="$?"

 if [ -f "${RESULT_FILE}" ]; then
     while read -r line; do
@@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then
             else
                echo "<TEST_CASE_ID=${test} RESULT=${result}
MEASUREMENT=${measurement} UNITS=${units}>"
             fi
+        elif echo "${line}" | egrep -iq "^lava-test-set.*";

then +            test_set_status="$(echo "${line}" | awk '{print $2}')" +            test_set_name="$(echo "${line}" | awk '{print $3}')" +            if [ "${lava_test_set}" -eq 0 ]; then +                lava-test-set "${test_set_status}" "${test_set_name}" +            else +                if [ "${test_set_status}" == "start" ]; then +                    echo "<LAVA_SIGNAL_TESTSET START ${test_set_name}>" +                else +                    echo "<LAVA_SIGNAL_TESTSET STOP>" +                fi +            fi          fi      done < "${RESULT_FILE}"  else

On 01.10.18 17:09, Anibal Limon wrote:  

...

Hi,

I was on vacation, that's the reason for the slow response,
comments below,

On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev
<oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

    Hi,

    I would like to discuss the following question.
    As it was said now we have to analyze pass/fail of
    every ptest. From my point of view there are couple

options.

    The first we can parse output and mark ptest as failed if
    there is even only one failed test found.


Right I will choice this approach changes needs to be done in

the ptest lava script [1] to fail when any of the package tests failed like [2].

    The second we can analyze each test within some packet and
    record corresponding results.
    I see a few issues here. First of all there will be a large
    number of test results as each ptest can run lots of tests.


Right but that need to be handled in every OE ptest script, I
mean if you want to fail if certain test inside a ptest fails
needs to be done in OE.


    Another thing is that we need somehow separate test results
    between particular packets.


Currently we use QA reports to see only the package test

result, now If you want to be able look at details you need to see the ptest log.

    As an option we can use lava-test-set feature for that. So
    each test within ptest will be marked as test case and

packet name we will see as test set.

    What do you think about that?


May be the lava-test-set is an option.

I would go to do the 1st option and then start to
review/implement the idea of use lava-test-feature.
Regards,
Anibal

[1]
https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/ptest.py
[2]
http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n87

    Regards,
    Alex

    On 23.08.18 16:10, Anibal Limon wrote:  

...

    On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev
    <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

        Thank you Anibal for the fast response


        On 22.08.18 19:50, Anibal Limon wrote:  

...

        On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev
        <oterenti@cisco.com <mailto:oterenti@cisco.com>>

wrote:

            Hi,

            I launched util-linux ptest using
            automated/linux/ptest/ptest.yaml from
            https://git.linaro.org/qa/test-definitions.git

and received the following results: https://pastebin.com/nj9PYQzE

            As you can see some tests failed. However, case
            util-linux marked as
            passed. It looks like ptest.py only analyze

return code of ptest-runner -d <ptest_dir> <ptest_name> command. And since ptest-runner finishes correctly exit code is 0. Therefore all tests are always marked as passed, and users never know when some of the tests fail.

            Maybe it worth to analyze each test?


        Talking about each ptest the result comes from the
        ptest script in the OE recipe [1], for convention if
        the OE ptest returns 0 means pass, so
        needs to be fixed in the OE ptest [2].  

        I’ve read https://wiki.yoctoproject.org/wiki/Ptest
        carefully a few times more. There are prescriptions
        about output format. But I didn’t find any mention

about return code processing or a reference to the convention you mentioned in the answer.

        I looked through some OE run-ptest scripts. I suspect
        they don’t verify if some of their tests failed, and
        exit with 0 even if all their tests failed.

        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest



    Right, looks that the OEQA test case was update since i
    worked on it [1], so now it takes into account the

pass/fail of every ptest. So the ptest.py needs to implement the same behavior.

    Regards,
    Anibal

    [1]
    http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80

...

        Regarding the LAVA ptest.py script, I mark the run as
        succeed if there is no critical error in the
        ptest-runner and we have a QA-reports tool to analyse
        pass/fails
        in detail for every ptest executed [3].  

            I heard about QA-reports tool but I’ve never used

it before, so maybe I missed something. From https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I see all ptests passed. Still, in log https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I found 54 failed tests and wasn’t able to find a report which indicates those failures.

            Is there such a report? It would be really useful

to know that some tests failed.

            Thanks

...

        [1]
        http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
        [2] https://wiki.yoctoproject.org/wiki/Ptest
        [3]
        https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/

        Regards,
        Anibal


            Best regards,
            Alex

On Wed, 28 Nov 2018 18:26:34 +0200 Oleksandr Terentiev oterenti@cisco.com wrote:

Hi,

I wasn't able to sign up at https://register.linaro.org/ with the following message: "Linaro employees and assignees, and people from Member companies cannot use this form"

In which case, you should be able to sign in to validation.linaro.org yourself using your Linaro LDAP details. If your linaro email address is oleksandr.terentiev@linaro.org, your LDAP username to use with LAVA is oleksandr.terentiev and then your LDAP password. Otherwise, contact Linaro support / your Tech Lead in Linaro.

Once logged in, just need to ask an admin to give you submission rights. (JIRA ticket is easiest: https://projects.linaro.org/secure/CreateIssue%21default.jspa and select LAB & System Software (LSS) - remember to specify validation.linaro.org as the server you want to access.

So I uploaded my changes to GitHub: https://github.com/oterenti/test-definitions/tree/ptest_modify

If https://validation.linaro.org/scheduler/job/1890442 can be good example of ptest run I'd suggest the following test action:

• test:

namespace: dragonboard-820c     name: qcomlt-ptest     timeout:       minutes: 160     definitions:     - repository: https://github.com/oterenti/test-definitions.git       from: git       path: automated/linux/ptest/ptest.yaml       name: linux-ptest       branch: ptest_modify       params:         EXCLUDE: bluez5 libxml2 parted python strace

I tried a couple of test jobs but the first failed trying to fastboot flash and another kernel panicked: https://lkft-staging.validation.linaro.org/scheduler/job/18795

See if this one completes - it looks hung to me: https://validation.linaro.org/scheduler/job/1899004

+ ./ptest.py -o ./result.txt -t -e bluez5 libxml2 parted python strace b'1+0 records in' b'1+0 records out' b'1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405323 s, 259 MB/s' b'mke2fs 1.43.8 (1-Jan-2018)' b'' b'Filesystem too small for a journal' [ 19.740484] sdd: sdd1 sdd2 sdd3 b'Makefile:20: ../include/builddefs: No such file or directory' b"make: *** No rule to make target '../include/builddefs'." b"make: Failed to remake makefile '../include/builddefs'."

Regards, Alex

On 28.11.18 10:38, Neil Williams wrote:

...

On Tue, 27 Nov 2018 20:22:42 +0200 Oleksandr Terentiev oterenti@cisco.com wrote:

...

Thanks for replay. Is there any public LAVA instance where I can run the example test? Maybe on staging.validation.linaro.org https://staging.validation.linaro.org/ ? Can I be registered there and how?

Staging (and other instances hosted by Linaro) share the same authentication support as validation.linaro.org, which is described in the LAVA documentation.

https://staging.validation.linaro.org/static/docs/v2/first_steps.html#linaro...

If you do not have a Linaro LDAP account, you can register at https://register.linaro.org/.

You could use validation.linaro.org for your test.

...

Or maybe there is another approach to run that.

Once the script is available in a public git repository instead of only as a patch, it can be executed from any suitable LAVA test job. So push the script and associated support to public git somewhere and post the URL here. Someone on this list can put that into a test job and run that. If you have a test job definition ready to go, put that alongside the script and test definition.

Any public git repo will be fine, it doesn't have to be the original one on git.linaro.org.

...

On 27.11.18 19:41, Anibal Limon wrote:

...

On Tue, 27 Nov 2018 at 08:29, Oleksandr Terentiev <oterenti@cisco.com mailto:oterenti@cisco.com> wrote:

 Hi Abibal,

 In our project we need to analyze a total number of passed

and failed tests for each packet. To distinguish packets we use lava-test-set feature. In order to implement that I modified ptest.py and send-to-lava.sh scripts. Could you please look at the patch and express your opinion? Maybe this code can be added to git.linaro.org/qa/test-definitions.git http://git.linaro.org/qa/test-definitions.git ?

Hi Oleksandr,

The code looks good, can you have an example of the LAVA test run to see the actual results?

Regards, Anibal

 Best regards,
 Alex

 automated/linux/ptest: Analyze each test in package tests

 Currently ptest.py analyze only exit code of each package

test to decide if it passed or not. However, ptest-runner can return success code even though some tests failed. So we need to parse test output and analyze it.

 It also quite useful to see exactly which tests failed. So

results are recorded for each particular test, and lava-test-set feature is used to distinguish packages.

 Signed-off-by: Oleksandr Terentiev <oterenti@cisco.com>
 <mailto:oterenti@cisco.com>

 diff --git a/automated/linux/ptest/ptest.py
 b/automated/linux/ptest/ptest.py
 index 13feb4d..a28d7f0 100755
 --- a/automated/linux/ptest/ptest.py
 +++ b/automated/linux/ptest/ptest.py
 @@ -84,20 +84,60 @@ def filter_ptests(ptests,

requested_ptests, exclude):

      return filter_ptests

 +def parse_line(line):
 +    test_status_list = {
 +        'pass': re.compile("^PASS:(.+)"),
 +        'fail': re.compile("^FAIL:(.+)"),
 +        'skip': re.compile("^SKIP:(.+)")
 +    }
 +
 +    for test_status, status_regex in

test_status_list.items(): +            test_name = status_regex.search(line) +            if test_name: +                return [test_name.group(1), test_status]

 -def check_ptest(ptest_dir, ptest_name, output_log):
 -    status = 'pass'
 +    return None

 -    try:
 -        output = subprocess.check_call('ptest-runner -d %s

%s' % -                                       (ptest_dir, ptest_name), shell=True, - stderr=subprocess.STDOUT) -    except subprocess.CalledProcessError: -        status = 'fail' +def parse_ptest(log_file): +    result = []

 -    with open(output_log, 'a+') as f:
 -        f.write("%s %s\n" % (ptest_name, status))
 +    with open(log_file, 'r') as f:
 +        for line in f:
 +            result_tuple = parse_line(line)
 +            if not result_tuple:
 +                continue
 +            print(result_tuple)
 +            result.append(result_tuple)
 +            continue

 +    return result
 +
 +def run_command(command, log_file):
 +    process = subprocess.Popen(command,
 +                               shell=True,
 + stdout=subprocess.PIPE,
 + stderr=subprocess.STDOUT)
 +    with open(log_file, 'w') as f:
 +        while True:
 +            output = process.stdout.readline()
 +            if output == '' and process.poll() is not None:
 +                break
 +            if output:
 +                print output.strip()
 +                f.write("%s\n" % output.strip())
 +    rc = process.poll()
 +    return rc
 +
 +def check_ptest(ptest_dir, ptest_name, output_log):
 +    log_name = os.path.join(os.getcwd(), '%s.log' %

ptest_name) +    status = run_command('ptest-runner -d %s %s' % (ptest_dir, ptest_name), log_name) + +    with open(output_log, 'a+') as f: +        f.write("lava-test-set start %s\n" % ptest_name) +        f.write("%s %s\n" % (ptest_name, "pass" if status == 0 else "fail")) +        for test, test_status in parse_ptest(log_name): +            f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test), test_status)) +        f.write("lava-test-set stop %s\n" % ptest_name)

  def main():
      parser = argparse.ArgumentParser(description="LAVA/OE

ptest script", diff --git a/automated/utils/send-to-lava.sh b/automated/utils/send-to-lava.sh index bf2a477..db4442c 100755 --- a/automated/utils/send-to-lava.sh +++ b/automated/utils/send-to-lava.sh @@ -4,6 +4,8 @@ RESULT_FILE="$1"

  which lava-test-case > /dev/null 2>&1
  lava_test_case="$?"
 +which lava-test-set > /dev/null 2>&1
 +lava_test_set="$?"

  if [ -f "${RESULT_FILE}" ]; then
      while read -r line; do
 @@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then
              else
                 echo "<TEST_CASE_ID=${test} RESULT=${result}
 MEASUREMENT=${measurement} UNITS=${units}>"
              fi
 +        elif echo "${line}" | egrep -iq "^lava-test-set.*";

then +            test_set_status="$(echo "${line}" | awk '{print $2}')" +            test_set_name="$(echo "${line}" | awk '{print $3}')" +            if [ "${lava_test_set}" -eq 0 ]; then +                lava-test-set "${test_set_status}" "${test_set_name}" +            else +                if [ "${test_set_status}" == "start" ]; then +                    echo "<LAVA_SIGNAL_TESTSET START ${test_set_name}>" +                else +                    echo "<LAVA_SIGNAL_TESTSET STOP>" +                fi +            fi          fi      done < "${RESULT_FILE}"  else

 On 01.10.18 17:09, Anibal Limon wrote:  

...

 Hi,

 I was on vacation, that's the reason for the slow response,
 comments below,

 On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev
 <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

     Hi,

     I would like to discuss the following question.
     As it was said now we have to analyze pass/fail of
     every ptest. From my point of view there are couple

options.

     The first we can parse output and mark ptest as failed

if there is even only one failed test found.

 Right I will choice this approach changes needs to be done

in the ptest lava script [1] to fail when any of the package tests failed like [2].

     The second we can analyze each test within some packet

and record corresponding results. I see a few issues here. First of all there will be a large number of test results as each ptest can run lots of tests.

 Right but that need to be handled in every OE ptest script,

I mean if you want to fail if certain test inside a ptest fails needs to be done in OE.

     Another thing is that we need somehow separate test

results between particular packets.

 Currently we use QA reports to see only the package test

result, now If you want to be able look at details you need to see the ptest log.

     As an option we can use lava-test-set feature for that.

So each test within ptest will be marked as test case and packet name we will see as test set.

     What do you think about that?


 May be the lava-test-set is an option.

 I would go to do the 1st option and then start to
 review/implement the idea of use lava-test-feature.
 Regards,
 Anibal

 [1]
 https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/ptest.py
 [2]
 http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n87

     Regards,
     Alex

     On 23.08.18 16:10, Anibal Limon wrote:  

...

     On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev
     <oterenti@cisco.com <mailto:oterenti@cisco.com>> wrote:

         Thank you Anibal for the fast response


         On 22.08.18 19:50, Anibal Limon wrote:  

> > On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev > <oterenti@cisco.com mailto:oterenti@cisco.com> > wrote: > > Hi, > > I launched util-linux ptest using > automated/linux/ptest/ptest.yaml from > https://git.linaro.org/qa/test-definitions.git > and received the > following results: > https://pastebin.com/nj9PYQzE > > As you can see some tests failed. However, > case util-linux marked as > passed. It looks like ptest.py only analyze > return code of ptest-runner > -d <ptest_dir> <ptest_name> command. And since > ptest-runner finishes > correctly exit code is 0. Therefore all tests > are always marked as > passed, and users never know when some of the > tests fail. > > Maybe it worth to analyze each test? > > > Talking about each ptest the result comes from the > ptest script in the OE recipe [1], for convention > if the OE ptest returns 0 means pass, so > needs to be fixed in the OE ptest [2]. I’ve read https://wiki.yoctoproject.org/wiki/Ptest carefully a few times more. There are prescriptions about output format. But I didn’t find any mention about return code processing or a reference to the convention you mentioned in the answer.

         I looked through some OE run-ptest scripts. I

suspect they don’t verify if some of their tests failed, and exit with 0 even if all their tests failed.

         http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
         http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest
         http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest
         http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest
         http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
         http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest



     Right, looks that the OEQA test case was update since i
     worked on it [1], so now it takes into account the

pass/fail of every ptest. So the ptest.py needs to implement the same behavior.

     Regards,
     Anibal

     [1]
     http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80

> Regarding the LAVA ptest.py script, I mark the > run as succeed if there is no critical error in the > ptest-runner and we have a QA-reports tool to > analyse pass/fails > in detail for every ptest executed [3]. I heard about QA-reports tool but I’ve never used it before, so maybe I missed something. From https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I see all ptests passed. Still, in log https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I found 54 failed tests and wasn’t able to find a report which indicates those failures.

             Is there such a report? It would be really

useful to know that some tests failed.

             Thanks

> [1] > http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-li... > [2] https://wiki.yoctoproject.org/wiki/Ptest > [3] > https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... > > Regards, > Anibal > > > Best regards, > Alex >