On 11/21/19 5:58 AM, Tim.Bird@sony.com wrote:
-----Original Message----- From: Cyril Hrubis
Hi!
One or more kernel tests failed:
ppc64le: ??? LTP lite ??? xfstests: ext4Both logs shows missing files, that may be an infrastructure problem as well.
Also can we include links to the logfiles here? Bonus points for showing the snippet with the actual failure in the email as well. I takes a fair amount of time locating them manually in the pipeline repository, it would be much much easier just with the links to the right logfile...
My preference would be to include the failure snippet somewhere in the e-mail as well (as opposed to just a link).
Thanks for the feedback Cyril, we did have links to each failure listed before but we were told it made the email look cluttered especially if there are multiple failures.
So it's exactly how Dmitry described it, you can't please everyone..,
The test logs are sorted by arch|host|TC, is there something we can do to make it easier to find related logs ? https://artifacts.cki-project.org/pipelines/296781/logs/
Maybe we can look into adding the linked logs to the bottom of the email with a reference id next to the failures in the summary, so for example:
ppc64le: ??? LTP lite [1] ??? xfstests: ext4 [2]That would work for me.
Maybe combine the 'footnote' idea with the 'inline' idea, and have the footnote include a link to the full log and a snippet with just the output from the failing testcase, from the full log?
We could also look into merging the ltp run logs into a single file as well.
That would make it too big I guess. Actually the only part I'm interested in most of the time is the part of the log with the failing test. I would be quite happy if we had logs/failures file on the pipelines sever that would contain only failures extracted from different logfiles. The question is if that's feasible with your framework.
Fuego has an LTP log-splitter and link generator. It's Fuego-specific and generates files referred to by links in the result tables that Fuego shows to users.
I don't know how CKI is generating it's data or storing it, but I can take a look and see if it could be applied to their use case. It's a python program that is fairly small.
There is a summary log which captures overall results: https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_l...
Then an individual log file for each LTP testsuite, e.g: https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_l...
See here: https://bitbucket.org/fuegotest/fuego-core/src/master/tests/Functional.LTP/p...
Thanks!
It might not be applicable, depending on whether CKI stores their LTP output similarly to how Fuego does, but IMHO it's worth taking a look. If there is sufficient interest, maybe this could be generalized and submitted to upstream LTP. The Fuego log-splitter produces individual files.
I think it's a good idea, as long as it can be generic enough where someone could modify a config file for example to indicate the log path and naming convention.
Another idea would be to write a program that takes an LTP log, and the name of a failing testcase, and outputs (on stdout) the snippet from the log for that testcase. I think this would be very easy to do, and might be suitable to use in multiple contexts: on the command line, in a report generator, or as a CGI script for a results server.
I logged a few tickets so our team can take a closer look and discuss for both failure snippets and linking LTP logs directly. I'm also checking to see if we have anything internally.
Thanks for all the feedback.
-- Tim