In the email, I like the info that is presented now, but would just add a summary per suite-type (eg, rollup results for each of boot, vts, cts). For cts, I'd be fine with a rollup on a per abi basis or combining the abis, whatever is easier. Seeing the rolled-up pass/fail numbers helps give confidence that all the tests actually ran and the results are in line with expectations without having scroll down and add things up in my head.

On Thu, Feb 1, 2018 at 12:13 PM, Dan Rue <dan.rue@linaro.org> wrote:

We can format it however you'd like.

Other projects did not want to see their persistent failures, so they're
hidden unless they're "new", in which case they show up in the
regressions list.

So, I can add in the list of failures below the list of regressions, and
above the "Boards, architectures and test suites" section. Any other
changes you'd like to see?

Dan

On Thu, Feb 01, 2018 at 12:05:29PM -0800, Todd Kjos wrote:
> Yes, that's better, but its a pretty long list so its hard to quickly tell
> if there are a lot of failures. How hard would it be to have a summary
> statement for cts and vts that give the overall pass/fail (so we don't have
> to scan down the whole list of tests to get a feel for how many failures
> there are)?
>
> On Thu, Feb 1, 2018 at 9:06 AM, Tom Gall <tom.gall@linaro.org> wrote:
>
> > Is this in keeping what you were thinking Todd from your prior email?
> >
> > Tom
> >
> > > On Feb 1, 2018, at 11:04 AM, Linaro QA (beta) <qa-reports@linaro.org>
> > wrote:
> > >
> > > Summary
> > > ------------------------------------------------------------------------
> > >
> > > git repo: https://android.googlesource.com/kernel/hikey-linaro
> > > git branch: android-hikey-linaro-4.4
> > > git commit: a01f7678d0a99aadc2d76e1ab9d3bc692d3b7ad8
> > > git describe: v4.4.114-568662-ga01f7678d0a9
> > > Test details: https://qa-reports.linaro.org/android-lkft/android-hikey-
> > linaro-4.4-aosp/build/v4.4.114-568662-ga01f7678d0a9
> > >
> > > Regressions (compared to build v4.4.113-568657-g7af7b22459e6)
> > > ------------------------------------------------------------------------
> > >
> > > hi6220-hikey - arm64 (AOSP master):
> > > vts-test/arm64-v8a.VtsKernelLtp:
> > > * VtsKernelLtp.syscalls.nanosleep01_64bit
> > >
> > > * test src: not informed
> > >
> > >
> > > Boards, architectures and test suites:
> > > -------------------------------------
> > >
> > > hi6220-hikey - arm64 (AOSP master)
> > > * boot - pass: 6,
> > > * boottime-test - pass: 3,
> > > * cts-lkft-arm64-v8a - pass: 2,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsAslrMallocTestCases - pass: 1,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsBionicTestCases - pass: 2618,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsBluetoothTestCases - pass: 42, fail: 3
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsCameraTestCases - pass: 224,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsDisplayTestCases - pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsDrmTestCases - pass: 42,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsGraphicsTestCases - pass: 1553,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases - pass: 12,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsJankDeviceTestCases - pass: 1,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsJniTestCases - pass: 65,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreLegacy22TestCases - pass: 5,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases - pass: 17845, fail:
> > 2
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Africa - pass:
> > 54,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[America -
> > pass: 140,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[America/Argentina
> > - pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[America/Indiana
> > - pass: 8,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[America/Kentucky
> > - pass: 2,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[America/North_Dakota
> > - pass: 3,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Antarctica -
> > pass: 12,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Arctic - pass:
> > 1,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Asia - pass:
> > 98,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Atlantic -
> > pass: 12,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Australia -
> > pass: 23,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Brazil - pass:
> > 4,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Canada - pass:
> > 9,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Chile - pass:
> > 2,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Etc - pass: 35,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Europe - pass:
> > 63,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Indian - pass:
> > 11,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Mexico - pass:
> > 3,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[Pacific -
> > pass: 43,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[SystemV -
> > pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testAllTransitions[US - pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.

> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Africa
> > - pass: 54,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[America
> > - pass: 140,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[America/Argentina
> > - pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[America/Indiana
> > - pass: 8,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[America/Kentucky
> > - pass: 2,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[America/North_Dakota
> > - pass: 3,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Antarctica
> > - pass: 12,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Arctic
> > - pass: 1,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Asia
> > - pass: 98,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Atlantic
> > - pass: 12,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Australia
> > - pass: 23,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Brazil
> > - pass: 4,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Canada
> > - pass: 9,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Chile
> > - pass: 2,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Etc -
> > pass: 35,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Europe
> > - pass: 63,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Indian
> > - pass: 11,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Mexico
> > - pass: 3,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[Pacific
> > - pass: 43,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.
> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[SystemV
> > - pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases/libcore.

> > java.time.zone.IcuZoneRulesProviderTest.testTransitionsNearInstants[US -
> > pass: 13,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsMonkeyTestCases - pass: 10,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsOsTestCases - pass: 455, fail: 5
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsSystemUiTestCases - pass: 3,
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsUsbTests - fail: 1
> > > * cts-lkft-arm64-v8a/arm64-v8a.CtsWebkitTestCases - pass: 221,
> > > * vts-test - pass: 8,
> > > * vts-test/arm64-v8a.KernelProcFileApiTest - pass: 24,
> > > * vts-test/arm64-v8a.SyscallExistenceTest - pass: 3,
> > > * vts-test/arm64-v8a.VtsKernelApiSysfsTest - pass: 3,
> > > * vts-test/arm64-v8a.VtsKernelBinderTest - pass: 53,
> > > * vts-test/arm64-v8a.VtsKernelConfig - pass: 1,
> > > * vts-test/arm64-v8a.VtsKernelHwBinder - pass: 1,
> > > * vts-test/arm64-v8a.VtsKernelLibcutilsTest - pass: 6,
> > > * vts-test/arm64-v8a.VtsKernelLinuxKselftest - pass: 20, fail: 2
> > > * vts-test/arm64-v8a.VtsKernelLtp - pass: 773, fail: 2
> > > * vts-test/arm64-v8a.VtsKernelNetdTest - pass: 34,
> > > * vts-test/arm64-v8a.VtsKernelSelinuxFileApi - pass: 3,
> > > * vts-test/arm64-v8a.VtsKernelTunTest/VtsKernelTunTest.Basic/VtsKernelTunTest.BasicRead
> > - pass: 2,
> > > * vts-test/arm64-v8a.VtsKernelTunTest/VtsKernelTunTest.Basic/VtsKernelTunTest.OpenAndClose
> > - pass: 2,
> > > * vts-test/arm64-v8a.VtsQtaguidTest - pass: 20,
> > > * vts-test/armeabi-v7a.KernelProcFileApiTest - pass: 24,
> > > * vts-test/armeabi-v7a.SyscallExistenceTest - pass: 3,
> > > * vts-test/armeabi-v7a.VtsKernelApiSysfsTest - pass: 3,
> > > * vts-test/armeabi-v7a.VtsKernelBinderTest - pass: 53,
> > > * vts-test/armeabi-v7a.VtsKernelConfig - pass: 1,
> > > * vts-test/armeabi-v7a.VtsKernelHwBinder - pass: 1,
> > > * vts-test/armeabi-v7a.VtsKernelLibcutilsTest - pass: 6,
> > > * vts-test/armeabi-v7a.VtsKernelLinuxKselftest - pass: 19,
> > > * vts-test/armeabi-v7a.VtsKernelLtp - pass: 782, fail: 2
> > > * vts-test/armeabi-v7a.VtsKernelNetdTest - pass: 34,
> > > * vts-test/armeabi-v7a.VtsKernelSelinuxFileApi - pass: 3,
> > > * vts-test/armeabi-v7a.VtsKernelTunTest/VtsKernelTunTest.Basic/VtsKernelTunTest.BasicRead
> > - pass: 2,
> > > * vts-test/armeabi-v7a.VtsKernelTunTest/VtsKernelTunTest.Basic/VtsKernelTunTest.OpenAndClose
> > - pass: 2,
> > > * vts-test/armeabi-v7a.VtsQtaguidTest - pass: 20,
> > >
> > >
> > >
> > > Documentation - https://collaborate.linaro.
> > org/display/LKFT/Email+Reports
> > >
> > > --
> > > Linaro QA (beta)
> > > https://qa-reports.linaro.org
> > > _______________________________________________
> > > Lts-dev mailing list
> > > Lts-dev@lists.linaro.org
> > > https://lists.linaro.org/mailman/listinfo/lts-dev
> >
> >

> _______________________________________________
> Lkft-maintainers mailing list
> Lkft-maintainers@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lkft-maintainers