On 5 February 2018 at 20:38, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:

On 3 February 2018 at 04:43, Dan Rue dan.rue@linaro.org wrote:

...

Looking through these kselftest failures. The large quantity were because the first run went without a skipfile. The second run still had a few failures that I was surprised by:

all architectures bpf/test_align bpf/test_verifier

x86 x86/ldt_gdt_64

The two bpf tests are passing on mainline and recently removed from our skiplist. The patches made 4.15, so I was surprised to see it failing. I believe the issue is that the test source was also updated (see i.e. 'selftests/bpf: fix test_align'), but we are running with an older version of kselftest on 4.15.

Right ! I have tested with no skipfile + default-in-kernel The above reported three tests PASSED.

...

I suspect the problem is the same with x86/ldt_gdt_64; mainline runs with mainline kselftest, but 4.15 is running with 4.14 selftest. So when the test gets fixed on mainline, mainline gets fixed, but then we see the regressions again on 4.15 when it's released.

Can someone remind me again why we run mismatched kselftest? It seems to cause more confusion than it is worth.

to change only one variable at a time. If kselftests were passing 100% of the time with matching kernel I would go for this setup. Right now it's not the case so I think we should stick to stable version of kselftests (+patches when tests/fixes are backported)

...

These issues are also difficult to explain on the mailing lists, and if any enterprising engineer were to try to investigate for themselves, they would be quite surprised.

I agree. However I think this is results from kernel testing policy (or lack of it).

...

I know we did it for good reasons, I just think we need to re-evaluate periodically to be sure it still makes sense.

When we get 100% of executed kselftests passing on matching kernel for all arches, I'm OK to switch to this version. LTSes will not get new tests added though, so there is a trade off.

If we want to test 4.15 kselftest on 4.15 we can change single variable. KSELFTEST_PATH: /opt/kselftests/default-in-kernel/

This is like mainline and next builds use default-in-kernel sources. default-in-kernel sources could changes for every build. We might end up with kselftest build issues and which varies the number of results in reports.

So lets switch kselftest sources to 4.15 (/opt/kselftests/mainline/ == 4.15) Anders, Any blockers here ? to upgrade to 4.15 ?

I have updated skipfiles ready for 4.15. https://review.linaro.org/#/c/23708/ https://review.linaro.org/#/c/23709/

- Naresh

milosz

...

Dan

On Sat, Feb 03, 2018 at 03:52:10AM +0000, Linaro QA wrote:

...

Summary

kernel: 4.15.1-rc1 git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git git branch: linux-4.15.y git commit: b01b3d9519f250398695c7cc6493ba1e8fb072f4 git describe: v4.15-56-gb01b3d9519f2 Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.15-oe/build/v4.15-56-gb...

No regressions (compared to build )

Boards, architectures and test suites:

hi6220-hikey - arm64

• boot - pass: 38,
• kselftest - pass: 98, skip: 14, fail: 12
• libhugetlbfs - pass: 180, skip: 2,
• ltp-cap_bounds-tests - pass: 4,
• ltp-containers-tests - pass: 128,
• ltp-fcntl-locktests-tests - pass: 4,
• ltp-filecaps-tests - pass: 4,
• ltp-fs-tests - pass: 120,
• ltp-fs_bind-tests - pass: 4,
• ltp-fs_perms_simple-tests - pass: 38,
• ltp-fsx-tests - pass: 4,
• ltp-hugetlb-tests - pass: 21, skip: 1,
• ltp-io-tests - pass: 6,
• ltp-ipc-tests - pass: 18,
• ltp-math-tests - pass: 22,
• ltp-nptl-tests - pass: 4,
• ltp-pty-tests - pass: 4,
• ltp-sched-tests - pass: 20,
• ltp-securebits-tests - pass: 8,
• ltp-syscalls-tests - pass: 1968, skip: 242,
• ltp-timers-tests - pass: 24,

juno-r2 - arm64

• boot - pass: 31,
• kselftest - pass: 111, skip: 28, fail: 12
• libhugetlbfs - pass: 90, skip: 1,
• ltp-cap_bounds-tests - pass: 4,
• ltp-containers-tests - pass: 128,
• ltp-fcntl-locktests-tests - pass: 2,
• ltp-filecaps-tests - pass: 4,
• ltp-fs-tests - pass: 120,
• ltp-fs_bind-tests - pass: 2,
• ltp-fs_perms_simple-tests - pass: 38,
• ltp-fsx-tests - pass: 2,
• ltp-hugetlb-tests - pass: 44,
• ltp-io-tests - pass: 3,
• ltp-ipc-tests - pass: 18,
• ltp-math-tests - pass: 11,
• ltp-nptl-tests - pass: 4,
• ltp-pty-tests - pass: 4,
• ltp-sched-tests - pass: 10,
• ltp-securebits-tests - pass: 4,
• ltp-syscalls-tests - pass: 987, skip: 121,
• ltp-timers-tests - pass: 24,

x15 - arm

• boot - pass: 41,
• kselftest - pass: 92, skip: 32, fail: 15
• libhugetlbfs - pass: 174, skip: 2,
• ltp-cap_bounds-tests - pass: 4,
• ltp-containers-tests - pass: 124, fail: 4
• ltp-fcntl-locktests-tests - pass: 4,
• ltp-filecaps-tests - pass: 4,
• ltp-fs-tests - pass: 120,
• ltp-fs_bind-tests - pass: 4,
• ltp-fs_perms_simple-tests - pass: 38,
• ltp-fsx-tests - pass: 4,
• ltp-hugetlb-tests - pass: 40, skip: 4,
• ltp-io-tests - pass: 6,
• ltp-ipc-tests - pass: 18,
• ltp-math-tests - pass: 22,
• ltp-nptl-tests - pass: 4,
• ltp-pty-tests - pass: 8,
• ltp-sched-tests - pass: 26, skip: 2,
• ltp-securebits-tests - pass: 8,
• ltp-syscalls-tests - pass: 2076, skip: 132,
• ltp-timers-tests - pass: 24,

x86_64

• boot - pass: 40,
• kselftest - pass: 121, skip: 16, fail: 14
• libhugetlbfs - pass: 180, skip: 2,
• ltp-cap_bounds-tests - pass: 4,
• ltp-containers-tests - pass: 128,
• ltp-fcntl-locktests-tests - pass: 4,
• ltp-filecaps-tests - pass: 4,
• ltp-fs-tests - pass: 122, skip: 2,
• ltp-fs_bind-tests - pass: 4,
• ltp-fs_perms_simple-tests - pass: 38,
• ltp-fsx-tests - pass: 4,
• ltp-hugetlb-tests - pass: 44,
• ltp-io-tests - pass: 6,
• ltp-ipc-tests - pass: 18,
• ltp-math-tests - pass: 22,
• ltp-nptl-tests - pass: 4,
• ltp-pty-tests - pass: 8,
• ltp-sched-tests - pass: 18, skip: 2,
• ltp-securebits-tests - pass: 8,
• ltp-syscalls-tests - pass: 2032, skip: 232,
• ltp-timers-tests - pass: 24,

-- Linaro QA (beta) https://qa-reports.linaro.org _______________________________________________ Lkft-triage mailing list Lkft-triage@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lkft-triage