Summary ------------------------------------------------------------------------
kernel: 4.4.147-rc1 git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git git branch: linux-4.4.y git commit: db3e08ea00d093d99d1ccbd3cf069db67967de00 git describe: v4.4.146-12-gdb3e08ea00d0 Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/build/v4.4.146-12-...
Regressions (compared to build v4.4.146) ------------------------------------------------------------------------
qemu_arm: ltp-fs-tests: * read_all_dev * runltp_fs
* test src: git://github.com/linux-test-project/ltp.git
qemu_x86_64: ltp-syscalls-tests: * inotify06 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.git
x15 - arm: ltp-syscalls-tests: * poll02 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.git
Boards, architectures and test suites: -------------------------------------
juno-r2 - arm64 * boot - pass: 21, * kselftest - pass: 30, skip: 43 * libhugetlbfs - pass: 89, fail: 1, skip: 1 * ltp-cap_bounds-tests - pass: 2, * ltp-containers-tests - pass: 44, skip: 37 * ltp-cve-tests - pass: 26, skip: 9 * ltp-fcntl-locktests-tests - pass: 2, * ltp-filecaps-tests - pass: 2, * ltp-fs-tests - pass: 60, skip: 6 * ltp-fs_bind-tests - pass: 2, * ltp-fs_perms_simple-tests - pass: 19, * ltp-fsx-tests - pass: 2, * ltp-hugetlb-tests - pass: 22, * ltp-ipc-tests - pass: 9, * ltp-math-tests - pass: 11, * ltp-nptl-tests - pass: 2, * ltp-open-posix-tests - pass: 1689, fail: 5, skip: 41 * ltp-pty-tests - pass: 4, * ltp-sched-tests - pass: 10, skip: 4 * ltp-securebits-tests - pass: 4, * ltp-syscalls-tests - pass: 1009, skip: 140 * ltp-timers-tests - pass: 13,
qemu_arm * boot - pass: 21, * kselftest - pass: 32, skip: 49 * libhugetlbfs - pass: 86, fail: 1, skip: 1 * ltp-cap_bounds-tests - pass: 2, * ltp-containers-tests - pass: 79, skip: 2 * ltp-cve-tests - pass: 23, skip: 12 * ltp-fcntl-locktests-tests - pass: 2, * ltp-filecaps-tests - pass: 2, * ltp-fs-tests - pass: 59, fail: 2, skip: 5 * ltp-fs_bind-tests - pass: 2, * ltp-fs_perms_simple-tests - pass: 19, * ltp-fsx-tests - pass: 2, * ltp-hugetlb-tests - pass: 21, skip: 1 * ltp-io-tests - pass: 3, * ltp-ipc-tests - pass: 9, * ltp-math-tests - pass: 11, * ltp-nptl-tests - pass: 2, * ltp-pty-tests - pass: 4, * ltp-sched-tests - pass: 7, skip: 7 * ltp-securebits-tests - pass: 4, * ltp-syscalls-tests - pass: 1049, skip: 100 * ltp-timers-tests - pass: 13,
qemu_x86_64 * boot - pass: 21, * kselftest - pass: 50, skip: 44 * libhugetlbfs - pass: 89, fail: 1, skip: 1 * ltp-cap_bounds-tests - pass: 2, * ltp-containers-tests - pass: 80, skip: 1 * ltp-cve-tests - pass: 30, skip: 5 * ltp-fcntl-locktests-tests - pass: 2, * ltp-filecaps-tests - pass: 2, * ltp-fs-tests - pass: 60, skip: 6 * ltp-fs_bind-tests - pass: 2, * ltp-fs_perms_simple-tests - pass: 19, * ltp-fsx-tests - pass: 2, * ltp-hugetlb-tests - pass: 22, * ltp-io-tests - pass: 3, * ltp-ipc-tests - pass: 9, * ltp-math-tests - pass: 11, * ltp-nptl-tests - pass: 2, * ltp-pty-tests - pass: 4, * ltp-sched-tests - pass: 13, skip: 1 * ltp-securebits-tests - pass: 4, * ltp-syscalls-tests - pass: 990, fail: 2, skip: 157 * ltp-timers-tests - pass: 13,
x15 - arm * boot - pass: 22, * kselftest - pass: 34, skip: 44 * libhugetlbfs - pass: 86, fail: 1, skip: 1 * ltp-cap_bounds-tests - pass: 2, * ltp-containers-tests - pass: 79, skip: 2 * ltp-cve-tests - pass: 25, skip: 10 * ltp-fcntl-locktests-tests - pass: 2, * ltp-filecaps-tests - pass: 2, * ltp-fs-tests - pass: 61, skip: 5 * ltp-fs_bind-tests - pass: 2, * ltp-fs_perms_simple-tests - pass: 19, * ltp-fsx-tests - pass: 2, * ltp-hugetlb-tests - pass: 20, skip: 2 * ltp-io-tests - pass: 3, * ltp-ipc-tests - pass: 9, * ltp-math-tests - pass: 11, * ltp-nptl-tests - pass: 2, * ltp-open-posix-tests - pass: 1690, fail: 5, skip: 40 * ltp-pty-tests - pass: 4, * ltp-sched-tests - pass: 13, skip: 1 * ltp-securebits-tests - pass: 4, * ltp-syscalls-tests - pass: 1068, fail: 2, skip: 79 * ltp-timers-tests - pass: 13,
x86_64 * boot - pass: 24, * kselftest - pass: 50, fail: 1, skip: 41 * kselftest-vsyscall-mode-native - pass: 49, fail: 1, skip: 41 * kselftest-vsyscall-mode-none - pass: 50, skip: 41 * libhugetlbfs - pass: 89, fail: 1, skip: 1 * ltp-cap_bounds-tests - pass: 2, * ltp-containers-tests - pass: 80, skip: 1 * ltp-cve-tests - pass: 30, skip: 5 * ltp-fcntl-locktests-tests - pass: 2, * ltp-filecaps-tests - pass: 2, * ltp-fs-tests - pass: 61, skip: 5 * ltp-fs_bind-tests - pass: 2, * ltp-fs_perms_simple-tests - pass: 19, * ltp-fsx-tests - pass: 2, * ltp-hugetlb-tests - pass: 22, * ltp-io-tests - pass: 3, * ltp-ipc-tests - pass: 9, * ltp-math-tests - pass: 11, * ltp-nptl-tests - pass: 2, * ltp-open-posix-tests - pass: 1686, fail: 5, skip: 44 * ltp-pty-tests - pass: 4, * ltp-sched-tests - pass: 9, skip: 5 * ltp-securebits-tests - pass: 4, * ltp-syscalls-tests - pass: 1030, skip: 119 * ltp-timers-tests - pass: 13,
Naresh,
This is not Greg's last push but I thought it would be good to do an initial check before last results so, if facing same issues, you could have something by tomorrow.
On Tue, Aug 07, 2018 at 10:26:16PM +0000, Linaro QA wrote:
Summary
kernel: 4.4.147-rc1 git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git git branch: linux-4.4.y git commit: db3e08ea00d093d99d1ccbd3cf069db67967de00 git describe: v4.4.146-12-gdb3e08ea00d0 Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/build/v4.4.146-12-...
Regressions (compared to build v4.4.146)
qemu_arm: ltp-fs-tests: * read_all_dev * runltp_fs
* test src: git://github.com/linux-test-project/ltp.git
read_all_dev:
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s read_all.c:321: BROK: queue_push(workers[j].q, path) timed out
This test is basically checking each mounted filesystem and checking if it is supposed to try to read or not (ignore list). If it is supposed to read directory contents it schedules a read callback function (that will read just a chunk of each file) in a thread from a thread pool created in the beginning of the test (1 per CPU).
This was a timeout (and stack frame was in a queue_push because that is where the code spend most of its time, most likely). Timeout could have happened because trying to read a /dev/XXX (to be discovered) could have caused one thread from the thread pool not to return from the read() attempt on the device (char/block).
If it continues in the next push, we should take a deeper look.
qemu_x86_64: ltp-syscalls-tests: * inotify06 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.git
safe_macros.c:225: BROK: inotify06.c:76: open(fname_1,66,0600) failed: ENOENT
tst_test.c:1015: INFO: Timeout per run is 0h 30m 00s inotify06.c:87: BROK: inotify_init failed: EMFILE
1) EMFILE: The user limit on the total number of inotify instances has been reached. 2) EMFILE: The per-process limit on the number of open file descriptors has been reached. 3) ENFILE: The system-wide limit on the total number of open files has been reached.
This test exhausts inotify file descriptors and watchers by creating groups of inotify file descriptors for 5 files and adding watchers to it. At the same time one child task keeps opening (O_CREAT), closing and unlinking the same files that are being monitored by the inotify logic.
This test tests an old bug of kernels 4.2, trying to cause a kernel crash because of a race condition. In our case it is likely that the test has stressed the logic of opening/closing files (and adding watchers to it). If the watcher logic got stuck and close could not cleanup the file descriptor reference, then we could achieve a file descriptor limit (thus the errors above). There is also a chance of inotify watcher logic (when adding) causing open() to missbehave (returning ENOENT when O_CREAT flag is given):
ENOENT: O_CREAT is not set (not our case) and the named file does not exist. Or, a directory component in pathname does not exist or is a dangling symbolic link.
ENOENT pathname refers to a nonexistent directory, O_TMPFILE and one of O_WRONLY or O_RDWR were specified in flags, but this kernel version does not provide the O_TMPFILE functionality.
Those 2 don't seem to be the case of this test.
@Naresh, mind opening a bug and pasting this small comment on this issue ? (Reminder: we have many inotify/fanotify missing patches for v4.4 and, likely, if this is something hard to be fixed it will be a wont fix. My fear is that this is something related to open() and inotify() watchers. Checking if this test has ever failed before would also be benefitial.
x15 - arm: ltp-syscalls-tests: * poll02 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.git
tst_timer_test.c:318: INFO: min 1004586us, max 1005156us, median 1004586us, trunc mean 1004586.00us (discarded 1) tst_timer_test.c:321: FAIL: poll() slept for too long
I think this is the one (this was hard to find on the output). I explained the issue with this in another situation:
For QEMU: https://bugs.linaro.org/show_bug.cgi?id=3852#c4 For X15: https://bugs.linaro.org/show_bug.cgi?id=3852#c8
Having paravirtulized clock source might help in this one as well (https://projects.linaro.org/browse/KV-114).
Thanks o/ - Rafael
On 8 August 2018 at 07:07, Rafael David Tinoco rafael.tinoco@linaro.org wrote:
Naresh,
This is not Greg's last push but I thought it would be good to do an initial check before last results so, if facing same issues, you could have something by tomorrow.
On Tue, Aug 07, 2018 at 10:26:16PM +0000, Linaro QA wrote:
Summary
kernel: 4.4.147-rc1 git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git git branch: linux-4.4.y git commit: db3e08ea00d093d99d1ccbd3cf069db67967de00 git describe: v4.4.146-12-gdb3e08ea00d0 Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/build/v4.4.146-12-...
Regressions (compared to build v4.4.146)
qemu_arm: ltp-fs-tests: * read_all_dev * runltp_fs
* test src: git://github.com/linux-test-project/ltp.gitread_all_dev:
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s read_all.c:321: BROK: queue_push(workers[j].q, path) timed out
This test is basically checking each mounted filesystem and checking if it is supposed to try to read or not (ignore list). If it is supposed to read directory contents it schedules a read callback function (that will read just a chunk of each file) in a thread from a thread pool created in the beginning of the test (1 per CPU).
This was a timeout (and stack frame was in a queue_push because that is where the code spend most of its time, most likely). Timeout could have happened because trying to read a /dev/XXX (to be discovered) could have caused one thread from the thread pool not to return from the read() attempt on the device (char/block).
If it continues in the next push, we should take a deeper look.
This is a new LTP test case and this is not a new failure. On the latest push this test case got PASS. https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-fs-tests...
qemu_x86_64: ltp-syscalls-tests: * inotify06 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.git
Test case history shows it was failed only once and PASS on latest push, https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-syscalls...
safe_macros.c:225: BROK: inotify06.c:76: open(fname_1,66,0600) failed: ENOENT
tst_test.c:1015: INFO: Timeout per run is 0h 30m 00s inotify06.c:87: BROK: inotify_init failed: EMFILE
OTOH, It was failed only on qemu_x86_64 and PASS on real hardware. We have to investigate why only failed on qemu.
- EMFILE: The user limit on the total number of inotify instances has been reached.
- EMFILE: The per-process limit on the number of open file descriptors has been reached.
- ENFILE: The system-wide limit on the total number of open files has been reached.
This test exhausts inotify file descriptors and watchers by creating groups of inotify file descriptors for 5 files and adding watchers to it. At the same time one child task keeps opening (O_CREAT), closing and unlinking the same files that are being monitored by the inotify logic.
This test tests an old bug of kernels 4.2, trying to cause a kernel crash because of a race condition. In our case it is likely that the test has stressed the logic of opening/closing files (and adding watchers to it). If the watcher logic got stuck and close could not cleanup the file descriptor reference, then we could achieve a file descriptor limit (thus the errors above). There is also a chance of inotify watcher logic (when adding) causing open() to missbehave (returning ENOENT when O_CREAT flag is given):
ENOENT: O_CREAT is not set (not our case) and the named file does not exist. Or, a directory component in pathname does not exist or is a dangling symbolic link.
ENOENT pathname refers to a nonexistent directory, O_TMPFILE and one of O_WRONLY or O_RDWR were specified in flags, but this kernel version does not provide the O_TMPFILE functionality.
Those 2 don't seem to be the case of this test.
@Naresh, mind opening a bug and pasting this small comment on this issue ? (Reminder: we have many inotify/fanotify missing patches for v4.4 and, likely, if this is something hard to be fixed it will be a wont fix. My fear is that this is something related to open() and inotify() watchers. Checking if this test has ever failed before would also be benefitial.
This was not a failure earlier, However, re-submitted test job to run inotify06 in loop for 50 iterations. If this re-producible will report bug. https://lkft.validation.linaro.org/scheduler/job/360271
x15 - arm: ltp-syscalls-tests: * poll02 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.gittst_timer_test.c:318: INFO: min 1004586us, max 1005156us, median 1004586us, trunc mean 1004586.00us (discarded 1) tst_timer_test.c:321: FAIL: poll() slept for too long
I think this is the one (this was hard to find on the output). I explained the issue with this in another situation:
For QEMU: https://bugs.linaro.org/show_bug.cgi?id=3852#c4 For X15: https://bugs.linaro.org/show_bug.cgi?id=3852#c8
Having paravirtulized clock source might help in this one as well (https://projects.linaro.org/browse/KV-114).
I have re-submitted the job to test poll02 for 50 iterations. https://lkft.validation.linaro.org/scheduler/job/360274
PASSED on the latest push. https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-syscalls...
Thank you.
- Naresh
Thanks o/
Rafael
On 8 August 2018 at 11:24, Naresh Kamboju naresh.kamboju@linaro.org wrote:
On 8 August 2018 at 07:07, Rafael David Tinoco rafael.tinoco@linaro.org wrote:
Naresh,
This is not Greg's last push but I thought it would be good to do an initial check before last results so, if facing same issues, you could have something by tomorrow.
On Tue, Aug 07, 2018 at 10:26:16PM +0000, Linaro QA wrote:
Summary
kernel: 4.4.147-rc1 git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git git branch: linux-4.4.y git commit: db3e08ea00d093d99d1ccbd3cf069db67967de00 git describe: v4.4.146-12-gdb3e08ea00d0 Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/build/v4.4.146-12-...
Regressions (compared to build v4.4.146)
qemu_x86_64: ltp-syscalls-tests: * inotify06 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.gitThis was not a failure earlier, However, re-submitted test job to run inotify06 in loop for 50 iterations. If this re-producible will report bug. https://lkft.validation.linaro.org/scheduler/job/360271
Test run for all 50 iterations and PASS every time on qemu_x86_64 https://lkft.validation.linaro.org/scheduler/job/360271#L604
x15 - arm: ltp-syscalls-tests: * poll02 * runltp_syscalls
https://bugs.linaro.org/show_bug.cgi?id=3852#c8Having paravirtulized clock source might help in this one as well (https://projects.linaro.org/browse/KV-114).
I have re-submitted the job to test poll02 for 50 iterations. https://lkft.validation.linaro.org/scheduler/job/360274
Test run for all 50 iterations and PASS every time on x15 device. https://lkft.validation.linaro.org/scheduler/job/360274#L1257
PASSED on the latest push. https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-syscalls...
- Naresh
read_all_dev:
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s read_all.c:321: BROK: queue_push(workers[j].q, path) timed out
This test is basically checking each mounted filesystem and checking if it is supposed to try to read or not (ignore list). If it is supposed to read directory contents it schedules a read callback function (that will read just a chunk of each file) in a thread from a thread pool created in the beginning of the test (1 per CPU).
This was a timeout (and stack frame was in a queue_push because that is where the code spend most of its time, most likely). Timeout could have happened because trying to read a /dev/XXX (to be discovered) could have caused one thread from the thread pool not to return from the read() attempt on the device (char/block).
If it continues in the next push, we should take a deeper look.
This is a new LTP test case and this is not a new failure. On the latest push this test case got PASS. https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-fs-tests...
qemu_x86_64: ltp-syscalls-tests: * inotify06 * runltp_syscalls
* test src: git://github.com/linux-test-project/ltp.gitTest case history shows it was failed only once and PASS on latest push, https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-syscalls...
safe_macros.c:225: BROK: inotify06.c:76: open(fname_1,66,0600) failed: ENOENT
tst_test.c:1015: INFO: Timeout per run is 0h 30m 00s inotify06.c:87: BROK: inotify_init failed: EMFILE
OTOH, It was failed only on qemu_x86_64 and PASS on real hardware. We have to investigate why only failed on qemu.
Yes.
FS test => read_all_dev read_all -d /dev -p -q -r 10
You could simulate this test doing:
root@readall:/dev$ find . -not -type d -exec echo {} ; -exec dd if={} of=/dev/null bs=1M count=1 ; ./vcsa5 0+1 records in 0+1 records out 4004 bytes (4.0 kB, 3.9 KiB) copied, 0.00404404 s, 990 kB/s ./vcs5 0+1 records in 0+1 records out 2000 bytes (2.0 kB, 2.0 KiB) copied, 0.00369915 s, 541 kB/s ...
./hvc3 dd: failed to open './hvc3': No such device ./hvc2 dd: failed to open './hvc2': No such device ./hvc1 dd: failed to open './hvc1': No such device ./hvc0 <stuck>
I'm stuck in /dev/hvc0 in a 4.17 kernel (from Debian). This is likely what is going on, we are stuck reading a char device (for example) and the thread waiting for the read() call never finishes its job, so the timeout triggers. (in my case this comes from CONFIG_HVC_XEN from the default config file from debian).
We should open a low priority bug for this one orelse we will face this in a near future depending on which modules are loaded and if the device being read is giving our read() call something back or not.
Not an issue with this kernel anyway.
Tks for checking Naresh!
@Naresh, mind opening a bug and pasting this small comment on this issue ? (Reminder: we have many inotify/fanotify missing patches for v4.4 and, likely, if this is something hard to be fixed it will be a wont fix. My fear is that this is something related to open() and inotify() watchers. Checking if this test has ever failed before would also be benefitial.
This was not a failure earlier, However, re-submitted test job to run inotify06 in loop for 50 iterations. If this re-producible will report bug. https://lkft.validation.linaro.org/scheduler/job/360271
For QEMU: https://bugs.linaro.org/show_bug.cgi?id=3852#c4 For X15: https://bugs.linaro.org/show_bug.cgi?id=3852#c8
Having paravirtulized clock source might help in this one as well (https://projects.linaro.org/browse/KV-114).
I have re-submitted the job to test poll02 for 50 iterations. https://lkft.validation.linaro.org/scheduler/job/360274
PASSED on the latest push. https://qa-reports.linaro.org/lkft/linux-stable-rc-4.4-oe/tests/ltp-syscalls...
Yep, this is timing related.
Thank you.
Tks for checking!! Cheers o/
-
Linaro QA -
Naresh Kamboju -
Rafael David Tinoco