Lkft-triage

lkft-triage@lists.linaro.org

2 participants
33162 discussions

Daily report for validation - 2024-Aug-26

by lava＠validation.linaro.org

Total jobs: 49 Total errors: 3 (6.12%) LAVA errors: 0 (0.00%) Test errors: 0 (0.00%) Job errors: 2 (4.08%) Infra errors: 1 (2.04%) Canceled jobs: 0 (0.00%) Device type: x86 Total jobs: 5 Total errors: 0 (0.00%) Device type: beaglebone-black Total jobs: 9 Total errors: 0 (0.00%) Device type: dragonboard-410c Total jobs: 3 Total errors: 2 (66.67%) Error type: Job Error count: 2 (66.67%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 2 (66.67%) IDs: dragonboard-410c-02: 4088886 4088926 Device type: synquacer Total jobs: 7 Total errors: 0 (0.00%) Device type: dragonboard-845c Total jobs: 5 Total errors: 0 (0.00%) Device type: bcm2837-rpi-3-b-32 Total jobs: 6 Total errors: 1 (16.67%) Error type: Infrastructure Error count: 1 (16.67%) Error: No match for error type 'Infrastructure', message 'apply-overlay-tftp timed out after 26 seconds' Count: 1 (16.67%) IDs: rpi3-b-32-02: 4088882 Device type: qemu Total jobs: 4 Total errors: 0 (0.00%) Device type: frdm-kw41z Total jobs: 2 Total errors: 0 (0.00%) Device type: frdm-k64f Total jobs: 3 Total errors: 0 (0.00%) Device type: docker Total jobs: 2 Total errors: 0 (0.00%) Device type: imx93-11x11-evk Total jobs: 2 Total errors: 0 (0.00%) Device type: juno Total jobs: 1 Total errors: 0 (0.00%)

2 years, 2 months

Daily cts/vts report for lkft - 2024-Aug-25

by lava＠validation.linaro.org

Total jobs: 23 Total errors: 5 (21.74%) LAVA errors: 0 (0.00%) Test errors: 5 (21.74%) Job errors: 0 (0.00%) Infra errors: 0 (0.00%) Canceled jobs: 0 (0.00%) Device type: dragonboard-845c Total jobs: 16 Total errors: 4 (25.00%) Error type: Test Error count: 4 (25.00%) Error: No match for error type 'Test', message 'tradefed - adb device lost[f3a58ae3]' Count: 1 (6.25%) IDs: db845c-04: 7818990 Error: No match for error type 'Test', message 'tradefed - adb device lost[3419ccc6]' Count: 2 (12.50%) IDs: db845c-06: 7818974 7818983 Error: No match for error type 'Test', message 'tradefed - adb device lost[db2a494d]' Count: 1 (6.25%) IDs: db845c-02: 7818959 Device type: qrb5165-rb5 Total jobs: 7 Total errors: 1 (14.29%) Error type: Test Error count: 1 (14.29%) Error: Device NOT found! Count: 1 (14.29%) IDs: rb5-03: 7818963

2 years, 2 months

Daily report for validation - 2024-Aug-25

by lava＠validation.linaro.org

Total jobs: 35 Total errors: 1 (2.86%) LAVA errors: 0 (0.00%) Test errors: 0 (0.00%) Job errors: 1 (2.86%) Infra errors: 0 (0.00%) Canceled jobs: 0 (0.00%) Device type: bcm2837-rpi-3-b-32 Total jobs: 3 Total errors: 0 (0.00%) Device type: dragonboard-410c Total jobs: 2 Total errors: 1 (50.00%) Error type: Job Error count: 1 (50.00%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 1 (50.00%) IDs: dragonboard-410c-02: 4088872 Device type: dragonboard-845c Total jobs: 4 Total errors: 0 (0.00%) Device type: synquacer Total jobs: 4 Total errors: 0 (0.00%) Device type: beaglebone-black Total jobs: 5 Total errors: 0 (0.00%) Device type: x86 Total jobs: 3 Total errors: 0 (0.00%) Device type: qemu Total jobs: 4 Total errors: 0 (0.00%) Device type: frdm-kw41z Total jobs: 2 Total errors: 0 (0.00%) Device type: frdm-k64f Total jobs: 3 Total errors: 0 (0.00%) Device type: docker Total jobs: 2 Total errors: 0 (0.00%) Device type: imx93-11x11-evk Total jobs: 2 Total errors: 0 (0.00%) Device type: juno Total jobs: 1 Total errors: 0 (0.00%)

2 years, 2 months

Daily cts/vts report for lkft - 2024-Aug-24

by lava＠validation.linaro.org

Total jobs: 96 Total errors: 16 (16.67%) LAVA errors: 0 (0.00%) Test errors: 15 (15.62%) Job errors: 0 (0.00%) Infra errors: 1 (1.04%) Canceled jobs: 0 (0.00%) Device type: dragonboard-845c Total jobs: 69 Total errors: 11 (15.94%) Error type: Test Error count: 11 (15.94%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 2 (2.90%) IDs: db845c-02: 7817794 db845c-09: 7818845 Error: No match for error type 'Test', message 'tradefed - adb device lost[fbf682a2]' Count: 2 (2.90%) IDs: db845c-09: 7817792 7818832 Error: Device NOT found! Count: 5 (7.25%) IDs: db845c-02: 7817777 7818770 db845c-04: 7817796 db845c-09: 7818718 7818744 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 595 seconds' Count: 1 (1.45%) IDs: db845c-02: 7817790 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 596 seconds' Count: 1 (1.45%) IDs: db845c-02: 7817779 Device type: qrb5165-rb5 Total jobs: 27 Total errors: 5 (18.52%) Error type: Test Error count: 4 (14.81%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 1 (3.70%) IDs: rb5-03: 7818837 Error: Device NOT found! Count: 3 (11.11%) IDs: rb5-03: 7817795 7818705 rb5-06: 7818722 Error type: Infrastructure Error count: 1 (3.70%) Error: Connection closed Count: 1 (3.70%) IDs: rb5-05: 7818788

2 years, 2 months

Daily report for validation - 2024-Aug-24

by lava＠validation.linaro.org

Total jobs: 179 Total errors: 23 (12.85%) LAVA errors: 0 (0.00%) Test errors: 1 (0.56%) Job errors: 13 (7.26%) Infra errors: 9 (5.03%) Canceled jobs: 0 (0.00%) Device type: bcm2837-rpi-3-b-32 Total jobs: 35 Total errors: 4 (11.43%) Error type: Infrastructure Error count: 1 (2.86%) Error: No match for error type 'Infrastructure', message 'prepare-tftp-overlay timed out after 132 seconds' Count: 1 (2.86%) IDs: rpi3-b-32-02: 4088823 Error type: Job Error count: 3 (8.57%) Error: compress-overlay timed out Count: 3 (8.57%) IDs: rpi3-b-32-02: 4088664 4088788 4088804 Device type: synquacer Total jobs: 29 Total errors: 0 (0.00%) Device type: dragonboard-845c Total jobs: 12 Total errors: 1 (8.33%) Error type: Infrastructure Error count: 1 (8.33%) Error: Reboot failed Count: 1 (8.33%) IDs: dragonboard-845c-02: 4088758 Device type: dragonboard-410c Total jobs: 10 Total errors: 9 (90.00%) Error type: Job Error count: 9 (90.00%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 9 (90.00%) IDs: dragonboard-410c-02: 4088681 4088686 4088714 4088727 4088736 4088754 4088792 4088814 4088830 Device type: beaglebone-black Total jobs: 37 Total errors: 0 (0.00%) Device type: x86 Total jobs: 19 Total errors: 0 (0.00%) Device type: qemu Total jobs: 4 Total errors: 0 (0.00%) Device type: frdm-kw41z Total jobs: 2 Total errors: 0 (0.00%) Device type: frdm-k64f Total jobs: 3 Total errors: 0 (0.00%) Device type: docker Total jobs: 2 Total errors: 0 (0.00%) Device type: imx93-11x11-evk Total jobs: 3 Total errors: 0 (0.00%) Device type: juno Total jobs: 1 Total errors: 0 (0.00%) Device type: x15 Total jobs: 22 Total errors: 9 (40.91%) Error type: Test Error count: 1 (4.55%) Error: No match for error type 'Test', message 'Network not able to ping; job exit' Count: 1 (4.55%) IDs: x15-01: 4088735 Error type: Infrastructure Error count: 7 (31.82%) Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 286 seconds' Count: 7 (31.82%) IDs: x15-01: 4061032 4061053 4061063 4061067 4061083 4061085 4061131 Error type: Job Error count: 1 (4.55%) Error: wait for prompt timed out Count: 1 (4.55%) IDs: x15-01: 4061120

2 years, 2 months

Daily cts/vts report for lkft - 2024-Aug-23

by lava＠validation.linaro.org

Total jobs: 82 Total errors: 11 (13.41%) LAVA errors: 0 (0.00%) Test errors: 10 (12.20%) Job errors: 0 (0.00%) Infra errors: 1 (1.22%) Canceled jobs: 0 (0.00%) Device type: dragonboard-845c Total jobs: 56 Total errors: 6 (10.71%) Error type: Test Error count: 6 (10.71%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 295 seconds' Count: 1 (1.79%) IDs: db845c-07: 7816836 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 597 seconds' Count: 2 (3.57%) IDs: db845c-02: 7816833 db845c-09: 7816834 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 582 seconds' Count: 1 (1.79%) IDs: db845c-07: 7816830 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 1 (1.79%) IDs: db845c-02: 7815580 Error: No match for error type 'Test', message 'tradefed - adb device lost[db2a494d]' Count: 1 (1.79%) IDs: db845c-02: 7815558 Device type: qrb5165-rb5 Total jobs: 24 Total errors: 4 (16.67%) Error type: Infrastructure Error count: 1 (4.17%) Error: No match for error type 'Infrastructure', message 'Unable to run pre-power: '/usr/local/lab-scripts/cbrxd_hub_control -n lkft-slave05-hub01 -m sync -u 14' failed' Count: 1 (4.17%) IDs: rb5-03: 7817786 Error type: Test Error count: 3 (12.50%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 298 seconds' Count: 1 (4.17%) IDs: rb5-06: 7817785 Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed' Count: 1 (4.17%) IDs: rb5-03: 7817767 Error: Device NOT found! Count: 1 (4.17%) IDs: rb5-03: 7815602 Device type: juno-r2 Total jobs: 2 Total errors: 1 (50.00%) Error type: Test Error count: 1 (50.00%) Error: lava-test-shell timed out Count: 1 (50.00%) IDs: juno-r2-01: 7817342

2 years, 2 months

Daily report for validation - 2024-Aug-23

by lava＠validation.linaro.org

Total jobs: 443 Total errors: 124 (27.99%) LAVA errors: 0 (0.00%) Test errors: 8 (1.81%) Job errors: 86 (19.41%) Infra errors: 30 (6.77%) Canceled jobs: 0 (0.00%) Device type: dragonboard-845c Total jobs: 22 Total errors: 3 (13.64%) Error type: Job Error count: 3 (13.64%) Error: No match for error type 'Job', message 'Unable to run: ['docker', 'run', '--rm', '--init', '--workdir=/lava-downloads', '--device=/dev/kvm', '--mount=type=bind,source=/var/lib/lava/dispatcher/tmp/4088634/downloads/common,destination=/lava-downloads', 'linaro/lava-android-postprocess:bulleye-2023.09.14-01', '/lava-downloads/postprocess.sh']' Count: 1 (4.55%) IDs: dragonboard-845c-02: 4088634 Error: No match for error type 'Job', message 'login-action timed out after 874 seconds' Count: 1 (4.55%) IDs: dragonboard-845c-06: 4088616 Error: No match for error type 'Job', message 'login-action timed out after 873 seconds' Count: 1 (4.55%) IDs: dragonboard-845c-06: 4088179 Device type: dragonboard-410c Total jobs: 18 Total errors: 17 (94.44%) Error type: Job Error count: 17 (94.44%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 17 (94.44%) IDs: dragonboard-410c-02: 4088306 4088309 4088316 4088335 4088364 4088374 4088418 4088438 4088460 4088477 4088514 4088519 4088533 4088575 4088598 4088615 4088633 Device type: synquacer Total jobs: 59 Total errors: 4 (6.78%) Error type: Infrastructure Error count: 1 (1.69%) Error: Connection closed Count: 1 (1.69%) IDs: synquacer-01: 4088476 Error type: Job Error count: 3 (5.08%) Error: Invalid job data Count: 3 (5.08%) IDs: synquacer-01: 4088182 4088192 4088195 Device type: beaglebone-black Total jobs: 109 Total errors: 25 (22.94%) Error type: Job Error count: 25 (22.94%) Error: Invalid job data Count: 25 (22.94%) IDs: beaglebone-black04: 4088152 4088154 4088156 4088158 4088160 4088162 4088164 4088170 4088173 4088186 4088188 4088194 4088198 4088200 4088202 4088204 4088206 4088208 4088210 4088212 4088214 4088219 4088221 4088223 4088225 Device type: bcm2837-rpi-3-b-32 Total jobs: 108 Total errors: 36 (33.33%) Error type: Infrastructure Error count: 4 (3.70%) Error: No match for error type 'Infrastructure', message 'apply-overlay-tftp timed out after 26 seconds' Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088588 Error: No match for error type 'Infrastructure', message 'compress-ramdisk timed out after 2 seconds' Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088569 Error: Connection closed Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088537 Error: No match for error type 'Infrastructure', message 'apply-overlay-tftp timed out after 3 seconds' Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088506 Error type: Job Error count: 32 (29.63%) Error: compress-overlay timed out Count: 3 (2.78%) IDs: rpi3-b-32-02: 4088295 4088427 4088500 Error: No match for error type 'Job', message 'test-definition timed out after 86 seconds' Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088414 Error: No match for error type 'Job', message 'test-definition timed out after 43 seconds' Count: 1 (0.93%) IDs: rpi3-b-32-02: 4088375 Error: Invalid job data Count: 27 (25.00%) IDs: rpi3-b-32-02: 4088139 4088141 4088143 4088145 4088147 4088149 4088151 4088153 4088155 4088157 4088159 4088161 4088163 4088165 4088177 4088183 4088187 4088190 4088193 4088196 4088199 4088201 4088203 4088205 4088207 4088209 4088211 Device type: x86 Total jobs: 46 Total errors: 9 (19.57%) Error type: Test Error count: 6 (13.04%) Error: lava-test-shell timed out Count: 6 (13.04%) IDs: x86-01: 4088167 4088189 4088422 4088558 4088560 4088602 Error type: Job Error count: 3 (6.52%) Error: No match for error type 'Job', message 'test-definition timed out after 263 seconds' Count: 1 (2.17%) IDs: x86-01: 4088320 Error: Invalid job data Count: 2 (4.35%) IDs: x86-01: 4088185 4088191 Device type: qemu Total jobs: 4 Total errors: 0 (0.00%) Device type: frdm-kw41z Total jobs: 2 Total errors: 0 (0.00%) Device type: frdm-k64f Total jobs: 3 Total errors: 0 (0.00%) Device type: docker Total jobs: 2 Total errors: 0 (0.00%) Device type: x15 Total jobs: 64 Total errors: 30 (46.88%) Error type: Test Error count: 2 (3.12%) Error: No match for error type 'Test', message 'Network not able to ping; job exit' Count: 1 (1.56%) IDs: x15-01: 4088549 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 584 seconds' Count: 1 (1.56%) IDs: x15-01: 4088383 Error type: Infrastructure Error count: 25 (39.06%) Error: No match for error type 'Infrastructure', message 'wait-device-boardid timed out after 849 seconds' Count: 1 (1.56%) IDs: x15-01: 4088545 Error: No match for error type 'Infrastructure', message 'Unable to run: ['docker', 'exec', 'lava-4088542-1.18.7-d29ec194-05f0-43ec-afa8-24d9f5c7c30f', 'fastboot', '-s', '1300d00902428922', 'flash', 'super', '/var/lib/lava/dispatcher/tmp/4088542/fastboot-deploy-sieefcp3/super/4.19-super.img']' Count: 1 (1.56%) IDs: x15-01: 4088542 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 286 seconds' Count: 21 (32.81%) IDs: x15-01: 4060789 4060791 4060808 4060818 4060825 4060826 4060835 4060838 4060840 4060875 4060905 4060917 4060925 4060938 4060949 4060955 4060957 4060973 4060985 4060988 4061026 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 285 seconds' Count: 2 (3.12%) IDs: x15-01: 4060886 4060974 Error type: Job Error count: 3 (4.69%) Error: wait for prompt timed out Count: 1 (1.56%) IDs: x15-01: 4060856 Error: Invalid job data Count: 2 (3.12%) IDs: x15-01: 4060762 4060773 Device type: imx93-11x11-evk Total jobs: 5 Total errors: 0 (0.00%) Device type: juno Total jobs: 1 Total errors: 0 (0.00%)

2 years, 2 months

next-20240822: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.

by Naresh Kamboju

The following kernel crash was noticed on arm64 Rock-pi 4b, Qcomm dragonboard 410c and 845c devices while booting / loading kernel modules on today's Linux next-20240822. First seen on next-20240822. Good: next-20240821 BAD: next-20240822 Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> We are bisecting this problem. Crash log: -------- [ 0.000000] Linux version 6.11.0-rc4-next-20240822 (tuxmake@tuxmake) (Debian clang version 18.1.8 (++20240731024919+3b5b5c1ec4a3-1~exp1~20240731145017.142), Debian LLD 18.1.8) #1 SMP PREEMPT @1724309008 [ 0.000000] KASLR disabled due to lack of seed [ 0.000000] Machine model: Radxa ROCK Pi 4B ... [ 16.923808] coresight-cpu-debug fe430000.debug: Coresight debug-CPU0 initialized [ 16.933919] coresight-cpu-debug fe432000.debug: Coresight debug-CPU1 initialized [ 16.935653] coresight-cpu-debug fe434000.debug: Coresight debug-CPU2 initialized [ 16.937846] coresight-cpu-debug fe436000.debug: Coresight debug-CPU3 initialized [ 16.939744] coresight-cpu-debug fe610000.debug: Coresight debug-CPU4 initialized [ 16.942132] coresight-cpu-debug fe710000.debug: Coresight debug-CPU5 initialized [ 37.987966] rcu: INFO: rcu_preempt self-detected stall on CPU [ 37.989037] rcu: 2-...!: (5248 ticks this GP) idle=d1f4/1/0x4000000000000000 softirq=1552/1553 fqs=0 [ 37.990715] rcu: (t=5250 jiffies g=1269 q=784 ncpus=6) [ 37.991667] rcu: rcu_preempt kthread timer wakeup didn't happen for 5249 jiffies! g1269 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 [ 37.993718] rcu: Possible timer handling issue on cpu=4 timer-softirq=296 [ 37.994966] rcu: rcu_preempt kthread starved for 5250 jiffies! g1269 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=4 [ 37.996844] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 37.998499] rcu: RCU grace-period kthread stack dump: [ 37.999419] task:rcu_preempt state:I stack:0 pid:18 tgid:18 ppid:2 flags:0x00000008 [ 38.001117] Call trace: [ 38.001565] __switch_to+0x15c/0x208 [ 38.002229] __schedule+0x54c/0x9d0 [ 38.002870] schedule+0x12c/0x1e0 [ 38.003479] schedule_timeout+0x9c/0x1b0 [ 38.004202] rcu_gp_fqs_loop+0x1e0/0x880 [ 38.004926] rcu_gp_kthread+0x70/0x230 [ 38.005616] kthread+0x104/0x198 [ 38.006209] ret_from_fork+0x10/0x20 [ 38.006877] CPU: 2 UID: 0 PID: 237 Comm: (udev-worker) Not tainted 6.11.0-rc4-next-20240822 #1 [ 38.008442] Hardware name: Radxa ROCK Pi 4B (DT) [ 38.009284] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 38.010551] pc : smp_call_function_many_cond+0x468/0x6b8 [ 38.011521] lr : smp_call_function_many_cond+0x428/0x6b8 [ 38.012490] sp : ffff800083d1bb50 [ 38.013094] x29: ffff800083d1bb90 x28: 0000000000000006 x27: 0000000000000000 [ 38.014399] x26: ffff80008269e000 x25: 0000000000000000 x24: 0000000000000001 [ 38.015702] x23: ffff8000801bc470 x22: ffff800074e75000 x21: ffff0000f74d8c20 [ 38.017005] x20: ffff0000f746cb80 x19: ffff8000826a3470 x18: ffff80007afc2000 [ 38.018308] x17: 000000000909008e x16: 0000000000000000 x15: ffff8000816092a0 [ 38.019611] x14: 060a090000000000 x13: 0000000000000003 x12: 000000000000003f [ 38.020914] x11: 0000000000000001 x10: 0000000000000000 x9 : 0000000000000011 [ 38.022218] x8 : ffff0000f74b6c28 x7 : 0000000000000000 x6 : 0000000000000080 [ 38.023520] x5 : 000000000000003f x4 : 0000000000000000 x3 : 0000000000000006 [ 38.024823] x2 : 0000000000000004 x1 : 0000000000000004 x0 : 0000000000000004 [ 38.026126] Call trace: [ 38.026574] smp_call_function_many_cond+0x468/0x6b8 [ 38.027481] kick_all_cpus_sync+0x44/0x78 [ 38.028216] flush_module_icache+0x38/0xe0 [ 38.028965] load_module+0x1374/0x1778 [ 38.029654] __arm64_sys_finit_module+0x238/0x348 [ 38.030516] invoke_syscall+0x4c/0x110 [ 38.031207] el0_svc_common+0x8c/0xf0 [ 38.031880] do_el0_svc+0x28/0x40 [ 38.032490] el0_svc+0x40/0x88 [ 38.033054] el0t_64_sync_handler+0x90/0x100 [ 38.033837] el0t_64_sync+0x190/0x198 [ 38.034509] Sending NMI from CPU 2 to CPUs 4: [ 48.036555] Sending NMI from CPU 2 to CPUs 5: [ 78.247966] sched: DL replenish lagged too much [ 121.047963] rcu: INFO: rcu_preempt self-detected stall on CPU [ 121.049009] rcu: 2-...!: (21002 ticks this GP) idle=d1f4/1/0x4000000000000000 softirq=1552/1553 fqs=0 [ 121.050702] rcu: (t=26015 jiffies g=1269 q=784 ncpus=6) [ 121.051669] rcu: rcu_preempt kthread timer wakeup didn't happen for 26014 jiffies! g1269 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x200 [ 121.053735] rcu: Possible timer handling issue on cpu=4 timer-softirq=296 [ 121.054982] rcu: rcu_preempt kthread starved for 26015 jiffies! g1269 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=4 [ 121.056876] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 121.058530] rcu: RCU grace-period kthread stack dump: [ 121.059447] task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00000008 [ 121.061140] Call trace: [ 121.061587] __switch_to+0x15c/0x208 [ 121.062245] __schedule+0x54c/0x9d0 [ 121.062885] schedule+0x12c/0x1e0 [ 121.063494] schedule_timeout+0x9c/0x1b0 [ 121.064214] rcu_gp_fqs_loop+0x1e0/0x880 [ 121.064936] rcu_gp_kthread+0x70/0x230 [ 121.065625] kthread+0x104/0x198 [ 121.066217] ret_from_fork+0x10/0x20 [ 121.066874] CPU: 2 UID: 0 PID: 237 Comm: (udev-worker) Not tainted 6.11.0-rc4-next-20240822 #1 ... [ 785.590550] Sending NMI from CPU 2 to CPUs 4: [ 795.592594] Sending NMI from CPU 2 to CPUs 5: boot Log links, -------- - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240822/te… - https://lkft.validation.linaro.org/scheduler/job/7816099#L523 - https://lkft.validation.linaro.org/scheduler/job/7815910#L2448 Boot failed comparison: - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240822/te… metadata: ---- git describe: next-20240822 git repo: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next git sha: 6a7917c89f219f09b1d88d09f376000914a52763 kernel config: https://storage.tuxsuite.com/public/linaro/lkft/builds/2l0EZh4ZCNgUovHichD9… kernel version: 6.11.0-rc4 artifact location: https://storage.tuxsuite.com/public/linaro/lkft/builds/2l0EZh4ZCNgUovHichD9… build url: https://storage.tuxsuite.com/public/linaro/lkft/builds/2l0EZh4ZCNgUovHichD9… toolchain: clang-18 and gcc-13 Steps to reproduce: --------- - https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2l0EbxmBUp… - https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2l0EbxmBUp… -- Linaro LKFT https://lkft.linaro.org

2 years, 2 months

Daily cts/vts report for lkft - 2024-Aug-22

by lava＠validation.linaro.org

Total jobs: 114 Total errors: 16 (14.04%) LAVA errors: 0 (0.00%) Test errors: 15 (13.16%) Job errors: 0 (0.00%) Infra errors: 0 (0.00%) Canceled jobs: 1 (0.88%) Device type: dragonboard-845c Total jobs: 93 Total errors: 12 (12.90%) Error type: Test Error count: 11 (11.83%) Error: No match for error type 'Test', message 'tradefed - adb device lost[db2a494d]' Count: 2 (2.15%) IDs: db845c-02: 7815425 7815624 Error: Device NOT found! Count: 5 (5.38%) IDs: db845c-02: 7815297 7815597 7815618 db845c-05: 7815429 db845c-09: 7814214 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 583 seconds' Count: 1 (1.08%) IDs: db845c-09: 7815596 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 1 (1.08%) IDs: db845c-09: 7815592 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 593 seconds' Count: 1 (1.08%) IDs: db845c-02: 7814288 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 596 seconds' Count: 1 (1.08%) IDs: db845c-02: 7814210 Error type: Canceled Error count: 1 (1.08%) Error: The job was canceled Count: 1 (1.08%) IDs: db845c-05: 7814265 Device type: qrb5165-rb5 Total jobs: 19 Total errors: 4 (21.05%) Error type: Test Error count: 4 (21.05%) Error: Device NOT found! Count: 3 (15.79%) IDs: rb5-03: 7815287 7815524 7815540 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 1 (5.26%) IDs: rb5-03: 7815292 Device type: juno-r2 Total jobs: 2 Total errors: 0 (0.00%)

2 years, 2 months

Daily report for validation - 2024-Aug-22

by lava＠validation.linaro.org

Total jobs: 296 Total errors: 116 (39.19%) LAVA errors: 0 (0.00%) Test errors: 0 (0.00%) Job errors: 114 (38.51%) Infra errors: 2 (0.68%) Canceled jobs: 0 (0.00%) Device type: dragonboard-410c Total jobs: 12 Total errors: 11 (91.67%) Error type: Job Error count: 11 (91.67%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 11 (91.67%) IDs: dragonboard-410c-02: 4087840 4087865 4087886 4087902 4087993 4087997 4088015 4088032 4088054 4088081 4088181 Device type: bcm2837-rpi-3-b-32 Total jobs: 105 Total errors: 43 (40.95%) Error type: Job Error count: 42 (40.00%) Error: No match for error type 'Job', message 'test-definition timed out after 63 seconds' Count: 1 (0.95%) IDs: rpi3-b-32-02: 4088176 Error: Invalid job data Count: 39 (37.14%) IDs: rpi3-b-32-02: 4088004 4088018 4088026 4088035 4088039 4088050 4088059 4088061 4088067 4088068 4088069 4088072 4088073 4088080 4088086 4088087 4088088 4088092 4088094 4088095 4088097 4088100 4088101 4088103 4088104 4088107 4088110 4088113 4088116 4088119 4088121 4088123 4088125 4088127 4088129 4088131 4088133 4088135 4088137 Error: compress-overlay timed out Count: 1 (0.95%) IDs: rpi3-b-32-02: 4088029 Error: No match for error type 'Job', message 'test-install-overlay timed out after 2 seconds' Count: 1 (0.95%) IDs: rpi3-b-32-02: 4087894 Error type: Infrastructure Error count: 1 (0.95%) Error: Connection closed Count: 1 (0.95%) IDs: rpi3-b-32-02: 4087910 Device type: beaglebone-black Total jobs: 72 Total errors: 36 (50.00%) Error type: Job Error count: 36 (50.00%) Error: Invalid job data Count: 36 (50.00%) IDs: beaglebone-black04: 4088001 4088010 4088016 4088022 4088028 4088037 4088043 4088047 4088057 4088065 4088070 4088078 4088090 4088093 4088099 4088108 4088111 4088114 4088117 4088120 4088122 4088124 4088126 4088128 4088130 4088132 4088134 4088136 4088138 4088140 4088142 4088144 4088146 4088148 4088150 4088168 Device type: x86 Total jobs: 34 Total errors: 13 (38.24%) Error type: Job Error count: 13 (38.24%) Error: Invalid job data Count: 13 (38.24%) IDs: x86-01: 4088007 4088017 4088019 4088036 4088041 4088063 4088066 4088091 4088096 4088098 4088102 4088105 4088166 Device type: synquacer Total jobs: 41 Total errors: 13 (31.71%) Error type: Job Error count: 12 (29.27%) Error: Invalid job data Count: 12 (29.27%) IDs: synquacer-01: 4088014 4088034 4088038 4088052 4088062 4088085 4088089 4088106 4088109 4088112 4088115 4088118 Error type: Infrastructure Error count: 1 (2.44%) Error: Connection closed Count: 1 (2.44%) IDs: synquacer-01: 4087889 Device type: dragonboard-845c Total jobs: 13 Total errors: 0 (0.00%) Device type: qemu Total jobs: 4 Total errors: 0 (0.00%) Device type: frdm-kw41z Total jobs: 2 Total errors: 0 (0.00%) Device type: frdm-k64f Total jobs: 3 Total errors: 0 (0.00%) Device type: docker Total jobs: 2 Total errors: 0 (0.00%) Device type: imx93-11x11-evk Total jobs: 7 Total errors: 0 (0.00%) Device type: juno Total jobs: 1 Total errors: 0 (0.00%)

2 years, 2 months

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Lkft-triage