Total jobs: 201
Total errors: 60 (29.85%)
LAVA errors: 2 (1.00%)
Test errors: 56 (27.86%)
Job errors: 1 (0.50%)
Infra errors: 1 (0.50%)
Canceled jobs: 0 (0.00%)
Device type: dragonboard-845c
Total jobs: 117
Total errors: 27 (23.08%)
Error type: Test
Error count: 26 (22.22%)
Error: No match for error type 'Test', message 'tradefed - adb device lost[927ecf10]'
Count: 4 (3.42%)
IDs:
db845c-11:
8288453 8288918 8288919 8289308
Error: No match for error type 'Test', message 'tradefed - adb device lost[db449025]'
Count: 2 (1.71%)
IDs:
db845c-14:
8288751 8289279
Error: No match for error type 'Test', message 'tradefed - adb device lost[f290bb24]'
Count: 3 (2.56%)
IDs:
db845c-12:
8288452 8288926 8288932
Error: No match for error type 'Test', message 'tradefed - adb device lost[dd3b965f]'
Count: 1 (0.85%)
IDs:
db845c-02:
8288841
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 4 (3.42%)
IDs:
db845c-02:
8288794 8288797
db845c-07:
8287325 8287328
Error: Device NOT found!
Count: 6 (5.13%)
IDs:
db845c-02:
8288601 8288608 8288678 8288708 8288715
8288776
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 599 seconds'
Count: 2 (1.71%)
IDs:
db845c-12:
8288753
db845c-14:
8288669
Error: No match for error type 'Test', message 'tradefed - adb device lost[c7cf1849]'
Count: 1 (0.85%)
IDs:
db845c-07:
8288718
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 593 seconds'
Count: 2 (1.71%)
IDs:
db845c-02:
8287811
db845c-07:
8288631
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 293 seconds'
Count: 1 (0.85%)
IDs:
db845c-07:
8288576
Error type: Infrastructure
Error count: 1 (0.85%)
Error: fastboot-flash-action timed out
Count: 1 (0.85%)
IDs:
db845c-03:
8287964
Device type: sm8550-hdk
Total jobs: 15
Total errors: 3 (20.00%)
Error type: Test
Error count: 3 (20.00%)
Error: Device NOT found!
Count: 1 (6.67%)
IDs:
sm8550-hdk-01:
8288763
Error: No match for error type 'Test', message 'tradefed - adb device lost[124d1c34]'
Count: 2 (13.33%)
IDs:
sm8550-hdk-01:
8288761 8288762
Device type: hi6220-hikey-r2
Total jobs: 10
Total errors: 9 (90.00%)
Error type: Test
Error count: 9 (90.00%)
Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed'
Count: 4 (40.00%)
IDs:
hikey-6220-r2-01:
8288894 8288914
hikey-6220-r2-05:
8288821
hikey-6220-r2-06:
8288772
Error: No match for error type 'Test', message 'tradefed - adb device lost[4595FE84003F72FC]'
Count: 2 (20.00%)
IDs:
hikey-6220-r2-04:
8288851 8288906
Error: No match for error type 'Test', message 'tradefed - adb device lost[235D989C003B0752]'
Count: 2 (20.00%)
IDs:
hikey-6220-r2-03:
8287987 8288880
Error: No match for error type 'Test', message 'tradefed - adb device lost[191FC75C003312EE]'
Count: 1 (10.00%)
IDs:
hikey-6220-r2-06:
8287980
Device type: qrb5165-rb5
Total jobs: 54
Total errors: 20 (37.04%)
Error type: Test
Error count: 18 (33.33%)
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 9 (16.67%)
IDs:
rb5-03:
8288747 8288800
rb5-07:
8288606 8288621 8288658 8288660 8288665
8288730 8288749
Error: No match for error type 'Test', message 'tradefed - adb device lost[74d67c95]'
Count: 5 (9.26%)
IDs:
rb5-03:
8287817 8288579 8288716 8288729 8288750
Error: No match for error type 'Test', message 'tradefed - adb device lost[d8a1879f]'
Count: 2 (3.70%)
IDs:
rb5-07:
8288623 8288724
Error: Device NOT found!
Count: 1 (1.85%)
IDs:
rb5-03:
8288600
Error: No match for error type 'Test', message 'tradefed - adb device lost[f11b68eb]'
Count: 1 (1.85%)
IDs:
rb5-05:
8287810
Error type: Bug
Error count: 2 (3.70%)
Error: No match for error type 'Bug', message 'Failed to share device '/dev/bus/usb/003/074' to docker container lava-8288743-2.10.3-c8875fa1-3c8c-4a55-8628-fc453551b876'
Count: 1 (1.85%)
IDs:
rb5-05:
8288743
Error: No match for error type 'Bug', message 'Failed to share device '/dev/bus/usb/003/094' to docker container lava-8288653-2.10.3-cdf2f49c-d73d-4275-a07a-04c43d0e966b'
Count: 1 (1.85%)
IDs:
rb5-05:
8288653
Device type: hi960-hikey
Total jobs: 1
Total errors: 0 (0.00%)
Device type: x15
Total jobs: 3
Total errors: 1 (33.33%)
Error type: Job
Error count: 1 (33.33%)
Error: No match for error type 'Job', message 'login-action timed out after 1774 seconds'
Count: 1 (33.33%)
IDs:
x15-06:
8288065
Device type: rk3399-rock-pi-4b
Total jobs: 1
Total errors: 0 (0.00%)
Total jobs: 89
Total errors: 9 (10.11%)
LAVA errors: 0 (0.00%)
Test errors: 0 (0.00%)
Job errors: 8 (8.99%)
Infra errors: 1 (1.12%)
Canceled jobs: 0 (0.00%)
Device type: beaglebone-black
Total jobs: 29
Total errors: 0 (0.00%)
Device type: bcm2837-rpi-3-b-32
Total jobs: 15
Total errors: 0 (0.00%)
Device type: x15
Total jobs: 23
Total errors: 1 (4.35%)
Error type: Job
Error count: 1 (4.35%)
Error: No match for error type 'Job', message 'login-action timed out after 273 seconds'
Count: 1 (4.35%)
IDs:
x15-01:
4150206
Device type: dragonboard-845c
Total jobs: 12
Total errors: 1 (8.33%)
Error type: Infrastructure
Error count: 1 (8.33%)
Error: Connection closed
Count: 1 (8.33%)
IDs:
dragonboard-845c-05:
4150180
Device type: dragonboard-410c
Total jobs: 8
Total errors: 7 (87.50%)
Error type: Job
Error count: 7 (87.50%)
Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.''
Count: 7 (87.50%)
IDs:
dragonboard-410c-01:
4150156 4150174 4150197 4150214 4150232
4150236 4150250
Device type: imx93-11x11-evk
Total jobs: 1
Total errors: 0 (0.00%)
Device type: hi6220-hikey-r2
Total jobs: 1
Total errors: 0 (0.00%)
Total jobs: 43
Total errors: 3 (6.98%)
LAVA errors: 0 (0.00%)
Test errors: 0 (0.00%)
Job errors: 3 (6.98%)
Infra errors: 0 (0.00%)
Canceled jobs: 0 (0.00%)
Device type: dragonboard-410c
Total jobs: 4
Total errors: 3 (75.00%)
Error type: Job
Error count: 3 (75.00%)
Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.''
Count: 3 (75.00%)
IDs:
dragonboard-410c-01:
4150108 4150123 4150141
Device type: dragonboard-845c
Total jobs: 7
Total errors: 0 (0.00%)
Device type: x15
Total jobs: 10
Total errors: 0 (0.00%)
Device type: bcm2837-rpi-3-b-32
Total jobs: 7
Total errors: 0 (0.00%)
Device type: beaglebone-black
Total jobs: 13
Total errors: 0 (0.00%)
Device type: imx93-11x11-evk
Total jobs: 1
Total errors: 0 (0.00%)
Device type: hi6220-hikey-r2
Total jobs: 1
Total errors: 0 (0.00%)
+cc Steve
On Sun, May 25, 2025 at 06:18:42PM +0100, David Laight wrote:
> On Fri, 23 May 2025 20:01:33 +0200
> "Arnd Bergmann" <arnd(a)arndb.de> wrote:
>
> > On Fri, May 23, 2025, at 19:11, Kent Overstreet wrote:
> > > On Fri, May 23, 2025 at 05:17:15PM +0200, Arnd Bergmann wrote:
> > >>
> > >> - KASAN_STACK adds extra redzones for each variable
> > >> - KASAN_STACK further prevents stack slots from getting
> > >> reused inside one function, in order to better pinpoint
> > >> which instance caused problems like out-of-scope access
> > >> - passing structures by value causes them to be put on
> > >> the stack on some architectures, even when the structure
> > >> size is only one or two registers
> > >
> > > We mainly do this with bkey_s_c, which is just two words: on x86_64,
> > > that gets passed in registers. Is riscv different?
> >
> > Not sure, I think it's mostly older ABIs that are limited,
> > either not passing structures in registers at all, or only
> > possibly one but not two of them.
> >
> > >> - sanitizers turn off optimizations that lead to better
> > >> stack usage
> > >> - in some cases, the missed optimization ends up causing
> > >> local variables to get spilled to the stack many times
> > >> because of a combination of all the above.
> > >
> > > Yeesh.
> > >
> > > I suspect we should be running with a larger stack when the sanitizers
> > > are running, and perhaps tweak the warnings accordingly. I did a bunch
> > > of stack usage work after I found a kmsan build was blowing out the
> > > stack, but then running with max stack usage tracing enabled showed it
> > > to be a largely non issue on non-sanitizer builds, IIRC.
> >
> > Enabling KASAN does double the available stack space. However, I don't
> > think we should use that as an excuse to raise the per-function
> > warning limit, because
> >
> > - the majority of all function stacks do not grow that much when
> > sanitizers are enabled
> > - allmodconfig enables KASAN and should still catch mistakes
> > where a driver accidentally puts a large structure on the stack
>
> That is rather annoying when you want to look at the generated code :-(
>
> > - 2KB on 64-bit targes is a really large limit. At some point
> > in the past I had a series that lowered the limit to 1536 byte
> > for 64-bit targets, but I never managed to get all the changes
> > merged.
>
> I've a cunning plan to do a proper static analysis of stack usage.
> It is a 'simple' matter of getting objtool to output all calls with
> the stack offset.
> Indirect calls need the function hashes from fine-ibt, but also need
> clang to support 'hash seeds' to disambiguate all the void (*)(void *)
> functions.
> That'll first barf at all recursion, and then, I expect, show a massive
> stack use inside snprintf() in some error path.
I suspect recursion will make the results you get with that approach
useless.
We already have "trace max stack", but that only checks at process exit,
so it doesn't tell you much.
We could do better with tracing - just inject a trampoline that checks
the current stack usage against the maximum stack usage we've seen, and
emits a trace event with a stack trace if it's greater.
(and now Steve's going to tell us he's already done this :)
Total jobs: 34
Total errors: 4 (11.76%)
LAVA errors: 0 (0.00%)
Test errors: 0 (0.00%)
Job errors: 2 (5.88%)
Infra errors: 2 (5.88%)
Canceled jobs: 0 (0.00%)
Device type: dragonboard-410c
Total jobs: 3
Total errors: 2 (66.67%)
Error type: Job
Error count: 2 (66.67%)
Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.''
Count: 2 (66.67%)
IDs:
dragonboard-410c-01:
4150062 4150083
Device type: dragonboard-845c
Total jobs: 7
Total errors: 1 (14.29%)
Error type: Infrastructure
Error count: 1 (14.29%)
Error: Connection closed
Count: 1 (14.29%)
IDs:
dragonboard-845c-08:
4150045
Device type: beaglebone-black
Total jobs: 10
Total errors: 1 (10.00%)
Error type: Infrastructure
Error count: 1 (10.00%)
Error: matched a bootloader error message
Count: 1 (10.00%)
IDs:
beaglebone-black03:
4150041
Device type: x15
Total jobs: 7
Total errors: 0 (0.00%)
Device type: bcm2837-rpi-3-b-32
Total jobs: 5
Total errors: 0 (0.00%)
Device type: imx93-11x11-evk
Total jobs: 1
Total errors: 0 (0.00%)
Device type: hi6220-hikey-r2
Total jobs: 1
Total errors: 0 (0.00%)
Total jobs: 294
Total errors: 91 (30.95%)
LAVA errors: 0 (0.00%)
Test errors: 76 (25.85%)
Job errors: 4 (1.36%)
Infra errors: 11 (3.74%)
Canceled jobs: 0 (0.00%)
Device type: hi960-hikey
Total jobs: 35
Total errors: 15 (42.86%)
Error type: Test
Error count: 14 (40.00%)
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 4 (11.43%)
IDs:
hi960-hikey-01:
8285848 8286197
hi960-hikey-02:
8285308 8285830
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 596 seconds'
Count: 1 (2.86%)
IDs:
hi960-hikey-05:
8285764
Error: Device NOT found!
Count: 2 (5.71%)
IDs:
hi960-hikey-02:
8285305
hi960-hikey-05:
8285333
Error: No match for error type 'Test', message 'lava-test-interactive timed out after 600 seconds'
Count: 3 (8.57%)
IDs:
hi960-hikey-03:
8285284
hi960-hikey-04:
8285282 8285285
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 593 seconds'
Count: 2 (5.71%)
IDs:
hi960-hikey-02:
8285190
hi960-hikey-05:
8285175
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 295 seconds'
Count: 1 (2.86%)
IDs:
hi960-hikey-02:
8285183
Error: No match for error type 'Test', message 'tradefed - adb device lost[57DD383701799F9A]'
Count: 1 (2.86%)
IDs:
hi960-hikey-05:
8285169
Error type: Job
Error count: 1 (2.86%)
Error: No match for error type 'Job', message 'login-action timed out after 863 seconds'
Count: 1 (2.86%)
IDs:
hi960-hikey-04:
8285170
Device type: dragonboard-845c
Total jobs: 143
Total errors: 30 (20.98%)
Error type: Test
Error count: 23 (16.08%)
Error: Device NOT found!
Count: 6 (4.20%)
IDs:
db845c-02:
8284115 8284221
db845c-11:
8285676 8285690 8285692
db845c-14:
8286176
Error: No match for error type 'Test', message 'tradefed - adb device lost[f290bb24]'
Count: 2 (1.40%)
IDs:
db845c-12:
8281108 8285842
Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed'
Count: 6 (4.20%)
IDs:
db845c-02:
8284662
db845c-03:
8285207
db845c-04:
8284661 8285208
db845c-10:
8284664 8285205
Error: No match for error type 'Test', message 'tradefed - adb device lost[dd3b965f]'
Count: 2 (1.40%)
IDs:
db845c-02:
8284083 8284657
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 5 (3.50%)
IDs:
db845c-02:
8284087 8284223 8284229
db845c-07:
8284105 8284117
Error: No match for error type 'Test', message 'tradefed - adb device lost[927ecf10]'
Count: 1 (0.70%)
IDs:
db845c-11:
8283234
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 579 seconds'
Count: 1 (0.70%)
IDs:
db845c-07:
8283187
Error type: Infrastructure
Error count: 6 (4.20%)
Error: Reboot failed
Count: 2 (1.40%)
IDs:
db845c-11:
8285675
db845c-13:
8285674
Error: Connection closed
Count: 1 (0.70%)
IDs:
db845c-14:
8285673
Error: Unable to fetch git repository
Count: 2 (1.40%)
IDs:
db845c-09:
8283216
db845c-14:
8283236
Error: fastboot-flash-action timed out
Count: 1 (0.70%)
IDs:
db845c-07:
8283212
Error type: Job
Error count: 1 (0.70%)
Error: No match for error type 'Job', message 'download-postprocess-docker timed out after 4492 seconds'
Count: 1 (0.70%)
IDs:
db845c-09:
8283186
Device type: hi6220-hikey-r2
Total jobs: 16
Total errors: 10 (62.50%)
Error type: Test
Error count: 9 (56.25%)
Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed'
Count: 6 (37.50%)
IDs:
hikey-6220-r2-05:
8285745
hikey-6220-r2-06:
8285271 8285738 8285740 8285746 8285768
Error: No match for error type 'Test', message 'lava-test-interactive timed out after 600 seconds'
Count: 1 (6.25%)
IDs:
hikey-6220-r2-01:
8285741
Error: No match for error type 'Test', message 'tradefed - adb device lost[7B1579B5003E507A]'
Count: 1 (6.25%)
IDs:
hikey-6220-r2-05:
8285270
Error: No match for error type 'Test', message 'tradefed - adb device lost[235D989C003B0752]'
Count: 1 (6.25%)
IDs:
hikey-6220-r2-03:
8285269
Error type: Infrastructure
Error count: 1 (6.25%)
Error: No match for error type 'Infrastructure', message 'wait-device-boardid timed out after 1169 seconds'
Count: 1 (6.25%)
IDs:
hikey-6220-r2-06:
8285272
Device type: qrb5165-rb5
Total jobs: 83
Total errors: 31 (37.35%)
Error type: Test
Error count: 25 (30.12%)
Error: Device NOT found!
Count: 9 (10.84%)
IDs:
rb5-03:
8284101 8284180 8284213 8284236 8284638
8285729
rb5-06:
8284235
rb5-07:
8284144 8284238
Error: No match for error type 'Test', message 'tradefed - adb device lost[74d67c95]'
Count: 1 (1.20%)
IDs:
rb5-03:
8285711
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 10 (12.05%)
IDs:
rb5-03:
8284636 8284679 8285664 8285669 8285709
rb5-07:
8284145 8284207 8284300 8285329 8285331
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 590 seconds'
Count: 1 (1.20%)
IDs:
rb5-03:
8285659
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 597 seconds'
Count: 1 (1.20%)
IDs:
rb5-03:
8285658
Error: No match for error type 'Test', message 'lava-docker-test timed out after 299 seconds'
Count: 1 (1.20%)
IDs:
rb5-07:
8284183
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 297 seconds'
Count: 1 (1.20%)
IDs:
rb5-06:
8284177
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 296 seconds'
Count: 1 (1.20%)
IDs:
rb5-06:
8284151
Error type: Infrastructure
Error count: 4 (4.82%)
Error: fastboot-flash-action timed out
Count: 2 (2.41%)
IDs:
rb5-06:
8284208
rb5-07:
8284234
Error: Unable to fetch git repository
Count: 2 (2.41%)
IDs:
rb5-01:
8284176
rb5-05:
8284179
Error type: Job
Error count: 2 (2.41%)
Error: No match for error type 'Job', message 'Post-processing of downloads failed'
Count: 1 (1.20%)
IDs:
rb5-01:
8284150
Error: No match for error type 'Job', message 'test-definition timed out after 607 seconds'
Count: 1 (1.20%)
IDs:
rb5-01:
8284146
Device type: x86
Total jobs: 2
Total errors: 0 (0.00%)
Device type: sm8550-hdk
Total jobs: 15
Total errors: 5 (33.33%)
Error type: Test
Error count: 5 (33.33%)
Error: No match for error type 'Test', message 'tradefed - adb device lost[124d1c34]'
Count: 3 (20.00%)
IDs:
sm8550-hdk-01:
8283145 8283148 8284186
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 7199 seconds'
Count: 1 (6.67%)
IDs:
sm8550-hdk-01:
8283138
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 599 seconds'
Count: 1 (6.67%)
IDs:
sm8550-hdk-01:
8282763
Total jobs: 123
Total errors: 13 (10.57%)
LAVA errors: 0 (0.00%)
Test errors: 0 (0.00%)
Job errors: 12 (9.76%)
Infra errors: 1 (0.81%)
Canceled jobs: 0 (0.00%)
Device type: dragonboard-410c
Total jobs: 11
Total errors: 10 (90.91%)
Error type: Job
Error count: 10 (90.91%)
Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.''
Count: 10 (90.91%)
IDs:
dragonboard-410c-01:
4149883 4149913 4149917 4149948 4149954
4149981 4149990 4150001 4150018 4150038
Device type: dragonboard-845c
Total jobs: 15
Total errors: 1 (6.67%)
Error type: Infrastructure
Error count: 1 (6.67%)
Error: Connection closed
Count: 1 (6.67%)
IDs:
dragonboard-845c-04:
4149947
Device type: bcm2837-rpi-3-b-32
Total jobs: 21
Total errors: 0 (0.00%)
Device type: x15
Total jobs: 33
Total errors: 2 (6.06%)
Error type: Job
Error count: 2 (6.06%)
Error: No match for error type 'Job', message 'login-action timed out after 274 seconds'
Count: 2 (6.06%)
IDs:
x15-01:
4149932 4149974
Device type: beaglebone-black
Total jobs: 41
Total errors: 0 (0.00%)
Device type: imx93-11x11-evk
Total jobs: 1
Total errors: 0 (0.00%)
Device type: hi6220-hikey-r2
Total jobs: 1
Total errors: 0 (0.00%)