[Eas-dev] WALT panic on Hikey960

Hi Vikram, Joonwoo,

[ + EAS mailing list ]

On Hikey960 with EASv1.3, I encountered many times for WALT panic, it reports the bug from below two functions; you also could see log in the below. Before I dig into this, could you give some suggestion for this? Or if there have some existed fixing for this? Thanks in advance.

void walt_dec_cumulative_runnable_avg(struct rq *rq, struct task_struct *p) { rq->cumulative_runnable_avg -= p->ravg.demand; BUG_ON((s64)rq->cumulative_runnable_avg < 0); }

static void fixup_cumulative_runnable_avg(struct rq *rq, struct task_struct *p, u64 new_task_load) { s64 task_load_delta = (s64)new_task_load - task_load(p);

rq->cumulative_runnable_avg += task_load_delta; if ((s64)rq->cumulative_runnable_avg < 0) panic("cra less than zero: tld: %lld, task_load(p) = %u\n", task_load_delta, task_load(p)); }

--- Panic Log ---

[ 1108.441865] init: Untracked pid 15425 exited with status 0 [ 1108.657107] ------------[ cut here ]------------ [ 1108.661746] kernel BUG at kernel/sched/walt.c:109! [ 1108.666538] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 1108.672026] CPU: 1 PID: 1248 Comm: kschedfreq:0 Not tainted 4.4.78-07635-g0255026 #45 [ 1108.679851] Hardware name: HiKey960 (DT) [ 1108.683770] task: ffffffc0b166c080 ti: ffffffc0b0e64000 task.ti: ffffffc0b0e64000 [ 1108.691261] PC is at walt_dec_cumulative_runnable_avg+0x40/0x44 [ 1108.697179] LR is at dequeue_task_rt+0x40/0x8c [ 1108.701617] pc : [<ffffff8008112428>] lr : [<ffffff800810c82c>] pstate: 60000185 [ 1108.709007] sp : ffffffc0b0e67b90 [ 1108.712315] x29: ffffffc0b0e67b90 x28: 0000000000000001 [ 1108.717633] x27: ffffff8008bc4fc4 x26: ffffffc0bff13400 [ 1108.722948] x25: ffffffc0b166c6c8 x24: 0000000000000000 [ 1108.728263] x23: ffffff8009095000 x22: ffffffc0b166c080 [ 1108.733579] x21: ffffffc0bff13be8 x20: ffffffc0b166c080 [ 1108.738895] x19: ffffffc0bff13400 x18: 0000000000000000 [ 1108.744209] x17: 0000000000000000 x16: 0000000000000000 [ 1108.749524] x15: 0000000000000000 x14: 0000000000000000 [ 1108.754839] x13: 0000000000000000 x12: 0000000034d5d91d [ 1108.760156] x11: ffffff8008be13cc x10: 00000000000009d0 [ 1108.765471] x9 : ffffffc0b0e64000 x8 : ffffffc0b0e67ce0 [ 1108.770786] x7 : ffffffc0ae6cfe30 x6 : ffffff8009095000 [ 1108.776101] x5 : 0000000000000001 x4 : 00000040b6ea8000 [ 1108.781415] x3 : 0000000000000002 x2 : 0000000000000000 [ 1108.786730] x1 : fffffffffffedce2 x0 : 00000000000bc75e [ 1108.792047] [ 1108.792047] SP: 0xffffffc0b0e67b10: [ 1108.797006] 7b10 b166c080 ffffffc0 09095000 ffffff80 00000000 00000000 b166c6c8 ffffffc0 [ 1108.805223] 7b30 bff13400 ffffffc0 08bc4fc4 ffffff80 00000001 00000000 b0e67b90 ffffffc0 [ 1108.813440] 7b50 0810c82c ffffff80 b0e67b90 ffffffc0 08112428 ffffff80 60000185 00000000 [ 1108.821656] 7b70 b0e67ba0 ffffffc0 0810c538 ffffff80 ffffffff ffffffff 0810c574 ffffff80 [ 1108.829872] 7b90 b0e67bc0 ffffffc0 0810c82c ffffff80 bff13400 ffffffc0 0810c820 ffffff80 [ 1108.838090] 7bb0 bff13400 ffffffc0 b166c080 ffffffc0 b0e67bf0 ffffffc0 080eead8 ffffff80 [ 1108.846306] 7bd0 bff13400 ffffffc0 0906b000 ffffff80 bff13400 ffffffc0 0906b000 ffffff80 [ 1108.854522] 7bf0 b0e67c20 ffffffc0 08bc4b80 ffffff80 bff13400 ffffffc0 08bc47e4 ffffff80 [ 1108.862741] [ 1108.862741] X1: 0xfffffffffffedc62: [ 1108.867700] dc60 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.875924] dc80 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.884140] dca0 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.892358] dcc0 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.900576] dce0 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.908793] dd00 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.917009] dd20 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.925227] dd40 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.933446] dd60 ******** ******** ******** ******** ******** ******** ******** ******** [ 1108.941666] [ 1108.941666] X7: 0xffffffc0ae6cfdb0: [ 1108.946625] fdb0 0000c350 00000000 00000001 00000000 00000000 00000000 ae6cfeb0 ffffffc0 [ 1108.954840] fdd0 00000000 00000000 00000001 00000000 00000000 00000000 0000c350 00000001 [ 1108.963055] fdf0 ae6cfe90 ffffffc0 0813dd98 ffffff80 b55c0418 0000007f 00000000 00000000 [ 1108.971271] fe10 ffffffff ffffffff b76b299c 0000007f ae6cfe60 ffffffc0 080efae0 00000001 [ 1108.979489] fe30 b0e67ce0 ffffffc0 00000000 00000000 00000000 00000000 0c0b8de9 00000102 [ 1108.987707] fe50 0c0aca99 00000102 0813c598 ffffff80 bff0ed40 ffffffc0 00000001 00000825 [ 1108.995923] fe70 08bc80e8 ffffff80 696c616d 6d656d2d 7275702d 00006567 b11f3100 ffffffc0 [ 1109.004140] fe90 00000000 00000000 08085f30 ffffff80 00000000 00000000 b6ee1020 0000007f [ 1109.012359] [ 1109.012359] X8: 0xffffffc0b0e67c60: [ 1109.017318] 7c60 b1092800 ffffffc0 091a8000 ffffff80 00000000 00000000 00009f4c 00000000 [ 1109.025536] 7c80 b0e67ca0 ffffffc0 08bc82a8 ffffff80 b0e67d98 ffffffc0 00000100 00000000 [ 1109.033752] 7ca0 b0e67d40 ffffffc0 08bc8348 ffffff80 b0e67d98 ffffffc0 00000064 00000000 [ 1109.041970] 7cc0 b11bb580 ffffffc0 b0e67d30 ffffffc0 0808e7e4 ffffff80 b166c080 00000001 [ 1109.050186] 7ce0 bff0f2d1 ffffffc0 ae6cfe30 ffffffc0 bff0f170 ffffffc0 09828528 00000102 [ 1109.058405] 7d00 0980fe88 00000102 0813c598 ffffff80 bff0ed40 ffffffc0 00000001 000004e0 [ 1109.066621] 7d20 08bc829c ffffff80 6863736b 72666465 303a7165 00000000 b166c080 ffffffc0 [ 1109.074839] 7d40 b0e67d70 ffffffc0 08bc8068 ffffff80 026e40e0 00000000 001a13c8 00000000 [ 1109.083058] [ 1109.083058] X9: 0xffffffc0b0e63f80: [ 1109.088015] 3f80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.096232] 3fa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.104448] 3fc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.112665] 3fe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.120883] 4000 00000000 00000000 ffffffff ffffffff b166c080 ffffffc0 00000003 00000001 [ 1109.129100] 4020 57ac6e9d 00000000 32273028 0d0f1c33 10233816 201d111a 3b013e3c 532f2f26 [ 1109.137317] 4040 ae7a3648 ffffffc0 ae354db8 ffffffc0 ae7a36c0 ffffffc0 ae7a36c0 ffffffc0 [ 1109.145535] 4060 00000001 00000000 aca585e0 ffffffc0 aca584e0 ffffffc0 07fb7c71 00000000 [ 1109.153753] [ 1109.153753] X19: 0xffffffc0bff13380: [ 1109.158799] 3380 00000000[ 1109.160774] mali e82c0000.mali: Reset interrupt didn't reach CPU. Check interrupt assignments.

[ 1109.169934] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.177103] 33a0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.185319] 33c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.193537] 33e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.201755] 3400 fab6faaf 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.209971] 3420 00000000 00000000 00000000 00000000 00031548 00000001 00000000 00000000 [ 1109.218187] 3440 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000 [ 1109.226402] 3460 00017d53 00000000 0002db81 00000000 00000000 00000000 00000000 00000000 [ 1109.234619] [ 1109.234619] X20: 0xffffffc0b166c000: [ 1109.239664] c000 0b030d00 08010211 100f0d05 07091214 040e0a0c ffffff00 00000001 00000000 [ 1109.247881] c020 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.256097] c040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.264313] c060 00000000 00000010 00000000 00000000 ffffffff 0000003f ffffffff 0000003f [ 1109.272528] c080 00000002 00000000 b0e64000 ffffffc0 00000003 04208040 00000000 00000000 [ 1109.280745] c0a0 00000000 00000000 00000001 00000000 00031435 00000001 84b47180 ffffffc0 [ 1109.288960] c0c0 00000001 00000001 00000031 00000078 00000031 00000032 08be15c8 ffffff80 [ 1109.297178] c0e0 00000400 00000000 00400000 00000000 00000001 00000000 00000000 00000000 [ 1109.305395] [ 1109.305395] X21: 0xffffffc0bff13b68: [ 1109.310440] 3b68 bff13b60 ffffffc0 bff13b70 ffffffc0 bff13b70 ffffffc0 bff13b80 ffffffc0 [ 1109.318658] 3b88 bff13b80 ffffffc0 bff13b90 ffffffc0 bff13b90 ffffffc0 bff13ba0 ffffffc0 [ 1109.326875] 3ba8 bff13ba0 ffffffc0 bff13bb0 ffffffc0 bff13bb0 ffffffc0 00000000 00000064 [ 1109.335092] 3bc8 00000064 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.343309] 3be8 bff13be8 ffffffc0 bff13be8 ffffffc0 00000000 00000000 00000000 00000000 [ 1109.351526] 3c08 00000000 00000000 0810cb88 ffffff80 00020002 00000000 00000000 00000000 [ 1109.359742] 3c28 006303e4 00000000 389fd980 00000000 f80df80d 00000000 00000000 00000000 [ 1109.367959] 3c48 bff13400 ffffffc0 091c42a0 ffffff80 00000000 00000000 00000000 00000000 [ 1109.376176] [ 1109.376176] X22: 0xffffffc0b166c000: [ 1109.381221] c000 0b030d00 08010211 100f0d05 07091214 040e0a0c ffffff00 00000001 00000000 [ 1109.389439] c020 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.397656] c040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.405873] c060 00000000 00000010 00000000 00000000 ffffffff 0000003f ffffffff 0000003f [ 1109.414090] c080 00000002 00000000 b0e64000 ffffffc0 00000003 04208040 00000000 00000000 [ 1109.422307] c0a0 00000000 00000000 00000001 00000000 00031435 00000001 84b47180 ffffffc0 [ 1109.430524] c0c0 00000001 00000001 00000031 00000078 00000031 00000032 08be15c8 ffffff80 [ 1109.438741] c0e0 00000400 00000000 00400000 00000000 00000001 00000000 00000000 00000000 [ 1109.446959] [ 1109.446959] X25: 0xffffffc0b166c648: [ 1109.452004] c648 b166c648 ffffffc0 b166c648 ffffffc0 b10c8910 ffffffc0 b10c8910 ffffffc0 [ 1109.460220] c668 b0e67ea0 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.468436] c688 00000073 00000000 00000000 00000000 00000073 00000000 00000000 00000000 [ 1109.476653] c6a8 00000000 00000000 00000000 00000000 00000000 00000000 00009814 00000000 [ 1109.484869] c6c8 00000004 00000000 09ec2856 00000001 09ec2856 00000001 00000000 00000000 [ 1109.493087] c6e8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.501305] c708 b166c708 ffffffc0 b166c708 ffffffc0 b166c718 ffffffc0 b166c718 ffffffc0 [ 1109.509521] c728 b166c728 ffffffc0 b166c728 ffffffc0 00000000 00000000 b0e0b880 ffffffc0 [ 1109.517740] [ 1109.517740] X26: 0xffffffc0bff13380: [ 1109.522785] 3380 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.531001] 33a0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.539217] 33c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.547434] 33e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.555651] 3400 fab6faaf 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1109.563868] 3420 00000000 00000000 00000000 00000000 00031548 00000001 00000000 00000000 [ 1109.572085] 3440 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000 [ 1109.580301] 3460 00017d53 00000000 0002db81 00000000 00000000 00000000 00000000 00000000 [ 1109.588521] [ 1109.588521] X29: 0xffffffc0b0e67b10: [ 1109.593567] 7b10 b166c080 ffffffc0 09095000 ffffff80 00000000 00000000 b166c6c8 ffffffc0 [ 1109.601784] 7b30 bff13400 ffffffc0 08bc4fc4 ffffff80 00000001 00000000 b0e67b90 ffffffc0 [ 1109.610001] 7b50 0810c82c ffffff80 b0e67b90 ffffffc0 08112428 ffffff80 60000185 00000000 [ 1109.618218] 7b70 b0e67ba0 ffffffc0 0810c538 ffffff80 ffffffff ffffffff 0810c574 ffffff80 [ 1109.626436] 7b90 b0e67bc0 ffffffc0 0810c82c ffffff80 bff13400 ffffffc0 0810c820 ffffff80 [ 1109.634653] 7bb0 bff13400 ffffffc0 b166c080 ffffffc0 b0e67bf0 ffffffc0 080eead8 ffffff80 [ 1109.642869] 7bd0 bff13400 ffffffc0 0906b000 ffffff80 bff13400 ffffffc0 0906b000 ffffff80 [ 1109.651086] 7bf0 b0e67c20 ffffffc0 08bc4b80 ffffff80 bff13400 ffffffc0 08bc47e4 ffffff80 [ 1109.659303] [ 1109.660788] Process kschedfreq:0 (pid: 1248, stack limit = 0xffffffc0b0e64020) [ 1109.668006] Stack: (0xffffffc0b0e67b90 to 0xffffffc0b0e68000) [ 1109.673748] 7b80: ffffffc0b0e67bc0 ffffff800810c82c [ 1109.681574] 7ba0: ffffffc0bff13400 ffffff800810c820 ffffffc0bff13400 ffffffc0b166c080 [ 1109.689399] 7bc0: ffffffc0b0e67bf0 ffffff80080eead8 ffffffc0bff13400 ffffff800906b000 [ 1109.697225] 7be0: ffffffc0bff13400 ffffff800906b000 ffffffc0b0e67c20 ffffff8008bc4b80 [ 1109.705052] 7c00: ffffffc0bff13400 ffffff8008bc47e4 ffffffc000000001 ffffffc0b166c080 [ 1109.712877] 7c20: ffffffc0b0e67c80 ffffff8008bc4fc4 ffffffc0b0e64000 0000000000000001 [ 1109.720703] 7c40: 00000000000186a0 ffffffc0b0e64000 ffffff8008be0000 00000000001a13c8 [ 1109.728529] 7c60: ffffffc0b1092800 ffffff80091a8000 0000000000000000 0000000000009f4c [ 1109.736355] 7c80: ffffffc0b0e67ca0 ffffff8008bc82a8 ffffffc0b0e67d98 0000000000000100 [ 1109.744181] 7ca0: ffffffc0b0e67d40 ffffff8008bc8348 ffffffc0b0e67d98 0000000000000064 [ 1109.752006] 7cc0: ffffffc0b11bb580 ffffffc0b0e67d30 ffffff800808e7e4 00000001b166c080 [ 1109.759832] 7ce0: ffffffc0bff0f2d1 ffffffc0ae6cfe30 ffffffc0bff0f170 0000010209828528 [ 1109.767658] 7d00: 000001020980fe88 ffffff800813c598 ffffffc0bff0ed40 000004e000000001 [ 1109.775483] 7d20: ffffff8008bc829c 726664656863736b 00000000303a7165 ffffffc0b166c080 [ 1109.783309] 7d40: ffffffc0b0e67d70 ffffff8008bc8068 00000000026e40e0 00000000001a13c8 [ 1109.791135] 7d60: 00000000000186a0 0000000108142a54 ffffffc0b0e67da0 ffffff800811a770 [ 1109.798961] 7d80: 000001020980f324 ffffffc0b0e64000 ffffffc0b11bb580 00000000026e40e0 [ 1109.806787] 7da0: ffffffc0b0e67e20 ffffff80080e1a14 ffffffc0b11bb400 ffffffc0b0e64000 [ 1109.814613] 7dc0: ffffff80091c41c8 ffffffc0b1092800 ffffff800811a6b0 0000000000000000 [ 1109.822438] 7de0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.830263] 7e00: ffffffc0b11bb400 ffffffc0b0e64000 ffffff80091c41c8 ffffffc000000032 [ 1109.838089] 7e20: 0000000000000000 ffffff8008085ed0 ffffff80080e192c ffffffc0b11bb400 [ 1109.845914] 7e40: 0000000000000000 0000000000000000 0000000000000000 ffffff80080efe18 [ 1109.853740] 7e60: 0000000000000000 0000000000000000 0000000000000000 ffffffc0b1092800 [ 1109.861565] 7e80: ffffffc000000000 ffffff8000000000 ffffffc0b0e67e90 ffffffc0b0e67e90 [ 1109.869392] 7ea0: 0000000000000000 ffffff8000000000 ffffffc0b0e67eb0 ffffffc0b0e67eb0 [ 1109.877217] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.885042] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.892867] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.900692] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.908518] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.916344] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.924169] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.931994] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.939820] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000 [ 1109.947646] 7fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.955470] Call trace: [ 1109.957911] Exception stack(0xffffffc0b0e679c0 to 0xffffffc0b0e67af0) [ 1109.964348] 79c0: ffffffc0bff13400 0000008000000000 ffffffc0b0e67b90 ffffff8008112428 [ 1109.972173] 79e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1109.979999] 7a00: 0000000000000000 0000000000000000 0000000000000000 0000000000000009 [ 1109.987824] 7a20: 0000000000000010 0000000000000010 0000000000000000 000000000000068a [ 1109.995650] 7a40: ffffffc0b0e67a90 ffffff8008bc8bac 0000000000000180 ffffff800928c688 [ 1110.003476] 7a60: 00000000000bc75e fffffffffffedce2 0000000000000000 0000000000000002 [ 1110.011301] 7a80: 00000040b6ea8000 0000000000000001 ffffff8009095000 ffffffc0ae6cfe30 [ 1110.019127] 7aa0: ffffffc0b0e67ce0 ffffffc0b0e64000 00000000000009d0 ffffff8008be13cc [ 1110.026952] 7ac0: 0000000034d5d91d 0000000000000000 0000000000000000 0000000000000000 [ 1110.034777] 7ae0: 0000000000000000 0000000000000000 [ 1110.039651] [<ffffff8008112428>] walt_dec_cumulative_runnable_avg+0x40/0x44 [ 1110.046609] [<ffffff800810c82c>] dequeue_task_rt+0x40/0x8c [ 1110.052093] [<ffffff80080eead8>] deactivate_task+0x98/0xbc [ 1110.057580] [<ffffff8008bc4b80>] __schedule+0x44c/0x7c0 [ 1110.062800] [<ffffff8008bc4fc4>] schedule+0x40/0xa0 [ 1110.067674] [<ffffff8008bc82a8>] schedule_hrtimeout_range_clock+0x94/0x100 [ 1110.074544] [<ffffff8008bc8348>] schedule_hrtimeout_range+0x34/0x40 [ 1110.080806] [<ffffff8008bc8068>] usleep_range+0x4c/0x58 [ 1110.086028] [<ffffff800811a770>] cpufreq_sched_thread+0xc0/0x1e4 [ 1110.092032] [<ffffff80080e1a14>] kthread+0xe8/0xfc [ 1110.096821] [<ffffff8008085ed0>] ret_from_fork+0x10/0x40 [ 1110.102129] Code: b7f80081 f9400bf3 a8c37bfd d65f03c0 (d4210000)