On 11/27/2012 03:20 AM, Dave Pigott wrote:

Hi all,

One overnight failure:

---

panda-es04

http://validation.linaro.org/lava-server/scheduler/job/40109

Failed to get into test image. Some sort of corruption. Went onto the board and booted test image again. It came up, and we got a command prompt, but it was still pretty flakey: https://pastebin.linaro.org/1129/

Although a reboot of the test image *might* have fixed this, it looks like either the test image was dodgy, or the board is genuinely failing. I've put back online to test.

This looks like a kernel issue we've seen before with IRQ unlock issue. Maybe this can be addressed with a newer test image?

[ 16.606079] [ BUG: bad unlock balance detected! ] [ 16.606079] ------------------------------------- [ 16.606109] syslink_daemon/1151 is trying to release lock (gatemp_module->gate_mutex) at: [ 16.606109] [<c0743b40>] mutex_unlock+0x18/0x1c [ 16.606109] but there are no more locks to release! [ 16.606109] [ 16.606170] other info that might help us debug this: [ 16.606170] no locks held by syslink_daemon/1151. [ 16.606170] [ 16.606170] stack backtrace: [ 16.606201] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from [<c0720fc8>] (dump_stack+0x20/0x24) [ 16.606201] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c072452c>] (print_unlock_inbalance_bug.part.19+0x84/0xac) [ 16.606201] [<c072452c>] (print_unlock_inbalance_bug.part.19+0x84/0xac) from [<c0094840>] (print_unlock_inbalance_bug+0x44/0x50) [ 16.606201] [<c0094840>] (print_unlock_inbalance_bug+0x44/0x50) from [<c0097388>] (__lock_release+0x9c/0xd8) [ 16.606201] [<c0097388>] (__lock_release+0x9c/0xd8) from [<c00974a0>] (lock_release+0xdc/0x100) [ 16.606201] [<c00974a0>] (lock_release+0xdc/0x100) from [<c0743a94>] (__mutex_unlock_slowpath+0xf8/0x18c) [ 16.606384] [<c0743a94>] (__mutex_unlock_slowpath+0xf8/0x18c) from [<c0743b40>] (mutex_unlock+0x18/0x1c) [ 16.606414] [<c0743b40>] (mutex_unlock+0x18/0x1c) from [<c0537f0c>] (gatehwspinlock_leave+0xc0/0xe0) [ 16.606414] [<c0537f0c>] (gatehwspinlock_leave+0xc0/0xe0) from [<c0538ea8>] (gatemp_leave+0x28/0x2c) [ 16.606414] [<c0538ea8>] (gatemp_leave+0x28/0x2c) from [<c053b21c>] (heapmemmp_alloc+0x1b8/0x200) [ 16.606414] [<c053b21c>] (heapmemmp_alloc+0x1b8/0x200) from [<c053c6e4>] (heapmemmp_ioctl+0x13c/0x4f0) [ 16.606506] [<c053c6e4>] (heapmemmp_ioctl+0x13c/0x4f0) from [<c0544dcc>] (ipc_ioc_router+0x164/0x194) [ 16.606506] [<c0544dcc>] (ipc_ioc_router+0x164/0x194) from [<c0544e64>] (ipc_ioctl+0x68/0x74) [ 16.606506] [<c0544e64>] (ipc_ioctl+0x68/0x74) from [<c0127af4>] (do_vfs_ioctl+0x2a8/0x2e4) [ 16.606597] [<c0127af4>] (do_vfs_ioctl+0x2a8/0x2e4) from [<c0127b90>] (sys_ioctl+0x60/0x84) [ 16.606628] [<c0127b90>] (sys_ioctl+0x60/0x84) from [<c00130e0>] (ret_fast_syscall+0x0/0x3c) [ 16.610687] gatehwspinlock_enter failed! status = 0xffffffed [ 16.612670] Registered user-space process for DEV_SYS_ERROR event in AppM3 [ 16.612670] Registered user-space process for DEV_WATCHDOG_ERROR event in AppM3 [ 16.612670] Registered user-space process for DEV_SYS_ERROR event in SysM3 [ 16.612670] Registered user-space process for DEV_WATCHDOG_ERROR event in SysM3 [ 17.077484] Unable to handle kernel NULL pointer dereference at virtual address 00000004 [ 17.080047] pgd = ea0e0000 [ 17.095672] [00000004] *pgd=00000000 [ 17.095672] Internal error: Oops: 805 [#1] PREEMPT SMP [ 17.119110] Modules linked in: wl12xx_sdio wl12xx [ 17.139221] CPU: 0 Tainted: G W (3.1.1-19-linaro-lt-omap #19~lt~ci~20120216183525+1329716076~4f41e9c2-Ubuntu) [ 17.158416] PC is at skb_dequeue+0x5c/0x74 [ 17.170257] LR is at _raw_spin_lock_irqsave+0x6c/0x78 [ 17.183166] pc : [<c0599324>] lr : [<c07449dc>] psr: a0000193 [ 17.183166] sp : ebb35cc8 ip : ebb35c98 fp : ebb35ce4 [ 17.197235] r10: 00000000 r9 : 0000076b r8 : ebb35df0 [ 17.223693] r7 : ee5b1ae0 r6 : ee5b1b84 r5 : ee5b1b78 r4 : ebca0200 [ 17.238250] r3 : 00000000 r2 : 00000000 r1 : 60000113 r0 : 00000000 [ 17.251831] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user [ 17.267364] Control: 10c5387d Table: aa0e004a DAC: 00000015 [ 17.275360] Process upstart-socket- (pid: 1407, stack limit = 0xebb342f8) [ 17.297088] Stack: (0xebb35cc8 to 0xebb36000) [ 17.300384] 5cc0: eea901e0 ebca0200 ebb35f64 00000095 ebb35d6c ebb35ce8 [ 17.326080] 5ce0: c0627764 c05992d4 eea90690 501520aa 00000009 ee5b1ddc 00000001 ebb34000 [ 17.326080] 5d00: ed2d10e0 ee5b1d7c ee5b1b78 00000001 00000000 00000000 00000000 ee08e4e0 [ 17.359191] 5d20: ee192ac0 00000000 00000001 00000000 00000000 00000000 60000193 c10b76bc [ 17.361938] 5d40: ebb35d5c ed2d10e0 40000040 ebb35f64 00000800 00000000 00000000 ebb35e64 [ 17.384552] 5d60: ebb35e54 ebb35d70 c059086c c062764c 40000040 ee5ad854 00000000 00000000 [ 17.409057] 5d80: 00000000 00000001 ffffffff 00000000 00000000 00000000 00000000 00000000 [ 17.415985] 5da0: eea901e0 c004ac64 00000000 00000000 a0000113 c0063898 ebb35df0 2e20b361 [ 17.415985] 5dc0: ebb35dfc ebb35e10 ee28bc00 ebb35de0 ebb35e04 ebb35de0 c0063898 c006348c [ 17.454864] 5de0: 50b99e70 2e20b361 c0098020 ee5ad840 00000001 00000001 40000040 00000800 [ 17.470703] 5e00: ed2d10e0 c0063868 ebb35d1c ebb35f64 50b99e70 ebb35e64 ebb35ea4 40000000 [ 17.470703] 5e20: bea319c0 ebb35e64 00000008 00000000 00000800 ebb35f64 ed2d10e0 40000040 [ 17.509063] 5e40: bea319c0 00000000 ebb35f54 ebb35e58 c0591b6c c05907b8 bea309a0 00000008 [ 17.509735] 5e60: ed01c710 b819e6ad 0000076b 00000000 60000113 00000002 ebb35ed4 ebb35e88 [ 17.533172] 5e80: c00975fc c0096858 00000002 ed01c710 c014ed30 c10bb6d0 ebb35ed4 ebb35ea8 [ 17.548797] 5ea0: c00974a0 c00972f8 00000000 00000002 ed01c710 00000000 00000000 00000000 [ 17.564422] 5ec0: 00000001 c014ed4c ebb35f3c ebb35ed8 c014ed4c c0085444 00000001 00000000 [ 17.580047] 5ee0: c014eb68 00000000 ebb35f3c ea1b94e0 00000000 00000001 00000000 ea1b94e8 [ 17.580047] 5f00: c011637c 00000000 00000000 c0590a0c c00974a0 00000001 ea1b94e8 00000002 [ 17.580047] 5f20: ebb35f84 271aec19 bea319c0 ed2d10e0 40000000 bea319c0 00000129 c0013368 [ 17.637512] 5f40: ebb34000 00000000 ebb35fa4 ebb35f58 c0592ea4 c0591a58 00000000 00000000 [ 17.651184] 5f60: ebb35fa4 00000000 00000000 ebb35e64 00000001 bea309a0 0000100c 40000000 [ 17.658172] 5f80: 00000000 fffffff7 00000000 00000004 b81995dc 00000800 00000000 ebb35fa8 [ 17.673797] 5fa0: c00130e0 c0592e64 00000004 b81995dc 00000004 bea319c0 40000000 bea319ac [ 17.689422] 5fc0: 00000004 b81995dc 00000800 00000129 00000000 bea31a2c 00000000 bea319dc [ 17.689422] 5fe0: 00000000 bea3099c b6ee1e39 b6ebbf6c 40000110 00000004 00000000 00000000 [ 17.733093] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>] (unix_stream_recvmsg+0x124/0x544) [ 17.744262] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from [<c059086c>] (sock_recvmsg+0xc0/0xdc) [ 17.763671] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) [ 17.783172] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from [<c0592ea4>] (sys_recvmsg+0x4c/0x70) [ 17.783172] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>] (ret_fast_syscall+0x0/0x3c) [ 17.806610] Code: e5853008 e894000c e5840004 e5840000 (e5823004) [ 17.830749] ---[ end trace da227214a82491b9 ]--- [ 17.830749] note: upstart-socket-[1407] exited with preempt_count 2 [ 17.853454] BUG: scheduling while atomic: upstart-socket-/1407/0x40000003 [ 17.872467] INFO: lockdep is turned off. [ 17.884063] Modules linked in: wl12xx_sdio wl12xx [ 17.896423] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from [<c0720fc8>] (dump_stack+0x20/0x24) [ 17.896423] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c072291c>] (__schedule_bug+0x84/0x94) [ 17.929504] [<c072291c>] (__schedule_bug+0x84/0x94) from [<c074170c>] (__schedule+0x88/0x5c8) [ 17.946105] [<c074170c>] (__schedule+0x88/0x5c8) from [<c00563dc>] (__cond_resched+0x30/0x40) [ 17.962493] [<c00563dc>] (__cond_resched+0x30/0x40) from [<c0741cf0>] (_cond_resched+0x40/0x50) [ 17.979675] [<c0741cf0>] (_cond_resched+0x40/0x50) from [<c00fe494>] (unmap_page_range+0x170/0x178) [ 17.997039] [<c00fe494>] (unmap_page_range+0x170/0x178) from [<c00fea30>] (unmap_vmas+0x88/0xb8) [ 18.014251] [<c00fea30>] (unmap_vmas+0x88/0xb8) from [<c0102c44>] (exit_mmap+0x138/0x244) [ 18.030914] [<c0102c44>] (exit_mmap+0x138/0x244) from [<c005c1e8>] (mmput.part.36+0x24/0xe0) [ 18.047698] [<c005c1e8>] (mmput.part.36+0x24/0xe0) from [<c005c2e4>] (mmput+0x40/0x44) [ 18.063964] [<c005c2e4>] (mmput+0x40/0x44) from [<c00606ec>] (exit_mm+0x158/0x160) [ 18.079833] [<c00606ec>] (exit_mm+0x158/0x160) from [<c00624b0>] (do_exit+0x200/0x39c) [ 18.096038] [<c00624b0>] (do_exit+0x200/0x39c) from [<c00176c4>] (die+0xf8/0x10c) [ 18.111816] [<c00176c4>] (die+0xf8/0x10c) from [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) [ 18.129119] [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) from [<c0747758>] (do_page_fault+0x334/0x358) [ 18.142852] [<c0747758>] (do_page_fault+0x334/0x358) from [<c07477a8>] (do_translation_fault+0x2c/0xb0) [ 18.165985] [<c07477a8>] (do_translation_fault+0x2c/0xb0) from [<c0008518>] (do_DataAbort+0x44/0xa8) [ 18.183898] [<c0008518>] (do_DataAbort+0x44/0xa8) from [<c074571c>] (__dabt_svc+0x3c/0x60) [ 18.200927] Exception stack(0xebb35c80 to 0xebb35cc8) [ 18.214416] 5c80: 00000000 60000113 00000000 00000000 ebca0200 ee5b1b78 ee5b1b84 ee5b1ae0 [ 18.227050] 5ca0: ebb35df0 0000076b 00000000 ebb35ce4 ebb35c98 ebb35cc8 c07449dc c0599324 [ 18.248596] 5cc0: a0000193 ffffffff [ 18.260498] [<c074571c>] (__dabt_svc+0x3c/0x60) from [<c0599324>] (skb_dequeue+0x5c/0x74) [ 18.277526] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>] (unix_stream_recvmsg+0x124/0x544) [ 18.295684] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from [<c059086c>] (sock_recvmsg+0xc0/0xdc) [ 18.306427] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) [ 18.331939] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from [<c0592ea4>] (sys_recvmsg+0x4c/0x70) [ 18.349853] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>] (ret_fast_syscall+0x0/0x3c) [ 18.367767] hci0: type 1 len 4 [ 18.551147] (stc): remove_channel_from_table: id 3 [ 18.564880] (stc): remove_channel_from_table: id 2 [ 18.578430] (stc): remove_channel_from_table: id 4 [ 18.591888] (stc): all chnl_ids unregistered [ 18.596588] (stk) :ldisc_install = 0 [ 18.609069] (stc): st_tty_close [ 36.869201] BUG: spinlock lockup on CPU#1, init/1, ee5b1ddc [ 37.578979] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from [<c0720fc8>] (dump_stack+0x20/0x24) [ 37.596435] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) [ 37.613983] [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) from [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) [ 37.632141] [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) from [<c07448a4>] (_raw_spin_lock+0x5c/0x64) [ 37.649597] [<c07448a4>] (_raw_spin_lock+0x5c/0x64) from [<c0627db0>] (unix_stream_sendmsg+0x22c/0x3fc) [ 37.668395] [<c0627db0>] (unix_stream_sendmsg+0x22c/0x3fc) from [<c0590980>] (sock_sendmsg+0xb4/0xd0) [ 37.686614] [<c0590980>] (sock_sendmsg+0xb4/0xd0) from [<c0590c40>] (__sys_sendmsg+0x1e4/0x29c) [ 37.704284] [<c0590c40>] (__sys_sendmsg+0x1e4/0x29c) from [<c0592d20>] (sys_sendmsg+0x4c/0x70) [ 37.721923] [<c0592d20>] (sys_sendmsg+0x4c/0x70) from [<c00130e0>] (ret_fast_syscall+0x0/0x3c) [ 38.345764] BUG: spinlock lockup on CPU#0, upstart-socket-/1407, ee5b1ddc [ 38.345764] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from [<c0720fc8>] (dump_stack+0x20/0x24) [ 39.153442] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) [ 39.171325] [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) from [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) [ 39.189819] [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) from [<c07448a4>] (_raw_spin_lock+0x5c/0x64) [ 39.208099] [<c07448a4>] (_raw_spin_lock+0x5c/0x64) from [<c0628828>] (unix_release_sock+0x44/0x220) [ 39.226593] [<c0628828>] (unix_release_sock+0x44/0x220) from [<c0628a34>] (unix_release+0x30/0x34) [ 39.244873] [<c0628a34>] (unix_release+0x30/0x34) from [<c05912f8>] (sock_release+0x30/0xd4) [ 39.262603] [<c05912f8>] (sock_release+0x30/0xd4) from [<c0591da8>] (sock_close+0x30/0x3c) [ 39.280090] [<c0591da8>] (sock_close+0x30/0x3c) from [<c0117bd8>] (__fput+0xf8/0x1bc) [ 39.296997] [<c0117bd8>] (__fput+0xf8/0x1bc) from [<c0117cdc>] (fput+0x40/0x44) [ 39.313354] [<c0117cdc>] (fput+0x40/0x44) from [<c0114dcc>] (filp_close+0x80/0x8c) [ 39.329956] [<c0114dcc>] (filp_close+0x80/0x8c) from [<c0060944>] (close_files+0xbc/0xec) [ 39.340393] [<c0060944>] (close_files+0xbc/0xec) from [<c0062190>] (put_files_struct+0x44/0xc8) [ 39.365173] [<c0062190>] (put_files_struct+0x44/0xc8) from [<c00622ac>] (exit_files+0x48/0x4c) [ 39.382995] [<c00622ac>] (exit_files+0x48/0x4c) from [<c0062560>] (do_exit+0x2b0/0x39c) [ 39.400146] [<c0062560>] (do_exit+0x2b0/0x39c) from [<c00176c4>] (die+0xf8/0x10c) [ 39.416687] [<c00176c4>] (die+0xf8/0x10c) from [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) [ 39.434600] [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) from [<c0747758>] (do_page_fault+0x334/0x358) [ 39.453582] [<c0747758>] (do_page_fault+0x334/0x358) from [<c07477a8>] (do_translation_fault+0x2c/0xb0) [ 39.472412] [<c07477a8>] (do_translation_fault+0x2c/0xb0) from [<c0008518>] (do_DataAbort+0x44/0xa8) [ 39.490844] [<c0008518>] (do_DataAbort+0x44/0xa8) from [<c074571c>] (__dabt_svc+0x3c/0x60) [ 39.508239] Exception stack(0xebb35c80 to 0xebb35cc8) [ 39.522155] 5c80: 00000000 60000113 00000000 00000000 ebca0200 ee5b1b78 ee5b1b84 ee5b1ae0 [ 39.539550] 5ca0: ebb35df0 0000076b 00000000 ebb35ce4 ebb35c98 ebb35cc8 c07449dc c0599324 [ 39.556884] 5cc0: a0000193 ffffffff [ 39.569091] [<c074571c>] (__dabt_svc+0x3c/0x60) from [<c0599324>] (skb_dequeue+0x5c/0x74) [ 39.586486] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>] (unix_stream_recvmsg+0x124/0x544) [ 39.604827] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from [<c059086c>] (sock_recvmsg+0xc0/0xdc) [ 39.623321] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) [ 39.641326] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from [<c0592ea4>] (sys_recvmsg+0x4c/0x70) [ 39.659271] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>] (ret_fast_syscall+0x0/0x3c)