On Thu, 22 Mar 2018 07:52:15 +0000 Mike Leach mike.leach@linaro.org wrote:
Hi, sorry, just jumping in with a couple of extra recommendations...
On 20 March 2018 at 15:34, Mike Bazov mike@perception-point.io wrote:
Thank you for your answers. I fixed the issue by applying the patches in the pull-request. I'm not
If this means you cherry-picked them without rebasing them onto a very late version of perf, then please do that, too. Preferably this one:
git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git
because it's perf/core branch contains some recent fixes in the area.
getting any segfaults.. but i have a different issue now. I'm recording my own program, super simple(int main() { printf("Sleep\n"); return 0; }). The recording seems to work:
That's an awfully small piece of userspace code: All that executable will do is issue a printf syscall with the fixed string, and nothing else: a very small amount of instructions, so maybe it's triggering a rarely tested recording corner-case.
$ sudo perf record -e cs_etm/@826000.etr/u --per-thread ./check Sleep [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.003 MB perf.data ]
But when i try to run the perf script, i'm not getting any trace events, even though i can see there are trace packets(using perf report --dump):
$ /home/mike/repo/trace/linux/tools/perf/perf --exec-path=/home/mike/repo/trace/linux/tools/perf script --script=python:/home/mike/repo/trace/linux/tools/perf/scripts/python/cs-trace-disasm.py -- -d /usr/bin/aarch64-linux-gnu-objdump /lib/aarch64-linux-gnu/ld-2.26.so with build id 6516ef8fa13fcb739834af9e87fb5fe9df612096 not found, continuing without symbols /lib/aarch64-linux-gnu/libc-2.26.so with build id 06e99d8d6acabab0643e0f525ac561cf73db6498 not found, continuing without symbols /home/linaro/check with build id 31694f29996e06da12f63d6088ec6eb23b3079c4 not found, continuing without symbols /lib/aarch64-linux-gnu/ld-2.26.so with build id 6516ef8fa13fcb739834af9e87fb5fe9df612096 not found, continuing without symbols /lib/aarch64-linux-gnu/libc-2.26.so with build id 06e99d8d6acabab0643e0f525ac561cf73db6498 not found, continuing without symbols /home/linaro/check with build id 31694f29996e06da12f63d6088ec6eb23b3079c4 not found, continuing without symbols
Without the memory images loaded above, the decoder cannot decode the trace packets into meaningful trace execution ranges - the process of decode requires that the decoder follow the executed program binary to determine which instructions were traced and which branches were taken. (the 'E' atoms in the dump below represent taken branches - the decode walks the code to find these branches).
So that's why the trace is not being decoded. The dump command simply shows the raw trace packets before decode.
I didn't know this, but also think it might be fixed with a perf update to the latest acme's perf/core branch. Not sure, just saying one more thing to check.
The output gives nothing but the "not found, continuing without symbols". I have dived into the code a little bit, and i can see that the only packet types that i get are CS_ETM_TRACE_ON. This switch case(cs-etm.c line 1010): 1009 switch (etmq->packet->sample_type) { 1010 case CS_ETM_RANGE: 1011 /* 1012 * If the packet contains an instruction 1013 * range, generate instruction sequence 1014 * events. 1015 */ 1016 cs_etm__sample(etmq); 1017 break; 1018 case CS_ETM_TRACE_ON: 1019 /* 1020 * Discontinuity in trace, flush 1021 * previous branch stack 1022 */ 1023 cs_etm__flush(etmq); 1024 break; 1025 default: 1026 break; 1027 }
Never! gets to CS_ETM_RANGE.. therefore i get no decoded instructions. perf report does show packets:
Idx:0; ID:16; I_ASYNC : Alignment Synchronisation. Idx:12; ID:16; I_TRACE_INFO : Trace Info.; INFO=0x1;CC_THRESHOLD=0x100 Idx:19; ID:16; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000; Idx:28; ID:16; I_TRACE_ON : Trace On. Idx:29; ID:16; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFF9D3> Idx:40; ID:16; I_ATOM_F1 : Atom format 1.; E Idx:41; ID:16; I_CCNT_F1 : Cycle Count format 1.; Count=0x0 Idx:42; ID:16; I_TIMESTAMP : Timestamp.; Updated val = 0x70fcdee7; CC=0x1 Idx:54; ID:16; I_ATOM_F1 : Atom format 1.; E Idx:55; ID:16; I_CCNT_F1 : Cycle Count format 1.; Count=0x1d4 Idx:58; ID:16; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE Idx:59; ID:16; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE Idx:60; ID:16; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE Idx:61; ID:16; I_ATOM_F3 : Atom format 3.; EEN Idx:62; ID:16; I_ATOM_F1 : Atom format 1.; E Idx:64; ID:16; I_EXCEPT : Exception.; Data Fault; Ret Addr Follows, Match Prev; Idx:66; ID:16; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000FFFF9D378770; Idx:75; ID:16; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF7DFFFE7> Idx:86; ID:16; I_TIMESTAMP : Timestamp.; Updated val = 0x70fcdf71; CC=0xf6 Idx:91; ID:16; I_TRACE_ON : Trace On. Idx:92; ID:16; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFF9D3>
I don't recognize these addresses, some are 32-bit printouts, some aren't, which is just confusing unless it's something like an aarch32 binary running on an aarch64 kernel. I can't tell whether they're userspace process, shared libraries, or kernelspace. The number of instructions in the example "Sleep" printing above tells me they might be kernel addresses, which might mean it needs to be run with superuser privileges? This is also needed, e.g., to read nonzero kallsyms pointers.
Idx:103; ID:16; I_ATOM_F1 : Atom format 1.; N Idx:104; ID:16; I_CCNT_F1 : Cycle Count format 1.; Count=0x0 Idx:105; ID:16; I_ATOM_F6 : Atom format 6.; EEEEN Idx:106; ID:16; I_ATOM_F2 : Atom format 2.; EE ... ... ... ... ... ...This is my .debug tree:
|-- [kernel.kallsyms] | `-- 1dc43d23817467d7717b19af07463af0d9a9bd83 | `-- kallsyms |-- [vdso] | `-- 18863444e4f3e2600f53e406421b2a0edd940888 | `-- vdso |-- bin | `-- check | `-- 31694f29996e06da12f63d6088ec6eb23b3079c4 | `-- elf `-- lib `-- aarch64-linux-gnu |-- ld-2.26.so | `-- 6516ef8fa13fcb739834af9e87fb5fe9df612096 | `-- elf `-- libc.so.6 `-- 06e99d8d6acabab0643e0f525ac561cf73db6498 `-- elf
This seems fine according to the HOWTO.md. The weird part is, that if i record "uname" and not my program, i do get trace events.
I agree that the .debug database looks valid. Not sure why perf cannot see it. I'll see if I can find out more.
uname does a lot more than the simple example above: was it meant to actually call sleep() in addition to printing "Sleep"?
Updating the perf executable may also fix bugs in the area of resolving paths to non-system binaries. Try sudo rm -fr ~/.debug, and re-run, since these may be results from a prior run.
Not sure if your target has it, but disabling address space randomization helps debugging, too (setarch linux64 -R <workload>).
Kim