Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static ssize_t data_read_offset(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { if (data_file_size(dso, machine)) return -1;
/* Check the offset sanity. */ if (offset > dso->data.file_size) return -1;
if (offset + size < offset) return -1;
return cached_read(dso, machine, offset, data, size); }
Thanks, Leo Yan
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.
What do you see in "perf report -D", do you see a PERF_RECORD_MMAP record for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in /lib/modules?
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Al IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Al,
On Fri, Nov 02, 2018 at 12:08:04PM +0000, Al Grant wrote:
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.
Agree, I also think this is not a specific issue only for Arm platform, this should be one common issue for how to parse kernel symbols with kallsyms file.
What do you see in "perf report -D", do you see a PERF_RECORD_MMAP record for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in /lib/modules?
Yes, I can see PERF_RECORD_MMAP for "[kernel.kallsyms]_text".
0x350 [0x50]: event: 1 . . ... raw event: size 80 bytes . 0000: 01 00 00 00 01 00 50 00 ff ff ff ff 00 00 00 00 ......P......... . 0010: 00 00 08 08 00 00 ff ff ff ff f7 f7 ff ff 00 00 ................ . 0020: 00 00 08 08 00 00 ff ff 5b 6b 65 72 6e 65 6c 2e ........[kernel. . 0030: 6b 61 6c 6c 73 79 6d 73 5d 5f 74 65 78 74 00 00 kallsyms]_text.. . 0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x350 [0x50]: PERF_RECORD_MMAP -1/0: [0xffff000008080000(0xfffff7f7ffff) @ 0xffff000008080000]: x [kernel.kallsyms]_text
Below is the infor for checking buildid:
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-2.27.so 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/[kernel.kallsyms]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:
root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 0
Thanks, Leo Yan
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld- 2.27.so 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux- gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/[kernel.kallsyms]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
What's in that last file? I've seen it happen that the copy of kallsyms in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.
Then, you can give yourself permissions - but perf's already cached the file and won't update it!
If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.
Perhaps nobody spotted this on Intel because perf report goes directly to /proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.
Al
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:
root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 0
Thanks, Leo Yan
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On Fri, Nov 02, 2018 at 02:12:28PM +0000, Al Grant wrote:
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld- 2.27.so 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux- gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/[kernel.kallsyms]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
What's in that last file?
I can see the correct symbol infos in this file:
root@debian:~/coresight_test# cat ~/.debug/[kernel.kallsyms]/0242d9154c78df1d8fe1d0512c36a236d0861a18/kallsyms | head -20 ffff000008080000 t _head ffff000008080000 T _text ffff000008080040 t pe_header ffff000008080044 t coff_header ffff000008080058 t optional_header ffff000008080070 t extra_header_fields ffff0000080800f8 t section_table ffff000008081000 T do_undefinstr ffff000008081000 t efi_header_end ffff000008081000 T _stext ffff000008081000 T __exception_text_start ffff000008081268 T do_cp15instr ffff000008081380 T do_sysinstr ffff000008081410 T do_debug_exception ffff000008081550 T do_mem_abort ffff000008081600 T do_el0_irq_bp_hardening ffff000008081650 T do_el0_ia_bp_hardening ffff0000080816f8 T do_sp_pc_abort ffff0000080817c4 T __exception_text_end ffff0000080817c8 t bcm2835_handle_irq
I've seen it happen that the copy of kallsyms in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.
Then, you can give yourself permissions - but perf's already cached the file and won't update it!
If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.
Perhaps nobody spotted this on Intel because perf report goes directly to /proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.
This is another issue related with permission for accessing kernel pointer values and this will bother much for cross platforms usage (e.g. 'perf record' on Arm platform and 'perf script' on x86 platform).
At my side, I executed all commands on Arm Juno board and simply to say the kallsyms support is broken in perf.
Thanks, Leo Yan
Hi Leo,
My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.
If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/[kernel.kallsyms]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms - which seems to me to be the relevant symbol file to use rather than the live one generated by /proc/kallsyms. I don't really want to use the local /proc/kallsyms on my x86 linux box when decoding an ARM trace captured elsewhere.
So perhaps the problem to be solved is not how to use /proc/kallsyms if no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?
Regards
Mike On Fri, 2 Nov 2018 at 02:55, leo.yan@linaro.org wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static ssize_t data_read_offset(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { if (data_file_size(dso, machine)) return -1;
/* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size);}
Thanks, Leo Yan _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Hi Mike,
On Fri, Nov 02, 2018 at 01:08:40PM +0000, Mike Leach wrote:
Hi Leo,
My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.
If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/[kernel.kallsyms]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than
the live one generated by /proc/kallsyms. I don't really want to use the local /proc/kallsyms on my x86 linux box when decoding an ARM trace captured elsewhere.
So perhaps the problem to be solved is not how to use /proc/kallsyms if no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?
Yes, this is good point, the subject should be changed to: "Question: perf dso support for kallsyms".
I think I made a mistake in my previous email, so clarify it:
~/.debug/[kernel.kallsyms]/$BUILDID/kallsyms should has higher priority than /proc/kallsyms, by default the perf tool should use kallsyms file under ~/.debug folder. I also tried to manually specify the kallsyms file with the command "perf script --kallsyms ./kallsyms", for both cases perf fails to parse symbols for any kernel address.
Thanks, Leo Yan
On Fri, Nov 02, 2018 at 10:55:16AM +0800, Leo Yan wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
Hi Jiri, Arnaldo,
Could you give some suggestion for this question? Thanks!
static ssize_t data_read_offset(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { if (data_file_size(dso, machine)) return -1;
/* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size);}
Thanks, Leo Yan
On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
thanks, jirka
I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static ssize_t data_read_offset(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { if (data_file_size(dso, machine)) return -1;
/* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size);}
Thanks, Leo Yan
Hi Jiri,
On Wed, Nov 07, 2018 at 03:10:06PM +0100, Jiri Olsa wrote:
On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.
For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling dso__data_read_offset() [1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.
Regarding to testing case, I can simply run 'uname' program so it can produce kernel and user space trace data, if I use 'perf script -k vmlinux' then I can see perf generate samples for kernel related traces, but if use 'perf script' then only can see samples for user space.
Thanks, Leo Yan
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tool...
I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static ssize_t data_read_offset(struct dso *dso, struct machine *machine, u64 offset, u8 *data, ssize_t size) { if (data_file_size(dso, machine)) return -1;
/* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size);}
Thanks, Leo Yan
On Thu, Nov 08, 2018 at 04:51:52PM +0800, leo.yan@linaro.org wrote:
Hi Jiri,
On Wed, Nov 07, 2018 at 03:10:06PM +0100, Jiri Olsa wrote:
On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux file. So when we don't specify kernel symbol files then perf tool will roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()
so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.
For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling dso__data_read_offset() [1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.
I haven't checked on this code for some time but AFAICS before you call dso__data_read_offset you need to check dso->data.status != DSO_DATA_STATUS_ERROR
map__load call ahead should take care on setting this and find the source of the data
check the intel_pt code (intel_pt_walk_next_insn) or for example grab_bb in perf script
I guess you need to have either vmlinux or kcore in place to get some data out of it
jirka
-
Al Grant -
Jiri Olsa -
leo.yan＠linaro.org -
Mike Leach