Hello, There is an issue related to perf which I am facing since 15 days. Hoping that the great minds here will help me to solve this. I have a requirement to make perf tool work on a device having ARM architecture. But, on recording the tracepoint events and then running ./perf report, it shows the shared objects name as [unknown] and Symbols as 00000, whereas for software and hardware events I do not experience this issue.I have cross compiled the perf tool available on mailine and ported it on a device having ARM architecture. Output on ARM based device with kernel 3.4. ./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ][ perf record: Captured and wrote 0.007 MB perf.data (~321 samples) ] ./perf report Overhead Command Shared Object Symbol ........................... ................................ ...........................................40.78% cal [unknown] [.]0000000031.6% cal [unknown] [.]00000000 [...] On ubuntu 12.04, system (kernel 3.10) it works perfectly fine. Output on x86 architecture ./perf record -e kmem:kmalloc gcalctool [ perf record: Woken up 3 times to write data ][ perf record: Captured and wrote 0.27 MB perf.data (~845 samples) ] ./perf report --stdio Overhead Command Shared Object Symbol ........................... ................................ ...........................................96.55% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace3.45% cal [kernel.kallsyms] [.]__kmalloc [...]
Keenly, awaiting for you help. Thanks. Sneha.
Hello,
On 15 May 2014 07:36, sneha priya sneha.cse@hotmail.com wrote:
Hello,
There is an issue related to perf which I am facing since 15 days. Hoping that the great minds here will help me to solve this.
I have a requirement to make perf tool work on a device having ARM architecture. But, on recording the tracepoint events and then running ./perf report, it shows the shared objects name as [unknown] and Symbols as 00000, whereas for software and hardware events I do not experience this issue. I have cross compiled the perf tool available on mailine and ported it on a device having ARM architecture.
Output on ARM based device with kernel 3.4.
./perf record -e kmem:kmalloc cal
[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.007 MB perf.data (~321 samples) ]
./perf report
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 40.78% cal [unknown] [.]00000000 31.6% cal [unknown] [.]00000000
[...]
On ubuntu 12.04, system (kernel 3.10) it works perfectly fine.
Output on x86 architecture
./perf record -e kmem:kmalloc gcalctool
[ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.27 MB perf.data (~845 samples) ]
./perf report --stdio
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 96.55% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.45% cal [kernel.kallsyms] [.]__kmalloc
[...]
Keenly, awaiting for you help.
There are a few things to check. Note that the 3.4 kernel is OK wrt perf and tracepoints but is quite old, the recent development now happens on 3.15.
- Do you have kallsyms enabled in your kernel (CONFIG_KALLSYMS=y)? - Are there warnings issued by perf record, e.g. access to kernel symbols etc? - perf report needs to know about your vmlinux image (the one that contains the debug symbols etc in the root directory of the kernel build). You can use '-k <file>' or '--vmlinux=<file>', cf. perf report --help. - You can dump the samples from perf.data using perf report -D. The entries with 'PERF_RECORD_SAMPLE(IP, 2)' are for the user space apps. - The '-v' option gives more info. This can be used multiple times ('-vv').
Thanks.
Sneha.
Regards, Jean
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Hello,
Indeed there is a problem in the ARM code for tracepoints. After a good discussion with the perf maintainers a solution has be found, cf. http://www.spinics.net/lists/arm-kernel/msg332293.html.
Can you check if this fixes the problem? It does on my side on 3.15-rc4.
The patch is under review by the ARM experts and hopefully should be merged soon.
Regards, Jean
On 16 May 2014 09:34, Jean Pihet jean.pihet@linaro.org wrote:
Hello,
On 15 May 2014 07:36, sneha priya sneha.cse@hotmail.com wrote:
Hello,
There is an issue related to perf which I am facing since 15 days. Hoping that the great minds here will help me to solve this.
I have a requirement to make perf tool work on a device having ARM architecture. But, on recording the tracepoint events and then running ./perf report, it shows the shared objects name as [unknown] and Symbols as 00000, whereas for software and hardware events I do not experience this issue. I have cross compiled the perf tool available on mailine and ported it on a device having ARM architecture.
Output on ARM based device with kernel 3.4.
./perf record -e kmem:kmalloc cal
[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.007 MB perf.data (~321 samples) ]
./perf report
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 40.78% cal [unknown] [.]00000000 31.6% cal [unknown] [.]00000000
[...]
On ubuntu 12.04, system (kernel 3.10) it works perfectly fine.
Output on x86 architecture
./perf record -e kmem:kmalloc gcalctool
[ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.27 MB perf.data (~845 samples) ]
./perf report --stdio
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 96.55% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.45% cal [kernel.kallsyms] [.]__kmalloc
[...]
Keenly, awaiting for you help.
There are a few things to check. Note that the 3.4 kernel is OK wrt perf and tracepoints but is quite old, the recent development now happens on 3.15.
- Do you have kallsyms enabled in your kernel (CONFIG_KALLSYMS=y)?
- Are there warnings issued by perf record, e.g. access to kernel symbols etc?
- perf report needs to know about your vmlinux image (the one that
contains the debug symbols etc in the root directory of the kernel build). You can use '-k <file>' or '--vmlinux=<file>', cf. perf report --help.
- You can dump the samples from perf.data using perf report -D. The
entries with 'PERF_RECORD_SAMPLE(IP, 2)' are for the user space apps.
- The '-v' option gives more info. This can be used multiple times ('-vv').
Thanks.
Sneha.
Regards, Jean
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Jean Pihet <jean.pihet@...> writes:
Hello,
Indeed there is a problem in the ARM code for tracepoints. After a good discussion with the perf maintainers a solution has be found, cf. http://www.spinics.net/lists/arm-kernel/msg332293.html.
Can you check if this fixes the problem? It does on my side on 3.15-rc4.
Hi Jean,
yes, it solved the problem but partially, i am getting different outputs for x86 and ARM.
On x86(3.10.28):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.009 MB perf.data (~381 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 96.77% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.23% cal [kernel.kallsyms] [k] __kmalloc
./perf report -v 96.77% cal /lib/modules/3.10.28+/build/vmlinux 0xffffffff81166f6d v [k] kmem_cache_alloc_trace 3.23% cal /lib/modules/3.10.28+/build/vmlinux 0xffffffff81166b24 v [k] __kmalloc
./perf buildid-list aef9a24fcddff67cd67bcc1fd27dbeaf86d35487 [kernel.kallsyms]
but on ARM(3.4.0):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.004 MB perf.data (~192 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 100.00% cal [ks8851] [k] 0x01247890
./perf report -v 100.00% cal /lib/modules/3.4.0/.../ks8851.ko 0x1247890 l [k] 0x01247890
./perf buildid-list d0898c01486575dddafc6686240186bdb4b4430a /lib/modules/3.4.0/.../ks8851.ko
that means on ARM, it is not getting the buildid for [kernel.kallsyms].
Are you getting the same output for ARM and x86 ??
regards, Prankul Garg
The patch is under review by the ARM experts and hopefully should be merged soon.
Regards, Jean
On 16 May 2014 09:34, Jean Pihet <jean.pihet@...> wrote:
Hello,
On 15 May 2014 07:36, sneha priya <sneha.cse@...> wrote:
Hello,
There is an issue related to perf which I am facing since 15 days.
Hoping
that the great minds here will help me to solve this.
I have a requirement to make perf tool work on a device having ARM architecture. But, on recording the tracepoint events and then running ./perf report, it shows the shared objects name as [unknown] and
Symbols as
00000, whereas for software and hardware events I do not experience
this
issue. I have cross compiled the perf tool available on mailine and ported it
on a
device having ARM architecture.
Output on ARM based device with kernel 3.4.
./perf record -e kmem:kmalloc cal
[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.007 MB perf.data (~321 samples) ]
./perf report
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 40.78% cal [unknown] [.]00000000 31.6% cal [unknown] [.]00000000
[...]
On ubuntu 12.04, system (kernel 3.10) it works perfectly fine.
Output on x86 architecture
./perf record -e kmem:kmalloc gcalctool
[ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.27 MB perf.data (~845 samples) ]
./perf report --stdio
Overhead Command
Shared
Object Symbol ........................... ................................ ........................................... 96.55% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.45% cal [kernel.kallsyms] [.]__kmalloc
[...]
Keenly, awaiting for you help.
There are a few things to check. Note that the 3.4 kernel is OK wrt perf and tracepoints but is quite old, the recent development now happens on 3.15.
- Do you have kallsyms enabled in your kernel (CONFIG_KALLSYMS=y)?
- Are there warnings issued by perf record, e.g. access to kernel
symbols etc?
- perf report needs to know about your vmlinux image (the one that
contains the debug symbols etc in the root directory of the kernel build). You can use '-k <file>' or '--vmlinux=<file>', cf. perf report --help.
- You can dump the samples from perf.data using perf report -D. The
entries with 'PERF_RECORD_SAMPLE(IP, 2)' are for the user space apps.
- The '-v' option gives more info. This can be used multiple times ('-
vv').
Thanks.
Sneha.
Regards, Jean
linaro-dev mailing list linaro-dev@... http://lists.linaro.org/mailman/listinfo/linaro-dev
Hi
On 19 May 2014 16:08, Prankul Garg prankul.iiitm@gmail.com wrote:
Jean Pihet <jean.pihet@...> writes:
Hello,
Indeed there is a problem in the ARM code for tracepoints. After a good discussion with the perf maintainers a solution has be found, cf. http://www.spinics.net/lists/arm-kernel/msg332293.html.
Can you check if this fixes the problem? It does on my side on 3.15-rc4.
Hi Jean,
yes, it solved the problem but partially, i am getting different outputs for x86 and ARM.
On x86(3.10.28):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.009 MB perf.data (~381 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 96.77% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.23% cal [kernel.kallsyms] [k] __kmalloc
./perf report -v 96.77% cal /lib/modules/3.10.28+/build/vmlinux 0xffffffff81166f6d v [k] kmem_cache_alloc_trace 3.23% cal /lib/modules/3.10.28+/build/vmlinux 0xffffffff81166b24 v [k] __kmalloc
./perf buildid-list aef9a24fcddff67cd67bcc1fd27dbeaf86d35487 [kernel.kallsyms]
but on ARM(3.4.0):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.004 MB perf.data (~192 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 100.00% cal [ks8851] [k] 0x01247890
./perf report -v 100.00% cal /lib/modules/3.4.0/.../ks8851.ko 0x1247890 l [k] 0x01247890
./perf buildid-list d0898c01486575dddafc6686240186bdb4b4430a /lib/modules/3.4.0/.../ks8851.ko
that means on ARM, it is not getting the buildid for [kernel.kallsyms].
Are you getting the same output for ARM and x86 ??
Yes. Here is the output I have:
root@axp-linaro:/home/linaro/linaro/kernel# ./linux/tools/perf/perf report -v build id event received for vmlinux: 6db809b8adab308f64037970799dc836f6b96422 Looking at the vmlinux_path (6 entries long) Using /proc/kallsyms for symbols # To display the perf.data header info, please use --header/--header-only option # # Samples: 12 of event 'kmem:kmalloc' # Event count (approx.): 12 # # Overhead Command Shared Object Sy # ........ ............ ................. ................................... # 91.67% stress_bt_v7 [kernel.kallsyms] 0xc00f1548 k [k] kmem_cache_alloc_t 8.33% stress_bt_v7 [kernel.kallsyms] 0xc00f1400 k [k] __kmalloc
# # (For a higher level overview, try: perf report --sort comm,dso) #
root@axp-linaro:/home/linaro/linaro/kernel/linux# ./tools/perf/perf buildid-list 6db809b8adab308f64037970799dc836f6b96422 vmlinux root@axp-linaro:/home/linaro/linaro/kernel/linux#
perf is looking for symbols in vmlinux (if present in the current dir or specified in the command line) and if not found from kallsyms. So if I run 'perf report' from the kernel root dir, the symbols are from vmlinux; if I run it from elsewhere the symbols are from kallsyms.
Also your build is quite old, a lot of changes went into perf since 3.4.
Regards, Jean
regards, Prankul Garg
The patch is under review by the ARM experts and hopefully should be merged soon.
Regards, Jean
On 16 May 2014 09:34, Jean Pihet <jean.pihet@...> wrote:
Hello,
On 15 May 2014 07:36, sneha priya <sneha.cse@...> wrote:
Hello,
There is an issue related to perf which I am facing since 15 days.
Hoping
that the great minds here will help me to solve this.
I have a requirement to make perf tool work on a device having ARM architecture. But, on recording the tracepoint events and then running ./perf report, it shows the shared objects name as [unknown] and
Symbols as
00000, whereas for software and hardware events I do not experience
this
issue. I have cross compiled the perf tool available on mailine and ported it
on a
device having ARM architecture.
Output on ARM based device with kernel 3.4.
./perf record -e kmem:kmalloc cal
[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.007 MB perf.data (~321 samples) ]
./perf report
Overhead Command Shared Object Symbol ........................... ................................ ........................................... 40.78% cal [unknown] [.]00000000 31.6% cal [unknown] [.]00000000
[...]
On ubuntu 12.04, system (kernel 3.10) it works perfectly fine.
Output on x86 architecture
./perf record -e kmem:kmalloc gcalctool
[ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.27 MB perf.data (~845 samples) ]
./perf report --stdio
Overhead Command
Shared
Object Symbol ........................... ................................ ........................................... 96.55% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.45% cal [kernel.kallsyms] [.]__kmalloc
[...]
Keenly, awaiting for you help.
There are a few things to check. Note that the 3.4 kernel is OK wrt perf and tracepoints but is quite old, the recent development now happens on 3.15.
- Do you have kallsyms enabled in your kernel (CONFIG_KALLSYMS=y)?
- Are there warnings issued by perf record, e.g. access to kernel
symbols etc?
- perf report needs to know about your vmlinux image (the one that
contains the debug symbols etc in the root directory of the kernel build). You can use '-k <file>' or '--vmlinux=<file>', cf. perf report --help.
- You can dump the samples from perf.data using perf report -D. The
entries with 'PERF_RECORD_SAMPLE(IP, 2)' are for the user space apps.
- The '-v' option gives more info. This can be used multiple times ('-
vv').
Thanks.
Sneha.
Regards, Jean
linaro-dev mailing list linaro-dev@... http://lists.linaro.org/mailman/listinfo/linaro-dev
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Jean Pihet <jean.pihet@...> writes:
Hi
On 19 May 2014 16:08, Prankul Garg <prankul.iiitm@...> wrote:
Jean Pihet <jean.pihet <at> ...> writes:
Hello,
Indeed there is a problem in the ARM code for tracepoints. After a good discussion with the perf maintainers a solution has be found, cf. http://www.spinics.net/lists/arm-kernel/msg332293.html.
Can you check if this fixes the problem? It does on my side on 3.15-
rc4.
Hi Jean,
yes, it solved the problem but partially, i am getting different outputs
for
x86 and ARM.
On x86(3.10.28):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.009 MB perf.data (~381 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 96.77% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.23% cal [kernel.kallsyms] [k] __kmalloc
./perf report -v 96.77% cal /lib/modules/3.10.28+/build/vmlinux
0xffffffff81166f6d
v [k] kmem_cache_alloc_trace 3.23% cal /lib/modules/3.10.28+/build/vmlinux
0xffffffff81166b24
v [k] __kmalloc
./perf buildid-list aef9a24fcddff67cd67bcc1fd27dbeaf86d35487 [kernel.kallsyms]
but on ARM(3.4.0):
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.004 MB perf.data (~192 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 100.00% cal [ks8851] [k] 0x01247890
./perf report -v 100.00% cal /lib/modules/3.4.0/.../ks8851.ko 0x1247890 l
[k]
0x01247890
./perf buildid-list d0898c01486575dddafc6686240186bdb4b4430a
/lib/modules/3.4.0/.../ks8851.ko
that means on ARM, it is not getting the buildid for [kernel.kallsyms].
Are you getting the same output for ARM and x86 ??
Yes. Here is the output I have:
root <at> axp-linaro:/home/linaro/linaro/kernel# ./linux/tools/perf/perf
report -v
build id event received for vmlinux:
6db809b8adab308f64037970799dc836f6b96422
Looking at the vmlinux_path (6 entries long) Using /proc/kallsyms for symbols # To display the perf.data header info, please use --header/--header-only
option
# # Samples: 12 of event 'kmem:kmalloc' # Event count (approx.): 12 # # Overhead Command Shared Object
Sy
# ........ ............ .................
...................................
# 91.67% stress_bt_v7 [kernel.kallsyms] 0xc00f1548 k [k]
kmem_cache_alloc_t
8.33% stress_bt_v7 [kernel.kallsyms] 0xc00f1400 k [k] __kmalloc# # (For a higher level overview, try: perf report --sort comm,dso) #
root <at> axp-linaro:/home/linaro/linaro/kernel/linux# ./tools/perf/perf buildid-list 6db809b8adab308f64037970799dc836f6b96422 vmlinux root <at> axp-linaro:/home/linaro/linaro/kernel/linux#
perf is looking for symbols in vmlinux (if present in the current dir or specified in the command line) and if not found from kallsyms. So if I run 'perf report' from the kernel root dir, the symbols are from vmlinux; if I run it from elsewhere the symbols are from kallsyms.
Hi Jean,
yes, I am getting your point regarding vmlinux or /proc/kallsyms. For cross checking it i also run command ./perf report -k /proc/kallsyms or ./perf report --vmlinux /path/to/vmlinux
but it didn't work for me.
when i debug the same and just print the statement, *__dsos__findnew: (function defined in "tools/perf/util/dso.c") which dsos it is finding.
So during perf record
# ./perf record -e kmem:kmalloc cal dsos__findnew: [kernel.kallsyms] dsos__findnew: [ks8851] dsos__findnew: [exfat_fs]
# ./perf report dsos__findnew: [ks8851] (say list of dsos 'd1') # cmdline : /opt/perf/perf record -e kmem:mm_page_alloc cal # event : name = kmem:mm_page_alloc, type = 2, config = 0x69, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0 # pmu mappings: cpu = 4, software = 1, tracepoint = 2, breakpoint = 5 # ======== #
dsos__findnew: [kernel.kallsyms] dsos__findnew: [ks8851] ..and many more (say list of dsos 'd2')
Now it matches the 'd1' and 'd2', and those dsos which matches,of those corresponding buildid is generated.
# Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 100.00% cal [ks8851] [k] 0x01247890
Also your build is quite old, a lot of changes went into perf since 3.4.
I have also tried the same on kernel 3.10.17 but having the same issue. Since i'm working for a specific product, it's not easy for me to move on 3.15. Is TRACEPOINT events working verified on 3.15 only before that, wasn't it working ??
Regards, Jean
regards, Prankul Garg
The patch is under review by the ARM experts and hopefully should be merged soon.
Regards, Jean
On 16 May 2014 09:34, Jean Pihet <jean.pihet <at> ...> wrote:
Hi,
On 21 May 2014 16:04, Prankul Garg prankul.iiitm@gmail.com wrote:
Jean Pihet <jean.pihet@...> writes:
Also your build is quite old, a lot of changes went into perf since 3.4.
Hi Jean,
finally, it works for me too. The problem i was facing due to the older kernel.
That is great!
The proposed patch is under review and should be mainlined soon.
Regards, Jean
Thanks & regards, Prankul Garg
Regards, Jean
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Jean Pihet <jean.pihet@...> writes:
./perf record -e kmem:kmalloc cal [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.009 MB perf.data (~381 samples) ]
./perf report # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... # 96.77% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace 3.23% cal [kernel.kallsyms] [k] __kmalloc
Hi Jean, Now it is able to detect [kernel.kallsyms] so it shows the symbol name, but it is not able to resolve all other symbols on call graph,
# ./perf record -g -e kmem:kmalloc cal # # Samples: 38 of event 'kmem:kmalloc' # Event count (approx.): 38 # # Overhead Command Shared Object Symbol # ........ ....... ................. .......................... #
97.37% cal [kernel.kallsyms] [k] kmem_cache_alloc_trace | --- kmem_cache_alloc_trace | |--48.65%-- 0x48 | |--10.81%-- 0xbebbe5bc | |--5.41%-- 0x8ec8 | |--5.41%-- 0xbebbe694 | |--2.70%-- 0xbebbe4cc | |--2.70%-- 0xbebbe4e4 | 0xb6ee9518 --24.32%-- [...]
2.63% cal [kernel.kallsyms] [k] __kmalloc | --- __kmalloc
showing only addresses.
Thanks, Prankul Garg
-
Jean Pihet -
Prankul Garg -
sneha priya