On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4ef0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb242b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks.
Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
Ok.
I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. So, are you saying that I only need to replace tool/perf directory?
Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 9:40 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Matt
Hi there,
> > > > We have Linux 4.9 running on our ARMv8 platform. I like to try > instruction tracing using perf-opencsd-4.9.
Ok, that should be an interesting project.
> > I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. > Looks like this tree consists of base > > Linux kernel 4.9 plus some some additions to support CoreSight > and instruction trace using Perf tool.
All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library.
> > > > For adding trace functionality, Is it possible to get a patch > that I can apply to my base Linux 4.9?
I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree.
> > > > Also which files must be changed to reflect specific SOC CoreSight topology?
CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house?
In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with.
I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions.
Regards, Mathieu
[1]. http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress-v2 p
c a 15_a7.dts [2]. http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/juno- b a s e .dtsi [3]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git / t r e e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4
> > > > Regards, Reza
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4ef 0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb242 b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks.
Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
Ok.
I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. So, are you saying that I only need to replace tool/perf directory?
Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 9:40 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Matt
Hi there,
> > > > We have Linux 4.9 running on our ARMv8 platform. I like to try > instruction tracing using perf-opencsd-4.9.
Ok, that should be an interesting project.
> > I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. > Looks like this tree consists of base > > Linux kernel 4.9 plus some some additions to support CoreSight > and instruction trace using Perf tool.
All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library.
> > > > For adding trace functionality, Is it possible to get a patch > that I can apply to my base Linux 4.9?
I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree.
> > > > Also which files must be changed to reflect specific SOC CoreSight topology?
CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house?
In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with.
I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions.
Regards, Mathieu
[1]. http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress-v 2 p
c a 15_a7.dts [2]. http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/juno
b a s e .dtsi [3]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gi t / t r e e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4
> > > > Regards, Reza
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4ef 0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb242 b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks.
Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
Ok.
> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. > So, are you saying that I only need to replace tool/perf directory?
Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, March 30, 2017 9:40 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Matt > > Hi there, > >> >> >> >> We have Linux 4.9 running on our ARMv8 platform. I like to try >> instruction tracing using perf-opencsd-4.9. > > Ok, that should be an interesting project. > >> >> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >> Looks like this tree consists of base >> >> Linux kernel 4.9 plus some some additions to support CoreSight >> and instruction trace using Perf tool. > > All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. > >> >> >> >> For adding trace functionality, Is it possible to get a patch >> that I can apply to my base Linux 4.9? > > I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. > >> >> >> >> Also which files must be changed to reflect specific SOC CoreSight topology? > > CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? > > In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. > > I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. > > Regards, > Mathieu > > [1]. > http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress-v > 2 > p > - > c > a > 15_a7.dts [2]. > http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/juno > - > b > a > s > e > .dtsi [3]. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gi > t > / > t > r > e > e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 > > >> >> >> >> Regards, Reza
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4e f 0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb24 2 b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks.
Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
Ok.
> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. > So, are you saying that I only need to replace tool/perf directory?
Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, March 30, 2017 9:40 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Matt > > Hi there, > >> >> >> >> We have Linux 4.9 running on our ARMv8 platform. I like to try >> instruction tracing using perf-opencsd-4.9. > > Ok, that should be an interesting project. > >> >> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >> Looks like this tree consists of base >> >> Linux kernel 4.9 plus some some additions to support CoreSight >> and instruction trace using Perf tool. > > All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. > >> >> >> >> For adding trace functionality, Is it possible to get a patch >> that I can apply to my base Linux 4.9? > > I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. > >> >> >> >> Also which files must be changed to reflect specific SOC CoreSight topology? > > CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? > > In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. > > I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. > > Regards, > Mathieu > > [1]. > http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress- > v > 2 > p > - > c > a > 15_a7.dts [2]. > http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/jun > o > - > b > a > s > e > .dtsi [3]. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g > i > t > / > t > r > e > e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 > > >> >> >> >> Regards, Reza
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4e f 0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb24 2 b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks.
Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
Ok.
> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. > So, are you saying that I only need to replace tool/perf directory?
Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, March 30, 2017 9:40 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Matt > > Hi there, > >> >> >> >> We have Linux 4.9 running on our ARMv8 platform. I like to try >> instruction tracing using perf-opencsd-4.9. > > Ok, that should be an interesting project. > >> >> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >> Looks like this tree consists of base >> >> Linux kernel 4.9 plus some some additions to support CoreSight >> and instruction trace using Perf tool. > > All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. > >> >> >> >> For adding trace functionality, Is it possible to get a patch >> that I can apply to my base Linux 4.9? > > I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. > >> >> >> >> Also which files must be changed to reflect specific SOC CoreSight topology? > > CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? > > In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. > > I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. > > Regards, > Mathieu > > [1]. > http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress- > v > 2 > p > - > c > a > 15_a7.dts [2]. > http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/jun > o > - > b > a > s > e > .dtsi [3]. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g > i > t > / > t > r > e > e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 > > >> >> >> >> Regards, Reza
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have taken all the patches. When building tools/perf we get the following compilation errors. Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
CC util/parse-events.o CC util/parse-events-flex.o CC util/pmu.o util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4e f 0 0 15c5d25c4b3
CC util/pmu-flex.o util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_SWTRACE: ^~~~~~~~~~~~~~~~~~~~~~~~~ util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared identifier is reported only once for each function it appears in util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) case OCSD_GEN_TRC_ELEM_CUSTOM: ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb24 2 b d 9f7328e7104
mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such file or directory
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, March 30, 2017 10:15 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks. > > Do you have links to these patches?
It's all there: https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, March 30, 2017 10:06 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. > > Ok. > >> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >> So, are you saying that I only need to replace tool/perf directory? > > Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. > >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, March 30, 2017 9:40 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Matt >> >> Hi there, >> >>> >>> >>> >>> We have Linux 4.9 running on our ARMv8 platform. I like to try >>> instruction tracing using perf-opencsd-4.9. >> >> Ok, that should be an interesting project. >> >>> >>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>> Looks like this tree consists of base >>> >>> Linux kernel 4.9 plus some some additions to support CoreSight >>> and instruction trace using Perf tool. >> >> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >> >>> >>> >>> >>> For adding trace functionality, Is it possible to get a patch >>> that I can apply to my base Linux 4.9? >> >> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >> >>> >>> >>> >>> Also which files must be changed to reflect specific SOC CoreSight topology? >> >> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >> >> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >> >> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >> >> Regards, >> Mathieu >> >> [1]. >> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress- >> v >> 2 >> p >> - >> c >> a >> 15_a7.dts [2]. >> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/jun >> o >> - >> b >> a >> s >> e >> .dtsi [3]. >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g >> i >> t >> / >> t >> r >> e >> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 >> >> >>> >>> >>> >>> Regards, Reza
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using master branch
git clone https://github.com/Linaro/OpenCSD.git my-opencsd git checkout -b master origin/master
Is this a correct branch for opencsd?
I am cross compiling both opencsd and tools/perf.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 2:34 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > > We have taken all the patches. When building tools/perf we get the following compilation errors. > Do you have some ideas if we are missing a patch?
The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
> > CC util/parse-events.o > CC util/parse-events-flex.o > CC util/pmu.o > util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined > but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { > ^~~~~~~~~~~~~~~~~~~~~~~~~
Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
[1]. https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4e f 0 0 15c5d25c4b3
> CC util/pmu-flex.o > util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: > util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) > case OCSD_GEN_TRC_ELEM_SWTRACE: > ^~~~~~~~~~~~~~~~~~~~~~~~~ > util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared > identifier is reported only once for each function it appears in > util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) > case OCSD_GEN_TRC_ELEM_CUSTOM: > ^~~~~~~~~~~~~~~~~~~~~~~~
Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
[2]. https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb24 2 b d 9f7328e7104
> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No > such file or directory > > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, March 30, 2017 10:15 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks. >> >> Do you have links to these patches? > > It's all there: > https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 > >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, March 30, 2017 10:06 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >> >> Ok. >> >>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>> So, are you saying that I only need to replace tool/perf directory? >> >> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 9:40 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Hello Matt >>> >>> Hi there, >>> >>>> >>>> >>>> >>>> We have Linux 4.9 running on our ARMv8 platform. I like to try >>>> instruction tracing using perf-opencsd-4.9. >>> >>> Ok, that should be an interesting project. >>> >>>> >>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>> Looks like this tree consists of base >>>> >>>> Linux kernel 4.9 plus some some additions to support CoreSight >>>> and instruction trace using Perf tool. >>> >>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>> >>>> >>>> >>>> >>>> For adding trace functionality, Is it possible to get a patch >>>> that I can apply to my base Linux 4.9? >>> >>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>> >>>> >>>> >>>> >>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>> >>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>> >>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>> >>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>> >>> Regards, >>> Mathieu >>> >>> [1]. >>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress- >>> v >>> 2 >>> p >>> - >>> c >>> a >>> 15_a7.dts [2]. >>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/jun >>> o >>> - >>> b >>> a >>> s >>> e >>> .dtsi [3]. >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g >>> i >>> t >>> / >>> t >>> r >>> e >>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 >>> >>> >>>> >>>> >>>> >>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch...
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I am using master branch > > git clone https://github.com/Linaro/OpenCSD.git my-opencsd > git checkout -b master origin/master > > Is this a correct branch for opencsd? > > I am cross compiling both opencsd and tools/perf. > > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 2:34 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Mathieu >> >> >> We have taken all the patches. When building tools/perf we get the following compilation errors. >> Do you have some ideas if we are missing a patch? > > The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. > >> >> CC util/parse-events.o >> CC util/parse-events-flex.o >> CC util/pmu.o >> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined >> but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >> ^~~~~~~~~~~~~~~~~~~~~~~~~ > > Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. > > [1]. > https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4e > f > 0 > 0 > 15c5d25c4b3 > >> CC util/pmu-flex.o >> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >> case OCSD_GEN_TRC_ELEM_SWTRACE: >> ^~~~~~~~~~~~~~~~~~~~~~~~~ >> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared >> identifier is reported only once for each function it appears in >> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >> case OCSD_GEN_TRC_ELEM_CUSTOM: >> ^~~~~~~~~~~~~~~~~~~~~~~~ > > Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. > > [2]. > https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb24 > 2 > b > d > 9f7328e7104 > >> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No >> such file or directory >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, March 30, 2017 10:15 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks. >>> >>> Do you have links to these patches? >> >> It's all there: >> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 10:06 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>> >>> Ok. >>> >>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>> So, are you saying that I only need to replace tool/perf directory? >>> >>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 9:40 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Matt >>>> >>>> Hi there, >>>> >>>>> >>>>> >>>>> >>>>> We have Linux 4.9 running on our ARMv8 platform. I like to try >>>>> instruction tracing using perf-opencsd-4.9. >>>> >>>> Ok, that should be an interesting project. >>>> >>>>> >>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>> Looks like this tree consists of base >>>>> >>>>> Linux kernel 4.9 plus some some additions to support CoreSight >>>>> and instruction trace using Perf tool. >>>> >>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>> >>>>> >>>>> >>>>> >>>>> For adding trace functionality, Is it possible to get a patch >>>>> that I can apply to my base Linux 4.9? >>>> >>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>> >>>>> >>>>> >>>>> >>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>> >>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>> >>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>> >>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>> >>>> Regards, >>>> Mathieu >>>> >>>> [1]. >>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress- >>>> v >>>> 2 >>>> p >>>> - >>>> c >>>> a >>>> 15_a7.dts [2]. >>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/jun >>>> o >>>> - >>>> b >>>> a >>>> s >>>> e >>>> .dtsi [3]. >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g >>>> i >>>> t >>>> / >>>> t >>>> r >>>> e >>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4 >>>> >>>> >>>>> >>>>> >>>>> >>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n11 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:27 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
Mathieu
On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I am using master branch > > git clone https://github.com/Linaro/OpenCSD.git my-opencsd > git checkout -b master origin/master > > Is this a correct branch for opencsd? > > I am cross compiling both opencsd and tools/perf. > > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 2:34 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Mathieu >> >> >> We have taken all the patches. When building tools/perf we get the following compilation errors. >> Do you have some ideas if we are missing a patch? > > The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. > >> >> CC util/parse-events.o >> CC util/parse-events-flex.o >> CC util/pmu.o >> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >> ^~~~~~~~~~~~~~~~~~~~~~~~~ > > Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. > > [1]. > https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87cea > a4e > f > 0 > 0 > 15c5d25c4b3 > >> CC util/pmu-flex.o >> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >> case OCSD_GEN_TRC_ELEM_SWTRACE: >> ^~~~~~~~~~~~~~~~~~~~~~~~~ >> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >> undeclared identifier is reported only once for each function >> it appears in >> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >> case OCSD_GEN_TRC_ELEM_CUSTOM: >> ^~~~~~~~~~~~~~~~~~~~~~~~ > > Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. > > [2]. > https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2b > b24 > 2 > b > d > 9f7328e7104 > >> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No >> such file or directory >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, March 30, 2017 10:15 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks. >>> >>> Do you have links to these patches? >> >> It's all there: >> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 10:06 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>> >>> Ok. >>> >>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>> So, are you saying that I only need to replace tool/perf directory? >>> >>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 9:40 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Matt >>>> >>>> Hi there, >>>> >>>>> >>>>> >>>>> >>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>> try instruction tracing using perf-opencsd-4.9. >>>> >>>> Ok, that should be an interesting project. >>>> >>>>> >>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>> Looks like this tree consists of base >>>>> >>>>> Linux kernel 4.9 plus some some additions to support >>>>> CoreSight and instruction trace using Perf tool. >>>> >>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>> >>>>> >>>>> >>>>> >>>>> For adding trace functionality, Is it possible to get a >>>>> patch that I can apply to my base Linux 4.9? >>>> >>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>> >>>>> >>>>> >>>>> >>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>> >>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>> >>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>> >>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>> >>>> Regards, >>>> Mathieu >>>> >>>> [1]. >>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpre >>>> ss- >>>> v >>>> 2 >>>> p >>>> - >>>> c >>>> a >>>> 15_a7.dts [2]. >>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/ >>>> jun >>>> o >>>> - >>>> b >>>> a >>>> s >>>> e >>>> .dtsi [3]. >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linu >>>> x.g >>>> i >>>> t >>>> / >>>> t >>>> r >>>> e >>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc >>>> 4 >>>> >>>> >>>>> >>>>> >>>>> >>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n11 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. > If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 3:27 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. > > For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). > > Mathieu > > On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I am using master branch >> >> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >> git checkout -b master origin/master >> >> Is this a correct branch for opencsd? >> >> I am cross compiling both opencsd and tools/perf. >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 2:34 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Mathieu >>> >>> >>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>> Do you have some ideas if we are missing a patch? >> >> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >> >>> >>> CC util/parse-events.o >>> CC util/parse-events-flex.o >>> CC util/pmu.o >>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >> >> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >> >> [1]. >> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87cea >> a4e >> f >> 0 >> 0 >> 15c5d25c4b3 >> >>> CC util/pmu-flex.o >>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>> undeclared identifier is reported only once for each function >>> it appears in >>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>> ^~~~~~~~~~~~~~~~~~~~~~~~ >> >> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >> >> [2]. >> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2b >> b24 >> 2 >> b >> d >> 9f7328e7104 >> >>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No >>> such file or directory >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 10:15 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks. >>>> >>>> Do you have links to these patches? >>> >>> It's all there: >>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 10:06 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>> >>>> Ok. >>>> >>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>> So, are you saying that I only need to replace tool/perf directory? >>>> >>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Matt >>>>> >>>>> Hi there, >>>>> >>>>>> >>>>>> >>>>>> >>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>> try instruction tracing using perf-opencsd-4.9. >>>>> >>>>> Ok, that should be an interesting project. >>>>> >>>>>> >>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>> Looks like this tree consists of base >>>>>> >>>>>> Linux kernel 4.9 plus some some additions to support >>>>>> CoreSight and instruction trace using Perf tool. >>>>> >>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> For adding trace functionality, Is it possible to get a >>>>>> patch that I can apply to my base Linux 4.9? >>>>> >>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>> >>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>> >>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>> >>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>> >>>>> Regards, >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpre >>>>> ss- >>>>> v >>>>> 2 >>>>> p >>>>> - >>>>> c >>>>> a >>>>> 15_a7.dts [2]. >>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/ >>>>> jun >>>>> o >>>>> - >>>>> b >>>>> a >>>>> s >>>>> e >>>>> .dtsi [3]. >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linu >>>>> x.g >>>>> i >>>>> t >>>>> / >>>>> t >>>>> r >>>>> e >>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc >>>>> 4 >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n1 1 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. > If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 3:27 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. > > For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). > > Mathieu > > On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I am using master branch >> >> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >> git checkout -b master origin/master >> >> Is this a correct branch for opencsd? >> >> I am cross compiling both opencsd and tools/perf. >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 2:34 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Mathieu >>> >>> >>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>> Do you have some ideas if we are missing a patch? >> >> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >> >>> >>> CC util/parse-events.o >>> CC util/parse-events-flex.o >>> CC util/pmu.o >>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >> >> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >> >> [1]. >> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ce >> a >> a4e >> f >> 0 >> 0 >> 15c5d25c4b3 >> >>> CC util/pmu-flex.o >>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>> undeclared identifier is reported only once for each function >>> it appears in >>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>> ^~~~~~~~~~~~~~~~~~~~~~~~ >> >> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >> >> [2]. >> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2 >> b >> b24 >> 2 >> b >> d >> 9f7328e7104 >> >>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>> No such file or directory >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 10:15 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks. >>>> >>>> Do you have links to these patches? >>> >>> It's all there: >>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 10:06 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>> >>>> Ok. >>>> >>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>> So, are you saying that I only need to replace tool/perf directory? >>>> >>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Matt >>>>> >>>>> Hi there, >>>>> >>>>>> >>>>>> >>>>>> >>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>> try instruction tracing using perf-opencsd-4.9. >>>>> >>>>> Ok, that should be an interesting project. >>>>> >>>>>> >>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>> Looks like this tree consists of base >>>>>> >>>>>> Linux kernel 4.9 plus some some additions to support >>>>>> CoreSight and instruction trace using Perf tool. >>>>> >>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> For adding trace functionality, Is it possible to get a >>>>>> patch that I can apply to my base Linux 4.9? >>>>> >>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>> >>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>> >>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>> >>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>> >>>>> Regards, >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpr >>>>> e >>>>> ss- >>>>> v >>>>> 2 >>>>> p >>>>> - >>>>> c >>>>> a >>>>> 15_a7.dts [2]. >>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm >>>>> / >>>>> jun >>>>> o >>>>> - >>>>> b >>>>> a >>>>> s >>>>> e >>>>> .dtsi [3]. >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin >>>>> u >>>>> x.g >>>>> i >>>>> t >>>>> / >>>>> t >>>>> r >>>>> e >>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-r >>>>> c >>>>> 4 >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n1 1 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We have changed the device tree to reflect all CoreSight components on our platform. Still I do not see any device listed in the following directory
/sys/bus/coresight/devices
Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, March 31, 2017 3:37 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Subject: Re: Perf-opencsd-4.9
On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. > If I see more problems, I may have to switch building them on the target.
Ok, let me know.
I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
> > Regards, Reza > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 3:27 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. > > For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). > > Mathieu > > On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I am using master branch >> >> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >> git checkout -b master origin/master >> >> Is this a correct branch for opencsd? >> >> I am cross compiling both opencsd and tools/perf. >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 2:34 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Mathieu >>> >>> >>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>> Do you have some ideas if we are missing a patch? >> >> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >> >>> >>> CC util/parse-events.o >>> CC util/parse-events-flex.o >>> CC util/pmu.o >>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >> >> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >> >> [1]. >> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ce >> a >> a4e >> f >> 0 >> 0 >> 15c5d25c4b3 >> >>> CC util/pmu-flex.o >>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>> undeclared identifier is reported only once for each function >>> it appears in >>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>> ^~~~~~~~~~~~~~~~~~~~~~~~ >> >> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >> >> [2]. >> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2 >> b >> b24 >> 2 >> b >> d >> 9f7328e7104 >> >>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>> No such file or directory >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, March 30, 2017 10:15 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks. >>>> >>>> Do you have links to these patches? >>> >>> It's all there: >>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 10:06 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>> >>>> Ok. >>>> >>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>> So, are you saying that I only need to replace tool/perf directory? >>>> >>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Matt >>>>> >>>>> Hi there, >>>>> >>>>>> >>>>>> >>>>>> >>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>> try instruction tracing using perf-opencsd-4.9. >>>>> >>>>> Ok, that should be an interesting project. >>>>> >>>>>> >>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>> Looks like this tree consists of base >>>>>> >>>>>> Linux kernel 4.9 plus some some additions to support >>>>>> CoreSight and instruction trace using Perf tool. >>>>> >>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> For adding trace functionality, Is it possible to get a >>>>>> patch that I can apply to my base Linux 4.9? >>>>> >>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>> >>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>> >>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>> >>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>> >>>>> Regards, >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpr >>>>> e >>>>> ss- >>>>> v >>>>> 2 >>>>> p >>>>> - >>>>> c >>>>> a >>>>> 15_a7.dts [2]. >>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm >>>>> / >>>>> jun >>>>> o >>>>> - >>>>> b >>>>> a >>>>> s >>>>> e >>>>> .dtsi [3]. >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin >>>>> u >>>>> x.g >>>>> i >>>>> t >>>>> / >>>>> t >>>>> r >>>>> e >>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-r >>>>> c >>>>> 4 >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/coresight-e...
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n1 1 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > We have changed the device tree to reflect all CoreSight components on our platform. > Still I do not see any device listed in the following directory > > /sys/bus/coresight/devices > > Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
> > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 3:37 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >> If I see more problems, I may have to switch building them on the target. > > Ok, let me know. > > I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. > >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 3:27 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >> >> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >> >> Mathieu >> >> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I am using master branch >>> >>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>> git checkout -b master origin/master >>> >>> Is this a correct branch for opencsd? >>> >>> I am cross compiling both opencsd and tools/perf. >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 2:34 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Hello Mathieu >>>> >>>> >>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>> Do you have some ideas if we are missing a patch? >>> >>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>> >>>> >>>> CC util/parse-events.o >>>> CC util/parse-events-flex.o >>>> CC util/pmu.o >>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>> >>> [1]. >>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ce >>> a >>> a4e >>> f >>> 0 >>> 0 >>> 15c5d25c4b3 >>> >>>> CC util/pmu-flex.o >>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>> undeclared identifier is reported only once for each function >>>> it appears in >>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>> >>> [2]. >>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2 >>> b >>> b24 >>> 2 >>> b >>> d >>> 9f7328e7104 >>> >>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>> No such file or directory >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 10:15 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks. >>>>> >>>>> Do you have links to these patches? >>>> >>>> It's all there: >>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>> >>>>> Ok. >>>>> >>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>> >>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Matt >>>>>> >>>>>> Hi there, >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>>> try instruction tracing using perf-opencsd-4.9. >>>>>> >>>>>> Ok, that should be an interesting project. >>>>>> >>>>>>> >>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>> Looks like this tree consists of base >>>>>>> >>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>> CoreSight and instruction trace using Perf tool. >>>>>> >>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> For adding trace functionality, Is it possible to get a >>>>>>> patch that I can apply to my base Linux 4.9? >>>>>> >>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>> >>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>> >>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>> >>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>> >>>>>> Regards, >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpr >>>>>> e >>>>>> ss- >>>>>> v >>>>>> 2 >>>>>> p >>>>>> - >>>>>> c >>>>>> a >>>>>> 15_a7.dts [2]. >>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm >>>>>> / >>>>>> jun >>>>>> o >>>>>> - >>>>>> b >>>>>> a >>>>>> s >>>>>> e >>>>>> .dtsi [3]. >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin >>>>>> u >>>>>> x.g >>>>>> i >>>>>> t >>>>>> / >>>>>> t >>>>>> r >>>>>> e >>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-r >>>>>> c >>>>>> 4 >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/coresight-e...
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
I fixed the power domain issue. Now, it can discover ETMs. For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0.
How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git / t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n 1 1 14
Regards, Reza
entering ETM4_probe
coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
Leaving ETMv4 Probe
entering ETMv4_probe sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x68/0x88 Modules linked in:
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 9:23 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > We have changed the device tree to reflect all CoreSight components on our platform. > Still I do not see any device listed in the following directory > > /sys/bus/coresight/devices > > Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and discoverable) on the bus. The first thing to do is make sure the CS devices show up at enumeration time. I suggest instrumenting function amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
> > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, March 31, 2017 3:37 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Subject: Re: Perf-opencsd-4.9 > > On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >> If I see more problems, I may have to switch building them on the target. > > Ok, let me know. > > I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. > >> >> Regards, Reza >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 3:27 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >> >> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >> >> Mathieu >> >> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I am using master branch >>> >>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>> git checkout -b master origin/master >>> >>> Is this a correct branch for opencsd? >>> >>> I am cross compiling both opencsd and tools/perf. >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 2:34 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Hello Mathieu >>>> >>>> >>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>> Do you have some ideas if we are missing a patch? >>> >>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>> >>>> >>>> CC util/parse-events.o >>>> CC util/parse-events-flex.o >>>> CC util/pmu.o >>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>> >>> [1]. >>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87c >>> e >>> a >>> a4e >>> f >>> 0 >>> 0 >>> 15c5d25c4b3 >>> >>>> CC util/pmu-flex.o >>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>> undeclared identifier is reported only once for each function >>>> it appears in >>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>> >>> [2]. >>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd >>> 2 >>> b >>> b24 >>> 2 >>> b >>> d >>> 9f7328e7104 >>> >>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>> No such file or directory >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Thursday, March 30, 2017 10:15 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks. >>>>> >>>>> Do you have links to these patches? >>>> >>>> It's all there: >>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>> >>>>> Ok. >>>>> >>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>> >>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Matt >>>>>> >>>>>> Hi there, >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>>> try instruction tracing using perf-opencsd-4.9. >>>>>> >>>>>> Ok, that should be an interesting project. >>>>>> >>>>>>> >>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>> Looks like this tree consists of base >>>>>>> >>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>> CoreSight and instruction trace using Perf tool. >>>>>> >>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> For adding trace functionality, Is it possible to get a >>>>>>> patch that I can apply to my base Linux 4.9? >>>>>> >>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>> >>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>> >>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>> >>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>> >>>>>> Regards, >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexp >>>>>> r >>>>>> e >>>>>> ss- >>>>>> v >>>>>> 2 >>>>>> p >>>>>> - >>>>>> c >>>>>> a >>>>>> 15_a7.dts [2]. >>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/ar >>>>>> m >>>>>> / >>>>>> jun >>>>>> o >>>>>> - >>>>>> b >>>>>> a >>>>>> s >>>>>> e >>>>>> .dtsi [3]. >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/li >>>>>> n >>>>>> u >>>>>> x.g >>>>>> i >>>>>> t >>>>>> / >>>>>> t >>>>>> r >>>>>> e >>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11- >>>>>> r >>>>>> c >>>>>> 4 >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/coresight-e...
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I fixed the power domain issue. Now, it can discover ETMs. > For some reason, it cannot create /sys/devices/cs_etm/cpu1 Rather > than creating cpu1, it tries to recreate cpu0. > > How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git / t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7#n 1 1 14
> > Regards, Reza > > entering ETM4_probe > > coresight-etm4x 8008c40000.etm: ETM 4.0 initialized > > Leaving ETMv4 Probe > > entering ETMv4_probe > sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 > sysfs_warn_dup+0x68/0x88 Modules linked in: > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Monday, April 17, 2017 9:23 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Mathieu >> >> We have changed the device tree to reflect all CoreSight components on our platform. >> Still I do not see any device listed in the following directory >> >> /sys/bus/coresight/devices >> >> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. > > When the system boots AMBA probes for devices connected (and > discoverable) on the bus. The first thing to do is make sure the > CS devices show up at enumeration time. I suggest instrumenting > function > amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. > While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. > > If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. > > Mathieu > > [1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 > > >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 3:37 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>> If I see more problems, I may have to switch building them on the target. >> >> Ok, let me know. >> >> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 3:27 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>> >>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>> >>> Mathieu >>> >>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I am using master branch >>>> >>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>> git checkout -b master origin/master >>>> >>>> Is this a correct branch for opencsd? >>>> >>>> I am cross compiling both opencsd and tools/perf. >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 2:34 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Mathieu >>>>> >>>>> >>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>> Do you have some ideas if we are missing a patch? >>>> >>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>> >>>>> >>>>> CC util/parse-events.o >>>>> CC util/parse-events-flex.o >>>>> CC util/pmu.o >>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>> >>>> [1]. >>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87c >>>> e >>>> a >>>> a4e >>>> f >>>> 0 >>>> 0 >>>> 15c5d25c4b3 >>>> >>>>> CC util/pmu-flex.o >>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>> undeclared identifier is reported only once for each function >>>>> it appears in >>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>> >>>> [2]. >>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd >>>> 2 >>>> b >>>> b24 >>>> 2 >>>> b >>>> d >>>> 9f7328e7104 >>>> >>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>> No such file or directory >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks. >>>>>> >>>>>> Do you have links to these patches? >>>>> >>>>> It's all there: >>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>> >>>>>> Ok. >>>>>> >>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>> >>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Matt >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to >>>>>>>> try instruction tracing using perf-opencsd-4.9. >>>>>>> >>>>>>> Ok, that should be an interesting project. >>>>>>> >>>>>>>> >>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>> Looks like this tree consists of base >>>>>>>> >>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>> >>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>> >>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>> >>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>> >>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>> >>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>> >>>>>>> Regards, >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexp >>>>>>> r >>>>>>> e >>>>>>> ss- >>>>>>> v >>>>>>> 2 >>>>>>> p >>>>>>> - >>>>>>> c >>>>>>> a >>>>>>> 15_a7.dts [2]. >>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/ar >>>>>>> m >>>>>>> / >>>>>>> jun >>>>>>> o >>>>>>> - >>>>>>> b >>>>>>> a >>>>>>> s >>>>>>> e >>>>>>> .dtsi [3]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/li >>>>>>> n >>>>>>> u >>>>>>> x.g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> r >>>>>>> e >>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11- >>>>>>> r >>>>>>> c >>>>>>> 4 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/cores ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" When I am running it on the target. Am I missing a package?
When building "perf": .. bpf: [ on ]
When running "perf" on the target. ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0-etf/' ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Monday, April 17, 2017 4:03 PM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, April 17, 2017 2:51 PM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > I fixed the power domain issue. Now, it can discover ETMs. > For some reason, it cannot create /sys/devices/cs_etm/cpu1 > Rather than creating cpu1, it tries to recreate cpu0. > > How does it map ETMs to CPUs?
The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
Instrumenting the code to understand what is going on is always a good idea.
Mathieu
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gi t / t ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7# n 1 1 14
> > Regards, Reza > > entering ETM4_probe > > coresight-etm4x 8008c40000.etm: ETM 4.0 initialized > > Leaving ETMv4 Probe > > entering ETMv4_probe > sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 > sysfs_warn_dup+0x68/0x88 Modules linked in: > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Monday, April 17, 2017 9:23 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Hello Mathieu >> >> We have changed the device tree to reflect all CoreSight components on our platform. >> Still I do not see any device listed in the following directory >> >> /sys/bus/coresight/devices >> >> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. > > When the system boots AMBA probes for devices connected (and > discoverable) on the bus. The first thing to do is make sure the > CS devices show up at enumeration time. I suggest instrumenting > function > amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. > While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. > > If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. > > Mathieu > > [1]. > http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 > > >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Friday, March 31, 2017 3:37 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Subject: Re: Perf-opencsd-4.9 >> >> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>> If I see more problems, I may have to switch building them on the target. >> >> Ok, let me know. >> >> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 3:27 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>> >>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>> >>> Mathieu >>> >>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I am using master branch >>>> >>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>> git checkout -b master origin/master >>>> >>>> Is this a correct branch for opencsd? >>>> >>>> I am cross compiling both opencsd and tools/perf. >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 2:34 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Mathieu >>>>> >>>>> >>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>> Do you have some ideas if we are missing a patch? >>>> >>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>> >>>>> >>>>> CC util/parse-events.o >>>>> CC util/parse-events-flex.o >>>>> CC util/pmu.o >>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>> >>>> [1]. >>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87 >>>> c >>>> e >>>> a >>>> a4e >>>> f >>>> 0 >>>> 0 >>>> 15c5d25c4b3 >>>> >>>>> CC util/pmu-flex.o >>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>> undeclared identifier is reported only once for each >>>>> function it appears in >>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>> >>>> [2]. >>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0af >>>> d >>>> 2 >>>> b >>>> b24 >>>> 2 >>>> b >>>> d >>>> 9f7328e7104 >>>> >>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>> No such file or directory >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks. >>>>>> >>>>>> Do you have links to these patches? >>>>> >>>>> It's all there: >>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>> >>>>>> Ok. >>>>>> >>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>> >>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Matt >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>> >>>>>>> Ok, that should be an interesting project. >>>>>>> >>>>>>>> >>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>> Looks like this tree consists of base >>>>>>>> >>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>> >>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>> >>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>> >>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>> >>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>> >>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>> >>>>>>> Regards, >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vex >>>>>>> p >>>>>>> r >>>>>>> e >>>>>>> ss- >>>>>>> v >>>>>>> 2 >>>>>>> p >>>>>>> - >>>>>>> c >>>>>>> a >>>>>>> 15_a7.dts [2]. >>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/a >>>>>>> r >>>>>>> m >>>>>>> / >>>>>>> jun >>>>>>> o >>>>>>> - >>>>>>> b >>>>>>> a >>>>>>> s >>>>>>> e >>>>>>> .dtsi [3]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/l >>>>>>> i >>>>>>> n >>>>>>> u >>>>>>> x.g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> r >>>>>>> e >>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11 >>>>>>> - >>>>>>> r >>>>>>> c >>>>>>> 4 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, Reza
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/cores ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > > Hello Mathieu > > BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" > When I am running it on the target. Am I missing a package? > > When building "perf": > .. bpf: [ on ] > > When running "perf" on the target. > ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname > event syntax error: 'cs_etm/8008010000.cluster0-etf/' > ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
> > > Regards, Reza > > -----Original Message----- > From: Etemadi, Mohammad > Sent: Monday, April 17, 2017 4:03 PM > To: Mathieu Poirier mathieu.poirier@linaro.org > Cc: coresight@lists.linaro.org > Subject: RE: Perf-opencsd-4.9 > > > Thanks Mathieu > > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Monday, April 17, 2017 2:51 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I fixed the power domain issue. Now, it can discover ETMs. >> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >> Rather than creating cpu1, it tries to recreate cpu0. >> >> How does it map ETMs to CPUs? > > The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. > > Instrumenting the code to understand what is going on is always a good idea. > > Mathieu > > [1]. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gi > t > / > t > ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7# > n > 1 > 1 > 14 > >> >> Regards, Reza >> >> entering ETM4_probe >> >> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >> >> Leaving ETMv4 Probe >> >> entering ETMv4_probe >> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >> ------------[ cut here ]------------ >> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >> sysfs_warn_dup+0x68/0x88 Modules linked in: >> >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Monday, April 17, 2017 9:23 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Mathieu >>> >>> We have changed the device tree to reflect all CoreSight components on our platform. >>> Still I do not see any device listed in the following directory >>> >>> /sys/bus/coresight/devices >>> >>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >> >> When the system boots AMBA probes for devices connected (and >> discoverable) on the bus. The first thing to do is make sure the >> CS devices show up at enumeration time. I suggest instrumenting >> function >> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >> >> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >> >> Mathieu >> >> [1]. >> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >> >> >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 3:37 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>> If I see more problems, I may have to switch building them on the target. >>> >>> Ok, let me know. >>> >>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 3:27 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>> >>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>> >>>> Mathieu >>>> >>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I am using master branch >>>>> >>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>> git checkout -b master origin/master >>>>> >>>>> Is this a correct branch for opencsd? >>>>> >>>>> I am cross compiling both opencsd and tools/perf. >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Mathieu >>>>>> >>>>>> >>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>> Do you have some ideas if we are missing a patch? >>>>> >>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>> >>>>>> >>>>>> CC util/parse-events.o >>>>>> CC util/parse-events-flex.o >>>>>> CC util/pmu.o >>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>> >>>>> [1]. >>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87 >>>>> c >>>>> e >>>>> a >>>>> a4e >>>>> f >>>>> 0 >>>>> 0 >>>>> 15c5d25c4b3 >>>>> >>>>>> CC util/pmu-flex.o >>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>> undeclared identifier is reported only once for each >>>>>> function it appears in >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>> >>>>> [2]. >>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0af >>>>> d >>>>> 2 >>>>> b >>>>> b24 >>>>> 2 >>>>> b >>>>> d >>>>> 9f7328e7104 >>>>> >>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>> No such file or directory >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks. >>>>>>> >>>>>>> Do you have links to these patches? >>>>>> >>>>>> It's all there: >>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>> >>>>>>> Ok. >>>>>>> >>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>> >>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Matt >>>>>>>> >>>>>>>> Hi there, >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>>> >>>>>>>> Ok, that should be an interesting project. >>>>>>>> >>>>>>>>> >>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>> Looks like this tree consists of base >>>>>>>>> >>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>> >>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>>> >>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>> >>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>> >>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>> >>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vex >>>>>>>> p >>>>>>>> r >>>>>>>> e >>>>>>>> ss- >>>>>>>> v >>>>>>>> 2 >>>>>>>> p >>>>>>>> - >>>>>>>> c >>>>>>>> a >>>>>>>> 15_a7.dts [2]. >>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/a >>>>>>>> r >>>>>>>> m >>>>>>>> / >>>>>>>> jun >>>>>>>> o >>>>>>>> - >>>>>>>> b >>>>>>>> a >>>>>>>> s >>>>>>>> e >>>>>>>> .dtsi [3]. >>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/l >>>>>>>> i >>>>>>>> n >>>>>>>> u >>>>>>>> x.g >>>>>>>> i >>>>>>>> t >>>>>>>> / >>>>>>>> t >>>>>>>> r >>>>>>>> e >>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11 >>>>>>>> - >>>>>>>> r >>>>>>>> c >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/core s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > > Hello Mathieu > > BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" > When I am running it on the target. Am I missing a package? > > When building "perf": > .. bpf: [ on ] > > When running "perf" on the target. > ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname > event syntax error: 'cs_etm/8008010000.cluster0-etf/' > ___ BPF support is not compiled
The perf command line parser is insanely complex. Here I suppose the extra "-etf" is causing the bison parser to hit a BPF rule, hence the error message. If you drop the "-etf" in the DT I'm pretty sure the message will go away. You can also enhance the flex/bison parser but to me the "address.name" convention is enough.
> > > Regards, Reza > > -----Original Message----- > From: Etemadi, Mohammad > Sent: Monday, April 17, 2017 4:03 PM > To: Mathieu Poirier mathieu.poirier@linaro.org > Cc: coresight@lists.linaro.org > Subject: RE: Perf-opencsd-4.9 > > > Thanks Mathieu > > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Monday, April 17, 2017 2:51 PM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> I fixed the power domain issue. Now, it can discover ETMs. >> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >> Rather than creating cpu1, it tries to recreate cpu0. >> >> How does it map ETMs to CPUs? > > The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. > > Instrumenting the code to understand what is going on is always a good idea. > > Mathieu > > [1]. > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g > i > t > / > t > ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7 > # > n > 1 > 1 > 14 > >> >> Regards, Reza >> >> entering ETM4_probe >> >> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >> >> Leaving ETMv4 Probe >> >> entering ETMv4_probe >> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >> ------------[ cut here ]------------ >> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >> sysfs_warn_dup+0x68/0x88 Modules linked in: >> >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Monday, April 17, 2017 9:23 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Hello Mathieu >>> >>> We have changed the device tree to reflect all CoreSight components on our platform. >>> Still I do not see any device listed in the following >>> directory >>> >>> /sys/bus/coresight/devices >>> >>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >> >> When the system boots AMBA probes for devices connected (and >> discoverable) on the bus. The first thing to do is make sure >> the CS devices show up at enumeration time. I suggest >> instrumenting function >> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >> >> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >> >> Mathieu >> >> [1]. >> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >> >> >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Friday, March 31, 2017 3:37 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>> If I see more problems, I may have to switch building them on the target. >>> >>> Ok, let me know. >>> >>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 3:27 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>> >>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>> >>>> Mathieu >>>> >>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I am using master branch >>>>> >>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>> git checkout -b master origin/master >>>>> >>>>> Is this a correct branch for opencsd? >>>>> >>>>> I am cross compiling both opencsd and tools/perf. >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Mathieu >>>>>> >>>>>> >>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>> Do you have some ideas if we are missing a patch? >>>>> >>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>> >>>>>> >>>>>> CC util/parse-events.o >>>>>> CC util/parse-events-flex.o >>>>>> CC util/pmu.o >>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>> >>>>> [1]. >>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b8 >>>>> 7 >>>>> c >>>>> e >>>>> a >>>>> a4e >>>>> f >>>>> 0 >>>>> 0 >>>>> 15c5d25c4b3 >>>>> >>>>>> CC util/pmu-flex.o >>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>> undeclared identifier is reported only once for each >>>>>> function it appears in >>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>> >>>>> [2]. >>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0a >>>>> f >>>>> d >>>>> 2 >>>>> b >>>>> b24 >>>>> 2 >>>>> b >>>>> d >>>>> 9f7328e7104 >>>>> >>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>> No such file or directory >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks. >>>>>>> >>>>>>> Do you have links to these patches? >>>>>> >>>>>> It's all there: >>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>> >>>>>>> Ok. >>>>>>> >>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>> >>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Matt >>>>>>>> >>>>>>>> Hi there, >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>>> >>>>>>>> Ok, that should be an interesting project. >>>>>>>> >>>>>>>>> >>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>> Looks like this tree consists of base >>>>>>>>> >>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>> >>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>>> >>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>> >>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>> >>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>> >>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/ve >>>>>>>> x >>>>>>>> p >>>>>>>> r >>>>>>>> e >>>>>>>> ss- >>>>>>>> v >>>>>>>> 2 >>>>>>>> p >>>>>>>> - >>>>>>>> c >>>>>>>> a >>>>>>>> 15_a7.dts [2]. >>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/ >>>>>>>> a >>>>>>>> r >>>>>>>> m >>>>>>>> / >>>>>>>> jun >>>>>>>> o >>>>>>>> - >>>>>>>> b >>>>>>>> a >>>>>>>> s >>>>>>>> e >>>>>>>> .dtsi [3]. >>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/ >>>>>>>> l >>>>>>>> i >>>>>>>> n >>>>>>>> u >>>>>>>> x.g >>>>>>>> i >>>>>>>> t >>>>>>>> / >>>>>>>> t >>>>>>>> r >>>>>>>> e >>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>>>> 1 >>>>>>>> - >>>>>>>> r >>>>>>>> c >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/core s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: > On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> >> Hello Mathieu >> >> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >> When I am running it on the target. Am I missing a package? >> >> When building "perf": >> .. bpf: [ on ] >> >> When running "perf" on the target. >> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >> ___ BPF support is not compiled > > The perf command line parser is insanely complex. Here I suppose > the extra "-etf" is causing the bison parser to hit a BPF rule, > hence the error message. If you drop the "-etf" in the DT I'm > pretty sure the message will go away. You can also enhance the > flex/bison parser but to me the "address.name" convention is enough. > >> >> >> Regards, Reza >> >> -----Original Message----- >> From: Etemadi, Mohammad >> Sent: Monday, April 17, 2017 4:03 PM >> To: Mathieu Poirier mathieu.poirier@linaro.org >> Cc: coresight@lists.linaro.org >> Subject: RE: Perf-opencsd-4.9 >> >> >> Thanks Mathieu >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Monday, April 17, 2017 2:51 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I fixed the power domain issue. Now, it can discover ETMs. >>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>> Rather than creating cpu1, it tries to recreate cpu0. >>> >>> How does it map ETMs to CPUs? >> >> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >> >> Instrumenting the code to understand what is going on is always a good idea. >> >> Mathieu >> >> [1]. >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.g >> i >> t >> / >> t >> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc7 >> # >> n >> 1 >> 1 >> 14 >> >>> >>> Regards, Reza >>> >>> entering ETM4_probe >>> >>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>> >>> Leaving ETMv4 Probe >>> >>> entering ETMv4_probe >>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>> ------------[ cut here ]------------ >>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>> >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Monday, April 17, 2017 9:23 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Hello Mathieu >>>> >>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>> Still I do not see any device listed in the following >>>> directory >>>> >>>> /sys/bus/coresight/devices >>>> >>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>> >>> When the system boots AMBA probes for devices connected (and >>> discoverable) on the bus. The first thing to do is make sure >>> the CS devices show up at enumeration time. I suggest >>> instrumenting function >>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>> >>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>> >>> Mathieu >>> >>> [1]. >>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>> >>> >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 3:37 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>> If I see more problems, I may have to switch building them on the target. >>>> >>>> Ok, let me know. >>>> >>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>> >>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>> >>>>> Mathieu >>>>> >>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I am using master branch >>>>>> >>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>> git checkout -b master origin/master >>>>>> >>>>>> Is this a correct branch for opencsd? >>>>>> >>>>>> I am cross compiling both opencsd and tools/perf. >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Mathieu >>>>>>> >>>>>>> >>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>> Do you have some ideas if we are missing a patch? >>>>>> >>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>> >>>>>>> >>>>>>> CC util/parse-events.o >>>>>>> CC util/parse-events-flex.o >>>>>>> CC util/pmu.o >>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>> >>>>>> [1]. >>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b8 >>>>>> 7 >>>>>> c >>>>>> e >>>>>> a >>>>>> a4e >>>>>> f >>>>>> 0 >>>>>> 0 >>>>>> 15c5d25c4b3 >>>>>> >>>>>>> CC util/pmu-flex.o >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>> undeclared identifier is reported only once for each >>>>>>> function it appears in >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>> >>>>>> [2]. >>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0a >>>>>> f >>>>>> d >>>>>> 2 >>>>>> b >>>>>> b24 >>>>>> 2 >>>>>> b >>>>>> d >>>>>> 9f7328e7104 >>>>>> >>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>> No such file or directory >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Do you have links to these patches? >>>>>>> >>>>>>> It's all there: >>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>> >>>>>>>> Ok. >>>>>>>> >>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>> >>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Hello Matt >>>>>>>>> >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>>>> >>>>>>>>> Ok, that should be an interesting project. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>> Looks like this tree consists of base >>>>>>>>>> >>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>> >>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>>>> >>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>> >>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>> >>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>> >>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mathieu >>>>>>>>> >>>>>>>>> [1]. >>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/ve >>>>>>>>> x >>>>>>>>> p >>>>>>>>> r >>>>>>>>> e >>>>>>>>> ss- >>>>>>>>> v >>>>>>>>> 2 >>>>>>>>> p >>>>>>>>> - >>>>>>>>> c >>>>>>>>> a >>>>>>>>> 15_a7.dts [2]. >>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/ >>>>>>>>> a >>>>>>>>> r >>>>>>>>> m >>>>>>>>> / >>>>>>>>> jun >>>>>>>>> o >>>>>>>>> - >>>>>>>>> b >>>>>>>>> a >>>>>>>>> s >>>>>>>>> e >>>>>>>>> .dtsi [3]. >>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/ >>>>>>>>> l >>>>>>>>> i >>>>>>>>> n >>>>>>>>> u >>>>>>>>> x.g >>>>>>>>> i >>>>>>>>> t >>>>>>>>> / >>>>>>>>> t >>>>>>>>> r >>>>>>>>> e >>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>>>>> 1 >>>>>>>>> - >>>>>>>>> r >>>>>>>>> c >>>>>>>>> 4 >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza > _______________________________________________ > CoreSight mailing list > CoreSight@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
Hello Mathieu
We are using perf-opencsd-4.9. Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/cor e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote:
Is it also possible that a leading @ is required? i.e. ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
Mike
On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: > On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> >> Hello Mathieu >> >> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >> When I am running it on the target. Am I missing a package? >> >> When building "perf": >> .. bpf: [ on ] >> >> When running "perf" on the target. >> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >> ___ BPF support is not compiled > > The perf command line parser is insanely complex. Here I > suppose the extra "-etf" is causing the bison parser to hit a > BPF rule, hence the error message. If you drop the "-etf" in > the DT I'm pretty sure the message will go away. You can also > enhance the flex/bison parser but to me the "address.name" convention is enough. > >> >> >> Regards, Reza >> >> -----Original Message----- >> From: Etemadi, Mohammad >> Sent: Monday, April 17, 2017 4:03 PM >> To: Mathieu Poirier mathieu.poirier@linaro.org >> Cc: coresight@lists.linaro.org >> Subject: RE: Perf-opencsd-4.9 >> >> >> Thanks Mathieu >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Monday, April 17, 2017 2:51 PM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu >>> >>> I fixed the power domain issue. Now, it can discover ETMs. >>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>> Rather than creating cpu1, it tries to recreate cpu0. >>> >>> How does it map ETMs to CPUs? >> >> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >> >> Instrumenting the code to understand what is going on is always a good idea. >> >> Mathieu >> >> [1]. >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >> g >> i >> t >> / >> t >> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc >> 7 >> # >> n >> 1 >> 1 >> 14 >> >>> >>> Regards, Reza >>> >>> entering ETM4_probe >>> >>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>> >>> Leaving ETMv4 Probe >>> >>> entering ETMv4_probe >>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>> ------------[ cut here ]------------ >>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>> >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Monday, April 17, 2017 9:23 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Hello Mathieu >>>> >>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>> Still I do not see any device listed in the following >>>> directory >>>> >>>> /sys/bus/coresight/devices >>>> >>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>> >>> When the system boots AMBA probes for devices connected (and >>> discoverable) on the bus. The first thing to do is make sure >>> the CS devices show up at enumeration time. I suggest >>> instrumenting function >>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>> >>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>> >>> Mathieu >>> >>> [1]. >>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>> >>> >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Friday, March 31, 2017 3:37 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>> If I see more problems, I may have to switch building them on the target. >>>> >>>> Ok, let me know. >>>> >>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>> >>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>> >>>>> Mathieu >>>>> >>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I am using master branch >>>>>> >>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>> git checkout -b master origin/master >>>>>> >>>>>> Is this a correct branch for opencsd? >>>>>> >>>>>> I am cross compiling both opencsd and tools/perf. >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Mathieu >>>>>>> >>>>>>> >>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>> Do you have some ideas if we are missing a patch? >>>>>> >>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>> >>>>>>> >>>>>>> CC util/parse-events.o >>>>>>> CC util/parse-events-flex.o >>>>>>> CC util/pmu.o >>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>> >>>>>> [1]. >>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b >>>>>> 8 >>>>>> 7 >>>>>> c >>>>>> e >>>>>> a >>>>>> a4e >>>>>> f >>>>>> 0 >>>>>> 0 >>>>>> 15c5d25c4b3 >>>>>> >>>>>>> CC util/pmu-flex.o >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>> undeclared identifier is reported only once for each >>>>>>> function it appears in >>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> >>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>> >>>>>> [2]. >>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0 >>>>>> a >>>>>> f >>>>>> d >>>>>> 2 >>>>>> b >>>>>> b24 >>>>>> 2 >>>>>> b >>>>>> d >>>>>> 9f7328e7104 >>>>>> >>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>> No such file or directory >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Do you have links to these patches? >>>>>>> >>>>>>> It's all there: >>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>> >>>>>>>> Ok. >>>>>>>> >>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>> >>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Hello Matt >>>>>>>>> >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>>>> >>>>>>>>> Ok, that should be an interesting project. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>> Looks like this tree consists of base >>>>>>>>>> >>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>> >>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>>>> >>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>> >>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>> >>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>> >>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mathieu >>>>>>>>> >>>>>>>>> [1]. >>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/v >>>>>>>>> e >>>>>>>>> x >>>>>>>>> p >>>>>>>>> r >>>>>>>>> e >>>>>>>>> ss- >>>>>>>>> v >>>>>>>>> 2 >>>>>>>>> p >>>>>>>>> - >>>>>>>>> c >>>>>>>>> a >>>>>>>>> 15_a7.dts [2]. >>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts >>>>>>>>> / >>>>>>>>> a >>>>>>>>> r >>>>>>>>> m >>>>>>>>> / >>>>>>>>> jun >>>>>>>>> o >>>>>>>>> - >>>>>>>>> b >>>>>>>>> a >>>>>>>>> s >>>>>>>>> e >>>>>>>>> .dtsi [3]. >>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds >>>>>>>>> / >>>>>>>>> l >>>>>>>>> i >>>>>>>>> n >>>>>>>>> u >>>>>>>>> x.g >>>>>>>>> i >>>>>>>>> t >>>>>>>>> / >>>>>>>>> t >>>>>>>>> r >>>>>>>>> e >>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>> 1 >>>>>>>>> 1 >>>>>>>>> - >>>>>>>>> r >>>>>>>>> c >>>>>>>>> 4 >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza > _______________________________________________ > CoreSight mailing list > CoreSight@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/cor e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: > Is it also possible that a leading @ is required? > i.e. > ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
> > Mike > > On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> >>> Hello Mathieu >>> >>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>> When I am running it on the target. Am I missing a package? >>> >>> When building "perf": >>> .. bpf: [ on ] >>> >>> When running "perf" on the target. >>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>> ___ BPF support is not compiled >> >> The perf command line parser is insanely complex. Here I >> suppose the extra "-etf" is causing the bison parser to hit a >> BPF rule, hence the error message. If you drop the "-etf" in >> the DT I'm pretty sure the message will go away. You can also >> enhance the flex/bison parser but to me the "address.name" convention is enough. >> >>> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Monday, April 17, 2017 4:03 PM >>> To: Mathieu Poirier mathieu.poirier@linaro.org >>> Cc: coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> >>> Thanks Mathieu >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Monday, April 17, 2017 2:51 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I fixed the power domain issue. Now, it can discover ETMs. >>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>> Rather than creating cpu1, it tries to recreate cpu0. >>>> >>>> How does it map ETMs to CPUs? >>> >>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>> >>> Instrumenting the code to understand what is going on is always a good idea. >>> >>> Mathieu >>> >>> [1]. >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>> g >>> i >>> t >>> / >>> t >>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc >>> 7 >>> # >>> n >>> 1 >>> 1 >>> 14 >>> >>>> >>>> Regards, Reza >>>> >>>> entering ETM4_probe >>>> >>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>> >>>> Leaving ETMv4 Probe >>>> >>>> entering ETMv4_probe >>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>> ------------[ cut here ]------------ >>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Monday, April 17, 2017 9:23 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Cc: coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Mathieu >>>>> >>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>> Still I do not see any device listed in the following >>>>> directory >>>>> >>>>> /sys/bus/coresight/devices >>>>> >>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>> >>>> When the system boots AMBA probes for devices connected (and >>>> discoverable) on the bus. The first thing to do is make sure >>>> the CS devices show up at enumeration time. I suggest >>>> instrumenting function >>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>> >>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>> >>>> Mathieu >>>> >>>> [1]. >>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>> >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>> If I see more problems, I may have to switch building them on the target. >>>>> >>>>> Ok, let me know. >>>>> >>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>> >>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>> >>>>>> Mathieu >>>>>> >>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I am using master branch >>>>>>> >>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>> git checkout -b master origin/master >>>>>>> >>>>>>> Is this a correct branch for opencsd? >>>>>>> >>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Mathieu >>>>>>>> >>>>>>>> >>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>> >>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>> >>>>>>>> >>>>>>>> CC util/parse-events.o >>>>>>>> CC util/parse-events-flex.o >>>>>>>> CC util/pmu.o >>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> >>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>> >>>>>>> [1]. >>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b >>>>>>> 8 >>>>>>> 7 >>>>>>> c >>>>>>> e >>>>>>> a >>>>>>> a4e >>>>>>> f >>>>>>> 0 >>>>>>> 0 >>>>>>> 15c5d25c4b3 >>>>>>> >>>>>>>> CC util/pmu-flex.o >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>> undeclared identifier is reported only once for each >>>>>>>> function it appears in >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> >>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>> >>>>>>> [2]. >>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0 >>>>>>> a >>>>>>> f >>>>>>> d >>>>>>> 2 >>>>>>> b >>>>>>> b24 >>>>>>> 2 >>>>>>> b >>>>>>> d >>>>>>> 9f7328e7104 >>>>>>> >>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>> No such file or directory >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Do you have links to these patches? >>>>>>>> >>>>>>>> It's all there: >>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9 >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>> >>>>>>>>> Ok. >>>>>>>>> >>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>> >>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Hello Matt >>>>>>>>>> >>>>>>>>>> Hi there, >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like >>>>>>>>>>> to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>> >>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>> >>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>> >>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For adding trace functionality, Is it possible to get a >>>>>>>>>>> patch that I can apply to my base Linux 4.9? >>>>>>>>>> >>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>> >>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>> >>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>> >>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> [1]. >>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/v >>>>>>>>>> e >>>>>>>>>> x >>>>>>>>>> p >>>>>>>>>> r >>>>>>>>>> e >>>>>>>>>> ss- >>>>>>>>>> v >>>>>>>>>> 2 >>>>>>>>>> p >>>>>>>>>> - >>>>>>>>>> c >>>>>>>>>> a >>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts >>>>>>>>>> / >>>>>>>>>> a >>>>>>>>>> r >>>>>>>>>> m >>>>>>>>>> / >>>>>>>>>> jun >>>>>>>>>> o >>>>>>>>>> - >>>>>>>>>> b >>>>>>>>>> a >>>>>>>>>> s >>>>>>>>>> e >>>>>>>>>> .dtsi [3]. >>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds >>>>>>>>>> / >>>>>>>>>> l >>>>>>>>>> i >>>>>>>>>> n >>>>>>>>>> u >>>>>>>>>> x.g >>>>>>>>>> i >>>>>>>>>> t >>>>>>>>>> / >>>>>>>>>> t >>>>>>>>>> r >>>>>>>>>> e >>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>> 1 >>>>>>>>>> 1 >>>>>>>>>> - >>>>>>>>>> r >>>>>>>>>> c >>>>>>>>>> 4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >> _______________________________________________ >> CoreSight mailing list >> CoreSight@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/coresight > > > > -- > Mike Leach > Principal Engineer, ARM Ltd. > Blackburn Design Centre. UK
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/co r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu and Mike
I rebuild the Linux image with name changes in the device tree and still see the same error.
Regards, Reza
# ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/8008010000.cluster0etf/' ___ BPF support is not compiled
This is not the proper syntax.
# ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname event syntax error: 'cs_etm/@8008010000.cluster0etf/' ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, April 19, 2017 10:58 AM To: Mike Leach mike.leach@linaro.org Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: > Is it also possible that a leading @ is required? > i.e. > ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
Definitely yes - that would be the first thing to try.
> > Mike > > On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> >>> Hello Mathieu >>> >>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>> When I am running it on the target. Am I missing a package? >>> >>> When building "perf": >>> .. bpf: [ on ] >>> >>> When running "perf" on the target. >>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>> ___ BPF support is not compiled >> >> The perf command line parser is insanely complex. Here I >> suppose the extra "-etf" is causing the bison parser to hit a >> BPF rule, hence the error message. If you drop the "-etf" in >> the DT I'm pretty sure the message will go away. You can also >> enhance the flex/bison parser but to me the "address.name" convention is enough. >> >>> >>> >>> Regards, Reza >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Monday, April 17, 2017 4:03 PM >>> To: Mathieu Poirier mathieu.poirier@linaro.org >>> Cc: coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> >>> Thanks Mathieu >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Monday, April 17, 2017 2:51 PM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu >>>> >>>> I fixed the power domain issue. Now, it can discover ETMs. >>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>> Rather than creating cpu1, it tries to recreate cpu0. >>>> >>>> How does it map ETMs to CPUs? >>> >>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>> >>> Instrumenting the code to understand what is going on is always a good idea. >>> >>> Mathieu >>> >>> [1]. >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>> g >>> i >>> t >>> / >>> t >>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-r >>> c >>> 7 >>> # >>> n >>> 1 >>> 1 >>> 14 >>> >>>> >>>> Regards, Reza >>>> >>>> entering ETM4_probe >>>> >>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>> >>>> Leaving ETMv4 Probe >>>> >>>> entering ETMv4_probe >>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>> ------------[ cut here ]------------ >>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Monday, April 17, 2017 9:23 AM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Cc: coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Hello Mathieu >>>>> >>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>> Still I do not see any device listed in the following >>>>> directory >>>>> >>>>> /sys/bus/coresight/devices >>>>> >>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>> >>>> When the system boots AMBA probes for devices connected (and >>>> discoverable) on the bus. The first thing to do is make sure >>>> the CS devices show up at enumeration time. I suggest >>>> instrumenting function >>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>> >>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>> >>>> Mathieu >>>> >>>> [1]. >>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>> >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>> If I see more problems, I may have to switch building them on the target. >>>>> >>>>> Ok, let me know. >>>>> >>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>> >>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>> >>>>>> Mathieu >>>>>> >>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I am using master branch >>>>>>> >>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>> git checkout -b master origin/master >>>>>>> >>>>>>> Is this a correct branch for opencsd? >>>>>>> >>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Mathieu >>>>>>>> >>>>>>>> >>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>> >>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>> >>>>>>>> >>>>>>>> CC util/parse-events.o >>>>>>>> CC util/parse-events-flex.o >>>>>>>> CC util/pmu.o >>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> >>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>> >>>>>>> [1]. >>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445 >>>>>>> b >>>>>>> 8 >>>>>>> 7 >>>>>>> c >>>>>>> e >>>>>>> a >>>>>>> a4e >>>>>>> f >>>>>>> 0 >>>>>>> 0 >>>>>>> 15c5d25c4b3 >>>>>>> >>>>>>>> CC util/pmu-flex.o >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>> undeclared identifier is reported only once for each >>>>>>>> function it appears in >>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> >>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>> >>>>>>> [2]. >>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d >>>>>>> 0 >>>>>>> a >>>>>>> f >>>>>>> d >>>>>>> 2 >>>>>>> b >>>>>>> b24 >>>>>>> 2 >>>>>>> b >>>>>>> d >>>>>>> 9f7328e7104 >>>>>>> >>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>> No such file or directory >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Do you have links to these patches? >>>>>>>> >>>>>>>> It's all there: >>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>> 9 >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>> >>>>>>>>> Ok. >>>>>>>>> >>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>> >>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Hello Matt >>>>>>>>>> >>>>>>>>>> Hi there, >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>> >>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>> >>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>> >>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For adding trace functionality, Is it possible to get >>>>>>>>>>> a patch that I can apply to my base Linux 4.9? >>>>>>>>>> >>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>> >>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>> >>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>> >>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> [1]. >>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/ >>>>>>>>>> v >>>>>>>>>> e >>>>>>>>>> x >>>>>>>>>> p >>>>>>>>>> r >>>>>>>>>> e >>>>>>>>>> ss- >>>>>>>>>> v >>>>>>>>>> 2 >>>>>>>>>> p >>>>>>>>>> - >>>>>>>>>> c >>>>>>>>>> a >>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dt >>>>>>>>>> s >>>>>>>>>> / >>>>>>>>>> a >>>>>>>>>> r >>>>>>>>>> m >>>>>>>>>> / >>>>>>>>>> jun >>>>>>>>>> o >>>>>>>>>> - >>>>>>>>>> b >>>>>>>>>> a >>>>>>>>>> s >>>>>>>>>> e >>>>>>>>>> .dtsi [3]. >>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvald >>>>>>>>>> s >>>>>>>>>> / >>>>>>>>>> l >>>>>>>>>> i >>>>>>>>>> n >>>>>>>>>> u >>>>>>>>>> x.g >>>>>>>>>> i >>>>>>>>>> t >>>>>>>>>> / >>>>>>>>>> t >>>>>>>>>> r >>>>>>>>>> e >>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>> 1 >>>>>>>>>> 1 >>>>>>>>>> - >>>>>>>>>> r >>>>>>>>>> c >>>>>>>>>> 4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >> _______________________________________________ >> CoreSight mailing list >> CoreSight@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/coresight > > > > -- > Mike Leach > Principal Engineer, ARM Ltd. > Blackburn Design Centre. UK
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54f71bf6...
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/co r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu and Mike > > I rebuild the Linux image with name changes in the device tree and still see the same error. > > Regards, Reza > > # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/8008010000.cluster0etf/' > ___ BPF support is not compiled >
This is not the proper syntax.
> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/@8008010000.cluster0etf/' > ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
> > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Wednesday, April 19, 2017 10:58 AM > To: Mike Leach mike.leach@linaro.org > Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >> Is it also possible that a leading @ is required? >> i.e. >> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname > > Definitely yes - that would be the first thing to try. > >> >> Mike >> >> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> >>>> Hello Mathieu >>>> >>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>> When I am running it on the target. Am I missing a package? >>>> >>>> When building "perf": >>>> .. bpf: [ on ] >>>> >>>> When running "perf" on the target. >>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>> ___ BPF support is not compiled >>> >>> The perf command line parser is insanely complex. Here I >>> suppose the extra "-etf" is causing the bison parser to hit a >>> BPF rule, hence the error message. If you drop the "-etf" in >>> the DT I'm pretty sure the message will go away. You can also >>> enhance the flex/bison parser but to me the "address.name" convention is enough. >>> >>>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Etemadi, Mohammad >>>> Sent: Monday, April 17, 2017 4:03 PM >>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>> Cc: coresight@lists.linaro.org >>>> Subject: RE: Perf-opencsd-4.9 >>>> >>>> >>>> Thanks Mathieu >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Monday, April 17, 2017 2:51 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Cc: coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>> >>>>> How does it map ETMs to CPUs? >>>> >>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>> >>>> Instrumenting the code to understand what is going on is always a good idea. >>>> >>>> Mathieu >>>> >>>> [1]. >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>> g >>>> i >>>> t >>>> / >>>> t >>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-r >>>> c >>>> 7 >>>> # >>>> n >>>> 1 >>>> 1 >>>> 14 >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> entering ETM4_probe >>>>> >>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>> >>>>> Leaving ETMv4 Probe >>>>> >>>>> entering ETMv4_probe >>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Mathieu >>>>>> >>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>> Still I do not see any device listed in the following >>>>>> directory >>>>>> >>>>>> /sys/bus/coresight/devices >>>>>> >>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>> >>>>> When the system boots AMBA probes for devices connected (and >>>>> discoverable) on the bus. The first thing to do is make sure >>>>> the CS devices show up at enumeration time. I suggest >>>>> instrumenting function >>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>> >>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>> >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>>> >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>> >>>>>> Ok, let me know. >>>>>> >>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>> >>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I am using master branch >>>>>>>> >>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>> git checkout -b master origin/master >>>>>>>> >>>>>>>> Is this a correct branch for opencsd? >>>>>>>> >>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> >>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>> >>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>> >>>>>>>>> >>>>>>>>> CC util/parse-events.o >>>>>>>>> CC util/parse-events-flex.o >>>>>>>>> CC util/pmu.o >>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>> >>>>>>>> [1]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445 >>>>>>>> b >>>>>>>> 8 >>>>>>>> 7 >>>>>>>> c >>>>>>>> e >>>>>>>> a >>>>>>>> a4e >>>>>>>> f >>>>>>>> 0 >>>>>>>> 0 >>>>>>>> 15c5d25c4b3 >>>>>>>> >>>>>>>>> CC util/pmu-flex.o >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>> function it appears in >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>> >>>>>>>> [2]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d >>>>>>>> 0 >>>>>>>> a >>>>>>>> f >>>>>>>> d >>>>>>>> 2 >>>>>>>> b >>>>>>>> b24 >>>>>>>> 2 >>>>>>>> b >>>>>>>> d >>>>>>>> 9f7328e7104 >>>>>>>> >>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>> No such file or directory >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Do you have links to these patches? >>>>>>>>> >>>>>>>>> It's all there: >>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>> 9 >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>> >>>>>>>>>> Ok. >>>>>>>>>> >>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>> >>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Matt >>>>>>>>>>> >>>>>>>>>>> Hi there, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>> >>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>> >>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>> >>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For adding trace functionality, Is it possible to get >>>>>>>>>>>> a patch that I can apply to my base Linux 4.9? >>>>>>>>>>> >>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>> >>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>> >>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>> >>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mathieu >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/ >>>>>>>>>>> v >>>>>>>>>>> e >>>>>>>>>>> x >>>>>>>>>>> p >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> ss- >>>>>>>>>>> v >>>>>>>>>>> 2 >>>>>>>>>>> p >>>>>>>>>>> - >>>>>>>>>>> c >>>>>>>>>>> a >>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dt >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> a >>>>>>>>>>> r >>>>>>>>>>> m >>>>>>>>>>> / >>>>>>>>>>> jun >>>>>>>>>>> o >>>>>>>>>>> - >>>>>>>>>>> b >>>>>>>>>>> a >>>>>>>>>>> s >>>>>>>>>>> e >>>>>>>>>>> .dtsi [3]. >>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvald >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> l >>>>>>>>>>> i >>>>>>>>>>> n >>>>>>>>>>> u >>>>>>>>>>> x.g >>>>>>>>>>> i >>>>>>>>>>> t >>>>>>>>>>> / >>>>>>>>>>> t >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>> 1 >>>>>>>>>>> 1 >>>>>>>>>>> - >>>>>>>>>>> r >>>>>>>>>>> c >>>>>>>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>> _______________________________________________ >>> CoreSight mailing list >>> CoreSight@lists.linaro.org >>> https://lists.linaro.org/mailman/listinfo/coresight >> >> >> >> -- >> Mike Leach >> Principal Engineer, ARM Ltd. >> Blackburn Design Centre. UK
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54f71bf6...
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/c o r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu and Mike > > I rebuild the Linux image with name changes in the device tree and still see the same error. > > Regards, Reza > > # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/8008010000.cluster0etf/' > ___ BPF support is not compiled >
This is not the proper syntax.
> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/@8008010000.cluster0etf/' > ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
> > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Wednesday, April 19, 2017 10:58 AM > To: Mike Leach mike.leach@linaro.org > Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >> Is it also possible that a leading @ is required? >> i.e. >> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname > > Definitely yes - that would be the first thing to try. > >> >> Mike >> >> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> >>>> Hello Mathieu >>>> >>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>> When I am running it on the target. Am I missing a package? >>>> >>>> When building "perf": >>>> .. bpf: [ on ] >>>> >>>> When running "perf" on the target. >>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>> ___ BPF support is not compiled >>> >>> The perf command line parser is insanely complex. Here I >>> suppose the extra "-etf" is causing the bison parser to hit a >>> BPF rule, hence the error message. If you drop the "-etf" in >>> the DT I'm pretty sure the message will go away. You can also >>> enhance the flex/bison parser but to me the "address.name" convention is enough. >>> >>>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Etemadi, Mohammad >>>> Sent: Monday, April 17, 2017 4:03 PM >>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>> Cc: coresight@lists.linaro.org >>>> Subject: RE: Perf-opencsd-4.9 >>>> >>>> >>>> Thanks Mathieu >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Monday, April 17, 2017 2:51 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Cc: coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>> >>>>> How does it map ETMs to CPUs? >>>> >>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>> >>>> Instrumenting the code to understand what is going on is always a good idea. >>>> >>>> Mathieu >>>> >>>> [1]. >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>> g >>>> i >>>> t >>>> / >>>> t >>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11- >>>> r >>>> c >>>> 7 >>>> # >>>> n >>>> 1 >>>> 1 >>>> 14 >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> entering ETM4_probe >>>>> >>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>> >>>>> Leaving ETMv4 Probe >>>>> >>>>> entering ETMv4_probe >>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Mathieu >>>>>> >>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>> Still I do not see any device listed in the following >>>>>> directory >>>>>> >>>>>> /sys/bus/coresight/devices >>>>>> >>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>> >>>>> When the system boots AMBA probes for devices connected (and >>>>> discoverable) on the bus. The first thing to do is make sure >>>>> the CS devices show up at enumeration time. I suggest >>>>> instrumenting function >>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>> >>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>> >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>>> >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>> >>>>>> Ok, let me know. >>>>>> >>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>> >>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I am using master branch >>>>>>>> >>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>> git checkout -b master origin/master >>>>>>>> >>>>>>>> Is this a correct branch for opencsd? >>>>>>>> >>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> >>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>> >>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>> >>>>>>>>> >>>>>>>>> CC util/parse-events.o >>>>>>>>> CC util/parse-events-flex.o >>>>>>>>> CC util/pmu.o >>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>> >>>>>>>> [1]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a344 >>>>>>>> 5 >>>>>>>> b >>>>>>>> 8 >>>>>>>> 7 >>>>>>>> c >>>>>>>> e >>>>>>>> a >>>>>>>> a4e >>>>>>>> f >>>>>>>> 0 >>>>>>>> 0 >>>>>>>> 15c5d25c4b3 >>>>>>>> >>>>>>>>> CC util/pmu-flex.o >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>> function it appears in >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>> >>>>>>>> [2]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3 >>>>>>>> d >>>>>>>> 0 >>>>>>>> a >>>>>>>> f >>>>>>>> d >>>>>>>> 2 >>>>>>>> b >>>>>>>> b24 >>>>>>>> 2 >>>>>>>> b >>>>>>>> d >>>>>>>> 9f7328e7104 >>>>>>>> >>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>> No such file or directory >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Do you have links to these patches? >>>>>>>>> >>>>>>>>> It's all there: >>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>> 9 >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>> >>>>>>>>>> Ok. >>>>>>>>>> >>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>> >>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Matt >>>>>>>>>>> >>>>>>>>>>> Hi there, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>> >>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>> >>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>> >>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For adding trace functionality, Is it possible to get >>>>>>>>>>>> a patch that I can apply to my base Linux 4.9? >>>>>>>>>>> >>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>> >>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>> >>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>> >>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mathieu >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts >>>>>>>>>>> / >>>>>>>>>>> v >>>>>>>>>>> e >>>>>>>>>>> x >>>>>>>>>>> p >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> ss- >>>>>>>>>>> v >>>>>>>>>>> 2 >>>>>>>>>>> p >>>>>>>>>>> - >>>>>>>>>>> c >>>>>>>>>>> a >>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/d >>>>>>>>>>> t >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> a >>>>>>>>>>> r >>>>>>>>>>> m >>>>>>>>>>> / >>>>>>>>>>> jun >>>>>>>>>>> o >>>>>>>>>>> - >>>>>>>>>>> b >>>>>>>>>>> a >>>>>>>>>>> s >>>>>>>>>>> e >>>>>>>>>>> .dtsi [3]. >>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torval >>>>>>>>>>> d >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> l >>>>>>>>>>> i >>>>>>>>>>> n >>>>>>>>>>> u >>>>>>>>>>> x.g >>>>>>>>>>> i >>>>>>>>>>> t >>>>>>>>>>> / >>>>>>>>>>> t >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>> 1 >>>>>>>>>>> 1 >>>>>>>>>>> - >>>>>>>>>>> r >>>>>>>>>>> c >>>>>>>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>> _______________________________________________ >>> CoreSight mailing list >>> CoreSight@lists.linaro.org >>> https://lists.linaro.org/mailman/listinfo/coresight >> >> >> >> -- >> Mike Leach >> Principal Engineer, ARM Ltd. >> Blackburn Design Centre. UK
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54f71bf6...
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/c o r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
I have 8008010000.etf in /sys/bus/coresight/devices
Reza
# ls /sys/bus/coresight/devices 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm #
-----Original Message----- From: Etemadi, Mohammad Sent: Thursday, April 20, 2017 10:50 AM To: 'Mathieu Poirier' mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu
Now, I am hitting another failure. It seems mmap has failed. Does it get the physical address from the name?
# ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname failed to mmap with 12 (Cannot allocate memory)
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 10:03 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu and Mike > > I rebuild the Linux image with name changes in the device tree and still see the same error. > > Regards, Reza > > # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/8008010000.cluster0etf/' > ___ BPF support is not compiled >
This is not the proper syntax.
> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname > event syntax error: 'cs_etm/@8008010000.cluster0etf/' > ___ BPF support is not compiled
This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
> > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Wednesday, April 19, 2017 10:58 AM > To: Mike Leach mike.leach@linaro.org > Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >> Is it also possible that a leading @ is required? >> i.e. >> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname > > Definitely yes - that would be the first thing to try. > >> >> Mike >> >> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>> >>>> Hello Mathieu >>>> >>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>> When I am running it on the target. Am I missing a package? >>>> >>>> When building "perf": >>>> .. bpf: [ on ] >>>> >>>> When running "perf" on the target. >>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>> ___ BPF support is not compiled >>> >>> The perf command line parser is insanely complex. Here I >>> suppose the extra "-etf" is causing the bison parser to hit a >>> BPF rule, hence the error message. If you drop the "-etf" in >>> the DT I'm pretty sure the message will go away. You can also >>> enhance the flex/bison parser but to me the "address.name" convention is enough. >>> >>>> >>>> >>>> Regards, Reza >>>> >>>> -----Original Message----- >>>> From: Etemadi, Mohammad >>>> Sent: Monday, April 17, 2017 4:03 PM >>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>> Cc: coresight@lists.linaro.org >>>> Subject: RE: Perf-opencsd-4.9 >>>> >>>> >>>> Thanks Mathieu >>>> >>>> Regards, Reza >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Monday, April 17, 2017 2:51 PM >>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>> Cc: coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> Thanks Mathieu >>>>> >>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>> >>>>> How does it map ETMs to CPUs? >>>> >>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>> >>>> Instrumenting the code to understand what is going on is always a good idea. >>>> >>>> Mathieu >>>> >>>> [1]. >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>> g >>>> i >>>> t >>>> / >>>> t >>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11- >>>> r >>>> c >>>> 7 >>>> # >>>> n >>>> 1 >>>> 1 >>>> 14 >>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> entering ETM4_probe >>>>> >>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>> >>>>> Leaving ETMv4 Probe >>>>> >>>>> entering ETMv4_probe >>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Hello Mathieu >>>>>> >>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>> Still I do not see any device listed in the following >>>>>> directory >>>>>> >>>>>> /sys/bus/coresight/devices >>>>>> >>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>> >>>>> When the system boots AMBA probes for devices connected (and >>>>> discoverable) on the bus. The first thing to do is make sure >>>>> the CS devices show up at enumeration time. I suggest >>>>> instrumenting function >>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>> >>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>> >>>>> Mathieu >>>>> >>>>> [1]. >>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>>> >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>> >>>>>> Ok, let me know. >>>>>> >>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>> >>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I am using master branch >>>>>>>> >>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>> git checkout -b master origin/master >>>>>>>> >>>>>>>> Is this a correct branch for opencsd? >>>>>>>> >>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> >>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>> >>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>> >>>>>>>>> >>>>>>>>> CC util/parse-events.o >>>>>>>>> CC util/parse-events-flex.o >>>>>>>>> CC util/pmu.o >>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>> >>>>>>>> [1]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a344 >>>>>>>> 5 >>>>>>>> b >>>>>>>> 8 >>>>>>>> 7 >>>>>>>> c >>>>>>>> e >>>>>>>> a >>>>>>>> a4e >>>>>>>> f >>>>>>>> 0 >>>>>>>> 0 >>>>>>>> 15c5d25c4b3 >>>>>>>> >>>>>>>>> CC util/pmu-flex.o >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>> function it appears in >>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> >>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>> >>>>>>>> [2]. >>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3 >>>>>>>> d >>>>>>>> 0 >>>>>>>> a >>>>>>>> f >>>>>>>> d >>>>>>>> 2 >>>>>>>> b >>>>>>>> b24 >>>>>>>> 2 >>>>>>>> b >>>>>>>> d >>>>>>>> 9f7328e7104 >>>>>>>> >>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>> No such file or directory >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Do you have links to these patches? >>>>>>>>> >>>>>>>>> It's all there: >>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>> 9 >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>> >>>>>>>>>> Ok. >>>>>>>>>> >>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>> >>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Matt >>>>>>>>>>> >>>>>>>>>>> Hi there, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>> >>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>> >>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>> >>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For adding trace functionality, Is it possible to get >>>>>>>>>>>> a patch that I can apply to my base Linux 4.9? >>>>>>>>>>> >>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>> >>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>> >>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>> >>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mathieu >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts >>>>>>>>>>> / >>>>>>>>>>> v >>>>>>>>>>> e >>>>>>>>>>> x >>>>>>>>>>> p >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> ss- >>>>>>>>>>> v >>>>>>>>>>> 2 >>>>>>>>>>> p >>>>>>>>>>> - >>>>>>>>>>> c >>>>>>>>>>> a >>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/d >>>>>>>>>>> t >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> a >>>>>>>>>>> r >>>>>>>>>>> m >>>>>>>>>>> / >>>>>>>>>>> jun >>>>>>>>>>> o >>>>>>>>>>> - >>>>>>>>>>> b >>>>>>>>>>> a >>>>>>>>>>> s >>>>>>>>>>> e >>>>>>>>>>> .dtsi [3]. >>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torval >>>>>>>>>>> d >>>>>>>>>>> s >>>>>>>>>>> / >>>>>>>>>>> l >>>>>>>>>>> i >>>>>>>>>>> n >>>>>>>>>>> u >>>>>>>>>>> x.g >>>>>>>>>>> i >>>>>>>>>>> t >>>>>>>>>>> / >>>>>>>>>>> t >>>>>>>>>>> r >>>>>>>>>>> e >>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>> 1 >>>>>>>>>>> 1 >>>>>>>>>>> - >>>>>>>>>>> r >>>>>>>>>>> c >>>>>>>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>> _______________________________________________ >>> CoreSight mailing list >>> CoreSight@lists.linaro.org >>> https://lists.linaro.org/mailman/listinfo/coresight >> >> >> >> -- >> Mike Leach >> Principal Engineer, ARM Ltd. >> Blackburn Design Centre. UK
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54f71bf6...
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/c o r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > > I have 8008010000.etf in /sys/bus/coresight/devices > > Reza > > # ls /sys/bus/coresight/devices > 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel > 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm # > > -----Original Message----- > From: Etemadi, Mohammad > Sent: Thursday, April 20, 2017 10:50 AM > To: 'Mathieu Poirier' mathieu.poirier@linaro.org > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: RE: Perf-opencsd-4.9 > > Thanks Mathieu > > Now, I am hitting another failure. It seems mmap has failed. > Does it get the physical address from the name? > > # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname > failed to mmap with 12 (Cannot allocate memory) > > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, April 20, 2017 10:03 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu and Mike >> >> I rebuild the Linux image with name changes in the device tree and still see the same error. >> >> Regards, Reza >> >> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname >> event syntax error: 'cs_etm/8008010000.cluster0etf/' >> ___ BPF support is not compiled >> > > This is not the proper syntax. > >> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname >> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >> ___ BPF support is not compiled > > This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem. > > I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. > On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works. > >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Wednesday, April 19, 2017 10:58 AM >> To: Mike Leach mike.leach@linaro.org >> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >>> Is it also possible that a leading @ is required? >>> i.e. >>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname >> >> Definitely yes - that would be the first thing to try. >> >>> >>> Mike >>> >>> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> >>>>> Hello Mathieu >>>>> >>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>>> When I am running it on the target. Am I missing a package? >>>>> >>>>> When building "perf": >>>>> .. bpf: [ on ] >>>>> >>>>> When running "perf" on the target. >>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>> ___ BPF support is not compiled >>>> >>>> The perf command line parser is insanely complex. Here I >>>> suppose the extra "-etf" is causing the bison parser to hit a >>>> BPF rule, hence the error message. If you drop the "-etf" in >>>> the DT I'm pretty sure the message will go away. You can also >>>> enhance the flex/bison parser but to me the "address.name" convention is enough. >>>> >>>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Etemadi, Mohammad >>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: RE: Perf-opencsd-4.9 >>>>> >>>>> >>>>> Thanks Mathieu >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>> >>>>>> How does it map ETMs to CPUs? >>>>> >>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>> >>>>> Instrumenting the code to understand what is going on is always a good idea. >>>>> >>>>> Mathieu >>>>> >>>>> [1]. >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>> g >>>>> i >>>>> t >>>>> / >>>>> t >>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11- >>>>> r >>>>> c >>>>> 7 >>>>> # >>>>> n >>>>> 1 >>>>> 1 >>>>> 14 >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> entering ETM4_probe >>>>>> >>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>> >>>>>> Leaving ETMv4 Probe >>>>>> >>>>>> entering ETMv4_probe >>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>> ------------[ cut here ]------------ >>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Mathieu >>>>>>> >>>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>>> Still I do not see any device listed in the following >>>>>>> directory >>>>>>> >>>>>>> /sys/bus/coresight/devices >>>>>>> >>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>>> >>>>>> When the system boots AMBA probes for devices connected (and >>>>>> discoverable) on the bus. The first thing to do is make sure >>>>>> the CS devices show up at enumeration time. I suggest >>>>>> instrumenting function >>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>> >>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344 >>>>>> >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>>> >>>>>>> Ok, let me know. >>>>>>> >>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>> >>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> I am using master branch >>>>>>>>> >>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>>> git checkout -b master origin/master >>>>>>>>> >>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>> >>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Hello Mathieu >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>> >>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> CC util/parse-events.o >>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>> CC util/pmu.o >>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> >>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>> >>>>>>>>> [1]. >>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a344 >>>>>>>>> 5 >>>>>>>>> b >>>>>>>>> 8 >>>>>>>>> 7 >>>>>>>>> c >>>>>>>>> e >>>>>>>>> a >>>>>>>>> a4e >>>>>>>>> f >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 15c5d25c4b3 >>>>>>>>> >>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>> function it appears in >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> >>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>> >>>>>>>>> [2]. >>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3 >>>>>>>>> d >>>>>>>>> 0 >>>>>>>>> a >>>>>>>>> f >>>>>>>>> d >>>>>>>>> 2 >>>>>>>>> b >>>>>>>>> b24 >>>>>>>>> 2 >>>>>>>>> b >>>>>>>>> d >>>>>>>>> 9f7328e7104 >>>>>>>>> >>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>> No such file or directory >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> Do you have links to these patches? >>>>>>>>>> >>>>>>>>>> It's all there: >>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>> 9 >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>> >>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>>> >>>>>>>>>>> Ok. >>>>>>>>>>> >>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>>> >>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Hello Matt >>>>>>>>>>>> >>>>>>>>>>>> Hi there, >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>> >>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>> >>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>>> >>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> For adding trace functionality, Is it possible to get >>>>>>>>>>>>> a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>> >>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>>> >>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>> >>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>> >>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Mathieu >>>>>>>>>>>> >>>>>>>>>>>> [1]. >>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts >>>>>>>>>>>> / >>>>>>>>>>>> v >>>>>>>>>>>> e >>>>>>>>>>>> x >>>>>>>>>>>> p >>>>>>>>>>>> r >>>>>>>>>>>> e >>>>>>>>>>>> ss- >>>>>>>>>>>> v >>>>>>>>>>>> 2 >>>>>>>>>>>> p >>>>>>>>>>>> - >>>>>>>>>>>> c >>>>>>>>>>>> a >>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/d >>>>>>>>>>>> t >>>>>>>>>>>> s >>>>>>>>>>>> / >>>>>>>>>>>> a >>>>>>>>>>>> r >>>>>>>>>>>> m >>>>>>>>>>>> / >>>>>>>>>>>> jun >>>>>>>>>>>> o >>>>>>>>>>>> - >>>>>>>>>>>> b >>>>>>>>>>>> a >>>>>>>>>>>> s >>>>>>>>>>>> e >>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torval >>>>>>>>>>>> d >>>>>>>>>>>> s >>>>>>>>>>>> / >>>>>>>>>>>> l >>>>>>>>>>>> i >>>>>>>>>>>> n >>>>>>>>>>>> u >>>>>>>>>>>> x.g >>>>>>>>>>>> i >>>>>>>>>>>> t >>>>>>>>>>>> / >>>>>>>>>>>> t >>>>>>>>>>>> r >>>>>>>>>>>> e >>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>> 1 >>>>>>>>>>>> 1 >>>>>>>>>>>> - >>>>>>>>>>>> r >>>>>>>>>>>> c >>>>>>>>>>>> 4 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>> _______________________________________________ >>>> CoreSight mailing list >>>> CoreSight@lists.linaro.org >>>> https://lists.linaro.org/mailman/listinfo/coresight >>> >>> >>> >>> -- >>> Mike Leach >>> Principal Engineer, ARM Ltd. >>> Blackburn Design Centre. UK
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). I have only defined 4 ETMs (only first cluster). After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
Regards, Reza
./perf record -e cs_etm/@8008010000.etf/ --per-thread uname cs_etm_get_ro: error reading: cpu4/mgmt/etmccer cs_etm_get_ro: error reading: cpu4/mgmt/etmidr cs_etm_get_ro: error reading: cpu5/mgmt/etmccer cs_etm_get_ro: error reading: cpu5/mgmt/etmidr cs_etm_get_ro: error reading: cpu6/mgmt/etmccer cs_etm_get_ro: error reading: cpu6/mgmt/etmidr cs_etm_get_ro: error reading: cpu7/mgmt/etmccer cs_etm_get_ro: error reading: cpu7/mgmt/etmidr cs_etm_get_ro: error reading: cpu8/mgmt/etmccer cs_etm_get_ro: error reading: cpu8/mgmt/etmidr cs_etm_get_ro: error reading: cpu9/mgmt/etmccer cs_etm_get_ro: error reading: cpu9/mgmt/etmidr cs_etm_get_ro: error reading: cpu10/mgmt/etmccer cs_etm_get_ro: error reading: cpu10/mgmt/etmidr cs_etm_get_ro: error reading: cpu11/mgmt/etmccer cs_etm_get_ro: error reading: cpu11/mgmt/etmidr cs_etm_get_ro: error reading: cpu12/mgmt/etmccer cs_etm_get_ro: error reading: cpu12/mgmt/etmidr cs_etm_get_ro: error reading: cpu13/mgmt/etmccer cs_etm_get_ro: error reading: cpu13/mgmt/etmidr cs_etm_get_ro: error reading: cpu14/mgmt/etmccer cs_etm_get_ro: error reading: cpu14/mgmt/etmidr cs_etm_get_ro: error reading: cpu15/mgmt/etmccer cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: Woken up 7 times to write data ] Warning: AUX data lost 6 times out of 16!
[ perf record: Captured and wrote 0.238 MB perf.data ] #
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Thursday, April 20, 2017 11:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
[1]. http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/ c o r e s ight-etm-perf.c
On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > > I have 8008010000.etf in /sys/bus/coresight/devices > > Reza > > # ls /sys/bus/coresight/devices > 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel > 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm # > > -----Original Message----- > From: Etemadi, Mohammad > Sent: Thursday, April 20, 2017 10:50 AM > To: 'Mathieu Poirier' mathieu.poirier@linaro.org > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: RE: Perf-opencsd-4.9 > > Thanks Mathieu > > Now, I am hitting another failure. It seems mmap has failed. > Does it get the physical address from the name? > > # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname > failed to mmap with 12 (Cannot allocate memory) > > Regards, Reza > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, April 20, 2017 10:03 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> Thanks Mathieu and Mike >> >> I rebuild the Linux image with name changes in the device tree and still see the same error. >> >> Regards, Reza >> >> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname >> event syntax error: 'cs_etm/8008010000.cluster0etf/' >> ___ BPF support is not compiled >> > > This is not the proper syntax. > >> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname >> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >> ___ BPF support is not compiled > > This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem. > > I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. > On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works. > >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Wednesday, April 19, 2017 10:58 AM >> To: Mike Leach mike.leach@linaro.org >> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >>> Is it also possible that a leading @ is required? >>> i.e. >>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname >> >> Definitely yes - that would be the first thing to try. >> >>> >>> Mike >>> >>> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>> >>>>> Hello Mathieu >>>>> >>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>>> When I am running it on the target. Am I missing a package? >>>>> >>>>> When building "perf": >>>>> .. bpf: [ on ] >>>>> >>>>> When running "perf" on the target. >>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>> ___ BPF support is not compiled >>>> >>>> The perf command line parser is insanely complex. Here I >>>> suppose the extra "-etf" is causing the bison parser to hit a >>>> BPF rule, hence the error message. If you drop the "-etf" in >>>> the DT I'm pretty sure the message will go away. You can >>>> also enhance the flex/bison parser but to me the "address.name" convention is enough. >>>> >>>>> >>>>> >>>>> Regards, Reza >>>>> >>>>> -----Original Message----- >>>>> From: Etemadi, Mohammad >>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: RE: Perf-opencsd-4.9 >>>>> >>>>> >>>>> Thanks Mathieu >>>>> >>>>> Regards, Reza >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>> Cc: coresight@lists.linaro.org >>>>> Subject: Re: Perf-opencsd-4.9 >>>>> >>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> Thanks Mathieu >>>>>> >>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>> >>>>>> How does it map ETMs to CPUs? >>>>> >>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>> >>>>> Instrumenting the code to understand what is going on is always a good idea. >>>>> >>>>> Mathieu >>>>> >>>>> [1]. >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>> g >>>>> i >>>>> t >>>>> / >>>>> t >>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11 >>>>> - >>>>> r >>>>> c >>>>> 7 >>>>> # >>>>> n >>>>> 1 >>>>> 1 >>>>> 14 >>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> entering ETM4_probe >>>>>> >>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>> >>>>>> Leaving ETMv4 Probe >>>>>> >>>>>> entering ETMv4_probe >>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>> ------------[ cut here ]------------ >>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Hello Mathieu >>>>>>> >>>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>>> Still I do not see any device listed in the following >>>>>>> directory >>>>>>> >>>>>>> /sys/bus/coresight/devices >>>>>>> >>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>>> >>>>>> When the system boots AMBA probes for devices connected >>>>>> (and >>>>>> discoverable) on the bus. The first thing to do is make >>>>>> sure the CS devices show up at enumeration time. I suggest >>>>>> instrumenting function >>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>> >>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L34 >>>>>> 4 >>>>>> >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>>> >>>>>>> Ok, let me know. >>>>>>> >>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>> >>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> I am using master branch >>>>>>>>> >>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>>> git checkout -b master origin/master >>>>>>>>> >>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>> >>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Hello Mathieu >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>> >>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> CC util/parse-events.o >>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>> CC util/pmu.o >>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> >>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>> >>>>>>>>> [1]. >>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a34 >>>>>>>>> 4 >>>>>>>>> 5 >>>>>>>>> b >>>>>>>>> 8 >>>>>>>>> 7 >>>>>>>>> c >>>>>>>>> e >>>>>>>>> a >>>>>>>>> a4e >>>>>>>>> f >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 15c5d25c4b3 >>>>>>>>> >>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>> function it appears in >>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> >>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>> >>>>>>>>> [2]. >>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc >>>>>>>>> 3 >>>>>>>>> d >>>>>>>>> 0 >>>>>>>>> a >>>>>>>>> f >>>>>>>>> d >>>>>>>>> 2 >>>>>>>>> b >>>>>>>>> b24 >>>>>>>>> 2 >>>>>>>>> b >>>>>>>>> d >>>>>>>>> 9f7328e7104 >>>>>>>>> >>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>> No such file or directory >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> Do you have links to these patches? >>>>>>>>>> >>>>>>>>>> It's all there: >>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>> 9 >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>> >>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>>> >>>>>>>>>>> Ok. >>>>>>>>>>> >>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>>> >>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Hello Matt >>>>>>>>>>>> >>>>>>>>>>>> Hi there, >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>> >>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>> >>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>>> >>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>> >>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>>> >>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>> >>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>> >>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Mathieu >>>>>>>>>>>> >>>>>>>>>>>> [1]. >>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dt >>>>>>>>>>>> s >>>>>>>>>>>> / >>>>>>>>>>>> v >>>>>>>>>>>> e >>>>>>>>>>>> x >>>>>>>>>>>> p >>>>>>>>>>>> r >>>>>>>>>>>> e >>>>>>>>>>>> ss- >>>>>>>>>>>> v >>>>>>>>>>>> 2 >>>>>>>>>>>> p >>>>>>>>>>>> - >>>>>>>>>>>> c >>>>>>>>>>>> a >>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/ >>>>>>>>>>>> d >>>>>>>>>>>> t >>>>>>>>>>>> s >>>>>>>>>>>> / >>>>>>>>>>>> a >>>>>>>>>>>> r >>>>>>>>>>>> m >>>>>>>>>>>> / >>>>>>>>>>>> jun >>>>>>>>>>>> o >>>>>>>>>>>> - >>>>>>>>>>>> b >>>>>>>>>>>> a >>>>>>>>>>>> s >>>>>>>>>>>> e >>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torva >>>>>>>>>>>> l >>>>>>>>>>>> d >>>>>>>>>>>> s >>>>>>>>>>>> / >>>>>>>>>>>> l >>>>>>>>>>>> i >>>>>>>>>>>> n >>>>>>>>>>>> u >>>>>>>>>>>> x.g >>>>>>>>>>>> i >>>>>>>>>>>> t >>>>>>>>>>>> / >>>>>>>>>>>> t >>>>>>>>>>>> r >>>>>>>>>>>> e >>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>> 1 >>>>>>>>>>>> 1 >>>>>>>>>>>> - >>>>>>>>>>>> r >>>>>>>>>>>> c >>>>>>>>>>>> 4 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>> _______________________________________________ >>>> CoreSight mailing list >>>> CoreSight@lists.linaro.org >>>> https://lists.linaro.org/mailman/listinfo/coresight >>> >>> >>> >>> -- >>> Mike Leach >>> Principal Engineer, ARM Ltd. >>> Blackburn Design Centre. UK
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab54 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). > I have only defined 4 ETMs (only first cluster). > After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. > He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
> > Regards, Reza > > ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname > cs_etm_get_ro: error reading: cpu4/mgmt/etmccer > cs_etm_get_ro: error reading: cpu4/mgmt/etmidr > cs_etm_get_ro: error reading: cpu5/mgmt/etmccer > cs_etm_get_ro: error reading: cpu5/mgmt/etmidr > cs_etm_get_ro: error reading: cpu6/mgmt/etmccer > cs_etm_get_ro: error reading: cpu6/mgmt/etmidr > cs_etm_get_ro: error reading: cpu7/mgmt/etmccer > cs_etm_get_ro: error reading: cpu7/mgmt/etmidr > cs_etm_get_ro: error reading: cpu8/mgmt/etmccer > cs_etm_get_ro: error reading: cpu8/mgmt/etmidr > cs_etm_get_ro: error reading: cpu9/mgmt/etmccer > cs_etm_get_ro: error reading: cpu9/mgmt/etmidr > cs_etm_get_ro: error reading: cpu10/mgmt/etmccer > cs_etm_get_ro: error reading: cpu10/mgmt/etmidr > cs_etm_get_ro: error reading: cpu11/mgmt/etmccer > cs_etm_get_ro: error reading: cpu11/mgmt/etmidr > cs_etm_get_ro: error reading: cpu12/mgmt/etmccer > cs_etm_get_ro: error reading: cpu12/mgmt/etmidr > cs_etm_get_ro: error reading: cpu13/mgmt/etmccer > cs_etm_get_ro: error reading: cpu13/mgmt/etmidr > cs_etm_get_ro: error reading: cpu14/mgmt/etmccer > cs_etm_get_ro: error reading: cpu14/mgmt/etmidr > cs_etm_get_ro: error reading: cpu15/mgmt/etmccer > cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: > Woken up 7 times to write data ] > Warning: > AUX data lost 6 times out of 16! > > [ perf record: Captured and wrote 0.238 MB perf.data ] # > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, April 20, 2017 11:19 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy. > > [1]. > http://lxr.free-electrons.com/source/drivers/hwtracing/coresight/ > c > o > r > e > s > ight-etm-perf.c > > On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> >> I have 8008010000.etf in /sys/bus/coresight/devices >> >> Reza >> >> # ls /sys/bus/coresight/devices >> 8008010000.etf 80080f0000.mainfunnel 8008810000.funnel >> 8008c40000.etm 8008d40000.etm 8008e40000.etm 8008f40000.etm # >> >> -----Original Message----- >> From: Etemadi, Mohammad >> Sent: Thursday, April 20, 2017 10:50 AM >> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: RE: Perf-opencsd-4.9 >> >> Thanks Mathieu >> >> Now, I am hitting another failure. It seems mmap has failed. >> Does it get the physical address from the name? >> >> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> failed to mmap with 12 (Cannot allocate memory) >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 10:03 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu and Mike >>> >>> I rebuild the Linux image with name changes in the device tree and still see the same error. >>> >>> Regards, Reza >>> >>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>> ___ BPF support is not compiled >>> >> >> This is not the proper syntax. >> >>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>> ___ BPF support is not compiled >> >> This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem. >> >> I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. >> On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works. >> >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Wednesday, April 19, 2017 10:58 AM >>> To: Mike Leach mike.leach@linaro.org >>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >>>> Is it also possible that a leading @ is required? >>>> i.e. >>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname >>> >>> Definitely yes - that would be the first thing to try. >>> >>>> >>>> Mike >>>> >>>> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> >>>>>> Hello Mathieu >>>>>> >>>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>>>> When I am running it on the target. Am I missing a package? >>>>>> >>>>>> When building "perf": >>>>>> .. bpf: [ on ] >>>>>> >>>>>> When running "perf" on the target. >>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>> ___ BPF support is not compiled >>>>> >>>>> The perf command line parser is insanely complex. Here I >>>>> suppose the extra "-etf" is causing the bison parser to hit a >>>>> BPF rule, hence the error message. If you drop the "-etf" in >>>>> the DT I'm pretty sure the message will go away. You can >>>>> also enhance the flex/bison parser but to me the "address.name" convention is enough. >>>>> >>>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Etemadi, Mohammad >>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>> >>>>>> >>>>>> Thanks Mathieu >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>> >>>>>>> How does it map ETMs to CPUs? >>>>>> >>>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>>> >>>>>> Instrumenting the code to understand what is going on is always a good idea. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>> g >>>>>> i >>>>>> t >>>>>> / >>>>>> t >>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11 >>>>>> - >>>>>> r >>>>>> c >>>>>> 7 >>>>>> # >>>>>> n >>>>>> 1 >>>>>> 1 >>>>>> 14 >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> entering ETM4_probe >>>>>>> >>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>> >>>>>>> Leaving ETMv4 Probe >>>>>>> >>>>>>> entering ETMv4_probe >>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>> ------------[ cut here ]------------ >>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Mathieu >>>>>>>> >>>>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>>>> Still I do not see any device listed in the following >>>>>>>> directory >>>>>>>> >>>>>>>> /sys/bus/coresight/devices >>>>>>>> >>>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>>>> >>>>>>> When the system boots AMBA probes for devices connected >>>>>>> (and >>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>> sure the CS devices show up at enumeration time. I suggest >>>>>>> instrumenting function >>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>>> >>>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L34 >>>>>>> 4 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>>>> >>>>>>>> Ok, let me know. >>>>>>>> >>>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>>> >>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>>>> >>>>>>>>> Mathieu >>>>>>>>> >>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using master branch >>>>>>>>>> >>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>>>> git checkout -b master origin/master >>>>>>>>>> >>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>> >>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Hello Mathieu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>> >>>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>> CC util/pmu.o >>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>> >>>>>>>>>> [1]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a34 >>>>>>>>>> 4 >>>>>>>>>> 5 >>>>>>>>>> b >>>>>>>>>> 8 >>>>>>>>>> 7 >>>>>>>>>> c >>>>>>>>>> e >>>>>>>>>> a >>>>>>>>>> a4e >>>>>>>>>> f >>>>>>>>>> 0 >>>>>>>>>> 0 >>>>>>>>>> 15c5d25c4b3 >>>>>>>>>> >>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>> function it appears in >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>>> >>>>>>>>>> [2]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc >>>>>>>>>> 3 >>>>>>>>>> d >>>>>>>>>> 0 >>>>>>>>>> a >>>>>>>>>> f >>>>>>>>>> d >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> b24 >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> d >>>>>>>>>> 9f7328e7104 >>>>>>>>>> >>>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>> No such file or directory >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>> >>>>>>>>>>> It's all there: >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>> 9 >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>>>> >>>>>>>>>>>> Ok. >>>>>>>>>>>> >>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>>>> >>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>> >>>>>>>>>>>>> Hi there, >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>> >>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to support >>>>>>>>>>>>>> CoreSight and instruction trace using Perf tool. >>>>>>>>>>>>> >>>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>> >>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>>>> >>>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>>> >>>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> [1]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dt >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> v >>>>>>>>>>>>> e >>>>>>>>>>>>> x >>>>>>>>>>>>> p >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> ss- >>>>>>>>>>>>> v >>>>>>>>>>>>> 2 >>>>>>>>>>>>> p >>>>>>>>>>>>> - >>>>>>>>>>>>> c >>>>>>>>>>>>> a >>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/ >>>>>>>>>>>>> d >>>>>>>>>>>>> t >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> a >>>>>>>>>>>>> r >>>>>>>>>>>>> m >>>>>>>>>>>>> / >>>>>>>>>>>>> jun >>>>>>>>>>>>> o >>>>>>>>>>>>> - >>>>>>>>>>>>> b >>>>>>>>>>>>> a >>>>>>>>>>>>> s >>>>>>>>>>>>> e >>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torva >>>>>>>>>>>>> l >>>>>>>>>>>>> d >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> l >>>>>>>>>>>>> i >>>>>>>>>>>>> n >>>>>>>>>>>>> u >>>>>>>>>>>>> x.g >>>>>>>>>>>>> i >>>>>>>>>>>>> t >>>>>>>>>>>>> / >>>>>>>>>>>>> t >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>> 1 >>>>>>>>>>>>> 1 >>>>>>>>>>>>> - >>>>>>>>>>>>> r >>>>>>>>>>>>> c >>>>>>>>>>>>> 4 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>> _______________________________________________ >>>>> CoreSight mailing list >>>>> CoreSight@lists.linaro.org >>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>> >>>> >>>> >>>> -- >>>> Mike Leach >>>> Principal Engineer, ARM Ltd. >>>> Blackburn Design Centre. UK
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Thanks Mathieu > > The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). > I have only defined 4 ETMs (only first cluster). > After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. > He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
> > Regards, Reza > > ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname > cs_etm_get_ro: error reading: cpu4/mgmt/etmccer > cs_etm_get_ro: error reading: cpu4/mgmt/etmidr > cs_etm_get_ro: error reading: cpu5/mgmt/etmccer > cs_etm_get_ro: error reading: cpu5/mgmt/etmidr > cs_etm_get_ro: error reading: cpu6/mgmt/etmccer > cs_etm_get_ro: error reading: cpu6/mgmt/etmidr > cs_etm_get_ro: error reading: cpu7/mgmt/etmccer > cs_etm_get_ro: error reading: cpu7/mgmt/etmidr > cs_etm_get_ro: error reading: cpu8/mgmt/etmccer > cs_etm_get_ro: error reading: cpu8/mgmt/etmidr > cs_etm_get_ro: error reading: cpu9/mgmt/etmccer > cs_etm_get_ro: error reading: cpu9/mgmt/etmidr > cs_etm_get_ro: error reading: cpu10/mgmt/etmccer > cs_etm_get_ro: error reading: cpu10/mgmt/etmidr > cs_etm_get_ro: error reading: cpu11/mgmt/etmccer > cs_etm_get_ro: error reading: cpu11/mgmt/etmidr > cs_etm_get_ro: error reading: cpu12/mgmt/etmccer > cs_etm_get_ro: error reading: cpu12/mgmt/etmidr > cs_etm_get_ro: error reading: cpu13/mgmt/etmccer > cs_etm_get_ro: error reading: cpu13/mgmt/etmidr > cs_etm_get_ro: error reading: cpu14/mgmt/etmccer > cs_etm_get_ro: error reading: cpu14/mgmt/etmidr > cs_etm_get_ro: error reading: cpu15/mgmt/etmccer > cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: > Woken up 7 times to write data ] > Warning: > AUX data lost 6 times out of 16! > > [ perf record: Captured and wrote 0.238 MB perf.data ] # > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Thursday, April 20, 2017 11:19 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy. > > [1]. > http://lxr.free-electrons.com/source/drivers/hwtracing/coresight > / > c > o > r > e > s > ight-etm-perf.c > > On 20 April 2017 at 10:11, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >> >> I have 8008010000.etf in /sys/bus/coresight/devices >> >> Reza >> >> # ls /sys/bus/coresight/devices 8008010000.etf >> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm >> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >> >> -----Original Message----- >> From: Etemadi, Mohammad >> Sent: Thursday, April 20, 2017 10:50 AM >> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: RE: Perf-opencsd-4.9 >> >> Thanks Mathieu >> >> Now, I am hitting another failure. It seems mmap has failed. >> Does it get the physical address from the name? >> >> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> failed to mmap with 12 (Cannot allocate memory) >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 10:03 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 19 April 2017 at 12:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>> Thanks Mathieu and Mike >>> >>> I rebuild the Linux image with name changes in the device tree and still see the same error. >>> >>> Regards, Reza >>> >>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>> ___ BPF support is not compiled >>> >> >> This is not the proper syntax. >> >>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>> ___ BPF support is not compiled >> >> This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem. >> >> I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. >> On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works. >> >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Wednesday, April 19, 2017 10:58 AM >>> To: Mike Leach mike.leach@linaro.org >>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org wrote: >>>> Is it also possible that a leading @ is required? >>>> i.e. >>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname >>> >>> Definitely yes - that would be the first thing to try. >>> >>>> >>>> Mike >>>> >>>> On 19 April 2017 at 16:07, Mathieu Poirier mathieu.poirier@linaro.org wrote: >>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>> >>>>>> Hello Mathieu >>>>>> >>>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>>>> When I am running it on the target. Am I missing a package? >>>>>> >>>>>> When building "perf": >>>>>> .. bpf: [ on ] >>>>>> >>>>>> When running "perf" on the target. >>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>> ___ BPF support is not compiled >>>>> >>>>> The perf command line parser is insanely complex. Here I >>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>> in the DT I'm pretty sure the message will go away. You can >>>>> also enhance the flex/bison parser but to me the "address.name" convention is enough. >>>>> >>>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Etemadi, Mohammad >>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>> >>>>>> >>>>>> Thanks Mathieu >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>> Cc: coresight@lists.linaro.org >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>> >>>>>>> How does it map ETMs to CPUs? >>>>>> >>>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>>> >>>>>> Instrumenting the code to understand what is going on is always a good idea. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>> g >>>>>> i >>>>>> t >>>>>> / >>>>>> t >>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>> 1 >>>>>> - >>>>>> r >>>>>> c >>>>>> 7 >>>>>> # >>>>>> n >>>>>> 1 >>>>>> 1 >>>>>> 14 >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> entering ETM4_probe >>>>>>> >>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>> >>>>>>> Leaving ETMv4 Probe >>>>>>> >>>>>>> entering ETMv4_probe >>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>> ------------[ cut here ]------------ >>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>> Hello Mathieu >>>>>>>> >>>>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>>>> Still I do not see any device listed in the following >>>>>>>> directory >>>>>>>> >>>>>>>> /sys/bus/coresight/devices >>>>>>>> >>>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>>>> >>>>>>> When the system boots AMBA probes for devices connected >>>>>>> (and >>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>> sure the CS devices show up at enumeration time. I >>>>>>> suggest instrumenting function >>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>>> >>>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3 >>>>>>> 4 >>>>>>> 4 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>>>> >>>>>>>> Ok, let me know. >>>>>>>> >>>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>>> >>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>>>> >>>>>>>>> Mathieu >>>>>>>>> >>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using master branch >>>>>>>>>> >>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>>>> git checkout -b master origin/master >>>>>>>>>> >>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>> >>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Hello Mathieu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>> >>>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>> CC util/pmu.o >>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>> >>>>>>>>>> [1]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3 >>>>>>>>>> 4 >>>>>>>>>> 4 >>>>>>>>>> 5 >>>>>>>>>> b >>>>>>>>>> 8 >>>>>>>>>> 7 >>>>>>>>>> c >>>>>>>>>> e >>>>>>>>>> a >>>>>>>>>> a4e >>>>>>>>>> f >>>>>>>>>> 0 >>>>>>>>>> 0 >>>>>>>>>> 15c5d25c4b3 >>>>>>>>>> >>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>> function it appears in >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>>> >>>>>>>>>> [2]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac >>>>>>>>>> c >>>>>>>>>> 3 >>>>>>>>>> d >>>>>>>>>> 0 >>>>>>>>>> a >>>>>>>>>> f >>>>>>>>>> d >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> b24 >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> d >>>>>>>>>> 9f7328e7104 >>>>>>>>>> >>>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>> No such file or directory >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>> >>>>>>>>>>> It's all there: >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>> 9 >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>>>> >>>>>>>>>>>> Ok. >>>>>>>>>>>> >>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>>>> >>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>> >>>>>>>>>>>>> Hi there, >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>> >>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>> support CoreSight and instruction trace using Perf tool. >>>>>>>>>>>>> >>>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>> >>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>>>> >>>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>>> >>>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> [1]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d >>>>>>>>>>>>> t >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> v >>>>>>>>>>>>> e >>>>>>>>>>>>> x >>>>>>>>>>>>> p >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> ss- >>>>>>>>>>>>> v >>>>>>>>>>>>> 2 >>>>>>>>>>>>> p >>>>>>>>>>>>> - >>>>>>>>>>>>> c >>>>>>>>>>>>> a >>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot >>>>>>>>>>>>> / >>>>>>>>>>>>> d >>>>>>>>>>>>> t >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> a >>>>>>>>>>>>> r >>>>>>>>>>>>> m >>>>>>>>>>>>> / >>>>>>>>>>>>> jun >>>>>>>>>>>>> o >>>>>>>>>>>>> - >>>>>>>>>>>>> b >>>>>>>>>>>>> a >>>>>>>>>>>>> s >>>>>>>>>>>>> e >>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv >>>>>>>>>>>>> a >>>>>>>>>>>>> l >>>>>>>>>>>>> d >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> l >>>>>>>>>>>>> i >>>>>>>>>>>>> n >>>>>>>>>>>>> u >>>>>>>>>>>>> x.g >>>>>>>>>>>>> i >>>>>>>>>>>>> t >>>>>>>>>>>>> / >>>>>>>>>>>>> t >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>> 1 >>>>>>>>>>>>> 1 >>>>>>>>>>>>> - >>>>>>>>>>>>> r >>>>>>>>>>>>> c >>>>>>>>>>>>> 4 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>> _______________________________________________ >>>>> CoreSight mailing list >>>>> CoreSight@lists.linaro.org >>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>> >>>> >>>> >>>> -- >>>> Mike Leach >>>> Principal Engineer, ARM Ltd. >>>> Blackburn Design Centre. UK
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it
will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com
wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does
not work.
I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there
that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores
and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com
wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com
wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux
kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the
perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same
application (uname)
AUX data lost X times out of Y!Yes that is normal. It depends on how the application being tested
has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come
back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a
problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com
wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not
return at all and perf takes lots of
CPU cycles. It looks "perf" is trying to process perf.data but it
cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a
wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
> Hello Mathieu > > There is a warning "AUX data lost 6 times out of 16!" what does
this warning imply?
> > Regards, Reza > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, April 21, 2017 9:21 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 20 April 2017 at 13:02, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>> Thanks Mathieu >> >> The issue was related to the number of CPUs. We have 4 clusters
each with 4 cores (total 16 cores).
>> I have only defined 4 ETMs (only first cluster). >> After setting maxcpus=4 in bootargs, I do not see mmap failure
anymore and I am getting the following logs.
>> He still somehow thinks there are 16 cores... > > The error message comes from the perf user space tool looking for
all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
> > [1]. /sys/bus/event_source/devices/cs_etm/ > >> >> Regards, Reza >> >> ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> cs_etm_get_ro: error reading: cpu4/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu4/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu5/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu5/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu6/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu6/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu7/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu7/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu8/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu8/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu9/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu9/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu10/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu10/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu11/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu11/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu12/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu12/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu13/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu13/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu14/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu14/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu15/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf
record:
>> Woken up 7 times to write data ] >> Warning: >> AUX data lost 6 times out of 16! >> >> [ perf record: Captured and wrote 0.238 MB perf.data ] # >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 11:19 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> The sink in the perf command line matches what is in sysFS so
that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
>> >> [1]. >> http://lxr.free-electrons.com/source/drivers/hwtracing/coresight >> / >> c >> o >> r >> e >> s >> ight-etm-perf.c >> >> On 20 April 2017 at 10:11, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>> >>> I have 8008010000.etf in /sys/bus/coresight/devices >>> >>> Reza >>> >>> # ls /sys/bus/coresight/devices 8008010000.etf >>> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm >>> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Thursday, April 20, 2017 10:50 AM >>> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> Thanks Mathieu >>> >>> Now, I am hitting another failure. It seems mmap has failed. >>> Does it get the physical address from the name? >>> >>> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >>> failed to mmap with 12 (Cannot allocate memory) >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, April 20, 2017 10:03 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 12:42, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>> Thanks Mathieu and Mike >>>> >>>> I rebuild the Linux image with name changes in the device tree
and still see the same error.
>>>> >>>> Regards, Reza >>>> >>>> # ./perf record -e cs_etm/8008010000.cluster0etf/
--per-thread uname
>>>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>>> >>> >>> This is not the proper syntax. >>> >>>> # ./perf record -e cs_etm/@8008010000.cluster0etf/
--per-thread uname
>>>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>> >>> This is the proper syntax and there is nothing wrong with this
command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
>>> >>> I don't know if the BPF related error message is real - it could
be a false positive. But we definitely have a feature interaction problem.
>>> On the flip side "cluster0etf" won't be accepted upstream so at
one point you will need to change it to simply "etf", and that works.
>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Wednesday, April 19, 2017 10:58 AM >>>> To: Mike Leach mike.leach@linaro.org >>>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>>> coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org
wrote:
>>>>> Is it also possible that a leading @ is required? >>>>> i.e. >>>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/
--per-thread uname
>>>> >>>> Definitely yes - that would be the first thing to try. >>>> >>>>> >>>>> Mike >>>>> >>>>> On 19 April 2017 at 16:07, Mathieu Poirier <
mathieu.poirier@linaro.org> wrote:
>>>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>> >>>>>>> Hello Mathieu >>>>>>> >>>>>>> BPF exists when I am building "perf" but I am getting an
error indicating "BPF support is not compiled"
>>>>>>> When I am running it on the target. Am I missing a package? >>>>>>> >>>>>>> When building "perf": >>>>>>> .. bpf: [ on ] >>>>>>> >>>>>>> When running "perf" on the target. >>>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/
--per-thread uname
>>>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>>> ___ BPF support is not compiled >>>>>> >>>>>> The perf command line parser is insanely complex. Here I >>>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>>> in the DT I'm pretty sure the message will go away. You can >>>>>> also enhance the flex/bison parser but to me the "address.name"
convention is enough.
>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Etemadi, Mohammad >>>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>>> >>>>>>> >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>>> >>>>>>>> How does it map ETMs to CPUs? >>>>>>> >>>>>>> The mapping between tracer and CPU is done in the device tree
- as such there seems to be a problem with the DT specification. If the
code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
>>>>>>> >>>>>>> Instrumenting the code to understand what is going on is
always a good idea.
>>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
linux.
>>>>>>> g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>>> 1 >>>>>>> - >>>>>>> r >>>>>>> c >>>>>>> 7 >>>>>>> # >>>>>>> n >>>>>>> 1 >>>>>>> 1 >>>>>>> 14 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> entering ETM4_probe >>>>>>>> >>>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>>> >>>>>>>> Leaving ETMv4 Probe >>>>>>>> >>>>>>>> entering ETMv4_probe >>>>>>>> sysfs: cannot create duplicate filename
'/devices/cs_etm/cpu0'
>>>>>>>> ------------[ cut here ]------------ >>>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Cc: coresight@lists.linaro.org >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> We have changed the device tree to reflect all CoreSight
components on our platform.
>>>>>>>>> Still I do not see any device listed in the following >>>>>>>>> directory >>>>>>>>> >>>>>>>>> /sys/bus/coresight/devices >>>>>>>>> >>>>>>>>> Am I missing something in Linux 4.9 configuration? I have
enabled CONFIG_CORESIGHT in .config.
>>>>>>>> >>>>>>>> When the system boots AMBA probes for devices connected >>>>>>>> (and >>>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>>> sure the CS devices show up at enumeration time. I >>>>>>>> suggest instrumenting function >>>>>>>> amba_device_try_add() [1] to see if CS devices are
discovered. If not then it is probably a matter of enabling the debug power domain.
>>>>>>>> While debugging I also suggest to boot with the "nohlt"
option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
>>>>>>>> >>>>>>>> If CS devices show up then you will need to instrument the
_probe() function of each CS driver to see what makes them unhappy.
>>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3 >>>>>>>> 4 >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot
to both opencsd and tools/perf makefiles.
>>>>>>>>>> If I see more problems, I may have to switch building them
on the target.
>>>>>>>>> >>>>>>>>> Ok, let me know. >>>>>>>>> >>>>>>>>> I will be travelling for the next two weeks. As such I may
be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> For my own sanity I tried doing the same thing and
everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
>>>>>>>>>> >>>>>>>>>> For the perf tools I am working with branch
perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
>>>>>>>>>> >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> I am using master branch >>>>>>>>>>> >>>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git
my-opencsd
>>>>>>>>>>> git checkout -b master origin/master >>>>>>>>>>> >>>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>>> >>>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>> Hello Mathieu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have taken all the patches. When building tools/perf
we get the following compilation errors.
>>>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>>> >>>>>>>>>>> The problem comes from mismatches between the kernel and
the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>>> CC util/pmu.o >>>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>>> defined but not used [-Werror=unused-const-variable=]
static const char * const cs_etm_global_header_fmts[] = {
>>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Right, you are building branch 4.9 with a gcc version
that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
>>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3 >>>>>>>>>>> 4 >>>>>>>>>>> 4 >>>>>>>>>>> 5 >>>>>>>>>>> b >>>>>>>>>>> 8 >>>>>>>>>>> 7 >>>>>>>>>>> c >>>>>>>>>>> e >>>>>>>>>>> a >>>>>>>>>>> a4e >>>>>>>>>>> f >>>>>>>>>>> 0 >>>>>>>>>>> 0 >>>>>>>>>>> 15c5d25c4b3 >>>>>>>>>>> >>>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function
?cs_etm_decoder__gen_trace_elem_printer?:
>>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error:
?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function)
>>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>>> function it appears in >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error:
?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function)
>>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Those two are related to the openCSD version. What
version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
>>>>>>>>>>> >>>>>>>>>>> [2]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac >>>>>>>>>>> c >>>>>>>>>>> 3 >>>>>>>>>>> d >>>>>>>>>>> 0 >>>>>>>>>>> a >>>>>>>>>>> f >>>>>>>>>>> d >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> b24 >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> d >>>>>>>>>>> 9f7328e7104 >>>>>>>>>>> >>>>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-
decoder.o.tmp':
>>>>>>>>>>>> No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>>> >>>>>>>>>>>> It's all there: >>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4
.
>>>>>>>>>>>> 9 >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is an in-house board and we had to do some BSP
changes to boot Linux 4.9 on our board.
>>>>>>>>>>>>> >>>>>>>>>>>>> Ok. >>>>>>>>>>>>> >>>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9
correctly with our source tree.
>>>>>>>>>>>>>> So, are you saying that I only need to replace
tool/perf directory?
>>>>>>>>>>>>> >>>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9,
that will make life easier on you.
>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad <
mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>>> like to try instruction tracing using
perf-opencsd-4.9.
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree
for perf-opencsd-4.9.
>>>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>>> support CoreSight and instruction trace using Perf
tool.
>>>>>>>>>>>>>> >>>>>>>>>>>>>> All the kernel side of the solution is already
upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" -
can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also which files must be changed to reflect specific
SOC CoreSight topology?
>>>>>>>>>>>>>> >>>>>>>>>>>>>> CoreSight is different on every platform, which is why
all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house?
>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the former case support for vexpress[1], juno
(R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with.
>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm always interested by what people are doing with
CoreSight. Get back to me if you have more questions.
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> v >>>>>>>>>>>>>> e >>>>>>>>>>>>>> x >>>>>>>>>>>>>> p >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> ss- >>>>>>>>>>>>>> v >>>>>>>>>>>>>> 2 >>>>>>>>>>>>>> p >>>>>>>>>>>>>> - >>>>>>>>>>>>>> c >>>>>>>>>>>>>> a >>>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot >>>>>>>>>>>>>> / >>>>>>>>>>>>>> d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> a >>>>>>>>>>>>>> r >>>>>>>>>>>>>> m >>>>>>>>>>>>>> / >>>>>>>>>>>>>> jun >>>>>>>>>>>>>> o >>>>>>>>>>>>>> - >>>>>>>>>>>>>> b >>>>>>>>>>>>>> a >>>>>>>>>>>>>> s >>>>>>>>>>>>>> e >>>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv >>>>>>>>>>>>>> a >>>>>>>>>>>>>> l >>>>>>>>>>>>>> d >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> l >>>>>>>>>>>>>> i >>>>>>>>>>>>>> n >>>>>>>>>>>>>> u >>>>>>>>>>>>>> x.g >>>>>>>>>>>>>> i >>>>>>>>>>>>>> t >>>>>>>>>>>>>> / >>>>>>>>>>>>>> t >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/
msm8916.dtsi?id=refs/tags/v4.
>>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> - >>>>>>>>>>>>>> r >>>>>>>>>>>>>> c >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, Reza >>>>>> _______________________________________________ >>>>>> CoreSight mailing list >>>>>> CoreSight@lists.linaro.org >>>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Leach >>>>> Principal Engineer, ARM Ltd. >>>>> Blackburn Design Centre. UK
Thanks Mathieu
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like “time: option in the perf script. Have you tried cycle accurate mode and time stamps?
Regards, Reza
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, June 12, 2017 9:36 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000tel:8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000tel:8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier <mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote:
Hello Mathieu
There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] Sent: Friday, April 21, 2017 9:21 AM To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; coresight@lists.linaro.orgmailto:coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 20 April 2017 at 13:02, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: > Thanks Mathieu > > The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores). > I have only defined 4 ETMs (only first cluster). > After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs. > He still somehow thinks there are 16 cores...
The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
[1]. /sys/bus/event_source/devices/cs_etm/
> > Regards, Reza > > ./perf record -e cs_etm/@8008010000tel:8008010000.etf/ --per-thread uname > cs_etm_get_ro: error reading: cpu4/mgmt/etmccer > cs_etm_get_ro: error reading: cpu4/mgmt/etmidr > cs_etm_get_ro: error reading: cpu5/mgmt/etmccer > cs_etm_get_ro: error reading: cpu5/mgmt/etmidr > cs_etm_get_ro: error reading: cpu6/mgmt/etmccer > cs_etm_get_ro: error reading: cpu6/mgmt/etmidr > cs_etm_get_ro: error reading: cpu7/mgmt/etmccer > cs_etm_get_ro: error reading: cpu7/mgmt/etmidr > cs_etm_get_ro: error reading: cpu8/mgmt/etmccer > cs_etm_get_ro: error reading: cpu8/mgmt/etmidr > cs_etm_get_ro: error reading: cpu9/mgmt/etmccer > cs_etm_get_ro: error reading: cpu9/mgmt/etmidr > cs_etm_get_ro: error reading: cpu10/mgmt/etmccer > cs_etm_get_ro: error reading: cpu10/mgmt/etmidr > cs_etm_get_ro: error reading: cpu11/mgmt/etmccer > cs_etm_get_ro: error reading: cpu11/mgmt/etmidr > cs_etm_get_ro: error reading: cpu12/mgmt/etmccer > cs_etm_get_ro: error reading: cpu12/mgmt/etmidr > cs_etm_get_ro: error reading: cpu13/mgmt/etmccer > cs_etm_get_ro: error reading: cpu13/mgmt/etmidr > cs_etm_get_ro: error reading: cpu14/mgmt/etmccer > cs_etm_get_ro: error reading: cpu14/mgmt/etmidr > cs_etm_get_ro: error reading: cpu15/mgmt/etmccer > cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: > Woken up 7 times to write data ] > Warning: > AUX data lost 6 times out of 16! > > [ perf record: Captured and wrote 0.238 MB perf.data ] # > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] > Sent: Thursday, April 20, 2017 11:19 AM > To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> > Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; > coresight@lists.linaro.orgmailto:coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy. > > [1]. > http://lxr.free-electrons.com/source/drivers/hwtracing/coresight > / > c > o > r > e > s > ight-etm-perf.c > > On 20 April 2017 at 10:11, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >> >> I have 8008010000tel:8008010000.etf in /sys/bus/coresight/devices >> >> Reza >> >> # ls /sys/bus/coresight/devices 8008010000tel:8008010000.etf >> 80080f0000.mainfunnel 8008810000tel:8008810000.funnel 8008c40000.etm >> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >> >> -----Original Message----- >> From: Etemadi, Mohammad >> Sent: Thursday, April 20, 2017 10:50 AM >> To: 'Mathieu Poirier' <mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; >> coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >> Subject: RE: Perf-opencsd-4.9 >> >> Thanks Mathieu >> >> Now, I am hitting another failure. It seems mmap has failed. >> Does it get the physical address from the name? >> >> # ./perf record -e cs_etm/@8008010000tel:8008010000.etf/ --per-thread uname >> failed to mmap with 12 (Cannot allocate memory) >> >> Regards, Reza >> >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 10:03 AM >> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >> Cc: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org>; >> coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> On 19 April 2017 at 12:42, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>> Thanks Mathieu and Mike >>> >>> I rebuild the Linux image with name changes in the device tree and still see the same error. >>> >>> Regards, Reza >>> >>> # ./perf record -e cs_etm/8008010000tel:8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/8008010000tel:8008010000.cluster0etf/' >>> ___ BPF support is not compiled >>> >> >> This is not the proper syntax. >> >>> # ./perf record -e cs_etm/@8008010000tel:8008010000.cluster0etf/ --per-thread uname >>> event syntax error: 'cs_etm/@8008010000tel:8008010000.cluster0etf/' >>> ___ BPF support is not compiled >> >> This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem. >> >> I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem. >> On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works. >> >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>> Sent: Wednesday, April 19, 2017 10:58 AM >>> To: Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org> >>> Cc: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com>; >>> coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 09:43, Mike Leach <mike.leach@linaro.orgmailto:mike.leach@linaro.org> wrote: >>>> Is it also possible that a leading @ is required? >>>> i.e. >>>> ./perf record -e cs_etm/@8008010000tel:8008010000.cluster0-etf/ --per-thread uname >>> >>> Definitely yes - that would be the first thing to try. >>> >>>> >>>> Mike >>>> >>>> On 19 April 2017 at 16:07, Mathieu Poirier <mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org> wrote: >>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>> >>>>>> Hello Mathieu >>>>>> >>>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled" >>>>>> When I am running it on the target. Am I missing a package? >>>>>> >>>>>> When building "perf": >>>>>> .. bpf: [ on ] >>>>>> >>>>>> When running "perf" on the target. >>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname >>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>> ___ BPF support is not compiled >>>>> >>>>> The perf command line parser is insanely complex. Here I >>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>> in the DT I'm pretty sure the message will go away. You can >>>>> also enhance the flex/bison parser but to me the "address.namehttp://address.name" convention is enough. >>>>> >>>>>> >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> -----Original Message----- >>>>>> From: Etemadi, Mohammad >>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>> To: Mathieu Poirier <mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org> >>>>>> Cc: coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>> >>>>>> >>>>>> Thanks Mathieu >>>>>> >>>>>> Regards, Reza >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>> Cc: coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>> >>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>> >>>>>>> How does it map ETMs to CPUs? >>>>>> >>>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>>> >>>>>> Instrumenting the code to understand what is going on is always a good idea. >>>>>> >>>>>> Mathieu >>>>>> >>>>>> [1]. >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>> g >>>>>> i >>>>>> t >>>>>> / >>>>>> t >>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>> 1 >>>>>> - >>>>>> r >>>>>> c >>>>>> 7 >>>>>> # >>>>>> n >>>>>> 1 >>>>>> 1 >>>>>> 14 >>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> entering ETM4_probe >>>>>>> >>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>> >>>>>>> Leaving ETMv4 Probe >>>>>>> >>>>>>> entering ETMv4_probe >>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>> ------------[ cut here ]------------ >>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>> Cc: coresight@lists.linaro.orgmailto:coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>> Hello Mathieu >>>>>>>> >>>>>>>> We have changed the device tree to reflect all CoreSight components on our platform. >>>>>>>> Still I do not see any device listed in the following >>>>>>>> directory >>>>>>>> >>>>>>>> /sys/bus/coresight/devices >>>>>>>> >>>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config. >>>>>>> >>>>>>> When the system boots AMBA probes for devices connected >>>>>>> (and >>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>> sure the CS devices show up at enumeration time. I >>>>>>> suggest instrumenting function >>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain. >>>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>>> >>>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3 >>>>>>> 4 >>>>>>> 4 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>> Thanks Mathieu >>>>>>>>> >>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles. >>>>>>>>> If I see more problems, I may have to switch building them on the target. >>>>>>>> >>>>>>>> Ok, let me know. >>>>>>>> >>>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.orgmailto:coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>>> >>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb). >>>>>>>>> >>>>>>>>> Mathieu >>>>>>>>> >>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using master branch >>>>>>>>>> >>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd >>>>>>>>>> git checkout -b master origin/master >>>>>>>>>> >>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>> >>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>>>> Hello Mathieu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors. >>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>> >>>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>> CC util/pmu.o >>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>> >>>>>>>>>> [1]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3 >>>>>>>>>> 4 >>>>>>>>>> 4 >>>>>>>>>> 5 >>>>>>>>>> b >>>>>>>>>> 8 >>>>>>>>>> 7 >>>>>>>>>> c >>>>>>>>>> e >>>>>>>>>> a >>>>>>>>>> a4e >>>>>>>>>> f >>>>>>>>>> 0 >>>>>>>>>> 0 >>>>>>>>>> 15c5d25c4b3 >>>>>>>>>> >>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>> function it appears in >>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> >>>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>>> >>>>>>>>>> [2]. >>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac >>>>>>>>>> c >>>>>>>>>> 3 >>>>>>>>>> d >>>>>>>>>> 0 >>>>>>>>>> a >>>>>>>>>> f >>>>>>>>>> d >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> b24 >>>>>>>>>> 2 >>>>>>>>>> b >>>>>>>>>> d >>>>>>>>>> 9f7328e7104 >>>>>>>>>> >>>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>> No such file or directory >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>> >>>>>>>>>>> It's all there: >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>> 9 >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board. >>>>>>>>>>>> >>>>>>>>>>>> Ok. >>>>>>>>>>>> >>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory? >>>>>>>>>>>> >>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.orgmailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad <mohammad.etemadi@intel.commailto:mohammad.etemadi@intel.com> wrote: >>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>> >>>>>>>>>>>>> Hi there, >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9. >>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>> >>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>> support CoreSight and instruction trace using Perf tool. >>>>>>>>>>>>> >>>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>> >>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology? >>>>>>>>>>>>> >>>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>>> >>>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Mathieu >>>>>>>>>>>>> >>>>>>>>>>>>> [1]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d >>>>>>>>>>>>> t >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> v >>>>>>>>>>>>> e >>>>>>>>>>>>> x >>>>>>>>>>>>> p >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> ss- >>>>>>>>>>>>> v >>>>>>>>>>>>> 2 >>>>>>>>>>>>> p >>>>>>>>>>>>> - >>>>>>>>>>>>> c >>>>>>>>>>>>> a >>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot >>>>>>>>>>>>> / >>>>>>>>>>>>> d >>>>>>>>>>>>> t >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> a >>>>>>>>>>>>> r >>>>>>>>>>>>> m >>>>>>>>>>>>> / >>>>>>>>>>>>> jun >>>>>>>>>>>>> o >>>>>>>>>>>>> - >>>>>>>>>>>>> b >>>>>>>>>>>>> a >>>>>>>>>>>>> s >>>>>>>>>>>>> e >>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv >>>>>>>>>>>>> a >>>>>>>>>>>>> l >>>>>>>>>>>>> d >>>>>>>>>>>>> s >>>>>>>>>>>>> / >>>>>>>>>>>>> l >>>>>>>>>>>>> i >>>>>>>>>>>>> n >>>>>>>>>>>>> u >>>>>>>>>>>>> x.g >>>>>>>>>>>>> i >>>>>>>>>>>>> t >>>>>>>>>>>>> / >>>>>>>>>>>>> t >>>>>>>>>>>>> r >>>>>>>>>>>>> e >>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>> 1 >>>>>>>>>>>>> 1 >>>>>>>>>>>>> - >>>>>>>>>>>>> r >>>>>>>>>>>>> c >>>>>>>>>>>>> 4 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>> _______________________________________________ >>>>> CoreSight mailing list >>>>> CoreSight@lists.linaro.orgmailto:CoreSight@lists.linaro.org >>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>> >>>> >>>> >>>> -- >>>> Mike Leach >>>> Principal Engineer, ARM Ltd. >>>> Blackburn Design Centre. UK
Hi,
A quick answer to your timestamp / cycle count question....
using the command line...
/home/mleach/perf-tools/perf record -e cs_etm/timestamp,cycacc,@20010000.etf/ --per-thread uname -a
I see the following for a perf report --stdio --dump
0: id[12] I_ASYNC : Alignment Synchronisation 12: id[12] I_TRACE_INFO : Trace Info.; INFO=0x1; CC_THRESHOLD=0x100 19: id[12] I_TRACE_ON : Trace On. 20: id[12] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000088FD8FC; Ctxt: AArch64,EL1, NS; 30: id[12] I_ATOM_F1 : Atom format 1.; N 32: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x0 33: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d617e00 43: id[12] I_ATOM_F1 : Atom format 1.; E 44: id[12] I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008902D9C; 54: id[12] I_ATOM_F2 : Atom format 2.; EE 55: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0xFFFF000008903488 ~[0x3488] 58: id[12] I_ATOM_F1 : Atom format 1.; E 59: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088FDD6C; 65: id[12] I_ATOM_F3 : Atom format 3.; NEE 66: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x088FE19C ~[0x1E19C] 69: id[12] I_ATOM_F1 : Atom format 1.; E 70: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B231C; 75: id[12] I_ATOM_F3 : Atom format 3.; NEE 76: id[12] I_ATOM_F3 : Atom format 3.; NNE 77: id[12] I_ATOM_F3 : Atom format 3.; EEE 78: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x126 81: id[12] I_ATOM_F2 : Atom format 2.; NE 82: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB5F8 ~[0xB5F8] 85: id[12] I_ATOM_F1 : Atom format 1.; E 86: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B16C8 ~[0x116C8] 89: id[12] I_ATOM_F1 : Atom format 1.; E 90: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2398 ~[0x12398] 93: id[12] I_ATOM_F1 : Atom format 1.; E 94: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24C8 ~[0x124C8] 98: id[12] I_ATOM_F3 : Atom format 3.; NNE 99: id[12] I_ATOM_F1 : Atom format 1.; E 100: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B0A60 ~[0x10A60] 103: id[12] I_ATOM_F1 : Atom format 1.; E 104: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B25E8 ~[0x125E8] 107: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x141 109: id[12] I_ATOM_F2 : Atom format 2.; NE 110: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2A64 ~[0x12A64] 114: id[12] I_ATOM_F3 : Atom format 3.; ENN 115: id[12] I_ATOM_F5 : Atom format 5.; NEEEE 116: id[12] I_ATOM_F4 : Atom format 4.; NENE 117: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB600 ~[0xB600] 120: id[12] I_ATOM_F1 : Atom format 1.; E 121: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24B4 ~[0x124B4] 124: id[12] I_ATOM_F3 : Atom format 3.; EEN 125: id[12] I_ATOM_F3 : Atom format 3.; ENN 126: id[12] I_ATOM_F3 : Atom format 3.; EEE 128: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B22D4 ~[0x122D4] 131: id[12] I_ATOM_F6 : Atom format 6.; EEEEE 132: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2308 ~[0x108] 134: id[12] I_ATOM_F1 : Atom format 1.; E 135: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7A40 ~[0x17A40] 138: id[12] I_ATOM_F1 : Atom format 1.; E 139: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x1af 144: id[12] I_ATOM_F6 : Atom format 6.; EEEN 145: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x102 146: id[12] I_ATOM_F3 : Atom format 3.; NEE 147: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7AE4 ~[0xE4] 149: id[12] I_ATOM_F3 : Atom format 3.; EEE 150: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x103 151: id[12] I_ATOM_F2 : Atom format 2.; EE 152: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B3C2C ~[0x13C2C] 155: id[12] I_ATOM_F3 : Atom format 3.; NEE 156: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088D2260; 162: id[12] I_ATOM_F1 : Atom format 1.; E 163: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d61c57a; CC=0x51 168: id[12] I_ATOM_F1 : Atom format 1.; E 169: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081492C8; 174: id[12] I_ATOM_F2 : Atom format 2.; NE
So, timestamps and cycle counts are being captured - the problem is with the perf report / script code not yet having been updated to use these values. i.e cs-etm-decoder.c does nothing with these packets at present. The issue here is to make these values meaningful in a perf context.
The timestamp is generated from a free running counter in the CoreSight hardware system and not related to the linux system time (for juno at least).
The cycle counts are for blocks of instructions. Note the CC_THRESHOLD=0x100 at the start of the trace run - this means that cycle count packets will be emitted only when the count reaches or exceeds 0x100 cycles, and by design for ETMv4 trace, will only be emitted when an atom or other waypoint packet is output so that the count is associated with that instruction. Given that with waypoint trace many executed instructions can be implied between two waypoints (E/N atoms above) then you can see that the counts cannot be precisely attributed to individual instructions, only blocks of code. It is possible to adjust the cycle count threshold value to get better resolution - but I am not sure if we have that option in the perf record command line as yet.
Regards
Mike
On 12 June 2017 at 15:41, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like “time: option in the perf script.
Have you tried cycle accurate mode and time stamps?
Regards, Reza
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, June 12, 2017 9:36 AM
To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > There is a warning "AUX data lost 6 times out of 16!" what does this > warning imply? > > Regards, Reza > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, April 21, 2017 9:21 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 20 April 2017 at 13:02, Etemadi, Mohammad > mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> The issue was related to the number of CPUs. We have 4 clusters each >> with 4 cores (total 16 cores). >> I have only defined 4 ETMs (only first cluster). >> After setting maxcpus=4 in bootargs, I do not see mmap failure >> anymore and I am getting the following logs. >> He still somehow thinks there are 16 cores... > > The error message comes from the perf user space tool looking for all > the processors known to the system [1]. It is a little chatty but doesn't > have an impact on trace collection. > > [1]. /sys/bus/event_source/devices/cs_etm/ > >> >> Regards, Reza >> >> ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> cs_etm_get_ro: error reading: cpu4/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu4/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu5/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu5/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu6/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu6/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu7/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu7/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu8/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu8/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu9/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu9/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu10/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu10/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu11/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu11/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu12/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu12/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu13/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu13/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu14/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu14/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu15/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: >> Woken up 7 times to write data ] >> Warning: >> AUX data lost 6 times out of 16! >> >> [ perf record: Captured and wrote 0.238 MB perf.data ] # >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 11:19 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> The sink in the perf command line matches what is in sysFS so that's >> good. At this point you will need to instrument etm_setup_aux [1] to >> understand what makes it unhappy. >> >> [1]. >> http://lxr.free-electrons.com/source/drivers/hwtracing/coresight >> / >> c >> o >> r >> e >> s >> ight-etm-perf.c >> >> On 20 April 2017 at 10:11, Etemadi, Mohammad >> mohammad.etemadi@intel.com wrote: >>> >>> I have 8008010000.etf in /sys/bus/coresight/devices >>> >>> Reza >>> >>> # ls /sys/bus/coresight/devices 8008010000.etf >>> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm >>> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Thursday, April 20, 2017 10:50 AM >>> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> Thanks Mathieu >>> >>> Now, I am hitting another failure. It seems mmap has failed. >>> Does it get the physical address from the name? >>> >>> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >>> failed to mmap with 12 (Cannot allocate memory) >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, April 20, 2017 10:03 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 12:42, Etemadi, Mohammad >>> mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu and Mike >>>> >>>> I rebuild the Linux image with name changes in the device tree and >>>> still see the same error. >>>> >>>> Regards, Reza >>>> >>>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread >>>> uname >>>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>>> >>> >>> This is not the proper syntax. >>> >>>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ >>>> --per-thread uname >>>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>> >>> This is the proper syntax and there is nothing wrong with this >>> command line. As such I did a few test on my side and was able to reproduce >>> the problem. Anything that starts with a 'c' after the '.' will trigger >>> this problem. >>> >>> I don't know if the BPF related error message is real - it could be >>> a false positive. But we definitely have a feature interaction problem. >>> On the flip side "cluster0etf" won't be accepted upstream so at one >>> point you will need to change it to simply "etf", and that works. >>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Wednesday, April 19, 2017 10:58 AM >>>> To: Mike Leach mike.leach@linaro.org >>>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>>> coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org >>>> wrote: >>>>> Is it also possible that a leading @ is required? >>>>> i.e. >>>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ >>>>> --per-thread uname >>>> >>>> Definitely yes - that would be the first thing to try. >>>> >>>>> >>>>> Mike >>>>> >>>>> On 19 April 2017 at 16:07, Mathieu Poirier >>>>> mathieu.poirier@linaro.org wrote: >>>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad >>>>>> mohammad.etemadi@intel.com wrote: >>>>>>> >>>>>>> Hello Mathieu >>>>>>> >>>>>>> BPF exists when I am building "perf" but I am getting an error >>>>>>> indicating "BPF support is not compiled" >>>>>>> When I am running it on the target. Am I missing a package? >>>>>>> >>>>>>> When building "perf": >>>>>>> .. bpf: [ on ] >>>>>>> >>>>>>> When running "perf" on the target. >>>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ >>>>>>> --per-thread uname >>>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>>> ___ BPF support is not compiled >>>>>> >>>>>> The perf command line parser is insanely complex. Here I >>>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>>> in the DT I'm pretty sure the message will go away. You can >>>>>> also enhance the flex/bison parser but to me the "address.name" >>>>>> convention is enough. >>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Etemadi, Mohammad >>>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>>> >>>>>>> >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad >>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>>> >>>>>>>> How does it map ETMs to CPUs? >>>>>>> >>>>>>> The mapping between tracer and CPU is done in the device tree - >>>>>>> as such there seems to be a problem with the DT specification. If the code >>>>>>> can't find a CPU for a tracer it will default to '0', which seems to be >>>>>>> happening here. Look at the dragonboard example [1]. >>>>>>> >>>>>>> Instrumenting the code to understand what is going on is always >>>>>>> a good idea. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>>> g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>>> 1 >>>>>>> - >>>>>>> r >>>>>>> c >>>>>>> 7 >>>>>>> # >>>>>>> n >>>>>>> 1 >>>>>>> 1 >>>>>>> 14 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> entering ETM4_probe >>>>>>>> >>>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>>> >>>>>>>> Leaving ETMv4 Probe >>>>>>>> >>>>>>>> entering ETMv4_probe >>>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>>> ------------[ cut here ]------------ >>>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Cc: coresight@lists.linaro.org >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad >>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> We have changed the device tree to reflect all CoreSight >>>>>>>>> components on our platform. >>>>>>>>> Still I do not see any device listed in the following >>>>>>>>> directory >>>>>>>>> >>>>>>>>> /sys/bus/coresight/devices >>>>>>>>> >>>>>>>>> Am I missing something in Linux 4.9 configuration? I have >>>>>>>>> enabled CONFIG_CORESIGHT in .config. >>>>>>>> >>>>>>>> When the system boots AMBA probes for devices connected >>>>>>>> (and >>>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>>> sure the CS devices show up at enumeration time. I >>>>>>>> suggest instrumenting function >>>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. >>>>>>>> If not then it is probably a matter of enabling the debug power domain. >>>>>>>> While debugging I also suggest to boot with the "nohlt" option >>>>>>>> or disable CPUidle completely. That way tracers (usually located in the >>>>>>>> CPU/cluster power domain) are guaranteed to be enabled. >>>>>>>> >>>>>>>> If CS devices show up then you will need to instrument the >>>>>>>> _probe() function of each CS driver to see what makes them unhappy. >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3 >>>>>>>> 4 >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad >>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to >>>>>>>>>> both opencsd and tools/perf makefiles. >>>>>>>>>> If I see more problems, I may have to switch building them >>>>>>>>>> on the target. >>>>>>>>> >>>>>>>>> Ok, let me know. >>>>>>>>> >>>>>>>>> I will be travelling for the next two weeks. As such I may >>>>>>>>> be slow to respond, if at all. While I am away you can always send your >>>>>>>>> questions to "coresight@lists.linaro.org". There is a lot of very >>>>>>>>> knowledgeable people on there that can help you. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> For my own sanity I tried doing the same thing and >>>>>>>>>> everything works as advertised. The only difference is that I'm not cross >>>>>>>>>> compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and >>>>>>>>>> the cross compilation isn't part of the equation. >>>>>>>>>> >>>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 >>>>>>>>>> (b50067a52cf3). On the openCSD side I am using the master branch >>>>>>>>>> (054c07caa2eb). >>>>>>>>>> >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad >>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> I am using master branch >>>>>>>>>>> >>>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git >>>>>>>>>>> my-opencsd >>>>>>>>>>> git checkout -b master origin/master >>>>>>>>>>> >>>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>>> >>>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad >>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Mathieu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have taken all the patches. When building tools/perf we >>>>>>>>>>>> get the following compilation errors. >>>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>>> >>>>>>>>>>> The problem comes from mismatches between the kernel and >>>>>>>>>>> the openCSD version. We try to keep them working for older versions but >>>>>>>>>>> that can't be guaranteed all the time. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>>> CC util/pmu.o >>>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] >>>>>>>>>>>> static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Right, you are building branch 4.9 with a gcc version that >>>>>>>>>>> is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3 >>>>>>>>>>> 4 >>>>>>>>>>> 4 >>>>>>>>>>> 5 >>>>>>>>>>> b >>>>>>>>>>> 8 >>>>>>>>>>> 7 >>>>>>>>>>> c >>>>>>>>>>> e >>>>>>>>>>> a >>>>>>>>>>> a4e >>>>>>>>>>> f >>>>>>>>>>> 0 >>>>>>>>>>> 0 >>>>>>>>>>> 15c5d25c4b3 >>>>>>>>>>> >>>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function >>>>>>>>>>>> ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>>> function it appears in >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Those two are related to the openCSD version. What version >>>>>>>>>>> of the library are you working with? In any case if you stick with the >>>>>>>>>>> latest revision (0.5.4) or TIP you should be fine. The commits that added >>>>>>>>>>> these two is here [2]. >>>>>>>>>>> >>>>>>>>>>> [2]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac >>>>>>>>>>> c >>>>>>>>>>> 3 >>>>>>>>>>> d >>>>>>>>>>> 0 >>>>>>>>>>> a >>>>>>>>>>> f >>>>>>>>>>> d >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> b24 >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> d >>>>>>>>>>> 9f7328e7104 >>>>>>>>>>> >>>>>>>>>>>> mv: cannot stat >>>>>>>>>>>> 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>>> No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad >>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>>> >>>>>>>>>>>> It's all there: >>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>>> 9 >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad >>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is an in-house board and we had to do some BSP >>>>>>>>>>>>>> changes to boot Linux 4.9 on our board. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok. >>>>>>>>>>>>> >>>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 >>>>>>>>>>>>>> correctly with our source tree. >>>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf >>>>>>>>>>>>>> directory? >>>>>>>>>>>>> >>>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, >>>>>>>>>>>>> that will make life easier on you. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad >>>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for >>>>>>>>>>>>>>> perf-opencsd-4.9. >>>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>>> support CoreSight and instruction trace using Perf >>>>>>>>>>>>>>> tool. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All the kernel side of the solution is already upstream. >>>>>>>>>>>>>> What is on gitHub it part of the user space perf tools that haven't been >>>>>>>>>>>>>> upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - >>>>>>>>>>>>>> can you rephrase of give me more details? All the patches on top of >>>>>>>>>>>>>> mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also which files must be changed to reflect specific >>>>>>>>>>>>>>> SOC CoreSight topology? >>>>>>>>>>>>>> >>>>>>>>>>>>>> CoreSight is different on every platform, which is why >>>>>>>>>>>>>> all the platform specific stuff has been pushed to the device tree. Is this >>>>>>>>>>>>>> a commercial board or something in-house? >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the former case support for vexpress[1], juno >>>>>>>>>>>>>> (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also >>>>>>>>>>>>>> something for HiKey that I could share with you if need be. If this is an >>>>>>>>>>>>>> in-house project you will need to make up your own device tree base on the >>>>>>>>>>>>>> topology you are working with. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm always interested by what people are doing with >>>>>>>>>>>>>> CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> v >>>>>>>>>>>>>> e >>>>>>>>>>>>>> x >>>>>>>>>>>>>> p >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> ss- >>>>>>>>>>>>>> v >>>>>>>>>>>>>> 2 >>>>>>>>>>>>>> p >>>>>>>>>>>>>> - >>>>>>>>>>>>>> c >>>>>>>>>>>>>> a >>>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot >>>>>>>>>>>>>> / >>>>>>>>>>>>>> d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> a >>>>>>>>>>>>>> r >>>>>>>>>>>>>> m >>>>>>>>>>>>>> / >>>>>>>>>>>>>> jun >>>>>>>>>>>>>> o >>>>>>>>>>>>>> - >>>>>>>>>>>>>> b >>>>>>>>>>>>>> a >>>>>>>>>>>>>> s >>>>>>>>>>>>>> e >>>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv >>>>>>>>>>>>>> a >>>>>>>>>>>>>> l >>>>>>>>>>>>>> d >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> l >>>>>>>>>>>>>> i >>>>>>>>>>>>>> n >>>>>>>>>>>>>> u >>>>>>>>>>>>>> x.g >>>>>>>>>>>>>> i >>>>>>>>>>>>>> t >>>>>>>>>>>>>> / >>>>>>>>>>>>>> t >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> - >>>>>>>>>>>>>> r >>>>>>>>>>>>>> c >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, Reza >>>>>> _______________________________________________ >>>>>> CoreSight mailing list >>>>>> CoreSight@lists.linaro.org >>>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Leach >>>>> Principal Engineer, ARM Ltd. >>>>> Blackburn Design Centre. UK
Thanks Mike.
I also see I_CCNT packets in the raw trace. Do you know if the decoder is updated to capture these packets? We need to correlate events on different cores. Having a notion of common time cross all the cores will help us to correlate events.
Regards, Reza
-----Original Message----- From: Mike Leach [mailto:mike.leach@linaro.org] Sent: Tuesday, June 13, 2017 6:06 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mathieu Poirier mathieu.poirier@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
Hi,
A quick answer to your timestamp / cycle count question....
using the command line...
/home/mleach/perf-tools/perf record -e cs_etm/timestamp,cycacc,@20010000.etf/ --per-thread uname -a
I see the following for a perf report --stdio --dump
0: id[12] I_ASYNC : Alignment Synchronisation 12: id[12] I_TRACE_INFO : Trace Info.; INFO=0x1; CC_THRESHOLD=0x100 19: id[12] I_TRACE_ON : Trace On. 20: id[12] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000088FD8FC; Ctxt: AArch64,EL1, NS; 30: id[12] I_ATOM_F1 : Atom format 1.; N 32: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x0 33: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d617e00 43: id[12] I_ATOM_F1 : Atom format 1.; E 44: id[12] I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008902D9C; 54: id[12] I_ATOM_F2 : Atom format 2.; EE 55: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0xFFFF000008903488 ~[0x3488] 58: id[12] I_ATOM_F1 : Atom format 1.; E 59: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088FDD6C; 65: id[12] I_ATOM_F3 : Atom format 3.; NEE 66: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x088FE19C ~[0x1E19C] 69: id[12] I_ATOM_F1 : Atom format 1.; E 70: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B231C; 75: id[12] I_ATOM_F3 : Atom format 3.; NEE 76: id[12] I_ATOM_F3 : Atom format 3.; NNE 77: id[12] I_ATOM_F3 : Atom format 3.; EEE 78: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x126 81: id[12] I_ATOM_F2 : Atom format 2.; NE 82: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB5F8 ~[0xB5F8] 85: id[12] I_ATOM_F1 : Atom format 1.; E 86: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B16C8 ~[0x116C8] 89: id[12] I_ATOM_F1 : Atom format 1.; E 90: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2398 ~[0x12398] 93: id[12] I_ATOM_F1 : Atom format 1.; E 94: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24C8 ~[0x124C8] 98: id[12] I_ATOM_F3 : Atom format 3.; NNE 99: id[12] I_ATOM_F1 : Atom format 1.; E 100: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B0A60 ~[0x10A60] 103: id[12] I_ATOM_F1 : Atom format 1.; E 104: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B25E8 ~[0x125E8] 107: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x141 109: id[12] I_ATOM_F2 : Atom format 2.; NE 110: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2A64 ~[0x12A64] 114: id[12] I_ATOM_F3 : Atom format 3.; ENN 115: id[12] I_ATOM_F5 : Atom format 5.; NEEEE 116: id[12] I_ATOM_F4 : Atom format 4.; NENE 117: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB600 ~[0xB600] 120: id[12] I_ATOM_F1 : Atom format 1.; E 121: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24B4 ~[0x124B4] 124: id[12] I_ATOM_F3 : Atom format 3.; EEN 125: id[12] I_ATOM_F3 : Atom format 3.; ENN 126: id[12] I_ATOM_F3 : Atom format 3.; EEE 128: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B22D4 ~[0x122D4] 131: id[12] I_ATOM_F6 : Atom format 6.; EEEEE 132: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2308 ~[0x108] 134: id[12] I_ATOM_F1 : Atom format 1.; E 135: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7A40 ~[0x17A40] 138: id[12] I_ATOM_F1 : Atom format 1.; E 139: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x1af 144: id[12] I_ATOM_F6 : Atom format 6.; EEEN 145: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x102 146: id[12] I_ATOM_F3 : Atom format 3.; NEE 147: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7AE4 ~[0xE4] 149: id[12] I_ATOM_F3 : Atom format 3.; EEE 150: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x103 151: id[12] I_ATOM_F2 : Atom format 2.; EE 152: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B3C2C ~[0x13C2C] 155: id[12] I_ATOM_F3 : Atom format 3.; NEE 156: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088D2260; 162: id[12] I_ATOM_F1 : Atom format 1.; E 163: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d61c57a; CC=0x51 168: id[12] I_ATOM_F1 : Atom format 1.; E 169: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081492C8; 174: id[12] I_ATOM_F2 : Atom format 2.; NE
So, timestamps and cycle counts are being captured - the problem is with the perf report / script code not yet having been updated to use these values. i.e cs-etm-decoder.c does nothing with these packets at present. The issue here is to make these values meaningful in a perf context.
The timestamp is generated from a free running counter in the CoreSight hardware system and not related to the linux system time (for juno at least).
The cycle counts are for blocks of instructions. Note the CC_THRESHOLD=0x100 at the start of the trace run - this means that cycle count packets will be emitted only when the count reaches or exceeds 0x100 cycles, and by design for ETMv4 trace, will only be emitted when an atom or other waypoint packet is output so that the count is associated with that instruction. Given that with waypoint trace many executed instructions can be implied between two waypoints (E/N atoms above) then you can see that the counts cannot be precisely attributed to individual instructions, only blocks of code. It is possible to adjust the cycle count threshold value to get better resolution - but I am not sure if we have that option in the perf record command line as yet.
Regards
Mike
On 12 June 2017 at 15:41, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like “time: option in the perf script.
Have you tried cycle accurate mode and time stamps?
Regards, Reza
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, June 12, 2017 9:36 AM
To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab 5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > There is a warning "AUX data lost 6 times out of 16!" what does > this warning imply? > > Regards, Reza > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, April 21, 2017 9:21 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 20 April 2017 at 13:02, Etemadi, Mohammad > mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> The issue was related to the number of CPUs. We have 4 clusters >> each with 4 cores (total 16 cores). >> I have only defined 4 ETMs (only first cluster). >> After setting maxcpus=4 in bootargs, I do not see mmap failure >> anymore and I am getting the following logs. >> He still somehow thinks there are 16 cores... > > The error message comes from the perf user space tool looking > for all the processors known to the system [1]. It is a little > chatty but doesn't have an impact on trace collection. > > [1]. /sys/bus/event_source/devices/cs_etm/ > >> >> Regards, Reza >> >> ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> cs_etm_get_ro: error reading: cpu4/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu4/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu5/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu5/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu6/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu6/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu7/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu7/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu8/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu8/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu9/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu9/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu10/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu10/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu11/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu11/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu12/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu12/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu13/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu13/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu14/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu14/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu15/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: >> Woken up 7 times to write data ] >> Warning: >> AUX data lost 6 times out of 16! >> >> [ perf record: Captured and wrote 0.238 MB perf.data ] # >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 11:19 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> The sink in the perf command line matches what is in sysFS so >> that's good. At this point you will need to instrument >> etm_setup_aux [1] to understand what makes it unhappy. >> >> [1]. >> http://lxr.free-electrons.com/source/drivers/hwtracing/coresigh >> t >> / >> c >> o >> r >> e >> s >> ight-etm-perf.c >> >> On 20 April 2017 at 10:11, Etemadi, Mohammad >> mohammad.etemadi@intel.com wrote: >>> >>> I have 8008010000.etf in /sys/bus/coresight/devices >>> >>> Reza >>> >>> # ls /sys/bus/coresight/devices 8008010000.etf >>> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm >>> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Thursday, April 20, 2017 10:50 AM >>> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> Thanks Mathieu >>> >>> Now, I am hitting another failure. It seems mmap has failed. >>> Does it get the physical address from the name? >>> >>> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >>> failed to mmap with 12 (Cannot allocate memory) >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, April 20, 2017 10:03 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 12:42, Etemadi, Mohammad >>> mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu and Mike >>>> >>>> I rebuild the Linux image with name changes in the device >>>> tree and still see the same error. >>>> >>>> Regards, Reza >>>> >>>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread >>>> uname >>>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>>> >>> >>> This is not the proper syntax. >>> >>>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ >>>> --per-thread uname >>>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>> >>> This is the proper syntax and there is nothing wrong with this >>> command line. As such I did a few test on my side and was >>> able to reproduce the problem. Anything that starts with a >>> 'c' after the '.' will trigger this problem. >>> >>> I don't know if the BPF related error message is real - it >>> could be a false positive. But we definitely have a feature interaction problem. >>> On the flip side "cluster0etf" won't be accepted upstream so >>> at one point you will need to change it to simply "etf", and that works. >>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Wednesday, April 19, 2017 10:58 AM >>>> To: Mike Leach mike.leach@linaro.org >>>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>>> coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org >>>> wrote: >>>>> Is it also possible that a leading @ is required? >>>>> i.e. >>>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ >>>>> --per-thread uname >>>> >>>> Definitely yes - that would be the first thing to try. >>>> >>>>> >>>>> Mike >>>>> >>>>> On 19 April 2017 at 16:07, Mathieu Poirier >>>>> mathieu.poirier@linaro.org wrote: >>>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad >>>>>> mohammad.etemadi@intel.com wrote: >>>>>>> >>>>>>> Hello Mathieu >>>>>>> >>>>>>> BPF exists when I am building "perf" but I am getting an >>>>>>> error indicating "BPF support is not compiled" >>>>>>> When I am running it on the target. Am I missing a package? >>>>>>> >>>>>>> When building "perf": >>>>>>> .. bpf: [ on ] >>>>>>> >>>>>>> When running "perf" on the target. >>>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ >>>>>>> --per-thread uname >>>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>>> ___ BPF support is not compiled >>>>>> >>>>>> The perf command line parser is insanely complex. Here I >>>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>>> in the DT I'm pretty sure the message will go away. You >>>>>> can also enhance the flex/bison parser but to me the "address.name" >>>>>> convention is enough. >>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Etemadi, Mohammad >>>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>>> >>>>>>> >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad >>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>>> For some reason, it cannot create >>>>>>>> /sys/devices/cs_etm/cpu1 Rather than creating cpu1, it tries to recreate cpu0. >>>>>>>> >>>>>>>> How does it map ETMs to CPUs? >>>>>>> >>>>>>> The mapping between tracer and CPU is done in the device >>>>>>> tree - as such there seems to be a problem with the DT >>>>>>> specification. If the code can't find a CPU for a tracer >>>>>>> it will default to '0', which seems to be happening here. Look at the dragonboard example [1]. >>>>>>> >>>>>>> Instrumenting the code to understand what is going on is >>>>>>> always a good idea. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>>> g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>> 1 >>>>>>> 1 >>>>>>> - >>>>>>> r >>>>>>> c >>>>>>> 7 >>>>>>> # >>>>>>> n >>>>>>> 1 >>>>>>> 1 >>>>>>> 14 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> entering ETM4_probe >>>>>>>> >>>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>>> >>>>>>>> Leaving ETMv4 Probe >>>>>>>> >>>>>>>> entering ETMv4_probe >>>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>>> ------------[ cut here ]------------ >>>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Cc: coresight@lists.linaro.org >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad >>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> We have changed the device tree to reflect all CoreSight >>>>>>>>> components on our platform. >>>>>>>>> Still I do not see any device listed in the following >>>>>>>>> directory >>>>>>>>> >>>>>>>>> /sys/bus/coresight/devices >>>>>>>>> >>>>>>>>> Am I missing something in Linux 4.9 configuration? I >>>>>>>>> have enabled CONFIG_CORESIGHT in .config. >>>>>>>> >>>>>>>> When the system boots AMBA probes for devices connected >>>>>>>> (and >>>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>>> sure the CS devices show up at enumeration time. I >>>>>>>> suggest instrumenting function >>>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. >>>>>>>> If not then it is probably a matter of enabling the debug power domain. >>>>>>>> While debugging I also suggest to boot with the "nohlt" >>>>>>>> option or disable CPUidle completely. That way tracers >>>>>>>> (usually located in the CPU/cluster power domain) are guaranteed to be enabled. >>>>>>>> >>>>>>>> If CS devices show up then you will need to instrument >>>>>>>> the >>>>>>>> _probe() function of each CS driver to see what makes them unhappy. >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L >>>>>>>> 3 >>>>>>>> 4 >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier >>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad >>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using Yocto tool chain. So, I have to add >>>>>>>>>> --sysroot to both opencsd and tools/perf makefiles. >>>>>>>>>> If I see more problems, I may have to switch building >>>>>>>>>> them on the target. >>>>>>>>> >>>>>>>>> Ok, let me know. >>>>>>>>> >>>>>>>>> I will be travelling for the next two weeks. As such I >>>>>>>>> may be slow to respond, if at all. While I am away you >>>>>>>>> can always send your questions to >>>>>>>>> "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> For my own sanity I tried doing the same thing and >>>>>>>>>> everything works as advertised. The only difference is >>>>>>>>>> that I'm not cross compiling and I have gcc 5.4.0. We >>>>>>>>>> know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation. >>>>>>>>>> >>>>>>>>>> For the perf tools I am working with branch >>>>>>>>>> perf-openscd-4.9 (b50067a52cf3). On the openCSD side I >>>>>>>>>> am using the master branch (054c07caa2eb). >>>>>>>>>> >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad >>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> I am using master branch >>>>>>>>>>> >>>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git >>>>>>>>>>> my-opencsd >>>>>>>>>>> git checkout -b master origin/master >>>>>>>>>>> >>>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>>> >>>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad >>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Mathieu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have taken all the patches. When building >>>>>>>>>>>> tools/perf we get the following compilation errors. >>>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>>> >>>>>>>>>>> The problem comes from mismatches between the kernel >>>>>>>>>>> and the openCSD version. We try to keep them working >>>>>>>>>>> for older versions but that can't be guaranteed all the time. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>>> CC util/pmu.o >>>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] >>>>>>>>>>>> static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Right, you are building branch 4.9 with a gcc version >>>>>>>>>>> that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a >>>>>>>>>>> 3 >>>>>>>>>>> 4 >>>>>>>>>>> 4 >>>>>>>>>>> 5 >>>>>>>>>>> b >>>>>>>>>>> 8 >>>>>>>>>>> 7 >>>>>>>>>>> c >>>>>>>>>>> e >>>>>>>>>>> a >>>>>>>>>>> a4e >>>>>>>>>>> f >>>>>>>>>>> 0 >>>>>>>>>>> 0 >>>>>>>>>>> 15c5d25c4b3 >>>>>>>>>>> >>>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function >>>>>>>>>>>> ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: >>>>>>>>>>>> each undeclared identifier is reported only once for >>>>>>>>>>>> each function it appears in >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Those two are related to the openCSD version. What >>>>>>>>>>> version of the library are you working with? In any >>>>>>>>>>> case if you stick with the latest revision (0.5.4) or >>>>>>>>>>> TIP you should be fine. The commits that added these two is here [2]. >>>>>>>>>>> >>>>>>>>>>> [2]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04a >>>>>>>>>>> c >>>>>>>>>>> c >>>>>>>>>>> 3 >>>>>>>>>>> d >>>>>>>>>>> 0 >>>>>>>>>>> a >>>>>>>>>>> f >>>>>>>>>>> d >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> b24 >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> d >>>>>>>>>>> 9f7328e7104 >>>>>>>>>>> >>>>>>>>>>>> mv: cannot stat >>>>>>>>>>>> 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>>> No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad >>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>>> >>>>>>>>>>>> It's all there: >>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>>> 9 >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad >>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is an in-house board and we had to do some BSP >>>>>>>>>>>>>> changes to boot Linux 4.9 on our board. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok. >>>>>>>>>>>>> >>>>>>>>>>>>>> I just want to make sure that I merge >>>>>>>>>>>>>> perf-openCSD-4.9 correctly with our source tree. >>>>>>>>>>>>>> So, are you saying that I only need to replace >>>>>>>>>>>>>> tool/perf directory? >>>>>>>>>>>>> >>>>>>>>>>>>> Simply apply the patches you find on >>>>>>>>>>>>> perf-opencsd-4.9, that will make life easier on you. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad >>>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree >>>>>>>>>>>>>>> for perf-opencsd-4.9. >>>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>>> support CoreSight and instruction trace using Perf >>>>>>>>>>>>>>> tool. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All the kernel side of the solution is already upstream. >>>>>>>>>>>>>> What is on gitHub it part of the user space perf >>>>>>>>>>>>>> tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can >>>>>>>>>>>>>> add" - can you rephrase of give me more details? >>>>>>>>>>>>>> All the patches on top of mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also which files must be changed to reflect >>>>>>>>>>>>>>> specific SOC CoreSight topology? >>>>>>>>>>>>>> >>>>>>>>>>>>>> CoreSight is different on every platform, which is >>>>>>>>>>>>>> why all the platform specific stuff has been pushed >>>>>>>>>>>>>> to the device tree. Is this a commercial board or something in-house? >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the former case support for vexpress[1], juno >>>>>>>>>>>>>> (R0/1/2)[2] and the dragonboard[3] are already >>>>>>>>>>>>>> upstream. There is also something for HiKey that I >>>>>>>>>>>>>> could share with you if need be. If this is an >>>>>>>>>>>>>> in-house project you will need to make up your own device tree base on the topology you are working with. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm always interested by what people are doing with >>>>>>>>>>>>>> CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/ >>>>>>>>>>>>>> d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> v >>>>>>>>>>>>>> e >>>>>>>>>>>>>> x >>>>>>>>>>>>>> p >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> ss- >>>>>>>>>>>>>> v >>>>>>>>>>>>>> 2 >>>>>>>>>>>>>> p >>>>>>>>>>>>>> - >>>>>>>>>>>>>> c >>>>>>>>>>>>>> a >>>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boo >>>>>>>>>>>>>> t >>>>>>>>>>>>>> / >>>>>>>>>>>>>> d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> a >>>>>>>>>>>>>> r >>>>>>>>>>>>>> m >>>>>>>>>>>>>> / >>>>>>>>>>>>>> jun >>>>>>>>>>>>>> o >>>>>>>>>>>>>> - >>>>>>>>>>>>>> b >>>>>>>>>>>>>> a >>>>>>>>>>>>>> s >>>>>>>>>>>>>> e >>>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/tor >>>>>>>>>>>>>> v >>>>>>>>>>>>>> a >>>>>>>>>>>>>> l >>>>>>>>>>>>>> d >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> l >>>>>>>>>>>>>> i >>>>>>>>>>>>>> n >>>>>>>>>>>>>> u >>>>>>>>>>>>>> x.g >>>>>>>>>>>>>> i >>>>>>>>>>>>>> t >>>>>>>>>>>>>> / >>>>>>>>>>>>>> t >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> - >>>>>>>>>>>>>> r >>>>>>>>>>>>>> c >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, Reza >>>>>> _______________________________________________ >>>>>> CoreSight mailing list >>>>>> CoreSight@lists.linaro.org >>>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Leach >>>>> Principal Engineer, ARM Ltd. >>>>> Blackburn Design Centre. UK
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
-
Etemadi, Mohammad -
Mathieu Poirier -
Mike Leach