Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
Thanks, Sebastian
Hello Sebastian,
On 27 April 2017 at 13:45, Sebastian Pop sebpop@gmail.com wrote:
Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
The problem with HiKey is the APB clock... But I had patches for that not too long ago and I did get trace runs going. I'll look for them tomorrow.
Mathieu
Thanks, Sebastian
On Thu, Apr 27, 2017 at 4:36 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
Hello Sebastian,
On 27 April 2017 at 13:45, Sebastian Pop sebpop@gmail.com wrote:
Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
The problem with HiKey is the APB clock... But I had patches for that not too long ago and I did get trace runs going. I'll look for them tomorrow.
George just pointed out that there is this patch: https://patchwork.kernel.org/patch/9669003/ "Add coresight DT nodes for hikey board." I will try to build the android 4.9 kernel with this patch tomorrow.
Mathieu, are you speaking about another set of patches that are needed on top of this for the APB clock?
Thanks, Sebastian
Hi Sebastian,
On Thu, Apr 27, 2017 at 09:16:07PM -0500, Sebastian Pop wrote:
On Thu, Apr 27, 2017 at 4:36 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
Hello Sebastian,
On 27 April 2017 at 13:45, Sebastian Pop sebpop@gmail.com wrote:
Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
The problem with HiKey is the APB clock... But I had patches for that not too long ago and I did get trace runs going. I'll look for them tomorrow.
George just pointed out that there is this patch: https://patchwork.kernel.org/patch/9669003/ "Add coresight DT nodes for hikey board." I will try to build the android 4.9 kernel with this patch tomorrow.
Mathieu, are you speaking about another set of patches that are needed on top of this for the APB clock?
I think Mathieu refers below two patches: http://people.linaro.org/~leo.yan/opencsd_hikey/0001-coresight-DTS-add-cores... http://people.linaro.org/~leo.yan/opencsd_hikey/0002-sctrl-coresight-test-op...
Thanks, Leo Yan
On Fri, Apr 28, 2017 at 3:04 AM, Leo Yan leo.yan@linaro.org wrote:
I think Mathieu refers below two patches: http://people.linaro.org/~leo.yan/opencsd_hikey/0001-coresight-DTS-add-cores... http://people.linaro.org/~leo.yan/opencsd_hikey/0002-sctrl-coresight-test-op...
Thanks for the pointers!
On 27 April 2017 at 20:16, Sebastian Pop sebpop@gmail.com wrote:
On Thu, Apr 27, 2017 at 4:36 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
Hello Sebastian,
On 27 April 2017 at 13:45, Sebastian Pop sebpop@gmail.com wrote:
Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
The problem with HiKey is the APB clock... But I had patches for that not too long ago and I did get trace runs going. I'll look for them tomorrow.
George just pointed out that there is this patch: https://patchwork.kernel.org/patch/9669003/ "Add coresight DT nodes for hikey board." I will try to build the android 4.9 kernel with this patch tomorrow.
I vaguely remember reviewing that patch...
Mathieu, are you speaking about another set of patches that are needed on top of this for the APB clock?
I've attached the patches used to get traces going on HiKey on a 4.11-rc3 kernel - simply compile boot and run.
Thanks, Sebastian
Awesome, thank you all!
I applied both patches to my local kernel (tried both 4.9 from AOSP, and 4.11 from upstream Linux), and can't seem to get my device to fully boot. :/
Looking at `dmesg` from failed runs, it seems that etm0 is being initialized cleanly, but initializing any other etm {1..7} causes my kernel to panic. All of the panics take the form "Unhandled fault: synchronous external abort (0x96000210) at $addr," where $addr is `drvdata->base + TRCIDR0` (from etm4_init_arch_data in drivers/hwtracing/coresight/ coresight-etm4x.c).
I've messed around a bit, and it seems that writes to `drvdata->base + multiple_offsets` are perfectly OK, but the moment that you try to read from it, you get a panic. Like said, this only occurs on ETMs that aren't etm0. If I just throw a `if (drvdata->cpu) return;` in before we try to read from `*drvdata->base`, the device boots happily with a single ETM.
Do you all have any tips for tracking down what's causing this? Linux isn't exactly my specialty, so my troubleshooting attempts haven't really gotten me very far. :)
If it matters, my coresight-related configuration options are: CONFIG_CORESIGHT=y CONFIG_CORESIGHT_ETBV10=y CONFIG_CORESIGHT_SOURCE_ETM4X=y CONFIG_CORESIGHT_LINKS_AND_SINKS=y CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y CONFIG_CORESIGHT_QCOM_REPLICATOR=y CONFIG_CORESIGHT_SINK_TPIU=y CONFIG_CORESIGHT_STM=y
(I just ticked whatever boxes looked even mildly relevant. ;) Cutting down on what's enabled doesn't seem to help, though...)
Thanks again, George
On Fri, Apr 28, 2017 at 7:27 AM, Mathieu Poirier <mathieu.poirier@linaro.org
wrote:
On 27 April 2017 at 20:16, Sebastian Pop sebpop@gmail.com wrote:
On Thu, Apr 27, 2017 at 4:36 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
Hello Sebastian,
On 27 April 2017 at 13:45, Sebastian Pop sebpop@gmail.com wrote:
Hi,
we started looking at how to enable collection of branch traces with coresight etm on the hikey boards that are the reference platform for the android linux-4.9 work. Does somebody from Linaro have access to the description of where the coresight components are located for the hikey devices? We would appreciate help on enabling linux-perf collection of traces on the hikey.
The problem with HiKey is the APB clock... But I had patches for that not too long ago and I did get trace runs going. I'll look for them tomorrow.
George just pointed out that there is this patch: https://patchwork.kernel.org/patch/9669003/ "Add coresight DT nodes for hikey board." I will try to build the android 4.9 kernel with this patch tomorrow.
I vaguely remember reviewing that patch...
Mathieu, are you speaking about another set of patches that are needed on top of this for the APB clock?
I've attached the patches used to get traces going on HiKey on a 4.11-rc3 kernel - simply compile boot and run.
Thanks, Sebastian
On Mon, May 15, 2017 at 8:38 PM, George Burgess gbiv@google.com wrote:
Awesome, thank you all!
I applied both patches to my local kernel (tried both 4.9 from AOSP, and 4.11 from upstream Linux), and can't seem to get my device to fully boot. :/
Looking at `dmesg` from failed runs, it seems that etm0 is being initialized cleanly, but initializing any other etm {1..7} causes my kernel to panic. All of the panics take the form "Unhandled fault: synchronous external abort (0x96000210) at $addr," where $addr is `drvdata->base + TRCIDR0` (from etm4_init_arch_data in drivers/hwtracing/coresight/coresight-etm4x.c).
I've messed around a bit, and it seems that writes to `drvdata->base + multiple_offsets` are perfectly OK, but the moment that you try to read from it, you get a panic. Like said, this only occurs on ETMs that aren't etm0. If I just throw a `if (drvdata->cpu) return;` in before we try to read from `*drvdata->base`, the device boots happily with a single ETM.
Do you all have any tips for tracking down what's causing this? Linux isn't exactly my specialty, so my troubleshooting attempts haven't really gotten me very far. :)
If it matters, my coresight-related configuration options are: CONFIG_CORESIGHT=y CONFIG_CORESIGHT_ETBV10=y CONFIG_CORESIGHT_SOURCE_ETM4X=y CONFIG_CORESIGHT_LINKS_AND_SINKS=y CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y CONFIG_CORESIGHT_QCOM_REPLICATOR=y CONFIG_CORESIGHT_SINK_TPIU=y CONFIG_CORESIGHT_STM=y
(I just ticked whatever boxes looked even mildly relevant. ;) Cutting down on what's enabled doesn't seem to help, though...)
Hi George,
I have also seen the hang on my device, though I was not able to identify what caused the device to reboot: I am still waiting for a UART device https://www.seeedstudio.com/96Boards-UART-p-2525.html to arrive to see the output of the console.
My setup is AOSP master with kernel 4.9 from https://android.googlesource.com/kernel/hikey-linaro and the two patches sent by Mathieu applied on top of the android kernel. The .config generated by "make hikey_defconfig" works fine, then the device keeps rebooting when configured with CoreSight with this change:
--- .config-0 2017-05-15 10:20:02.152499418 -0500 +++ .config-1 2017-05-15 10:21:48.157466820 -0500 @@ -4010,7 +4010,10 @@ # CONFIG_LIBNVDIMM is not set # CONFIG_DEV_DAX is not set # CONFIG_NVMEM is not set -# CONFIG_STM is not set +CONFIG_STM=y +# CONFIG_STM_DUMMY is not set +# CONFIG_STM_SOURCE_CONSOLE is not set +# CONFIG_STM_SOURCE_HEARTBEAT is not set # CONFIG_INTEL_TH is not set
# @@ -4484,7 +4487,14 @@ # CONFIG_PID_IN_CONTEXTIDR is not set # CONFIG_ARM64_RANDOMIZE_TEXT_OFFSET is not set # CONFIG_DEBUG_ALIGN_RODATA is not set -# CONFIG_CORESIGHT is not set +CONFIG_CORESIGHT=y +CONFIG_CORESIGHT_LINKS_AND_SINKS=y +CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y +CONFIG_CORESIGHT_SINK_TPIU=y +CONFIG_CORESIGHT_SINK_ETBV10=y +CONFIG_CORESIGHT_SOURCE_ETM4X=y +CONFIG_CORESIGHT_QCOM_REPLICATOR=y +CONFIG_CORESIGHT_STM=y
# # Security options
Hi Sebastian, George,
On Tue, May 16, 2017 at 08:24:56AM -0500, Sebastian Pop wrote:
On Mon, May 15, 2017 at 8:38 PM, George Burgess gbiv@google.com wrote:
Awesome, thank you all!
I applied both patches to my local kernel (tried both 4.9 from AOSP, and 4.11 from upstream Linux), and can't seem to get my device to fully boot. :/
Looking at `dmesg` from failed runs, it seems that etm0 is being initialized cleanly, but initializing any other etm {1..7} causes my kernel to panic. All of the panics take the form "Unhandled fault: synchronous external abort (0x96000210) at $addr," where $addr is `drvdata->base + TRCIDR0` (from etm4_init_arch_data in drivers/hwtracing/coresight/coresight-etm4x.c).
I've messed around a bit, and it seems that writes to `drvdata->base + multiple_offsets` are perfectly OK, but the moment that you try to read from it, you get a panic. Like said, this only occurs on ETMs that aren't etm0. If I just throw a `if (drvdata->cpu) return;` in before we try to read from `*drvdata->base`, the device boots happily with a single ETM.
Do you all have any tips for tracking down what's causing this? Linux isn't exactly my specialty, so my troubleshooting attempts haven't really gotten me very far. :)
If it matters, my coresight-related configuration options are: CONFIG_CORESIGHT=y CONFIG_CORESIGHT_ETBV10=y CONFIG_CORESIGHT_SOURCE_ETM4X=y CONFIG_CORESIGHT_LINKS_AND_SINKS=y CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y CONFIG_CORESIGHT_QCOM_REPLICATOR=y CONFIG_CORESIGHT_SINK_TPIU=y CONFIG_CORESIGHT_STM=y
(I just ticked whatever boxes looked even mildly relevant. ;) Cutting down on what's enabled doesn't seem to help, though...)
Hi George,
I have also seen the hang on my device, though I was not able to identify what caused the device to reboot: I am still waiting for a UART device https://www.seeedstudio.com/96Boards-UART-p-2525.html to arrive to see the output of the console.
When enable coresight and hang on Hikey, could you try to disable CPUIdle totally? You can remove CPUIdle related configurations in .config or you can add 'nohlt' into kernel command line.
My setup is AOSP master with kernel 4.9 from https://android.googlesource.com/kernel/hikey-linaro and the two patches sent by Mathieu applied on top of the android kernel. The .config generated by "make hikey_defconfig" works fine, then the device keeps rebooting when configured with CoreSight with this change:
--- .config-0 2017-05-15 10:20:02.152499418 -0500 +++ .config-1 2017-05-15 10:21:48.157466820 -0500 @@ -4010,7 +4010,10 @@ # CONFIG_LIBNVDIMM is not set # CONFIG_DEV_DAX is not set # CONFIG_NVMEM is not set -# CONFIG_STM is not set +CONFIG_STM=y +# CONFIG_STM_DUMMY is not set +# CONFIG_STM_SOURCE_CONSOLE is not set +# CONFIG_STM_SOURCE_HEARTBEAT is not set # CONFIG_INTEL_TH is not set
# @@ -4484,7 +4487,14 @@ # CONFIG_PID_IN_CONTEXTIDR is not set # CONFIG_ARM64_RANDOMIZE_TEXT_OFFSET is not set # CONFIG_DEBUG_ALIGN_RODATA is not set -# CONFIG_CORESIGHT is not set +CONFIG_CORESIGHT=y +CONFIG_CORESIGHT_LINKS_AND_SINKS=y +CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y +CONFIG_CORESIGHT_SINK_TPIU=y +CONFIG_CORESIGHT_SINK_ETBV10=y +CONFIG_CORESIGHT_SOURCE_ETM4X=y +CONFIG_CORESIGHT_QCOM_REPLICATOR=y +CONFIG_CORESIGHT_STM=y
# # Security options _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On 16 May 2017 at 08:02, Leo Yan leo.yan@linaro.org wrote:
Hi Sebastian, George,
On Tue, May 16, 2017 at 08:24:56AM -0500, Sebastian Pop wrote:
On Mon, May 15, 2017 at 8:38 PM, George Burgess gbiv@google.com wrote:
Awesome, thank you all!
I applied both patches to my local kernel (tried both 4.9 from AOSP, and 4.11 from upstream Linux), and can't seem to get my device to fully boot. :/
Looking at `dmesg` from failed runs, it seems that etm0 is being initialized cleanly, but initializing any other etm {1..7} causes my kernel to panic. All of the panics take the form "Unhandled fault: synchronous external abort (0x96000210) at $addr," where $addr is `drvdata->base + TRCIDR0` (from etm4_init_arch_data in drivers/hwtracing/coresight/coresight-etm4x.c).
I've messed around a bit, and it seems that writes to `drvdata->base + multiple_offsets` are perfectly OK, but the moment that you try to read from it, you get a panic. Like said, this only occurs on ETMs that aren't etm0. If I just throw a `if (drvdata->cpu) return;` in before we try to read from `*drvdata->base`, the device boots happily with a single ETM.
Do you all have any tips for tracking down what's causing this? Linux isn't exactly my specialty, so my troubleshooting attempts haven't really gotten me very far. :)
If it matters, my coresight-related configuration options are: CONFIG_CORESIGHT=y CONFIG_CORESIGHT_ETBV10=y CONFIG_CORESIGHT_SOURCE_ETM4X=y CONFIG_CORESIGHT_LINKS_AND_SINKS=y CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y CONFIG_CORESIGHT_QCOM_REPLICATOR=y CONFIG_CORESIGHT_SINK_TPIU=y CONFIG_CORESIGHT_STM=y
(I just ticked whatever boxes looked even mildly relevant. ;) Cutting down on what's enabled doesn't seem to help, though...)
Hi George,
I have also seen the hang on my device, though I was not able to identify what caused the device to reboot: I am still waiting for a UART device https://www.seeedstudio.com/96Boards-UART-p-2525.html to arrive to see the output of the console.
When enable coresight and hang on Hikey, could you try to disable CPUIdle totally? You can remove CPUIdle related configurations in .config or you can add 'nohlt' into kernel command line.
Leo is correct - this is the first thing to look at.
My setup is AOSP master with kernel 4.9 from https://android.googlesource.com/kernel/hikey-linaro and the two patches sent by Mathieu applied on top of the android kernel. The .config generated by "make hikey_defconfig" works fine, then the device keeps rebooting when configured with CoreSight with this change:
--- .config-0 2017-05-15 10:20:02.152499418 -0500 +++ .config-1 2017-05-15 10:21:48.157466820 -0500 @@ -4010,7 +4010,10 @@ # CONFIG_LIBNVDIMM is not set # CONFIG_DEV_DAX is not set # CONFIG_NVMEM is not set -# CONFIG_STM is not set +CONFIG_STM=y +# CONFIG_STM_DUMMY is not set +# CONFIG_STM_SOURCE_CONSOLE is not set +# CONFIG_STM_SOURCE_HEARTBEAT is not set # CONFIG_INTEL_TH is not set
# @@ -4484,7 +4487,14 @@ # CONFIG_PID_IN_CONTEXTIDR is not set # CONFIG_ARM64_RANDOMIZE_TEXT_OFFSET is not set # CONFIG_DEBUG_ALIGN_RODATA is not set -# CONFIG_CORESIGHT is not set +CONFIG_CORESIGHT=y +CONFIG_CORESIGHT_LINKS_AND_SINKS=y +CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y +CONFIG_CORESIGHT_SINK_TPIU=y +CONFIG_CORESIGHT_SINK_ETBV10=y +CONFIG_CORESIGHT_SOURCE_ETM4X=y +CONFIG_CORESIGHT_QCOM_REPLICATOR=y +CONFIG_CORESIGHT_STM=y
# # Security options _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On Tue, May 16, 2017 at 10:23 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 16 May 2017 at 08:02, Leo Yan leo.yan@linaro.org wrote:
When enable coresight and hang on Hikey, could you try to disable CPUIdle totally? You can remove CPUIdle related configurations in .config or you can add 'nohlt' into kernel command line.
Leo is correct - this is the first thing to look at.
Thanks Leo and Mathieu for the suggestion. With that change my board still does not boot correctly. Here is how I disabled CPUIDLE in my config file:
--- .config-1 2017-05-15 10:21:48.157466820 -0500 +++ .config 2017-05-16 15:26:55.955816933 -0500 @@ -559,15 +559,7 @@ # # CPU Idle # -CONFIG_CPU_IDLE=y -CONFIG_CPU_IDLE_GOV_LADDER=y -CONFIG_CPU_IDLE_GOV_MENU=y -CONFIG_DT_IDLE_STATES=y - -# -# ARM CPU Idle Drivers -# -CONFIG_ARM_CPUIDLE=y +# CONFIG_CPU_IDLE is not set # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
#
Seconding Sebastian's observations.
I'll also note that I see a different error every now and again (maybe in ~15% or so of my boots): "Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP". I'm assuming it's similar to the "synchronous external abort" error, since it's triggered at almost the exactly same instruction address (2 bytes off), and it only happens if etm4_probe/amba_probe/... are in the call trace the kernel dumps. On the off chance that it's helpful, though... :)
I'll play around with config changes a bit more.
On Tue, May 16, 2017 at 1:31 PM, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 16, 2017 at 10:23 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 16 May 2017 at 08:02, Leo Yan leo.yan@linaro.org wrote:
When enable coresight and hang on Hikey, could you try to disable CPUIdle totally? You can remove CPUIdle related configurations in .config or you can add 'nohlt' into kernel command line.
Leo is correct - this is the first thing to look at.
Thanks Leo and Mathieu for the suggestion. With that change my board still does not boot correctly. Here is how I disabled CPUIDLE in my config file:
--- .config-1 2017-05-15 10:21:48.157466820 -0500 +++ .config 2017-05-16 15:26:55.955816933 -0500 @@ -559,15 +559,7 @@ # # CPU Idle # -CONFIG_CPU_IDLE=y -CONFIG_CPU_IDLE_GOV_LADDER=y -CONFIG_CPU_IDLE_GOV_MENU=y -CONFIG_DT_IDLE_STATES=y
-# -# ARM CPU Idle Drivers -# -CONFIG_ARM_CPUIDLE=y +# CONFIG_CPU_IDLE is not set # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
#
On 16 May 2017 at 14:31, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 16, 2017 at 10:23 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 16 May 2017 at 08:02, Leo Yan leo.yan@linaro.org wrote:
When enable coresight and hang on Hikey, could you try to disable CPUIdle totally? You can remove CPUIdle related configurations in .config or you can add 'nohlt' into kernel command line.
Leo is correct - this is the first thing to look at.
Thanks Leo and Mathieu for the suggestion. With that change my board still does not boot correctly. Here is how I disabled CPUIDLE in my config file:
--- .config-1 2017-05-15 10:21:48.157466820 -0500 +++ .config 2017-05-16 15:26:55.955816933 -0500 @@ -559,15 +559,7 @@ # # CPU Idle # -CONFIG_CPU_IDLE=y -CONFIG_CPU_IDLE_GOV_LADDER=y -CONFIG_CPU_IDLE_GOV_MENU=y -CONFIG_DT_IDLE_STATES=y
This will definitly disable CPUidle - and you are using a perf-opencsd-v4.11 kernel?
-# -# ARM CPU Idle Drivers -# -CONFIG_ARM_CPUIDLE=y +# CONFIG_CPU_IDLE is not set # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
#
On Wed, May 17, 2017 at 9:12 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
This will definitly disable CPUidle - and you are using a perf-opencsd-v4.11 kernel?
George was mentioning that he tried the upstream linux kernel 4.11. I am trying to enable the ETM on the 4.9 kernel coming from the AOSP on top of a build from AOSP master. I know that I will be on my own there.
So let me do a smaller step: I will try to enable ETM on the hikey running Debian with the kernel from https://github.com/Linaro/OpenCSD/tree/perf-opencsd-master.
Sebastian
On Wed, May 17, 2017 at 09:49:52AM -0500, Sebastian Pop wrote:
On Wed, May 17, 2017 at 9:12 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
This will definitly disable CPUidle - and you are using a perf-opencsd-v4.11 kernel?
George was mentioning that he tried the upstream linux kernel 4.11. I am trying to enable the ETM on the 4.9 kernel coming from the AOSP on top of a build from AOSP master. I know that I will be on my own there.
So let me do a smaller step: I will try to enable ETM on the hikey running Debian with the kernel from https://github.com/Linaro/OpenCSD/tree/perf-opencsd-master.
I think one thing should meantion is ARM-TF has another fixing: https://github.com/96boards-hikey/arm-trusted-firmware/commit/543f1db7fd7e0a...
If you are using latest boot images, you should have included this change int ARM-TF. This change is merged into ARM-TF at last year, but just ensure this is not a block issue at your side.
Thanks, Leo Yan
Looks like that ARM-TF patch was precisely what I was missing. :)
When I set the bits that patch sets, I can now get Hikey to fully boot with OpenCSD's Linux branch 4.11+ETMs, and can capture perf traces like so: perf record -e cs_etm/@f6404000.etr/u --per-thread -- ./foo
Thanks again!
On Wed, May 17, 2017 at 8:13 AM, Leo Yan leo.yan@linaro.org wrote:
On Wed, May 17, 2017 at 09:49:52AM -0500, Sebastian Pop wrote:
On Wed, May 17, 2017 at 9:12 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
This will definitly disable CPUidle - and you are using a perf-opencsd-v4.11 kernel?
George was mentioning that he tried the upstream linux kernel 4.11. I am trying to enable the ETM on the 4.9 kernel coming from the AOSP on top of a build from AOSP master. I know that I will be on my own
there.
So let me do a smaller step: I will try to enable ETM on the hikey
running
Debian with the kernel from https://github.com/Linaro/OpenCSD/tree/perf-opencsd-master.
I think one thing should meantion is ARM-TF has another fixing: https://github.com/96boards-hikey/arm-trusted-firmware/commi t/543f1db7fd7e0aadcf947adedd63eaed817ef402
If you are using latest boot images, you should have included this change int ARM-TF. This change is merged into ARM-TF at last year, but just ensure this is not a block issue at your side.
Thanks, Leo Yan
Hi George,
On Tue, May 23, 2017 at 12:32 AM, George Burgess gbiv@google.com wrote:
Looks like that ARM-TF patch was precisely what I was missing. :)
When I set the bits that patch sets, I can now get Hikey to fully boot with OpenCSD's Linux branch 4.11+ETMs, and can capture perf traces like so: perf record -e cs_etm/@f6404000.etr/u --per-thread -- ./foo
Could you please send a patch or more explanations with what is needed? I still have not figured out how to use the arm-tf patch that Leo pointed out.
Thanks, Sebastian
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch, since I'm using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip the bits that the ARM patch does. It seems that this works just as well (and is convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the device to boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line 241ish). Trying to apply a patch to make that work gave me a build error, which I didn't look too much into; I'm perfectly fine with my kernel not being near AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play nicely with AOSP and put them in patch form tomorrow. It was mainly just me adding bits and pieces to Makefiles and crossing my fingers.
On Mon, May 22, 2017 at 10:42 PM, Sebastian Pop sebpop@gmail.com wrote:
Hi George,
On Tue, May 23, 2017 at 12:32 AM, George Burgess gbiv@google.com wrote:
Looks like that ARM-TF patch was precisely what I was missing. :)
When I set the bits that patch sets, I can now get Hikey to fully boot
with
OpenCSD's Linux branch 4.11+ETMs, and can capture perf traces like so: perf record -e cs_etm/@f6404000.etr/u --per-thread -- ./foo
Could you please send a patch or more explanations with what is needed? I still have not figured out how to use the arm-tf patch that Leo pointed out.
Thanks, Sebastian
On Tue, May 23, 2017 at 1:44 AM, George Burgess gbiv@google.com wrote:
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch, since I'm using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip the bits that the ARM patch does. It seems that this works just as well (and is convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the device to boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line 241ish). Trying to apply a patch to make that work gave me a build error, which I didn't look too much into; I'm perfectly fine with my kernel not being near AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play nicely with AOSP and put them in patch form tomorrow. It was mainly just me adding bits and pieces to Makefiles and crossing my fingers.
Thanks George for the patch. I got the AOSP linux kernel 4.9 working with CoreSight with the attached patches. I can see the ETM components:
hikey:/ # ls /sys/bus/coresight/devices/ amba:replicator@0/ f6501000.funnel/ f659f000.etm/ f65df000.etm/ f6401000.funnel/ f659c000.etm/ f65dc000.etm/ f6402000.etf/ f659d000.etm/ f65dd000.etm/ f6404000.etr/ f659e000.etm/ f65de000.etm/
However perf does not collect traces yet: it fails with some malloc problem (maybe the same problem as what you reported above)
# perf record -e cs_etm/@f6404000.etr/u --per-thread failed to mmap with 12 (Cannot allocate memory)
I will try to run perf under gdb and see where and why this fails.
Sebastian
On 31 May 2017 at 11:09, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 23, 2017 at 1:44 AM, George Burgess gbiv@google.com wrote:
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch, since I'm using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip the bits that the ARM patch does. It seems that this works just as well (and is convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the device to boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line 241ish). Trying to apply a patch to make that work gave me a build error, which I didn't look too much into; I'm perfectly fine with my kernel not being near AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play nicely with AOSP and put them in patch form tomorrow. It was mainly just me adding bits and pieces to Makefiles and crossing my fingers.
Thanks George for the patch. I got the AOSP linux kernel 4.9 working with CoreSight with the attached patches. I can see the ETM components:
hikey:/ # ls /sys/bus/coresight/devices/ amba:replicator@0/ f6501000.funnel/ f659f000.etm/ f65df000.etm/ f6401000.funnel/ f659c000.etm/ f65dc000.etm/ f6402000.etf/ f659d000.etm/ f65dd000.etm/ f6404000.etr/ f659e000.etm/ f65de000.etm/
However perf does not collect traces yet: it fails with some malloc problem (maybe the same problem as what you reported above)
# perf record -e cs_etm/@f6404000.etr/u --per-thread failed to mmap with 12 (Cannot allocate memory)
The perf interface for ETR isn't part of mainline yet. There is a driver available on github (also for kernel 4.9) that should apply cleanly on top of AOSP. ETF support is available in mainline and should work out of the box.
I will try to run perf under gdb and see where and why this fails.
Sebastian
Great to hear, Sebastian! :) The "failed to mmap" was the same error I got, as well.
Is this a good place to talk about perf inject machinery, or should I start a new thread for that?
If this is the right place, it looks like the traces from my hikey might not be getting read properly by `perf inject` when I'm trying to generate LBR traces. A bit of debugging has shown that many of the traces can be decoded, but we eventually hit one of type ETM4_PKT_I_COND_I_F2, which TrcPktDecodeEtmV4I::decodePacket currently doesn't support.
I'm reading the docs, and it looks like that kind of tracing needs to be explicitly enabled. Doing `cat /sys/bus/coresight/devices/*.etm/mode`, the bits to enable that all appear clear on my setup. So, I'm unsure how it's sneaking into the traces. I found a convenient way to dump the traces (cs_etm__dump_event), and it looks like there may be some invalid traces in perf.data, as well?
In any case, I've attached three things in one tarball: - perf.data from running a bubble sort program on Hikey for a few ms; recorded using `taskset 1 ./perf record -e cs_etm/@f6404000.etr/u --per-thread ./gbiv-tool` - etm-dump.txt, which is stdout from running `perf --debug verbose=9 inject -i ./perf.data -o lbr_trace --itrace=i100usl64 --strip` (with a call to cs_etm__dump_event baked in) - lbr_trace, the (nearly empty?) output of the trace.
When trying to convert lbr_trace to an LLVM profile, I get errors about 0% of the samples being mappable onto the executable.
I'm running perf on my x86 machine from the opencsd 4.11 branch; opencsd was freshly pulled+compiled earlier today.
I'll continue trying to figure out the root cause tomorrow.
On Wed, May 31, 2017 at 10:56 AM, Mathieu Poirier < mathieu.poirier@linaro.org> wrote:
On 31 May 2017 at 11:09, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 23, 2017 at 1:44 AM, George Burgess gbiv@google.com wrote:
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch,
since I'm
using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip the
bits
that the ARM patch does. It seems that this works just as well (and is convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the
device to
boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line
241ish).
Trying to apply a patch to make that work gave me a build error, which I didn't look too much into; I'm perfectly fine with my kernel not being
near
AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play
nicely
with AOSP and put them in patch form tomorrow. It was mainly just me
adding
bits and pieces to Makefiles and crossing my fingers.
Thanks George for the patch. I got the AOSP linux kernel 4.9 working with CoreSight with the attached patches. I can see the ETM components:
hikey:/ # ls /sys/bus/coresight/devices/ amba:replicator@0/ f6501000.funnel/ f659f000.etm/
f65df000.etm/
f6401000.funnel/ f659c000.etm/ f65dc000.etm/ f6402000.etf/ f659d000.etm/ f65dd000.etm/ f6404000.etr/ f659e000.etm/ f65de000.etm/
However perf does not collect traces yet: it fails with some malloc
problem
(maybe the same problem as what you reported above)
# perf record -e cs_etm/@f6404000.etr/u --per-thread failed to mmap with 12 (Cannot allocate memory)
The perf interface for ETR isn't part of mainline yet. There is a driver available on github (also for kernel 4.9) that should apply cleanly on top of AOSP. ETF support is available in mainline and should work out of the box.
I will try to run perf under gdb and see where and why this fails.
Sebastian
Hi George,
I have looked at the trace you supplied using perf report --stdio --dump - the output of the first few lines is below.
My immediate view is that this trace is hopelessly corrupt. The size of trace would suggest that the buffer has not wrapped - which can cause issues in the current perf record implementation.
The small excerpt below is riddles with reserved trace packet tokens and invalid sequences. The address and context packet at index 26951 shows a VMID and ContextID entry - even though the perf setup cannot enable these features. The addresses look a little unusual too.
The trace ID is consistent, suggesting that the output from the ETR frame formatter is correct - making me think that something upstream (ETM, funnel etc) is causing the problem - though I would not rely on this. The other possibility is that the ETM is not programmed according to the settings perf enabled it with - which could cause a mismatch between the decoder expectations and the trace received.
I would suggest capturing a trace session using ETB / ETF if that is possible on your system.
With bad data any attempt to use --inject is doomed to failure
Regards
Mike
. ... CoreSight ETM Trace data: size 121856 bytes 2868: id[10] I_NOT_SYNC : I Stream not synchronised 26872: id[10] I_ASYNC : Alignment Synchronisation. 26885: id[10] I_TRACE_INFO : Trace Info.; PCTL=0x0 26887: id[10] I_BAD_SEQUENCE : Invalid Sequence in packet.[I_EXTENSION] 26889: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x3000017EB7F43258; Ctxt: AArch32, EL2, NS; 26900: id[10] I_ATOM_F2 : Atom format 2.; EE 26901: id[10] I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0001007FB7EB17C4; 26910: id[10] I_ATOM_F4 : Atom format 4.; ENEN 26912: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB17D0 ~[0x1D0] 26914: id[10] I_ATOM_F3 : Atom format 3.; ENE 26915: id[10] I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEE 26916: id[10] I_ATOM_F3 : Atom format 3.; NNE 26917: id[10] I_ATOM_F3 : Atom format 3.; EEN 26918: id[10] I_ATOM_F3 : Atom format 3.; ENE 26919: id[10] I_ATOM_F1 : Atom format 1.; E 26920: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB22E8 ~[0x122E8] 26923: id[10] I_ATOM_F3 : Atom format 3.; NEE 26924: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB17E0 ~[0x117E0] 26928: id[10] I_ATOM_F4 : Atom format 4.; NENE 26929: id[10] I_ATOM_F3 : Atom format 3.; NNE 26930: id[10] I_EXCEPT : Exception.; Reserved; 26932: id[10] I_RESERVED : Reserved Packet Header 26933: id[10] I_ADDR_L_32IS1 : Address, Long, 32 bit, IS1.; Addr=0xB6F418F2; 26938: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFE88FEFF80082400; Ctxt: AArch64,EL1, NS; 26950: id[10] I_TRACE_ON : Trace On. 26951: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x3000007EB7F431E0; Ctxt: AArch64,EL2, S; CID=0xea0c0d95; VMID=0x009d; 26967: id[10] I_RESERVED : Reserved Packet Header 26968: id[10] I_EVENT : Trace Event. 26969: id[10] I_BAD_SEQUENCE : Invalid Sequence in packet.[I_ASYNC] 26973: id[10] I_ATOM_F3 : Atom format 3.; NNN 26974: id[10] I_ATOM_F1 : Atom format 1.; N 26976: id[10] I_RESERVED : Reserved Packet Header 26977: id[10] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0xF419EDE4; 26982: id[10] I_RESERVED : Reserved Packet Header 26983: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x30FFFF8108092404; Ctxt: AArch32, EL0, S; 26994: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x00007EB7F518F388; Ctxt: AArch32, EL0, S;
On 1 June 2017 at 05:59, George Burgess gbiv@google.com wrote:
Great to hear, Sebastian! :) The "failed to mmap" was the same error I got, as well.
Is this a good place to talk about perf inject machinery, or should I start a new thread for that?
If this is the right place, it looks like the traces from my hikey might not be getting read properly by `perf inject` when I'm trying to generate LBR traces. A bit of debugging has shown that many of the traces can be decoded, but we eventually hit one of type ETM4_PKT_I_COND_I_F2, which TrcPktDecodeEtmV4I::decodePacket currently doesn't support.
I'm reading the docs, and it looks like that kind of tracing needs to be explicitly enabled. Doing `cat /sys/bus/coresight/devices/*.etm/mode`, the bits to enable that all appear clear on my setup. So, I'm unsure how it's sneaking into the traces. I found a convenient way to dump the traces (cs_etm__dump_event), and it looks like there may be some invalid traces in perf.data, as well?
In any case, I've attached three things in one tarball:
- perf.data from running a bubble sort program on Hikey for a few ms;
recorded using `taskset 1 ./perf record -e cs_etm/@f6404000.etr/u --per-thread ./gbiv-tool`
- etm-dump.txt, which is stdout from running `perf --debug verbose=9 inject
-i ./perf.data -o lbr_trace --itrace=i100usl64 --strip` (with a call to cs_etm__dump_event baked in)
- lbr_trace, the (nearly empty?) output of the trace.
When trying to convert lbr_trace to an LLVM profile, I get errors about 0% of the samples being mappable onto the executable.
I'm running perf on my x86 machine from the opencsd 4.11 branch; opencsd was freshly pulled+compiled earlier today.
I'll continue trying to figure out the root cause tomorrow.
On Wed, May 31, 2017 at 10:56 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 31 May 2017 at 11:09, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 23, 2017 at 1:44 AM, George Burgess gbiv@google.com wrote:
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch, since I'm using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip the bits that the ARM patch does. It seems that this works just as well (and is convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the device to boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line 241ish). Trying to apply a patch to make that work gave me a build error, which I didn't look too much into; I'm perfectly fine with my kernel not being near AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play nicely with AOSP and put them in patch form tomorrow. It was mainly just me adding bits and pieces to Makefiles and crossing my fingers.
Thanks George for the patch. I got the AOSP linux kernel 4.9 working with CoreSight with the attached patches. I can see the ETM components:
hikey:/ # ls /sys/bus/coresight/devices/ amba:replicator@0/ f6501000.funnel/ f659f000.etm/ f65df000.etm/ f6401000.funnel/ f659c000.etm/ f65dc000.etm/ f6402000.etf/ f659d000.etm/ f65dd000.etm/ f6404000.etr/ f659e000.etm/ f65de000.etm/
However perf does not collect traces yet: it fails with some malloc problem (maybe the same problem as what you reported above)
# perf record -e cs_etm/@f6404000.etr/u --per-thread failed to mmap with 12 (Cannot allocate memory)
The perf interface for ETR isn't part of mainline yet. There is a driver available on github (also for kernel 4.9) that should apply cleanly on top of AOSP. ETF support is available in mainline and should work out of the box.
I will try to run perf under gdb and see where and why this fails.
Sebastian
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Thanks, Mike!
The size of trace would suggest that the buffer has not wrapped - which
can
cause issues in the current perf record implementation.
Looks like bad traces still pop up early on even if I run the program for ~65 seconds (which generates a 150MB perf.data file), sadly.
I'll poke around and see what else I can figure out. :)
On Thu, Jun 1, 2017 at 6:55 AM, Mike Leach mike.leach@linaro.org wrote:
Hi George,
I have looked at the trace you supplied using perf report --stdio --dump - the output of the first few lines is below.
My immediate view is that this trace is hopelessly corrupt. The size of trace would suggest that the buffer has not wrapped - which can cause issues in the current perf record implementation.
The small excerpt below is riddles with reserved trace packet tokens and invalid sequences. The address and context packet at index 26951 shows a VMID and ContextID entry - even though the perf setup cannot enable these features. The addresses look a little unusual too.
The trace ID is consistent, suggesting that the output from the ETR frame formatter is correct - making me think that something upstream (ETM, funnel etc) is causing the problem - though I would not rely on this. The other possibility is that the ETM is not programmed according to the settings perf enabled it with - which could cause a mismatch between the decoder expectations and the trace received.
I would suggest capturing a trace session using ETB / ETF if that is possible on your system.
With bad data any attempt to use --inject is doomed to failure
Regards
Mike
. ... CoreSight ETM Trace data: size 121856 bytes 2868: id[10] I_NOT_SYNC : I Stream not synchronised 26872: id[10] I_ASYNC : Alignment Synchronisation. 26885: id[10] I_TRACE_INFO : Trace Info.; PCTL=0x0 26887: id[10] I_BAD_SEQUENCE : Invalid Sequence in packet.[I_EXTENSION] 26889: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x3000017EB7F43258; Ctxt: AArch32, EL2, NS; 26900: id[10] I_ATOM_F2 : Atom format 2.; EE 26901: id[10] I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0001007FB7EB17C4; 26910: id[10] I_ATOM_F4 : Atom format 4.; ENEN 26912: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB17D0 ~[0x1D0] 26914: id[10] I_ATOM_F3 : Atom format 3.; ENE 26915: id[10] I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEE 26916: id[10] I_ATOM_F3 : Atom format 3.; NNE 26917: id[10] I_ATOM_F3 : Atom format 3.; EEN 26918: id[10] I_ATOM_F3 : Atom format 3.; ENE 26919: id[10] I_ATOM_F1 : Atom format 1.; E 26920: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB22E8 ~[0x122E8] 26923: id[10] I_ATOM_F3 : Atom format 3.; NEE 26924: id[10] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x0001007FB7EB17E0 ~[0x117E0] 26928: id[10] I_ATOM_F4 : Atom format 4.; NENE 26929: id[10] I_ATOM_F3 : Atom format 3.; NNE 26930: id[10] I_EXCEPT : Exception.; Reserved; 26932: id[10] I_RESERVED : Reserved Packet Header 26933: id[10] I_ADDR_L_32IS1 : Address, Long, 32 bit, IS1.; Addr=0xB6F418F2; 26938: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFE88FEFF80082400; Ctxt: AArch64,EL1, NS; 26950: id[10] I_TRACE_ON : Trace On. 26951: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x3000007EB7F431E0; Ctxt: AArch64,EL2, S; CID=0xea0c0d95; VMID=0x009d; 26967: id[10] I_RESERVED : Reserved Packet Header 26968: id[10] I_EVENT : Trace Event. 26969: id[10] I_BAD_SEQUENCE : Invalid Sequence in packet.[I_ASYNC] 26973: id[10] I_ATOM_F3 : Atom format 3.; NNN 26974: id[10] I_ATOM_F1 : Atom format 1.; N 26976: id[10] I_RESERVED : Reserved Packet Header 26977: id[10] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0xF419EDE4; 26982: id[10] I_RESERVED : Reserved Packet Header 26983: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x30FFFF8108092404; Ctxt: AArch32, EL0, S; 26994: id[10] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x00007EB7F518F388; Ctxt: AArch32, EL0, S;
On 1 June 2017 at 05:59, George Burgess gbiv@google.com wrote:
Great to hear, Sebastian! :) The "failed to mmap" was the same error I
got,
as well.
Is this a good place to talk about perf inject machinery, or should I
start
a new thread for that?
If this is the right place, it looks like the traces from my hikey might
not
be getting read properly by `perf inject` when I'm trying to generate LBR traces. A bit of debugging has shown that many of the traces can be
decoded,
but we eventually hit one of type ETM4_PKT_I_COND_I_F2, which TrcPktDecodeEtmV4I::decodePacket currently doesn't support.
I'm reading the docs, and it looks like that kind of tracing needs to be explicitly enabled. Doing `cat /sys/bus/coresight/devices/*.etm/mode`,
the
bits to enable that all appear clear on my setup. So, I'm unsure how it's sneaking into the traces. I found a convenient way to dump the traces (cs_etm__dump_event), and it looks like there may be some invalid traces
in
perf.data, as well?
In any case, I've attached three things in one tarball:
- perf.data from running a bubble sort program on Hikey for a few ms;
recorded using `taskset 1 ./perf record -e cs_etm/@f6404000.etr/u --per-thread ./gbiv-tool`
- etm-dump.txt, which is stdout from running `perf --debug verbose=9
inject
-i ./perf.data -o lbr_trace --itrace=i100usl64 --strip` (with a call to cs_etm__dump_event baked in)
- lbr_trace, the (nearly empty?) output of the trace.
When trying to convert lbr_trace to an LLVM profile, I get errors about
0%
of the samples being mappable onto the executable.
I'm running perf on my x86 machine from the opencsd 4.11 branch; opencsd
was
freshly pulled+compiled earlier today.
I'll continue trying to figure out the root cause tomorrow.
On Wed, May 31, 2017 at 10:56 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On 31 May 2017 at 11:09, Sebastian Pop sebpop@gmail.com wrote:
On Tue, May 23, 2017 at 1:44 AM, George Burgess gbiv@google.com
wrote:
Right; sorry, I'm tired. :) I was a bit sneaky with the ARM patch, since I'm using Android+fastboot with this kernel, as I think you are.
In particular, I modified Li Pengcheng's sysconf patch to also flip
the
bits that the ARM patch does. It seems that this works just as well (and
is
convenient, since that patch already ioremaps the region that the arm-tf patch writes to). New hi6220-sysconfig.c patch is attached.
I tried applying this patch to 4.9, as well. I was able to get the device to boot and recognize its ETMs, but perf dies seemingly because `sink_ops(sink)->alloc_buffer == NULL` (coresight-etm-perf.c; line 241ish). Trying to apply a patch to make that work gave me a build error,
which
I didn't look too much into; I'm perfectly fine with my kernel not
being
near AOSP's HEAD for the moment. :)
If you'd like, I'm happy to gather the hacks I made to make 4.11 play nicely with AOSP and put them in patch form tomorrow. It was mainly just me adding bits and pieces to Makefiles and crossing my fingers.
Thanks George for the patch. I got the AOSP linux kernel 4.9 working with CoreSight with the attached patches. I can see the ETM components:
hikey:/ # ls /sys/bus/coresight/devices/ amba:replicator@0/ f6501000.funnel/ f659f000.etm/ f65df000.etm/ f6401000.funnel/ f659c000.etm/ f65dc000.etm/ f6402000.etf/ f659d000.etm/ f65dd000.etm/ f6404000.etr/ f659e000.etm/ f65de000.etm/
However perf does not collect traces yet: it fails with some malloc problem (maybe the same problem as what you reported above)
# perf record -e cs_etm/@f6404000.etr/u --per-thread failed to mmap with 12 (Cannot allocate memory)
The perf interface for ETR isn't part of mainline yet. There is a driver available on github (also for kernel 4.9) that should apply cleanly on top of AOSP. ETF support is available in mainline and should work out of the box.
I will try to run perf under gdb and see where and why this fails.
Sebastian
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK
-
George Burgess -
Leo Yan -
Mathieu Poirier -
Mike Leach -
Sebastian Pop