Hi Leo,

I have been running the following scripts after boot:

for i in 0 1 2 3; do for j in 0 1 2 ; do echo 1 > /sys/devices/system/cpu/cpu$i/cpuidle/state$j/disable; done; done

for i in 4 5 6 7; do for j in 0 1 2 3 ; do echo 1 > /sys/devices/system/cpu/cpu$i/cpuidle/state$j/disable; done; done

Thanks for the recommendations, I will look into them!

Best, Daniel

On Mar 8, 2021, at 15:35, Leo Yan leo.yan@linaro.org wrote:

Hi Daniel,

On Mon, Mar 08, 2021 at 02:40:04PM +0100, Daniel Su wrote:

...

Thanks for the response, I have spent more time investigating and I have confirmed SNID and NSNID are properly set to 0b11.

I have also managed to see secure world execution in the trace captures, however I am having an issue where unless I read out the buffer quickly, all I see is a repetitive pattern of:

I_ASYNC I_TRACE_INFO I_ADDR_L_64IS0 ADDR

and ADDR looks to be inside cpu_do_idle

I am having issues filtering this out and it appears the exception level filter has no effect. Do you have any idea why this might be occurring?

Before I enable trace I also disable CPUidle as recommended for all cores, and also ensure that execution occurs on a single core by taking the rest offline.

Could you confirm how you disable CPUIdle?

Just remind, even you use the kernel command option "nohlt" to disable CPU idle, it doesn't really disable the kernel's idle flow, so this is why you can see the function cpu_do_idle(), which execute infinitely for instructions dsb(), wfi().

Another thing should note, if you don't use "nohlt" command and set the ETM registers, if I understand correctly, since the CPU will be powered off, so all your setting will be lost after the CPU is powered on again. This is because the CPU will be powered off and in the next time, it's return back from Arm trusted firmare, all the context for ETM will be lost. One main reason is the DT property "arm,coresight-loses-context-with-cpu;" for ETM for Hikey960 has not been enabled. So you could try below patch.

...

I also noticed that trctraceid occasionally changes when I deactivate and then reactivate the sink/source.

If you don't want to try below patch, another testing approach is suggested as:

• Add "nohlt" in the kernel command line;
• Boot the system;
• Set the ETM registers;

In theory, you should not see any context losing issue for ETM.

Thanks, Leo

---8<---

diff --git a/arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi b/arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi index d607f2f6698c..9f99f1c53f9d 100644 --- a/arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi +++ b/arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi @@ -17,6 +17,7 @@ etm@ecc40000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu0>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -34,6 +35,7 @@ etm@ecd40000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu1>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -51,6 +53,7 @@ etm@ece40000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu2>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -68,6 +71,7 @@ etm@ecf40000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu3>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -160,6 +164,7 @@ etm@ed440000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu4>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -177,6 +182,7 @@ etm@ed540000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu5>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -194,6 +200,7 @@ etm@ed640000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu6>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  

@@ -211,6 +218,7 @@ etm@ed740000 { clocks = <&crg_ctrl HI3660_PCLK>; clock-names = "apb_pclk"; cpu = <&cpu7>;

• 	arm,coresight-loses-context-with-cpu;
  
  out-ports {
  	port {
  
  
  

...

Best, Daniel

...

On Mar 2, 2021, at 13:57, Mike Leach mike.leach@linaro.org wrote:

Hi Daniel,

Looking at the sequence I would agree that it should work. I did check the driver to see if there were any issues but could not see anything after a code inspection. It may be worth firstly checkout TRCAUTHSTATUS to ensure that Secure mode tracing is actually permitted for your target.

On Thu, 25 Feb 2021 at 15:27, Daniel Su <daniel.sun.su@gmail.com mailto:daniel.sun.su@gmail.com> wrote:

...

Currently I am executing:

# ETM_MODE_EXCL_USER + ETM_MODE_EXCL_KERN + ETM_MODE_VIEWINST_STARTSTOP echo 0xc800000 > mode

# Exclude ALL Normal World exception levels echo 7 > ns_exlevel_vinst

# Exclude SEL1 echo 2 > s_exlevel_vinst

# Activate Trace echo 1 > /sys/bus/coresight/devices/ec036000.etf/enable_sink echo 1 > /sys/bus/coresight/devices/ecc40000.etm/enable_source

However the circular buffer is still being constantly filled, for example I am able to see that it is executing in the kernel idle loop. I should only be seeing instructions executing at SEL0 with this setup right?

Are you seeing both addressess and atoms here - or just addresses and trace on / info packets?

Finally you may want to check the actual values being programmed into the viewinst control register. First chekc after you halt trace, and perhaps add some logging to the driver to confirm the expected settings are being programmed.

Regards

Mike

...

Any tips on what I am missing here?

Best, Daniel

On Feb 19, 2021, at 17:11, Mike Leach mike.leach@linaro.org wrote:

Hi Daniel,

The CoreSight mailing list is the best place to ask these questions - I've added it to the to: section of this mail.

On Fri, 19 Feb 2021 at 13:43, Daniel Su daniel.sun.su@gmail.com wrote:

Sorry I made a mistake, I need to switch those bits but those aren't in the ETM mode sysFS parameter.

On Feb 19, 2021, at 14:21, Daniel Su daniel.sun.su@gmail.com wrote:

Thanks,

To just give a bit of background I am doing an internship to improve tracing and debugging for secure world applications in TrustZone. Originally the idea was to come up with a software solution but I thought using the hardware tracing features might be very powerful as the secure world trusted execution environments typically have less debugging and tracing functionalities built into the trusted operating system.

I have been doing further reading and I believe I need to configure the ETM mode sysFS parameter.

To switch these bits:

/* secure state access levels - TRCACATRn */ #define ETM_EXLEVEL_S_APP BIT(8) #define ETM_EXLEVEL_S_OS BIT(9) #define ETM_EXLEVEL_S_HYP BIT(10) #define ETM_EXLEVEL_S_MON BIT(11) /* non-secure state access levels - TRCACATRn */ #define ETM_EXLEVEL_NS_APP BIT(12) #define ETM_EXLEVEL_NS_OS BIT(13) #define ETM_EXLEVEL_NS_HYP BIT(14) #define ETM_EXLEVEL_NS_NA BIT(15)

ns_exlevel_vinst and s_exlevel_vinst control these bits in the TRCVICTLR.

addr_exlevel_s_ns sets both S and NS bits in the TRCACATRn register currently addressed by addr_idx.

In the kernel tree - Documentation/trace/coresight/coresight-etm4x-reference.rst - has lots of information on programming these sysfs files.

I would advise setting mode first - which will set up some default values for TRCVITLR, then make any further adjustments you want.

Right now I am able to capture the traces and read them out thanks to Leo Yan's command examples and presentation.

I still need to figure out how to only trace branching instructions, and also how to best decode the traces. Right now even though I can read out /dev/ec036000.etf , I am unsure how to best go about interpreting the raw trace data.

Hardware trace will trace everything - subject to certain filters - such as address filtering, exception filtering etc. It is not possible to trace only branch instructions. You will need to capture trace, decode then run an analysis on that decode to determine what branches have been taken. ETMv4 is program flow trace - so all branches are marked as taken or not taken. Consider the program:-

=============== 0x1000 start: < some code > ... 0x1100 B func1

0x1200 func1: <some code> .... 0x1240 mov r0, &func2 B r0

0x2000 func2: <some code>

This will result in trace as follows:- TRACE_ON ADDR(0x1000) ATOM(E)

I believe ptm2human and OpenCSD are two open source libraries that allow for decoding of the ETMv4 trace data. I understand there are paid solutions as well such as the DS-5 and Trace32 from Lauterbach to decode.

Do you have any recommendations on other decoders or any tips in general as to how to decode the raw trace data into human readable format?

Best, Daniel

On Feb 18, 2021, at 18:38, Suzuki K Poulose suzuki.poulose@arm.com wrote:

Hi Daniel

On 2/18/21 4:12 PM, Daniel Su wrote:

Hello, First of all I want to say thanks for your work on the Coresight support as well as the presentation slides. Currently I am working with the Hikey960 which I see is supported and has been used as a reference platform in your slides. I believe it should be possible to setup self-hosted Coresight to trace only for branch instructions in the secure world. Currently I am investigating how to set this up on the Hikey960 without access to a hardware debugger. I am wondering if any of you have done this before, if there is additional documentation, or if there are any good support groups/chats/channels to ask questions. I have some experience with Coresight mainly from reading the ARM reference manuals but it is still a lot to parse.

I haven't done anything similar. You should be able to imitate how the CoreSight drivers program the components ( ETM, Funnels, Replicator and ETR).

For ETM, you just need to make sure you don't filter out the Secure EL1/EL2/EL0. You would need to read the manuals for the components or user the kernel driver as the reference.

Cheers Suzuki

I believe in Linux I should be using `ns_exlevel_vinst` to filter out Non-Secure instructions. Also since the Hikey960 is multicore, I guess I would need to configure the top level Coresight in order to trace all cores? Would this be the ETR then? Best, Daniel

-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK

-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK