Hi Mike,

Thanks for your patience and exceptional response!

---

From: Mike Leach <mike.leach@linaro.org>
Sent: 11 July 2019 17:04
To: Student - Ng Yi Zher Jeremy <jeremy_ng@mymail.sutd.edu.sg>
Cc: Leo Yan <leo.yan@linaro.org>; Coresight ML <coresight@lists.linaro.org>
Subject: Re: Request for Guidance in Using sysFS for Coresight in HiKey960 (Cortex A53/A73)

 

Hi Jeremy,

Unfortunately my previous mail appears to have been affected by some
formatting problems by my webmail client, so is incomplete.

Thank you for getting back to me! 🙂

On Wed, 10 Jul 2019 at 16:07, Mike Leach <mike.leach@linaro.org> wrote:
>
>  retHello Jeremy,
>
> On Wed, 10 Jul 2019 at 10:06, Student - Ng Yi Zher Jeremy
> <jeremy_ng@mymail.sutd.edu.sg> wrote:
> >
> > Dear Sir,
> >
> > Please look at inline comments for my reply.
> >
> > My apologies for such a late reply.
> >
> > ________________________________
> > From: Leo Yan <leo.yan@linaro.org>
> > Sent: 10 June 2019 17:32
> > To: Student - Ng Yi Zher Jeremy
> > Cc: Mathieu Poirier; Suzuki K Poulose; Coresight ML
> > Subject: Re: Request for Guidance in Using sysFS for Coresight in HiKey960 (Cortex A53/A73)
> >
> >
> > Hi Jeremy,
> >
> > Please see several quick inline comments.
> >
> > On Mon, Jun 10, 2019 at 08:49:48AM +0000, Student - Ng Yi Zher Jeremy wrote:
> >
> > [...]
> >
> > > ```bash
> > > -rw-r--r-- 1 root root 4096 1970-01-01 00:11 ecc40000.etm/addr_start
> > > hikey960:/sys/bus/coresight/devices # echo d10583ff > ecc40000.etm/addr_start
> > > 1|hikey960:/sys/bus/coresight/devices # cat ecc40000.etm/addr_start
> > > cat: ecc40000.etm/addr_start: Operation not permitted
> > > ```
> >
> > After read the code, I think you need firstly to specify addr_idx:
> >
> >   echo 3 > ecc40000.etm/addr_idx
> >   echo d10583ff > ecc40000.etm/addr_start
> >   cat ecc40000.etm/addr_start
> >
> > I did as per your instruction. I was wondering if it is normal that once I
> > specified a value for addr_start, I am unable to insert a value into addr_stop.
> >
> > An example of the snippet is found here:
> >
> > hikey960: # echo 3 > addr_idx
> > hikey960: # cat addr*
> > addr_context:      0x0
> > addr_ctxtype:      none
> > addr_idx:          0x3
> > addr_instdatatype: instr
> > cat: addr_range:   Operation not permitted
> > addr_single:       0x0
> > addr_start:        0x0
> > addr_stop:         0x0
> >
> > hikey960: # echo 0x55555557b0 > addr_start
> > hikey960: # cat addr*
> > addr_context:      0x0
> > addr_ctxtype:      none
> > addr_idx:          0x3
> > addr_instdatatype:   instr
> > cat: addr_range:   Operation not permitted
> > addr_single:       Operation not permitted
> > addr_start:        0x55555557b0
> > addr_stop:         Operation not permitted
> >
> > > I thought it was not important that the Operation was 'not permitted'. Nonetheless, I continued with the tutorial as depicted [here](https://github.com/torvalds/linux/blob/master/Documentation/trace/coresight.txt). One thing I will like to note is that my `rwp` did move when i initially activated the sink and source. However, the 2nd time i checked the `rwp`, it went back `0x0`, the same position as that of my `rrp`.
> > >
> > > Regardless, I run the program in another kernel, performed CPU affinity to the program to point at the CPU in which the ETM is activated (in this case, CPU0). While the program is running, i looked at the `rwp/rrp` and see that neither of the pointers are moving. I thought perhaps if i perform `dd` operation, I might be able to get something.
> > >
> > > I performed `dd if=/dev/ec036000.etf of=/data/cs.bin bs=1M` but my device crashed and reboot itself without saving any files.
> >
> > Could you try below commands before dd command?
> >
> >   echo 1 > /sys/bus/coresight/devices/ec036000.etf
> >   # Enable ETM for CPU0
> >   echo 1 > /sys/bus/coresight/devices/ecc40000.etm/enable_source
> >   dd if=/dev/ec036000.etf of=/data/cs.bin bs=1M
> >
> > Please note, you need firstly configure the address range, and then
> > enable the ETM and sink.  If afterwards change the address range
> > again, you need disable etf/etm and re-enable etf/etm so can set
> > address range configuration into registers properly.
> >
> > I have also tried your above suggestion. Currently, I find
> > addr_start and addr_stop traces to better fit what I want rather
> > than addr_range. addr_start and addr_stop will be able to
> > capture traces of what the system perform in terms of context
> > switches, what syscalls it makes, etc.
> >
> > Regardless, I have tried addr_range and I was able to
> > acquire some traces from my program. I will like to share
> > with you some of my findings because I find the result of
> > the traces to be peculiar in what it has chosen to trace and
> > ignore.
> >
> > I have switched off ASLR, so I am under the assumption
> > that the instruction address from trace dumps should
> > be similar to that of GDB
> >
> >
> > === simple_math_function1.c ===
> > void math(int *i){
> >     (*i)++;
> >     (*i)++;
> >     (*i)++;
> >     (*i)++;
> > }
> >
> > main(int i){
> >     // many function calls, loop once
> >     while (i <= 20){
> >             math(&i);
> >             math(&i);
> >             math(&i);
> >             math(&i);
> >             math(&i);
> >             printf("%d\n",i);
> >     }
> > }
> > =======================
> >
> > this is a snippet of my decoded trace log:
> >     Address - Instruction address 0x0000005555555cf4, Instruction set Aarch32 (Thumb)
> >     Address - Instruction address 0x0000005555555cfc, Instruction set Aarch32 (Thumb)
> >     Address - Instruction address 0x0000005555555d04, Instruction set Aarch32 (Thumb)
> >     Address - Instruction address 0x0000005555555d0c, Instruction set Aarch32 (Thumb)
> >     Address - Instruction address 0x0000005555555d14, Instruction set Aarch32 (Thumb)
> >
> > compared against GDB:
> >     |0x5555555cf4 <main+972>         ldr    x0, [sp,#64]
> >     |0x5555555cf8 <main+976>         bl     0x55555558d8 <math>
> >     |0x5555555cfc <main+980>         ldr    x0, [sp,#64]
> >     |0x5555555d00 <main+984>         bl     0x55555558d8 <math>
> >     |0x5555555d04 <main+988>         ldr    x0, [sp,#64]
> >     |0x5555555d08 <main+992>         bl     0x55555558d8 <math>
> >     |0x5555555d0c <main+996>         ldr    x0, [sp,#64]
> >     |0x5555555d10 <main+1000>        bl     0x55555558d8 <math>
> >     |0x5555555d14 <main+1004>        ldr    x0, [sp,#64]
> >
>
> Please remember that the decode you are seeing is the decode of trace
> packets, not full trace decode.
> Addresses that appear in the packets represent addresses that cannot
> be deduced otherwise from analysing the program image.
> No other addresses will appear as part of the trace.
> Therefore the series of addresses that appear to refer to the LDR
> instructions above, tell the trace decoder where execution resumes
> after the return from the "math" routine.
> The BL instruction requires no address element in the trace as the
> branch target can be deduced directly from the instructions.

>

Thank you for explaining this to me. I thought it was rather peculiar and I have 

been reading the ETMv4 TRM over and over again but I could not understand how

the ETM was choosing what instructions to trace. 

Unfortunately, I have never done tracing before this and the 

learning curve might have been extraordinary for me at this stage.

> > comparing the gdb and trace dumps, it seems to only trace ldr and ignore bl instructions.
> >
> > this is the 2nd program, just a slight tweak from the above
> > program:
> >
> > === simple_math_function2.c ===
> > void math(int *i){
> >     (*i)++;
> >     (*i)++;
> >     (*i)++;
> >     (*i)++;
> > }
> >
> > main(int i){
> >     // 1 function call, looped many times
> >     while (i <= 20){
> >             math(&i);
> >             printf("%d\n",i);
> >     }
> > }
> > =======================
> > trace dump:
> >
> > Address - Instruction address 0x0000005555555b9c, Instruction set Aarch32 (Thumb)
> > ATOM - E
> > Commit - 1
> > Address - Instruction address 0x0000005555555b9c, Instruction set Aarch32 (Thumb)
> > ATOM - E
> > Commit - 1
> > Address - Instruction address 0x0000005555555b9c, Instruction set Aarch32 (Thumb)
> > ATOM - E
> > Commit - 1
> > Address - Instruction address 0x0000005555555b9c, Instruction set Aarch32 (Thumb)
> > ATOM - E
> > Commit - 1
> > Address - Instruction address 0x0000005555555b9c, Instruction set Aarch32 (Thumb)
> > ATOM - E
> > Commit - 1
> >
> > gdb log:
> >
> > |0x5555555b94 <main+620> ldr    x0, [sp,#64]
> > |0x5555555b98 <main+624> bl     0x55555558d8 <math>
> > |0x5555555b9c <main+628> b      0x5555555b78 <main+592>
> > |0x5555555ba0 <main+632> ldr    x8, [sp,#64]
> >
>
> Again the 0x0000005555555b9c addresses represent the return from the
> math routine, and not trace for the b  0x5555555b78 <main+592>
> instruction
> This b 0x0x5555555b78 is likely represented by the E atom
>
> There is some concern regarding the classification of the AArch32 code

> as thumb - it would seem from the assembly code that the

What this should have said was that the assembly code appears to be

AArch64 code - registers Xn are an indicator of this. However the
decoder you are using appears to be classifying that addresses as
AArch32 & Thumb.
This could be an issue with the decoder - how are you compiling the

code? Is it AArch64?

Yes, it is actually AArch64. I cross-compiled my programs with android-ndk-20r.

I was puzzled with the decoder and I have had my

suspicions on the decoding tool as well. I am recently looking closer at the 

source code of ptm2human in case it might have wrongly classified my trace packets.

> These relationships are explained in the ETMv4 TRM - Appendix A,
> examples of trace.
>
> Regards
>
>
> Mike
>
>
>
> > This time, i notice that the etm only trace the `b`
> > instructions but not the `ldr` instructions.
> >
> > I am rather puzzled by the outcome of the traces as I could not understand why the result might be different for a somewhat similar code...
> >
> > I don't try at my side, if have any question or issue, please report
> > back so I can test on Hikey960.
> >
> > Thanks,
> > Leo Yan
> >
> > Thank you very much for your kindness and assistance the
> > last time!
> >
> > Also, one last question: I will like to ask how I might be
> > able to activate global timestamp for instruction trace? i
> > noticed trcidr0 supports timestamp (tssize of 64 bits).
> > However, trcconfigr have disabled all my options. Is there
> > a workaround this?
> >
> > Since perf is still not supported in android, i have been
> > using ptm2human to decode etm dumps. I am also
> > wondering if there are other decoders that I have not
> > explored? From what I have seen, DS-5 seems to request
> > DSTREAM to activate ETM trace and decode the trace
> > dumps. I tried to do it without the DSTREAM and DS-5 was
> > unable to recognise the trace dumps.
> >

Have you looked at OpenCSD (https://github.com/Linaro/OpenCSD) ?
This is a trace decode library that will do full decode as well as the
packet decode provided by ptm2human.


Regards

Mike

I have taken a peek at OpenCSD previously. But there seems to be more documentations

on using OpenCSD integrated with perf than using the libraries standalone.

I might also have confused myself when I saw that the trace decoder will not work

if OpenCSD is not integrated with perf from a conference slide made by Mathieu Poirier.

I have attached an image grab from the slide with the line highlighted for easy reference.

reference: https://elinux.org/images/b/b3/Hardware_Assisted_Tracing_on_ARM.pdf

I will love to explore OpenCSD as well. I tinkered it slightly this 

morning and it seems that a snapshot of the device is 

required. I am still trying to understand how I might be able

to trace and decode with OpenCSD and also how I might be 

able to obtain a snapshot of my device.

With ptm2human, I am merely extracting the etm dumps 

from the sinks by using the dd command, extract the trace dump 

from target device (hikey960 - android) to host device (linux-x86_64) 

and decode the binary files into a decoded trace file.

> > Any help will be greatly appreciated!
> >
> > Sincerely,
> > Jeremy
> >
> > The above message may contain confidential and/or proprietary information that is exempt from disclosure under applicable law and is intended for receipt and use solely by the addressee(s) named above. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this message, or any attachment, is strictly prohibited. If you have received this email in error, please inform the sender immediately by reply e-mail or telephone, reversing the charge if necessary. Please delete the message thereafter. Thank you.
> >
> > _______________________________________________
> > CoreSight mailing list
> > CoreSight@lists.linaro.org
> > https://lists.linaro.org/mailman/listinfo/coresight
>
>
>
> --
> Mike Leach
> Principal Engineer, ARM Ltd.
> Manchester Design Centre. UK



--
Mike Leach
Principal Engineer, ARM Ltd.

Manchester Design Centre. UK

I will like to also ask how I might be able to change some trcconfig settings.

it is found in *.etm/mgmt/trcconfig and it is a read-only file.

I have tried to change the settings through memory mapping 

and searched for other ways through tweaking with the driver 

files in drivers/hwtracing/coresight/coresight-etm4x.c

I have been wanting to activate timestamps and/or branch broadcast.

In all honesty, I am still trying to understand what exactly branch broadcast

is as well.

Really sorry to have bothered you with mediocre and amateurish questions.

Yours Sincerely,

Jeremy

The above message may contain confidential and/or proprietary information that is exempt from disclosure under applicable law and is intended for receipt and use solely by the addressee(s) named above. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this message, or any attachment, is strictly prohibited. If you have received this email in error, please inform the sender immediately by reply e-mail or telephone, reversing the charge if necessary. Please delete the message thereafter. Thank you.