...

if it's return branch instruction (OCSD_S_INSTR_V8_RET), then it's return instruction;

True for v8 - but other ISAs (A32 / T32) there is no explicit marking of "ret" - these can be simply mov PC. [rX] or any other instruction that updates the PC.

In A32 / T32, we define some specific instructions as being returns for the purpose of incrementing the return perf event (0x0E, perf stat -e re):

- BX R14 - MOV PC, LR - POP {…, PC} - LDR PC, [SP], #offset

Generally, CPU implementations will take this as the list of instructions that should use a special stack-based return predictor instead of the regular branch predictor. This in turn means that compilers are careful to use these instructions when they want a return and some other instruction when they want some other indirect branch.

So this is the appropriate list to flag as returns in A32 / T32. Hopefully you then find the count of returns in the trace matches what's counted by perf stat -e re.

Al IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Hi Mike,

On Wed, Oct 03, 2018 at 04:04:26PM +0100, Mike Leach wrote:

[...]

...

I did some investigation for these, CoreSight can set partial sample flags for A64 branch instructions with packet infos:

• ocsd_generic_trace_elem::last_i_type can be used to check if the instruction is immediate branch instruction or indirect branch instruction;

• ocsd_generic_trace_elem::last_i_subtype can be used to distinguish if it's link branch instruction (OCSD_S_INSTR_BR_LINK), if it's link branch instruction then this means this branch insturction is for function call;

A branch + link will be a function call, but all function calls do not have to be br+link.

Yeah, so just want to confirm how we can distinguish the function call for 'b func_name'? E.g. I checked a bit in ARM-TF, there have some function calls are using 'b func_name' for function call and don't need to jump back to caller [1].

Do I miss any other types of function call? I checked doc 'ARMv8 Instruction Set Overview', program counter cannot be directly changed for function call in ARMv8.

...

if it's return branch instruction (OCSD_S_INSTR_V8_RET), then it's return instruction;

True for v8 - but other ISAs (A32 / T32) there is no explicit marking of "ret" - these can be simply mov PC. [rX] or any other instruction that updates the PC.

Yeah, we will discuss in another email for this.

...

Otherwise (OCSD_S_INSTR_NONE), it's a simple branch instruction within the function.

No - this indicates the end of a trace range that is _not_ a branch. Could happen prior to an exception or simply due to trace filtering.

Thanks for pointing out this error, will fix.

...

But there still have several things are not clear for me:

• How can we distinguish the exception is for system call, or interrupt. The reason is we can see Intel-pt set sample flags for different values for different exception types:

  PERF_IP_FLAG_INTERRUPT PERF_IP_FLAG_SYSCALLRET

  I think we might can use ocsd_generic_trace_elem::exception_number to distinguish the different exception types, but I don't find any doc to clear define for this value.

The ETMv4 specification defines exception numbers - these are simply passed through directly to the output elements. So with this information the exception types flags could be set.

Let me check more detailed info for this, especially after looked into in ETMv4 spec and in the OpenCSD code.

In OpenCSD lib it has arraies to present the exception type [2], this array is consistent with ETMv4 spec:

static const char *ARv8Excep[] = { "PE Reset", "Debug Halt", "Call", "Trap", "System Error", "Reserved", "Inst Debug", "Data Debug", "Reserved", "Reserved", "Alignment", "Inst Fault", "Data Fault", "Reserved", "IRQ", "FIQ" };

But I'm not sure if the ocsd_generic_trace_elem::exception_number has the consistent meaning with upper array; e.g. if exception_number = 14, means the exception is an IRQ; I tried to dump the log, I even found there have logs including exception_number = 15, if connecting with upper array, this exception is FIQ but this doesn't make sense for me due I don't enable Trustzone related features on system so the system should have no user for FIQ.

[...]

Thanks, Leo Yan

[1] https://github.com/ARM-software/arm-trusted-firmware/blob/master/bl1/aarch64... [2] https://github.com/Linaro/OpenCSD/blob/master/decoder/source/etmv4/trc_pkt_e...

On Mon, Oct 08, 2018 at 09:49:34AM +0100, Mike Leach wrote:

[...]

...

...

...

• ocsd_generic_trace_elem::last_i_subtype can be used to distinguish if it's link branch instruction (OCSD_S_INSTR_BR_LINK), if it's link branch instruction then this means this branch insturction is for function call;

A branch + link will be a function call, but all function calls do not have to be br+link.

Yeah, so just want to confirm how we can distinguish the function call for 'b func_name'? E.g. I checked a bit in ARM-TF, there have some function calls are using 'b func_name' for function call and don't need to jump back to caller [1].

Do I miss any other types of function call? I checked doc 'ARMv8 Instruction Set Overview', program counter cannot be directly changed for function call in ARMv8.

For v8 that is true. Not so sure for v7. I think you can assume that any b+link is a function call. For B <address> and B.<cond> <address> it is impossible to be certain from just the trace data - these could be function calls or simply jumps for loop operations in a single function. You can either assume all of these are not function calls, or use other information (e.g. symbol infomation ) from the source files.

Thanks for suggestion, using symbol info will be more accurate, will look into this.

[...]

...

...

...

I think we might can use ocsd_generic_trace_elem::exception_number to distinguish the different exception types, but I don't find any doc to clear define for this value.

The ETMv4 specification defines exception numbers - these are simply passed through directly to the output elements. So with this information the exception types flags could be set.

Let me check more detailed info for this, especially after looked into in ETMv4 spec and in the OpenCSD code.

In OpenCSD lib it has arraies to present the exception type [2], this array is consistent with ETMv4 spec:

 static const char *ARv8Excep[] = {
     "PE Reset", "Debug Halt", "Call", "Trap",
     "System Error", "Reserved", "Inst Debug", "Data Debug",
     "Reserved", "Reserved", "Alignment", "Inst Fault",
     "Data Fault", "Reserved", "IRQ", "FIQ"
 };

But I'm not sure if the ocsd_generic_trace_elem::exception_number has the consistent meaning with upper array; e.g. if exception_number = 14, means the exception is an IRQ; I tried to dump the log, I even found there have logs including exception_number = 15, if connecting with upper array, this exception is FIQ but this doesn't make sense for me due I don't enable Trustzone related features on system so the system should have no user for FIQ.

Looking at the examples supplied with the library, I see 0x2 (call), 0xc (data fault) and 0xe (IRQ) exception types. I have checked the code in the decoder and cannot see anything that would indicate this is not operating correctly. Therefore it would suggest that if you are seeing FIQ, these are really happening, or the decode is not correctly synchronised to the start of the trace data. If not synchronised then I would expect to see lots of errors, not just unexpected FIQs.

At my side, I do see the exception number is mess from the dumped elements as below; you also could check complete dump log from [1], which is a big size log file (~764MB).

elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=4 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=9 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=1 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=15 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=7 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=7 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0

[1] http://people.linaro.org/~leo.yan/opencsd_juno/elem_dump_for_exception_numbe...

Thanks, Leo Yan

On Mon, Oct 08, 2018 at 02:15:35PM +0100, Mike Leach wrote:

[...]

...

At my side, I do see the exception number is mess from the dumped elements as below; you also could check complete dump log from [1], which is a big size log file (~764MB).

elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=6 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=4 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=0 elem: elem_type=9 last_i_type=0 last_i_subtype=0 exception_number=0 exc=1 exc_ret=1 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=15 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=7 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=33 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=7 exc=0 exc_ret=0 elem: elem_type=5 last_i_type=1 last_i_subtype=0 exception_number=13 exc=0 exc_ret=0

Exception number in the packet is only valid if the element type is an exception. It is part of this structure within the input packet:-

typedef struct _ocsd_generic_trace_elem { ... //! packet specific payloads union { uint32_t exception_number; /**< exception number for exception type packets */ trace_event_t trace_event; /**< Trace event - trigger etc */ trace_on_reason_t trace_on_reason; /**< reason for the trace on packet */ ocsd_swt_info_t sw_trace_info; /**< software trace packet info */ uint32_t num_instr_range; /**< number of instructions covered by range packet (for T32 this cannot be calculated from en-st/i_size) */ }; ... }

This reduces the bytes needed for the output packet by overlapping mutually exclusive features.

Ah, sorry for noise. I go through all elem_type=8, then I can see exception numbers are (2, 11, 14). Exception 11 ("Inst Fault") also is reasonable for me if connect with kernel's insn_emulation module.

leoy@leoy-ThinkPad-X240s:~$ grep elem_type=8 /tmp/out2.log | sort -t' ' -u elem: elem_type=8 last_i_type=0 last_i_subtype=0 exception_number=14 exc=1 exc_ret=0 elem: elem_type=8 last_i_type=0 last_i_subtype=0 exception_number=2 exc=1 exc_ret=0 elem: elem_type=8 last_i_type=1 last_i_subtype=0 exception_number=14 exc=1 exc_ret=0 elem: elem_type=8 last_i_type=1 last_i_subtype=1 exception_number=11 exc=1 exc_ret=0

Thanks a lot for helping, will firstly prepare patches for A64 related fixing based on the discussion.

Thanks, Leo Yan