Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Thierry,
One way of getting packet dump If you have built the full OpenCSD project is to use the trc_pkt_lister(.exe) program in your decoder\tests\bin<platform><rel|dbg> directory. This operates on trace snapshot directories - these contain trace binary data and a bunch of .ini files describing the trace configuration - see tests\snapshots\juno_r1_1 for an example.
Create a directory for your cstrace.bin - copy all the .ini files from the above mentioned Juno example into the same directory. Run trc_pkt_lister -ss_dir <your_data_directory> and you should get output as follows:-
================================================================================ Trace Packet Lister: CS Decode library testing -----------------------------------------------
** Library Version : 0.7.0
Test Command Line:- trc_pkt_lister.exe -ss_dir c:\work\ocsd-issues\thierry
Trace Packet Lister : reading snapshot from path c:\work\ocsd-issues\thierry Using ETB_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x11 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x13 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x15 Idx:0; ID:12;I_ASYNC : Alignment Synchronisation. Idx:12; ID:12;I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:12;I_TRACE_ON : Trace On. Idx:18; ID:12;I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000089071EC; Ctxt: AArch64,EL1, NS; Idx:28; ID:12;I_ATOM_F2 : Atom format 2.; NE Idx:29; ID:12;I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF00000890C72C; Idx:39; ID:12;I_ATOM_F2 : Atom format 2.; EE Idx:40; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABDC; Idx:45; ID:12;I_ATOM_F4 : Atom format 4.; NENE Idx:46; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:48; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:49; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B7E14; Idx:54; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:55; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:56; ID:12;I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7E24 ~[0x24] Idx:58; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:59; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABC8; Idx:65; ID:12;I_ATOM_F1 : Atom format 1.; E ================================================================================
That's just a small excerpt from the start of the packet decode.
This works because your TRACEIDR value matches one of the TRACEIDR values in the ETMv4 metadata files from the original example - see device_6.ini to device_11.ini - and the name of your data file is cstrace.bin - see trace.ini
The full snapshot directory format is described by the document in decoder\docs\specs
Regards
Mike
-----Original Message----- From: Thierry Laviron Sent: 07 July 2017 15:12 To: Mike Leach Mike.Leach@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the
disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Thanks Mike, that's exactly what I was looking for.
With this I should be able to compare the sysfs trace output with the perf trace output, to make sure they are (mostly) tracing the same thing.
I'll let you know if I run into any other issues.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 10 July 2017 12:00 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org; Al Grant Al.Grant@arm.com Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
One way of getting packet dump If you have built the full OpenCSD project is to use the trc_pkt_lister(.exe) program in your decoder\tests\bin<platform><rel|dbg> directory. This operates on trace snapshot directories - these contain trace binary data and a bunch of .ini files describing the trace configuration - see tests\snapshots\juno_r1_1 for an example.
Create a directory for your cstrace.bin - copy all the .ini files from the above mentioned Juno example into the same directory. Run trc_pkt_lister -ss_dir <your_data_directory> and you should get output as follows:-
================================================================================ Trace Packet Lister: CS Decode library testing -----------------------------------------------
** Library Version : 0.7.0
Test Command Line:- trc_pkt_lister.exe -ss_dir c:\work\ocsd-issues\thierry
Trace Packet Lister : reading snapshot from path c:\work\ocsd-issues\thierry Using ETB_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x11 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x13 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x15 Idx:0; ID:12;I_ASYNC : Alignment Synchronisation. Idx:12; ID:12;I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:12;I_TRACE_ON : Trace On. Idx:18; ID:12;I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000089071EC; Ctxt: AArch64,EL1, NS; Idx:28; ID:12;I_ATOM_F2 : Atom format 2.; NE Idx:29; ID:12;I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF00000890C72C; Idx:39; ID:12;I_ATOM_F2 : Atom format 2.; EE Idx:40; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABDC; Idx:45; ID:12;I_ATOM_F4 : Atom format 4.; NENE Idx:46; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:48; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:49; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B7E14; Idx:54; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:55; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:56; ID:12;I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7E24 ~[0x24] Idx:58; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:59; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABC8; Idx:65; ID:12;I_ATOM_F1 : Atom format 1.; E ================================================================================
That's just a small excerpt from the start of the packet decode.
This works because your TRACEIDR value matches one of the TRACEIDR values in the ETMv4 metadata files from the original example - see device_6.ini to device_11.ini - and the name of your data file is cstrace.bin - see trace.ini
The full snapshot directory format is described by the document in decoder\docs\specs
Regards
Mike
-----Original Message----- From: Thierry Laviron Sent: 07 July 2017 15:12 To: Mike Leach Mike.Leach@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the
disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hello again,
I have been successfully decoding trace produced by the ETM on one of the big core, but for some reason the same process does not work when I try it on the little cores.
I get something like the following: Trace Packet Lister: CS Decode library testing -----------------------------------------------
** Library Version : 0.6.0
Test Command Line:- ./trc_pkt_lister -ss_dir /home/thierry/trace_decode/
Trace Packet Lister : reading snapshot from path /home/thierry/trace_decode/ Using ETR_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x16 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x18 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x1a Idx:0; ID:18; I_ASYNC : Alignment Synchronisation. Idx:12; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000000 00000; Idx:26; ID:18; I_ASYNC : Alignment Synchronisation. Idx:39; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:42; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000040 2A6E4; Idx:52; ID:18; I_TRACE_ON : Trace On. Idx:53; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF000008A6A5E8; Ctxt: AArch64,EL1, NS; Idx:64; ID:18; I_ATOM_F1 : Atom format 1.; E Idx:65; ID:18; I_TRACE_ON : Trace On. Idx:66; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF0000080F91E4; Ctxt: AArch64,EL1, NS; ... Idx:513971; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513972; ID:18; I_TRACE_ON : Trace On. Idx:513973; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF000008A65714; Ctxt: AArch64,EL1, NS; Idx:513984; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513985; ID:18; I_ASYNC : Alignment Synchronisation. Idx:513997; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 ID:10 END OF TRACE DATA ID:12 END OF TRACE DATA ID:14 END OF TRACE DATA ID:16 END OF TRACE DATA Idx:514001; ID:18; I_INCOMPLETE_EOT : Incomplete packet at end of trace.[I_ADDR_L_64IS0] ID:18 END OF TRACE DATA ID:1a END OF TRACE DATA Trace Packet Lister : Trace buffer done, processed 514032 bytes.
In the middle there is mostly a stream of Address w/ context, atom, and trace on packets (with a few variations in the addresses and atoms).
Also, when afterwards I try to read the value of the TRCPDSR (e.g., with 'less'), the kernel reports an unhandled fault (see attached log).
I tried to see what were the differences between an ETM attached to a big core and one attached to a little one, to see if the problem could come from here.
Here are the differences between sysfs entries: NameValue, big core (22040000.etm)Value, little core (23140000.etm) cpu13 mgmt/trcpidr00x5a0x5d mgmt/trcpidr20xb0x3b nr_ext_inp0x6e0x1e trcidr/trcidr10x4100f4000x4100f403 trcidr/trcidr30x17b00040xd7b0004 trcidr/trcidr50x2847086e0x28c7081e
I looked up the corresponding registers description, but do not know if a different set of those values could cause such an issue.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 10 July 2017 12:00 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org; Al Grant Al.Grant@arm.com Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
One way of getting packet dump If you have built the full OpenCSD project is to use the trc_pkt_lister(.exe) program in your decoder\tests\bin<platform><rel|dbg> directory. This operates on trace snapshot directories - these contain trace binary data and a bunch of .ini files describing the trace configuration - see tests\snapshots\juno_r1_1 for an example.
Create a directory for your cstrace.bin - copy all the .ini files from the above mentioned Juno example into the same directory. Run trc_pkt_lister -ss_dir <your_data_directory> and you should get output as follows:-
================================================================================ Trace Packet Lister: CS Decode library testing -----------------------------------------------
** Library Version : 0.7.0
Test Command Line:- trc_pkt_lister.exe -ss_dir c:\work\ocsd-issues\thierry
Trace Packet Lister : reading snapshot from path c:\work\ocsd-issues\thierry Using ETB_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x11 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x13 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x15 Idx:0; ID:12;I_ASYNC : Alignment Synchronisation. Idx:12; ID:12;I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:12;I_TRACE_ON : Trace On. Idx:18; ID:12;I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000089071EC; Ctxt: AArch64,EL1, NS; Idx:28; ID:12;I_ATOM_F2 : Atom format 2.; NE Idx:29; ID:12;I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF00000890C72C; Idx:39; ID:12;I_ATOM_F2 : Atom format 2.; EE Idx:40; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABDC; Idx:45; ID:12;I_ATOM_F4 : Atom format 4.; NENE Idx:46; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:48; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:49; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B7E14; Idx:54; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:55; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:56; ID:12;I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7E24 ~[0x24] Idx:58; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:59; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABC8; Idx:65; ID:12;I_ATOM_F1 : Atom format 1.; E ================================================================================
That's just a small excerpt from the start of the packet decode.
This works because your TRACEIDR value matches one of the TRACEIDR values in the ETMv4 metadata files from the original example - see device_6.ini to device_11.ini - and the name of your data file is cstrace.bin - see trace.ini
The full snapshot directory format is described by the document in decoder\docs\specs
Regards
Mike
-----Original Message----- From: Thierry Laviron Sent: 07 July 2017 15:12 To: Mike Leach Mike.Leach@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the
disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Thierry,
On 13 July 2017 at 12:42, Thierry Laviron Thierry.Laviron@arm.com wrote:
Hello again,
I have been successfully decoding trace produced by the ETM on one of the big core, but for some reason the same process does not work when I try it on the little cores.
I get something like the following: Trace Packet Lister: CS Decode library testing
** Library Version : 0.6.0
Test Command Line:- ./trc_pkt_lister -ss_dir /home/thierry/trace_decode/
Trace Packet Lister : reading snapshot from path /home/thierry/trace_decode/ Using ETR_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x16 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x18 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x1a Idx:0; ID:18; I_ASYNC : Alignment Synchronisation. Idx:12; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000000 00000; Idx:26; ID:18; I_ASYNC : Alignment Synchronisation. Idx:39; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:42; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000040 2A6E4; Idx:52; ID:18; I_TRACE_ON : Trace On. Idx:53; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF000008A6A5E8; Ctxt: AArch64,EL1, NS; Idx:64; ID:18; I_ATOM_F1 : Atom format 1.; E Idx:65; ID:18; I_TRACE_ON : Trace On. Idx:66; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF0000080F91E4; Ctxt: AArch64,EL1, NS; ... Idx:513971; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513972; ID:18; I_TRACE_ON : Trace On. Idx:513973; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF000008A65714; Ctxt: AArch64,EL1, NS; Idx:513984; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513985; ID:18; I_ASYNC : Alignment Synchronisation. Idx:513997; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 ID:10 END OF TRACE DATA ID:12 END OF TRACE DATA ID:14 END OF TRACE DATA ID:16 END OF TRACE DATA Idx:514001; ID:18; I_INCOMPLETE_EOT : Incomplete packet at end of trace.[I_ADDR_L_64IS0] ID:18 END OF TRACE DATA ID:1a END OF TRACE DATA Trace Packet Lister : Trace buffer done, processed 514032 bytes.
In the middle there is mostly a stream of Address w/ context, atom, and trace on packets (with a few variations in the addresses and atoms).
OK - I'm not seeing any problem here - what makes you think that decode is not working. You should know that we have successfully decoded trace from all the cores on juno in the past.
Also, when afterwards I try to read the value of the TRCPDSR (e.g., with 'less'), the kernel reports an unhandled fault (see attached log).
If you are using sysfs be careful to only access etms for cores that are actually powered up. In the Juno system, powering down a core will power down the relevent debug and cause a fault if you try to access it. Disable cpu_idle in this case.
Mike
I tried to see what were the differences between an ETM attached to a big core and one attached to a little one, to see if the problem could come from here.
Here are the differences between sysfs entries: NameValue, big core (22040000.etm)Value, little core (23140000.etm) cpu13 mgmt/trcpidr00x5a0x5d mgmt/trcpidr20xb0x3b nr_ext_inp0x6e0x1e trcidr/trcidr10x4100f4000x4100f403 trcidr/trcidr30x17b00040xd7b0004 trcidr/trcidr50x2847086e0x28c7081e
I looked up the corresponding registers description, but do not know if a different set of those values could cause such an issue.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 10 July 2017 12:00 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org; Al Grant Al.Grant@arm.com Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
One way of getting packet dump If you have built the full OpenCSD project is to use the trc_pkt_lister(.exe) program in your decoder\tests\bin<platform><rel|dbg> directory. This operates on trace snapshot directories - these contain trace binary data and a bunch of .ini files describing the trace configuration - see tests\snapshots\juno_r1_1 for an example.
Create a directory for your cstrace.bin - copy all the .ini files from the above mentioned Juno example into the same directory. Run trc_pkt_lister -ss_dir <your_data_directory> and you should get output as follows:-
================================================================================ Trace Packet Lister: CS Decode library testing
** Library Version : 0.7.0
Test Command Line:- trc_pkt_lister.exe -ss_dir c:\work\ocsd-issues\thierry
Trace Packet Lister : reading snapshot from path c:\work\ocsd-issues\thierry Using ETB_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x11 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x13 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x15 Idx:0; ID:12;I_ASYNC : Alignment Synchronisation. Idx:12; ID:12;I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:12;I_TRACE_ON : Trace On. Idx:18; ID:12;I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000089071EC; Ctxt: AArch64,EL1, NS; Idx:28; ID:12;I_ATOM_F2 : Atom format 2.; NE Idx:29; ID:12;I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF00000890C72C; Idx:39; ID:12;I_ATOM_F2 : Atom format 2.; EE Idx:40; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABDC; Idx:45; ID:12;I_ATOM_F4 : Atom format 4.; NENE Idx:46; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:48; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:49; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B7E14; Idx:54; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:55; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:56; ID:12;I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7E24 ~[0x24] Idx:58; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:59; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABC8; Idx:65; ID:12;I_ATOM_F1 : Atom format 1.; E ================================================================================
That's just a small excerpt from the start of the packet decode.
This works because your TRACEIDR value matches one of the TRACEIDR values in the ETMv4 metadata files from the original example - see device_6.ini to device_11.ini - and the name of your data file is cstrace.bin - see trace.ini
The full snapshot directory format is described by the document in decoder\docs\specs
Regards
Mike
-----Original Message----- From: Thierry Laviron Sent: 07 July 2017 15:12 To: Mike Leach Mike.Leach@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the
disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Hi Mike,
On further investigation, it appears my issue was related with my use of the context ID comparator: I was selecting it with a resource selector, for the ViewInst Enabling event. Apparently, as the enabling event is imprecise, this might have the effect that tracing would turn on and off repeatedly, and the reason I was not observing this on the big core would be that they have a slightly different implementation. The correct way to use the context ID comparator, I have been told, would be to associate it with address range comparators directly, which I assume is done using 'addr_context' and 'addr_ctxtype' in the sysFS interface. I am currently looking at the driver code for the ETM to understand how these are used.
Thanks for your help.
Best regards,
Thierry.
-----Original Message----- From: Mike Leach [mailto:mike.leach@linaro.org] Sent: 13 July 2017 16:31 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mike Leach Mike.Leach@arm.com; coresight@lists.linaro.org Subject: Re: Using Coresight in SysFS mode on Juno board
Hi Thierry,
On 13 July 2017 at 12:42, Thierry Laviron Thierry.Laviron@arm.com wrote:
Hello again,
I have been successfully decoding trace produced by the ETM on one of the big core, but for some reason the same process does not work when I try it on the little cores.
I get something like the following: Trace Packet Lister: CS Decode library testing
** Library Version : 0.6.0
Test Command Line:- ./trc_pkt_lister -ss_dir /home/thierry/trace_decode/
Trace Packet Lister : reading snapshot from path /home/thierry/trace_decode/ Using ETR_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x16 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x18 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x1a Idx:0; ID:18; I_ASYNC : Alignment Synchronisation. Idx:12; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000000 00000; Idx:26; ID:18; I_ASYNC : Alignment Synchronisation. Idx:39; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:42; ID:18; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x00000000040 2A6E4; Idx:52; ID:18; I_TRACE_ON : Trace On. Idx:53; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF000008A6A5E8; Ctxt: AArch64,EL1, NS; Idx:64; ID:18; I_ATOM_F1 : Atom format 1.; E Idx:65; ID:18; I_TRACE_ON : Trace On. Idx:66; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Add r=0xFFFF0000080F91E4; Ctxt: AArch64,EL1, NS; ... Idx:513971; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513972; ID:18; I_TRACE_ON : Trace On. Idx:513973; ID:18; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF000008A65714; Ctxt: AArch64,EL1, NS; Idx:513984; ID:18; I_ATOM_F1 : Atom format 1.; N Idx:513985; ID:18; I_ASYNC : Alignment Synchronisation. Idx:513997; ID:18; I_TRACE_INFO : Trace Info.; INFO=0x0 ID:10 END OF TRACE DATA ID:12 END OF TRACE DATA ID:14 END OF TRACE DATA ID:16 END OF TRACE DATA Idx:514001; ID:18; I_INCOMPLETE_EOT : Incomplete packet at end of trace.[I_ADDR_L_64IS0] ID:18 END OF TRACE DATA ID:1a END OF TRACE DATA Trace Packet Lister : Trace buffer done, processed 514032 bytes.
In the middle there is mostly a stream of Address w/ context, atom, and trace on packets (with a few variations in the addresses and atoms).
OK - I'm not seeing any problem here - what makes you think that decode is not working. You should know that we have successfully decoded trace from all the cores on juno in the past.
Also, when afterwards I try to read the value of the TRCPDSR (e.g., with 'less'), the kernel reports an unhandled fault (see attached log).
If you are using sysfs be careful to only access etms for cores that are actually powered up. In the Juno system, powering down a core will power down the relevent debug and cause a fault if you try to access it. Disable cpu_idle in this case.
Mike
I tried to see what were the differences between an ETM attached to a big core and one attached to a little one, to see if the problem could come from here.
Here are the differences between sysfs entries: NameValue, big core (22040000.etm)Value, little core (23140000.etm) cpu13 mgmt/trcpidr00x5a0x5d mgmt/trcpidr20xb0x3b nr_ext_inp0x6e0x1e trcidr/trcidr10x4100f4000x4100f403 trcidr/trcidr30x17b00040xd7b0004 trcidr/trcidr50x2847086e0x28c7081e
I looked up the corresponding registers description, but do not know if a different set of those values could cause such an issue.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 10 July 2017 12:00 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org; Al Grant Al.Grant@arm.com Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
One way of getting packet dump If you have built the full OpenCSD project is to use the trc_pkt_lister(.exe) program in your decoder\tests\bin<platform><rel|dbg> directory. This operates on trace snapshot directories - these contain trace binary data and a bunch of .ini files describing the trace configuration - see tests\snapshots\juno_r1_1 for an example.
Create a directory for your cstrace.bin - copy all the .ini files from the above mentioned Juno example into the same directory. Run trc_pkt_lister -ss_dir <your_data_directory> and you should get output as follows:-
====================================================================== ========== Trace Packet Lister: CS Decode library testing
** Library Version : 0.7.0
Test Command Line:- trc_pkt_lister.exe -ss_dir c:\work\ocsd-issues\thierry
Trace Packet Lister : reading snapshot from path c:\work\ocsd-issues\thierry Using ETB_0 as trace source Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x10 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x11 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x12 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x13 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x14 Trace Packet Lister : Protocol printer ETMV4I on Trace ID 0x15 Idx:0; ID:12;I_ASYNC : Alignment Synchronisation. Idx:12; ID:12;I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:17; ID:12;I_TRACE_ON : Trace On. Idx:18; ID:12;I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000089071EC; Ctxt: AArch64,EL1, NS; Idx:28; ID:12;I_ATOM_F2 : Atom format 2.; NE Idx:29; ID:12;I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF00000890C72C; Idx:39; ID:12;I_ATOM_F2 : Atom format 2.; EE Idx:40; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABDC; Idx:45; ID:12;I_ATOM_F4 : Atom format 4.; NENE Idx:46; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:48; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:49; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B7E14; Idx:54; ID:12;I_ATOM_F3 : Atom format 3.; ENE Idx:55; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:56; ID:12;I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7E24 ~[0x24] Idx:58; ID:12;I_ATOM_F1 : Atom format 1.; E Idx:59; ID:12;I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x0815ABC8; Idx:65; ID:12;I_ATOM_F1 : Atom format 1.; E ====================================================================== ==========
That's just a small excerpt from the start of the packet decode.
This works because your TRACEIDR value matches one of the TRACEIDR values in the ETMv4 metadata files from the original example - see device_6.ini to device_11.ini - and the name of your data file is cstrace.bin - see trace.ini
The full snapshot directory format is described by the document in decoder\docs\specs
Regards
Mike
-----Original Message----- From: Thierry Laviron Sent: 07 July 2017 15:12 To: Mike Leach Mike.Leach@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi,
I thought about my initial question, and I think this current limitation is not an issue for my purposes because I am trying to evaluate the effect on performance of turning trace on, both with sysFS and with perf. This means I am not that interested about the trace data itself, but for my measurements to make sense I have to be sure that I am tracing the correct program.
So I have been running short programs, that should generate a small enough amount of trace that it fits in the ETR buffer. The problem is, when I look at the raw trace data and try to decode it manually using the ETM Architecture Spec, the data quickly stops making sense.
I have attached an example trace data that I generated, (and the script I am using to run it/ configure the CoreSight components). If I follow the description of the encoding of Instruction trace packets, this sample starts with the following packets: At 0x0, a Numbered Data Sync Mark (numbered 5) At 0x1, an A-Sync packet At 0xd, a Trace Info packet. The next byte is 0, so that means there is no more info after that At 0xf, a context packet, with no payload Which means that starting at 0x10, I have an undefined sequence: 00 04 is not one of the known extension packets.
I have the feeling that there is more than one instruction stream in this file, because at 0x1218 I can see a sequence that looks at lot like an A-Sync packet with a 0e byte mixed in.
As I have checked that other ETMs and the STM all have 'enable_source' set to 0, I don't know where do these stray bytes come from.
Is there something that I am missing? Is this behaviour expected?
Additionally, is there an easy way to decode the trace from the raw binary data? I do have the OpenCSD library installed, and perf uses it when displaying ETM trace data. The documentation mentions a 'ptm2human' program, but I haven't found it (and don't know if it is still up to date). This last part is not vital, but if there is a simple wrapper for the decoder around, it would make checking the validity of the trace easier.
Best regards,
Thierry
-----Original Message----- From: Mike Leach Sent: 29 June 2017 17:17 To: Thierry Laviron Thierry.Laviron@arm.com Cc: Mathieu Poirier (mathieu.poirier@linaro.org) mathieu.poirier@linaro.org; 'coresight@lists.linaro.org' coresight@lists.linaro.org Subject: RE: Using Coresight in SysFS mode on Juno board
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
-----Original Message----- From: Thierry Laviron Sent: 29 June 2017 15:45 To: Mike Leach Subject: Using Coresight in SysFS mode on Juno board
Hi Mike,
I am currently trying to get trace data using the CoreSight system in SysFS mode on my Juno r2 board.
I found some documentation on how to use it in the Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the OpenCSD repository.
This document says that I can retrieve the trace data from /dev/ using dd, for example in my case that would be
root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
However, I am assuming this produces a dump of the memory buffer as it was when I stopped trace collection,
And that I do not have the full trace data generated (because it does not fit on the buffer).
I would like to be able to capture a continuous stream of data from the ETR, but did not find how should I do that.
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
I am writing a C program. Can I open a read access to the ETR buffer like this?
open("/dev/20070000.etr", O_RDONLY);and then read its content, to write somewhere else? (e.g. to a file on the
disc)?
As a second step, I am also trying to filter the trace generated. I found some useful documentation in
Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
However, while this is very useful to understand what are the purpose of the different files that appear in the
/sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format to put stuff in.
For example, I want to use the Context ID comparator, so the ETM traces only the process I am interested in.
I assume I need to write the PID of my process in ctxid_pid, probably write 0x1 in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
according to the ETM v4.3 architecture specification.
But I feel that I am missing something else, as it seems the ETM is not taking the filter into account.
i) you will need to have enabled PID=>context idr tracking in your kernel. ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators. Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs. The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
If there is more relevant documentation on this that I have not found, I would appreciate if you could point me to it.
If not, and what I am trying to do will not work, I would welcome some advice on how to do it properly.
Thanks in advance.
Best regards,
Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Blackburn Design Centre. UK IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-
Mike Leach -
Mike Leach -
Thierry Laviron