+CoreSight ML and Mathieu
---------- Forwarded message ---------- From: Mike Leach mike.leach@linaro.org Date: 3 September 2018 at 17:39 Subject: Re: Failed for ETM decoding with db410c snapshot mode To: Leo Yan leo.yan@linaro.org
HI Leo,
Short summary - there is a problem with the trace collected - not the decoder. See below for details
On 3 September 2018 at 08:06, leo.yan@linaro.org wrote:
Hi Mike, Mathieu,
[ + CoreSight ML ]
When I work on the CoreSight + perf tool and used crash extension program to extract the tracing data from perf aux buffer, finally I can get the trace data for about 1.6MB from ETF sink from DB410c board.
To verify the extracted trace data, I used 'snapshot' mode under OpenCSD code base, you could see the tar file for this [1]. After you download this file, you could place it under OpenCSD folder:
$ cp db410c_snapshot_kdump.tgz my_opencsd/decoder/tests/snapshots $ cd my_opencsd/decoder/tests/snapshots $ tar zxvf db410c_snapshot_kdump.tgz $ cd db410c_snapshot_kdump
$ ../../bin/builddir/trc_pkt_lister
This will print raw trace packets as it finds them without attempting any sort of interpretation.
$ ../../bin/builddir/trc_pkt_lister -decode
This will try to decode the raw trace packets into a sequence of instructions executed (alongside the raw packets)
This is where the packets are being flagged as incorrect.
If I use the command 'trc_pkt_lister' without any extra options, it can print out trace packets successfully; but if I add the extra option '-decode' it uses 'decode all' mode and it reports the errors as:
483710 Idx:53086; ID:10; [0xf8 ]; I_ATOM_F3 : Atom format 3.; NNN 483711 Idx:53086; ID:10; OCSD_GEN_TRC_ELEM_ADDR_NACC( 0xffff000008abc9f0 ) 483712 Idx:53088; ID:10; [0xdb ]; I_ATOM_F2 : Atom format 2.; EE 483713 Idx:53194; ID:10; [0x6b 0x8c 0x08 0xfa 0xdc 0x95 0x5c ]; I_COND_RES_F1 : Conditional Result, format 1.
This is a conditional result trace packet - however as far as I am aware the trace unit on an A53 (i.e. DB410 core) cannot produce these.
Additionally in the entire file I see 2 I_COND packets and 1 I_NUM_DS_MKR - a data synchronisation marker packet.
Now Data sync can only ever occur if data trace is supported and enabled. Data trace is architecturally prohibited for A class v8 cores (and unimplemented on most A class v7 cores).
If there were tracing of conditional elements occurring, and it were enabled, then the packets should match up - a cond instruction should match with one cond result element.
But in the end - event without these inconsistencies - the TRACE_INFO element at the top of the listing tells me that conditional instruction trace is disabled.
Thus you are seeing what I believe is the effect of concatenating trace data buffers together (you mention you have 1.6MB of data from the ETF - which is not that large), without inserting barrier packets in between. The decoder cannot spot the boundaries, and will carry on and be out of sync so can mis-read trace packet payload data as header data which will throw off the decode process.
When I look at the raw byte data I am seeing this at the top of the listing:-
Frame Data; Index 0; ID_DATA[????]; ff Frame Data; Index 0; ID_DATA[0x7f]; 7f ff 7f ff 7f ff
This does not look valid at all to me.
483714 DCD_ETMV4_0016 : 0x0018 (OCSD_ERR_BAD_DECODE_PKT) [Reserved or unknown packet in decoder.]; Unsupported packet type.Trace Packet Lister : Data Path fatal error 483715 0x0018 (OCSD_ERR_BAD_DECODE_PKT) [Reserved or unknown packet in decoder.]; Unsupported packet type.Trace Packet Lister : Trace buffer done, processed 53216 bytes.
You also could check detailed log trc_pkt_lister.ppl in the shared tar packet; After searched for the OpenCSD code and found this error is due it cannot support some types of packets [2].
So want to check what's the best for this issue; seems to me we need to fix this so it can support well to complete the decoding?
The reason we have not implemented support for these packets, is that we have never seen an implementation that generates them.
regards
Mike
Thanks in advance for suggestion. Leo Yan
[1] http://people.linaro.org/~leo.yan/opencsd_db410c/db410c_snapshot_kdump.tgz [2] https://github.com/Linaro/OpenCSD/blob/master/decoder/source/etmv4/trc_pkt_d...
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
Hi Mike,
HI Leo,
Short summary - there is a problem with the trace collected - not the decoder. See below for details
Thanks for quick replying.
If I use the command 'trc_pkt_lister' without any extra options, it can print out trace packets successfully; but if I add the extra option '-decode' it uses 'decode all' mode and it reports the errors as:
483710 Idx:53086; ID:10; [0xf8 ]; I_ATOM_F3 : Atom format 3.; NNN 483711 Idx:53086; ID:10; OCSD_GEN_TRC_ELEM_ADDR_NACC( 0xffff000008abc9f0 ) 483712 Idx:53088; ID:10; [0xdb ]; I_ATOM_F2 : Atom format 2.; EE 483713 Idx:53194; ID:10; [0x6b 0x8c 0x08 0xfa 0xdc 0x95 0x5c ]; I_COND_RES_F1 : Conditional Result, format 1.
This is a conditional result trace packet - however as far as I am aware the trace unit on an A53 (i.e. DB410 core) cannot produce these.
Additionally in the entire file I see 2 I_COND packets and 1 I_NUM_DS_MKR - a data synchronisation marker packet.
Now Data sync can only ever occur if data trace is supported and enabled. Data trace is architecturally prohibited for A class v8 cores (and unimplemented on most A class v7 cores).
If there were tracing of conditional elements occurring, and it were enabled, then the packets should match up - a cond instruction should match with one cond result element.
But in the end - event without these inconsistencies - the TRACE_INFO element at the top of the listing tells me that conditional instruction trace is disabled.
Thus you are seeing what I believe is the effect of concatenating trace data buffers together (you mention you have 1.6MB of data from the ETF - which is not that large), without inserting barrier packets in between. The decoder cannot spot the boundaries, and will carry on and be out of sync so can mis-read trace packet payload data as header data which will throw off the decode process.
I am suspecting this is caused by the barrier insertion in CoreSight driver. When the driver detects the ETB RAM is full, it will do below two things:
- Insert barrier packet 0x7fffffff;
- Due the barrier packet has occupied the buffer space, so the driver will read trace data from TMC_RRD but the trace data is discarded when insert barrier packet.
I read the code [1][2] and the code delibrately discards trace data so can leave the memory for barrier packets; after that it can fill continous seqential trace data.
I think the discarding trace data is incorrect, for most case the buffer is sufficient to store both barrier packets and trace data so we can keep complete trace data; from testing I also can see after save trace data from the beginning of ETB, the decoding error can dismiss.
I am working on minor fixing patches for this and will send out for reviewing.
When I look at the raw byte data I am seeing this at the top of the listing:-
Frame Data; Index 0; ID_DATA[????]; ff Frame Data; Index 0; ID_DATA[0x7f]; 7f ff 7f ff 7f ff
This does not look valid at all to me.
I think this is related with barrier packet 0x7fffffff which introduced by CoreSight driver [3].
I found there have another important flag for decoding: OCSD_DFRMTR_RESET_ON_4X_FSYNC. I think after set this flag, the decoder can recognize barrier packet and don't take it as normal trace data?
Thanks, Leo Yan
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv...
Hi Leo,
On 5 September 2018 at 03:22, leo.yan@linaro.org wrote:
Hi Mike,
HI Leo,
Short summary - there is a problem with the trace collected - not the decoder. See below for details
Thanks for quick replying.
If I use the command 'trc_pkt_lister' without any extra options, it can print out trace packets successfully; but if I add the extra option '-decode' it uses 'decode all' mode and it reports the errors as:
483710 Idx:53086; ID:10; [0xf8 ]; I_ATOM_F3 : Atom format 3.; NNN 483711 Idx:53086; ID:10; OCSD_GEN_TRC_ELEM_ADDR_NACC( 0xffff000008abc9f0 ) 483712 Idx:53088; ID:10; [0xdb ]; I_ATOM_F2 : Atom format 2.; EE 483713 Idx:53194; ID:10; [0x6b 0x8c 0x08 0xfa 0xdc 0x95 0x5c ]; I_COND_RES_F1 : Conditional Result, format 1.
This is a conditional result trace packet - however as far as I am aware the trace unit on an A53 (i.e. DB410 core) cannot produce these.
Additionally in the entire file I see 2 I_COND packets and 1 I_NUM_DS_MKR - a data synchronisation marker packet.
Now Data sync can only ever occur if data trace is supported and enabled. Data trace is architecturally prohibited for A class v8 cores (and unimplemented on most A class v7 cores).
If there were tracing of conditional elements occurring, and it were enabled, then the packets should match up - a cond instruction should match with one cond result element.
But in the end - event without these inconsistencies - the TRACE_INFO element at the top of the listing tells me that conditional instruction trace is disabled.
Thus you are seeing what I believe is the effect of concatenating trace data buffers together (you mention you have 1.6MB of data from the ETF - which is not that large), without inserting barrier packets in between. The decoder cannot spot the boundaries, and will carry on and be out of sync so can mis-read trace packet payload data as header data which will throw off the decode process.
I am suspecting this is caused by the barrier insertion in CoreSight driver. When the driver detects the ETB RAM is full, it will do below two things:
Insert barrier packet 0x7fffffff;
Due the barrier packet has occupied the buffer space, so the driver will read trace data from TMC_RRD but the trace data is discarded when insert barrier packet.
I read the code [1][2] and the code delibrately discards trace data so can leave the memory for barrier packets; after that it can fill continous seqential trace data.
I think the discarding trace data is incorrect, for most case the buffer is sufficient to store both barrier packets and trace data so we can keep complete trace data; from testing I also can see after save trace data from the beginning of ETB, the decoding error can dismiss.
I am working on minor fixing patches for this and will send out for reviewing.
When I look at the raw byte data I am seeing this at the top of the listing:-
Frame Data; Index 0; ID_DATA[????]; ff Frame Data; Index 0; ID_DATA[0x7f]; 7f ff 7f ff 7f ff
This does not look valid at all to me.
I think this is related with barrier packet 0x7fffffff which introduced by CoreSight driver [3].
I found there have another important flag for decoding: OCSD_DFRMTR_RESET_ON_4X_FSYNC. I think after set this flag, the decoder can recognize barrier packet and don't take it as normal trace data?
This flag tells the decoder to treat the barrier packet as a decoder reset, clearing all state and restarting from an unsynchronised state. This is used in perf for decode, but not in the trc_pkt_lister test program.
Mike
Thanks, Leo Yan
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv...
Following on from that - it is possible that we could add a command line option to enable the flag for the trc_pkt_lister program.
Mike
On 5 September 2018 at 14:34, Mike Leach mike.leach@linaro.org wrote:
Hi Leo,
On 5 September 2018 at 03:22, leo.yan@linaro.org wrote:
Hi Mike,
HI Leo,
Short summary - there is a problem with the trace collected - not the decoder. See below for details
Thanks for quick replying.
If I use the command 'trc_pkt_lister' without any extra options, it can print out trace packets successfully; but if I add the extra option '-decode' it uses 'decode all' mode and it reports the errors as:
483710 Idx:53086; ID:10; [0xf8 ]; I_ATOM_F3 : Atom format 3.; NNN 483711 Idx:53086; ID:10; OCSD_GEN_TRC_ELEM_ADDR_NACC( 0xffff000008abc9f0 ) 483712 Idx:53088; ID:10; [0xdb ]; I_ATOM_F2 : Atom format 2.; EE 483713 Idx:53194; ID:10; [0x6b 0x8c 0x08 0xfa 0xdc 0x95 0x5c ]; I_COND_RES_F1 : Conditional Result, format 1.
This is a conditional result trace packet - however as far as I am aware the trace unit on an A53 (i.e. DB410 core) cannot produce these.
Additionally in the entire file I see 2 I_COND packets and 1 I_NUM_DS_MKR - a data synchronisation marker packet.
Now Data sync can only ever occur if data trace is supported and enabled. Data trace is architecturally prohibited for A class v8 cores (and unimplemented on most A class v7 cores).
If there were tracing of conditional elements occurring, and it were enabled, then the packets should match up - a cond instruction should match with one cond result element.
But in the end - event without these inconsistencies - the TRACE_INFO element at the top of the listing tells me that conditional instruction trace is disabled.
Thus you are seeing what I believe is the effect of concatenating trace data buffers together (you mention you have 1.6MB of data from the ETF - which is not that large), without inserting barrier packets in between. The decoder cannot spot the boundaries, and will carry on and be out of sync so can mis-read trace packet payload data as header data which will throw off the decode process.
I am suspecting this is caused by the barrier insertion in CoreSight driver. When the driver detects the ETB RAM is full, it will do below two things:
Insert barrier packet 0x7fffffff;
Due the barrier packet has occupied the buffer space, so the driver will read trace data from TMC_RRD but the trace data is discarded when insert barrier packet.
I read the code [1][2] and the code delibrately discards trace data so can leave the memory for barrier packets; after that it can fill continous seqential trace data.
I think the discarding trace data is incorrect, for most case the buffer is sufficient to store both barrier packets and trace data so we can keep complete trace data; from testing I also can see after save trace data from the beginning of ETB, the decoding error can dismiss.
I am working on minor fixing patches for this and will send out for reviewing.
When I look at the raw byte data I am seeing this at the top of the listing:-
Frame Data; Index 0; ID_DATA[????]; ff Frame Data; Index 0; ID_DATA[0x7f]; 7f ff 7f ff 7f ff
This does not look valid at all to me.
I think this is related with barrier packet 0x7fffffff which introduced by CoreSight driver [3].
I found there have another important flag for decoding: OCSD_DFRMTR_RESET_ON_4X_FSYNC. I think after set this flag, the decoder can recognize barrier packet and don't take it as normal trace data?
This flag tells the decoder to treat the barrier packet as a decoder reset, clearing all state and restarting from an unsynchronised state. This is used in perf for decode, but not in the trc_pkt_lister test program.
Mike
Thanks, Leo Yan
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv... [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/driv...
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
Hi Mike,
On Wed, Sep 05, 2018 at 04:41:30PM +0100, Mike Leach wrote:
Following on from that - it is possible that we could add a command line option to enable the flag for the trc_pkt_lister program.
Just want to confirm, will you do this or you would like me to send PR in github?
[...]
Thanks, Leo Yan
Hi Leo,
I can do this, but it probably won't happen before connect. If you want to quickly test something look at line 90 in decoder\tests\snapshot_parser_lib\source\ss_to_dcdtree.cpp :-
m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format,OCSD_DFRMTR_FRAME_MEM_ALIGN);
Or in the flag there and we can add the additional command line option code later
Regards
Mike
-----Original Message----- From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of leo.yan@linaro.org Sent: 06 September 2018 02:56 To: Mike Leach mike.leach@linaro.org Cc: coresight@lists.linaro.org Subject: Re: Fwd: Failed for ETM decoding with db410c snapshot mode
Hi Mike,
On Wed, Sep 05, 2018 at 04:41:30PM +0100, Mike Leach wrote:
Following on from that - it is possible that we could add a command line option to enable the flag for the trc_pkt_lister program.
Just want to confirm, will you do this or you would like me to send PR in github?
[...]
Thanks, Leo Yan _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Mike,
On Thu, Sep 06, 2018 at 08:43:59AM +0000, Mike Leach wrote:
Hi Leo,
I can do this, but it probably won't happen before connect.
Thanks!
If you want to quickly test something look at line 90 in decoder\tests\snapshot_parser_lib\source\ss_to_dcdtree.cpp :-
m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format,OCSD_DFRMTR_FRAME_MEM_ALIGN);Or in the flag there and we can add the additional command line option code later
I did quick change as below, but it still fails to decode with one trace data with command: ../../bin/builddir/trc_pkt_lister -decode; I also uploaded the complete snapshot tar file [1].
Just to clarify this new tar file includes a new trace data which I captured later than shared with you ahead in this thread.
diff --git a/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp b/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp index 81ae82b..27982d5 100644 --- a/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp +++ b/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp @@ -87,7 +87,7 @@ bool CreateDcdTreeFromSnapShot::createDecodeTree(const std::string &SourceName,
/* create the initial device tree */ // TBD: handle syncs / hsyncs data from TPIU - m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format,OCSD_DFRMTR_FRAME_MEM_ALIGN); + m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format,OCSD_DFRMTR_FRAME_MEM_ALIGN|OCSD_DFRMTR_RESET_ON_4X_FSYNC); if(m_pDecodeTree == 0) { LogError("Failed to create decode tree object\n");
Thanks, Leo Yan
[1] http://people.linaro.org/~leo.yan/opencsd_db410c/db410c_snapshot_kdump_0906....
-
leo.yan＠linaro.org -
Mike Leach -
Mike Leach