Thank you Mike - I'm getting a way better understanding of the whole system.
Sorry for sharing a wrong command, unfortunately on 5.5.0-rc7 I see the same error persisting when running:
$ perf record --per-thread -e cs_etm/@tmc_etr0/u -- ls failed to set sink "" on event cs_etm/@tmc_etr0/u with 21 (Is a directory)
I'm using perf built as an Arm binary against the same kernel source code while linked against openCSD.
When I use the same command on my DB410 I get:
linaro@linaro-developer:~/perf-tests$ ./perf record --per-thread -e cs_etm/@tmc_etr0/u -- ls perf perf-save-data.bash test_170428_1048.tgz test_170502_1550.tgz perf-200302_1356-cti-v10-lnk perf.data test_170428_1107.tgz test_170920_1514.tgz perf-4.11-rc1-arm64 perf.data.old test_170501_1307.tgz test_171101_1726.tgz perf-4.9-rc1-arm64 sort test_170501_1427.tgz test_190311_1240.tgz perf-5.0-rc6-cpu-wide sort_3k test_170501_1442.tgz trace-sysfs.bash perf-clean-data.bash sort_any test_170501_2033.tgz trace-sysfs.bash~ perf-gcc5.3-4.13-cs-etr-arm64 test_170307_1304.tgz test_170501_2116.tgz perf-gcc5.3-rel-ocsd-4-13-base test_170307_1308.tgz test_170501_2121.tgz [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.110 MB perf.data ]
This is interesting! Do you get any warning when tracing, with the same perf command, any single-threaded workload which takes several seconds to run? How big is your perf.data file?
When running `perf report -D -i perf.data` I'm able to see the trace data correctly:
0: I_ASYNC : Alignment Synchronisation. 12: I_TRACE_INFO : Trace Info. 17: I_TRACE_ON : Trace On. 18: I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000007F89F24D80; Ctxt: AArch64,EL0, NS; 28: I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 29: I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 30: I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 32: I_ATOM_F6 : Atom format 6.; EEEEN 33: I_ATOM_F1 : Atom format 1.; E 34: I_EXCEPT : Exception.; Data Fault; Ret Addr Follows; 36: I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000007F89F2832C; 45: I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFFFFC000083400; Ctxt: AArch64,EL1, NS; 56: I_TRACE_ON : Trace On. 57: I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000007F89F2832C; Ctxt: AArch64,EL0, NS; 68: I_ATOM_F3 : Atom format 3.; NEE 69: I_ATOM_F3 : Atom format 3.; NEN
Is there any reference about the format in which the trace is here represented? I would just like to know what I_ASYNC, I_TRACE_INFO, I_TRACE_ON, I_ADDR_CTXT_L_64IS0 ,I_ATOM_F3 and all the other labels exactly mean
Thanks, Andrea
________________________________ From: Mike Leach mike.leach@linaro.org Sent: 04 March 2020 11:14 To: Andrea Brunato Andrea.Brunato@arm.com Cc: Mathieu Poirier mathieu.poirier@linaro.org; leo.yan@linaro.org leo.yan@linaro.org; guermazi.zied@gmail.com guermazi.zied@gmail.com; coresight@lists.linaro.org coresight@lists.linaro.org Subject: Re: perf record cs_etm: data lost
Hi Andrea,
On Wed, 4 Mar 2020 at 09:50, Andrea Brunato Andrea.Brunato@arm.com wrote:
Hi,
Is this a linux distribution or an android running on a linux kernel for android?
I'm running a linux distribution, Debian Buster
What is "sinkid=0xa6509eae" ?
It's the sinkid corresponding to tmc_etr0. When casting the perf command using @tmc_etr0 unfortunately I'm displayed the following error:
$ perf record --per-thread -e cs_etm/@tmc_etr/u -- ls
failed to set sink "" on event cs_etm/@tmc_etr/u with 21 (Is a directory)
The command above is wrong - you have missed the '0' off tmc_etr0. However I would expect the error to be "failed to set sink "tmc_etr" on event cs_etm/@tmc_etr/u with 2 (No such file or directory)" if this was the only problem.
When I use the same command on my DB410 I get:
linaro@linaro-developer:~/perf-tests$ ./perf record --per-thread -e cs_etm/@tmc_etr0/u -- ls perf perf-save-data.bash test_170428_1048.tgz test_170502_1550.tgz perf-200302_1356-cti-v10-lnk perf.data test_170428_1107.tgz test_170920_1514.tgz perf-4.11-rc1-arm64 perf.data.old test_170501_1307.tgz test_171101_1726.tgz perf-4.9-rc1-arm64 sort test_170501_1427.tgz test_190311_1240.tgz perf-5.0-rc6-cpu-wide sort_3k test_170501_1442.tgz trace-sysfs.bash perf-clean-data.bash sort_any test_170501_2033.tgz trace-sysfs.bash~ perf-gcc5.3-4.13-cs-etr-arm64 test_170307_1304.tgz test_170501_2116.tgz perf-gcc5.3-rel-ocsd-4-13-base test_170307_1308.tgz test_170501_2121.tgz [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.110 MB perf.data ]
Note - I have a local perf version built against the same tree as the kernel I am running. Ensure that you are doing the same otherwise odd things can happen.
so I found myself forced to manually specify the identifier. I think this problem has been solved in the latest kernel versions but I still have to double check.
My CoreSight subsystem is the following:
$ ls /sys/bus/coresight/devices/
etm0 etm1 etm2 etm3 etm4 etm5 etm6 etm7 funnel0 funnel1 funnel2 funnel3 funnel4 replicator0 stm0 tmc_etf0 tmc_etr0
$ grep . /sys/devices/cs_etm/sinks/* /sys/devices/cs_etm/sinks/tmc_etf0:0x97979549 /sys/devices/cs_etm/sinks/tmc_etr0:0xa6509eae
To allow the "--mmap-pages" to work with big buffer, the sink needs to directly output trace data to DDR, you also need to take care what's the buffer mode you are using (flat, sg, CATU).
Thanks for the suggestion! Is there a way for me to enforce this?
These are hardware dependent features. CATU is a CS-SoC600 hardware component that takes over the physical memory mapping from the ETR - which in SoC600 uses VA only. The ETR in CS-Soc400 had the mapping h/w build in that could in theory operate in Scatter-Gather (sg) or Contiguous (flat) memory mapping mode. However, some hardware implementations got the interconnect design wrong, and did not support certain read protocols that the SG mode in ETR requited, so the drivers were updated to have a device tree flag to indicate if sg was permitted. As I recall, sg was permitted in Juno, but others were limited to flat only.
Is it possible to check out how much data I'm losing/overwriting?
No, this is a hardware controlled circular buffer. The only information available to software is wrapped / not full.
Do you know what is the best buffer mode for avoiding losing data?
In general, CS-Soc400 based devices cannot halt the capture on full, as there is no available interrupt - hence buffers wrap and trace can be lost.
CS-SoC600 base devices could have this feature (which would match a similar feature that intel-pt has) assuming the hardware has the relevant designed connections. However, at present the drivers do not have the capability to use such an interrupt - as devices with the hardware have not been available to developers to date.
How can I switch between a buffer mode and another one?
See above - this is largely a hardware feature.
Is `--per-thread` the best way to trace a single-threaded workload?
Given the limited buffer size I would say probably yes. Using CPU wide would trace everything on the CPU(s) - including any other tasks running concurrently with that of interest.
Regards
Mike
Thanks everybody for the interest on this! Andrea ________________________________ From: Mathieu Poirier mathieu.poirier@linaro.org Sent: 04 March 2020 02:43 To: leo.yan@linaro.org leo.yan@linaro.org Cc: Andrea Brunato Andrea.Brunato@arm.com; coresight@lists.linaro.org coresight@lists.linaro.org Subject: Re: perf record cs_etm: data lost
On Tue, 3 Mar 2020 at 18:54, Leo Yan leo.yan@linaro.org wrote:
Hi Andrea,
On Tue, Mar 03, 2020 at 11:23:29AM -0700, Mathieu Poirier wrote:
Hi Andrea,
On Tue, 3 Mar 2020 at 09:34, Andrea Brunato Andrea.Brunato@arm.com wrote:
Good afternoon,
When using perf to record a simple benchmark I see the following warning:
$ perf record --per-thread -e cs_etm/sinkid=0xa6509eae/u ~/afdo/coremark/coremark/coremark.exe 0x3415 0x3415 0x66 0 7 1 2000 > run2.log
What is "sinkid=0xa6509eae" ?
Sink might be ETF/ETR/ETB, I think this is why Mathieu asked the meaning for 'sinkid', which we don't which sink you are using.
[ perf record: Woken up 6 times to write data ] Warning: AUX data lost 6 times out of 6!
[ perf record: Captured and wrote 0.378 MB perf.data ]
This is on DB845c (https://www.96boards.org/product/rb3-platform/), kernel 5.5.0-rc7.
It looks like perf is losing data and the reported perf.data file is very tiny, given that the application runs for several seconds. Does anybody have any idea about what it may be causing this?
Just give some info based on my understanding.
For the log "AUX data lost 6 times out of 6!", since you are using per-thread mode, so every time when the task switching out, the trace data in the sink will be read out and save into aux buffer. But usually the sink's buffer is not big (maximum to MBs), so it's easily to overwrite the hardware's cycle buffer and the previous trace data will be lost. For this reason, it reports "AUX data lost".
I've tried adding --mmap-pages=32K,64K to the command for increasing the buffer size, but it's not actually solving the problem
If it's the overflow in the hardware buffer in SRAM, "--mmap-pages" option doesn't work, since it only enlarges aux buffer page size but cannot change the SRAM size in CoreSight.
To allow the "--mmap-pages" to work with big buffer, the sink needs to directly output trace data to DDR, you also need to take care what's the buffer mode you are using (flat, sg, CATU).
Everything Leo said above is accurate.
Thanks, Leo
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.