On 13.08.2018 18:47, Mathieu Poirier wrote:
On Wed, 8 Aug 2018 at 01:59, Tomasz Nowicki tnowicki@caviumnetworks.com wrote:
Hi Mathieu,
It's been a while but I am back to Coresight.
Let me remind my setup and the issue I am struggling with now.
Kernel baseline: https://github.com/Linaro/perf-opencsd (perf-opencsd-v4.16) OpenCSD: https://github.com/Linaro/OpenCSD.git (master)
The simplest Coresight components path I used as a start point: ETMv4.1 -> TDR -> FUNNEL -> ETF
As I mentioned TDR is built by Cavium and it was added to aggregate 128 inputs into one output rather than cascading funnels. TDR has its own driver just to keep path connected in Linux Coresight framework.
Here is how I catch some trace data: sudo perf record -C 0 -e cs_etm/@etf0/ --per-thread test_app
The above command line tells perf to trace everything that is happening on CPU0 for as long as "test_app" is executing. In this case the "--per-thread" option is ignored. This is called a CPU-wide trace scenario and is currently not supported for CS (I am currently working on it).
If you want to make sure "test_app" executes on CPU0 and that you trace just that you will need to use the "taskset" utility:
sudo perf record -e cs_etm/@etf0/ --per-thread taskset 0x1 test_app
My apologise, I used 'taskset' part in my command but forgot to mention. Anyway, thank you for clarifying that CPU-wide trace scenario is not supported for now.
An alternative to the above would be to CPU-hotplug out CPU128-255 while you are testing.
Yes, this helped to setup path between source and sink. Finally I am able to decode data with: # sudo perf record -e cs_etm/@etf0/ --per-thread test_app # sudo perf report --stdio
Now I am trying to see exactly the path a processor took through the code according to these instructions: https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md#trace-decoding-with-p... However, on perf-opencsd 'perf-opencsd-v4.16' branch there is no 'cs-trace-disasm.py' script. Are you planning to add this script to mainline kernel or this was just for development purpose?
Thanks, Tomasz
Let's start with that before going further.
Thanks, Mathieu
I need to use -C because my machines has 2 nodes, 32 cores (128 threads) each and each node has different ETF. So I have to specify which CPU is the source and for specified ETF sink (EFT0 can be a sink for CPU0-CPU127, ETF1 can be a sink for CPU128-CPU255). Otherwise Linux cannot find path for ETMs related to CPU128-CPU255 if I specify ETF0 as a sink.
Overall, I can see some data using: # sudo perf report --stdio --dump [...] . ... CoreSight ETM Trace data: size 16384 bytes Frame deformatter: Found 4 FSYNCS ID:12 RESET operation on trace decode path Idx:108; ID:12; I_NOT_SYNC : I Stream not synchronised Idx:455; ID:12; I_ASYNC : Alignment Synchronisation. Idx:468; ID:12; I_TRACE_INFO : Trace Info.; INFO=0x0 Idx:470; ID:12; I_TRACE_ON : Trace On. Idx:471; ID:12; I_CTXT : Context Packet.; Ctxt: AArch64,EL0, NS; Idx:473; ID:12; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000AAABE0B09584; Idx:483; ID:12; I_ATOM_F1 : Atom format 1.; N Idx:484; ID:12; I_TIMESTAMP : Timestamp.; Updated val = 0x1b6a5d937cc1 Idx:492; ID:12; I_ATOM_F3 : Atom format 3.; NNE Idx:493; ID:12; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000AAABE0B0D210; Idx:504; ID:12; I_ATOM_F3 : Atom format 3.; NEE Idx:505; ID:12; I_ATOM_F3 : Atom format 3.; NEN Idx:506; ID:12; I_ATOM_F6 : Atom format 6.; EEEN Idx:507; ID:12; I_ATOM_F3 : Atom format 3.; NNE Idx:508; ID:12; I_ATOM_F1 : Atom format 1.; N Idx:509; ID:12; I_ATOM_F3 : Atom format 3.; NNN Idx:510; ID:12; I_ATOM_F3 : Atom format 3.; EEN Idx:512; ID:12; I_ATOM_F1 : Atom format 1.; E [...]
However, I still see errors while using: # sudo perf report --stdio 0x1e8 [0x60]: failed to process type: 1 Error: failed to process sample # To display the perf.data header info, please use --header/--header-only options.
The reason is that cs_etm__process_event() is failing on: if (!etm->timeless_decoding) return -EINVAL;
and etm->timeless_decoding is setup in cs_etm__is_timeless_decoding(). For some events time bit set and so far I failed to figure out what is going on. Have you met similar issue so far? Any pointers or hints are very appreciated.
One more comment below.
On 10.01.2018 21:10, Mathieu Poirier wrote:
On 10 January 2018 at 06:57, Tomasz Nowicki tnowicki@caviumnetworks.com wrote:
Hello Mathieu,
Thank you for your response. Please see comments below.
On 08.01.2018 17:53, Mathieu Poirier wrote:
Good day Tomasz,
On 5 January 2018 at 05:51, tn Tomasz.Nowicki@caviumnetworks.com wrote:
Hi Mathieu,
I am bringing up Coresight functiproject zeroonality on ThunderX2. While ramping up I come across your Connect session:
which I found very helpful.
Perfect - a few things have changed since then, see below.
During my research I had to create new Coresight component driver for Linux, here is the story. For ThunderX2, we aggregate data trace from all 128 ETMs into one funnel inport using so called TDR (Trace Data Ring) component. This should be transparent to software and does not require configuration at all. However, Linux Coresight framework requires components to be connected each other so we cannot leave funnel and ETMs disconnected in DT. I decided to create pure software component i.e. TDR which is meant to connect chain only, no actions on registers.
Is this TDR an ARM IP or built in-house by Cavium?
This is Cavium specific component which I am going to upstream once I test the whole functionality.
And I suppose itwas added there to aggregate 128 input into one output rather than cascading funnels?
Correct.
Now I am able to enable ETF sink and path from ETM via TDR via FUNNEL up to ETF and gather some data. To be sure things work properly I want to decode data using Linaro OpenCSD library following instructions from here:
https://community.arm.com/tools/b/blog/posts/do-a-coresight-trace-on-linux-w...
Thanks for pointing this out, I didn't know about it.
but still got error while doing 'perf report' step. Kernel perf tool support for OpenCSD is out of tree for now so I may miss some patches.
Can you get me a pastebin of the errors you're getting?
Sure, see: https://pastebin.com/6YDq8KfC As you see there is not much info about error cause.
Here is my setup: https://github.com/Linaro/perf-opencsd/commits/upstream-v1 (+ ThunderX2 specific patches)
Oh boy... I wasn't expecting people to use that but I suppose it is the right thing to do. Keep going with that code.
This, in combination with the upstream-v1 branch should work properly. That's how I test things on my Juno and Dragon board.
# echo 1 > etf0/enable_sink # perf record -C 0 -e cs_etm// sleep 2
Ok, that won't work as the -C option is currently not supported (I am working on it). I also suggest to make sure you have the very latest TIP [1] on branch [2] and to carefully read the README.md. We recently updated the instructions to fit the newest development. Lastly we have deprecated enabling the sink from the sysFS interface - it can still work but no guarantees are provided. It is better to specify the sink as part of the perf record command line, as shown in the most recent HOWTO.md.
I am able to specify sink as part of the perf record command line only for Linux Perf master branch: https://github.com/Linaro/perf-opencsd/commits/master
For upstream-v1 branch I am getting: $ perf record -vvv -e cs_etm/@etf0/ --per-thread uname Using CPUID 0x00000000420f5160 perf: util/evsel.c:783: apply_config_terms: Assertion `!(1)' failed. Aborted (core dumped)
Ok, I've uploaded upstream-v2. With that branch everything works fine on my side, no changes needed. I added a fix for a regression in the perf tip tree and the code required to use the ETR from the perf interface.
One thing about the above: "@etf0". Is this really the name you gave to the device in the DT? Look under /sys/bus/coresight/devices/ for an etf entry. What is listed there should is the name of the ETF as it is known to the system.
Indeed, the name is different but for perf command clarity I use shortcut.
Thanks, Tomasz