Hi,
A quick answer to your timestamp / cycle count question....
using the command line...
/home/mleach/perf-tools/perf record -e cs_etm/timestamp,cycacc,@20010000.etf/ --per-thread uname -a
I see the following for a perf report --stdio --dump
0: id[12] I_ASYNC : Alignment Synchronisation 12: id[12] I_TRACE_INFO : Trace Info.; INFO=0x1; CC_THRESHOLD=0x100 19: id[12] I_TRACE_ON : Trace On. 20: id[12] I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0xFFFF0000088FD8FC; Ctxt: AArch64,EL1, NS; 30: id[12] I_ATOM_F1 : Atom format 1.; N 32: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x0 33: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d617e00 43: id[12] I_ATOM_F1 : Atom format 1.; E 44: id[12] I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008902D9C; 54: id[12] I_ATOM_F2 : Atom format 2.; EE 55: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0xFFFF000008903488 ~[0x3488] 58: id[12] I_ATOM_F1 : Atom format 1.; E 59: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088FDD6C; 65: id[12] I_ATOM_F3 : Atom format 3.; NEE 66: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x088FE19C ~[0x1E19C] 69: id[12] I_ATOM_F1 : Atom format 1.; E 70: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081B231C; 75: id[12] I_ATOM_F3 : Atom format 3.; NEE 76: id[12] I_ATOM_F3 : Atom format 3.; NNE 77: id[12] I_ATOM_F3 : Atom format 3.; EEE 78: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x126 81: id[12] I_ATOM_F2 : Atom format 2.; NE 82: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB5F8 ~[0xB5F8] 85: id[12] I_ATOM_F1 : Atom format 1.; E 86: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B16C8 ~[0x116C8] 89: id[12] I_ATOM_F1 : Atom format 1.; E 90: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2398 ~[0x12398] 93: id[12] I_ATOM_F1 : Atom format 1.; E 94: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24C8 ~[0x124C8] 98: id[12] I_ATOM_F3 : Atom format 3.; NNE 99: id[12] I_ATOM_F1 : Atom format 1.; E 100: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B0A60 ~[0x10A60] 103: id[12] I_ATOM_F1 : Atom format 1.; E 104: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B25E8 ~[0x125E8] 107: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x141 109: id[12] I_ATOM_F2 : Atom format 2.; NE 110: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2A64 ~[0x12A64] 114: id[12] I_ATOM_F3 : Atom format 3.; ENN 115: id[12] I_ATOM_F5 : Atom format 5.; NEEEE 116: id[12] I_ATOM_F4 : Atom format 4.; NENE 117: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081AB600 ~[0xB600] 120: id[12] I_ATOM_F1 : Atom format 1.; E 121: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B24B4 ~[0x124B4] 124: id[12] I_ATOM_F3 : Atom format 3.; EEN 125: id[12] I_ATOM_F3 : Atom format 3.; ENN 126: id[12] I_ATOM_F3 : Atom format 3.; EEE 128: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B22D4 ~[0x122D4] 131: id[12] I_ATOM_F6 : Atom format 6.; EEEEE 132: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B2308 ~[0x108] 134: id[12] I_ATOM_F1 : Atom format 1.; E 135: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7A40 ~[0x17A40] 138: id[12] I_ATOM_F1 : Atom format 1.; E 139: id[12] I_CCNT_F1 : Cycle Count format 1.; Count=0x1af 144: id[12] I_ATOM_F6 : Atom format 6.; EEEN 145: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x102 146: id[12] I_ATOM_F3 : Atom format 3.; NEE 147: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B7AE4 ~[0xE4] 149: id[12] I_ATOM_F3 : Atom format 3.; EEE 150: id[12] I_CCNT_F3 : Cycle Count format 3.; Count=0x103 151: id[12] I_ATOM_F2 : Atom format 2.; EE 152: id[12] I_ADDR_S_IS0 : Address, Short, IS0.; Addr=0x081B3C2C ~[0x13C2C] 155: id[12] I_ATOM_F3 : Atom format 3.; NEE 156: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x088D2260; 162: id[12] I_ATOM_F1 : Atom format 1.; E 163: id[12] I_TIMESTAMP : Timestamp.; Updated val = 0x3a41d61c57a; CC=0x51 168: id[12] I_ATOM_F1 : Atom format 1.; E 169: id[12] I_ADDR_L_32IS0 : Address, Long, 32 bit, IS0.; Addr=0x081492C8; 174: id[12] I_ATOM_F2 : Atom format 2.; NE
So, timestamps and cycle counts are being captured - the problem is with the perf report / script code not yet having been updated to use these values. i.e cs-etm-decoder.c does nothing with these packets at present. The issue here is to make these values meaningful in a perf context.
The timestamp is generated from a free running counter in the CoreSight hardware system and not related to the linux system time (for juno at least).
The cycle counts are for blocks of instructions. Note the CC_THRESHOLD=0x100 at the start of the trace run - this means that cycle count packets will be emitted only when the count reaches or exceeds 0x100 cycles, and by design for ETMv4 trace, will only be emitted when an atom or other waypoint packet is output so that the count is associated with that instruction. Given that with waypoint trace many executed instructions can be implied between two waypoints (E/N atoms above) then you can see that the counts cannot be precisely attributed to individual instructions, only blocks of code. It is possible to adjust the cycle count threshold value to get better resolution - but I am not sure if we have that option in the perf record command line as yet.
Regards
Mike
On 12 June 2017 at 15:41, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like “time: option in the perf script.
Have you tried cycle accurate mode and time stamps?
Regards, Reza
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Monday, June 12, 2017 9:36 AM
To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option, It works as expected. I used control-c to terminate the perf and stop the trace. Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ] Warning: AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ] #perf script -f --ns -F pid,comm,sym,symoff,ip,cpu 0x368 [0x70]: failed to process type: 10 #
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 10:19 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you also know if there is a plan to support Perf APIs for configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Friday, June 09, 2017 9:00 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 8 June 2017 at 23:42, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
Does "perf" support snapshot mode? I have tried -S option and it does not work. I am using signal to stop the trace but it does not work. How should I stop the trace "snapshot" mode?
I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
Also "perf record" with -C does not work for me as well. I like to enable the trace only On one core but it does not work for me.
I tried the following
perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c 2 uname
That doesn't work right now. I am working on it but need to have a chat with Peter Z. at Linux Plumbers in
September to see how we're going to address the problem.
Mathieu
Regards, Reza
-----Original Message----- From: Etemadi, Mohammad Sent: Wednesday, May 03, 2017 10:20 AM To: Mathieu Poirier mathieu.poirier@linaro.org Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: RE: Perf-opencsd-4.9
Thanks Mathieu.
I also have another question about the usage of "perf"
We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 10:02 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:58, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Do you have a patch for Mike's fix? I can apply only that one and see if it helps.
https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5 4 f71bf6907d4
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
On 3 May 2017 at 08:48, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Hello Mathieu
We are using perf-opencsd-4.9.
Right, that is too old and you will need to upgrade.
Currently we are using Linux 4.9 on our target. I think the latest is perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
"perf record" shows different X and Y each time I run it for the same application (uname) AUX data lost X times out of Y!
Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
Is this expected? The values are quite different for each run.
As I mentioned earlier, in some runs, "perf report" does not come back at all.
Right, let's hope that Mike's fix will help you.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Wednesday, May 03, 2017 9:29 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
Mathieu
On 2 May 2017 at 09:20, Etemadi, Mohammad mohammad.etemadi@intel.com wrote:
Thanks Mathieu
Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
Regards, Reza
-----Original Message----- From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] Sent: Tuesday, May 02, 2017 9:56 AM To: Etemadi, Mohammad mohammad.etemadi@intel.com Cc: Mike Leach mike.leach@linaro.org; coresight@lists.linaro.org Subject: Re: Perf-opencsd-4.9
It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
On 1 May 2017 at 15:18, Etemadi, Mohammad mohammad.etemadi@intel.com wrote: > Hello Mathieu > > There is a warning "AUX data lost 6 times out of 16!" what does this > warning imply? > > Regards, Reza > > > > -----Original Message----- > From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] > Sent: Friday, April 21, 2017 9:21 AM > To: Etemadi, Mohammad mohammad.etemadi@intel.com > Cc: Mike Leach mike.leach@linaro.org; > coresight@lists.linaro.org > Subject: Re: Perf-opencsd-4.9 > > On 20 April 2017 at 13:02, Etemadi, Mohammad > mohammad.etemadi@intel.com wrote: >> Thanks Mathieu >> >> The issue was related to the number of CPUs. We have 4 clusters each >> with 4 cores (total 16 cores). >> I have only defined 4 ETMs (only first cluster). >> After setting maxcpus=4 in bootargs, I do not see mmap failure >> anymore and I am getting the following logs. >> He still somehow thinks there are 16 cores... > > The error message comes from the perf user space tool looking for all > the processors known to the system [1]. It is a little chatty but doesn't > have an impact on trace collection. > > [1]. /sys/bus/event_source/devices/cs_etm/ > >> >> Regards, Reza >> >> ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >> cs_etm_get_ro: error reading: cpu4/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu4/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu5/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu5/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu6/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu6/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu7/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu7/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu8/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu8/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu9/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu9/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu10/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu10/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu11/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu11/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu12/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu12/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu13/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu13/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu14/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu14/mgmt/etmidr >> cs_etm_get_ro: error reading: cpu15/mgmt/etmccer >> cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record: >> Woken up 7 times to write data ] >> Warning: >> AUX data lost 6 times out of 16! >> >> [ perf record: Captured and wrote 0.238 MB perf.data ] # >> >> -----Original Message----- >> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >> Sent: Thursday, April 20, 2017 11:19 AM >> To: Etemadi, Mohammad mohammad.etemadi@intel.com >> Cc: Mike Leach mike.leach@linaro.org; >> coresight@lists.linaro.org >> Subject: Re: Perf-opencsd-4.9 >> >> The sink in the perf command line matches what is in sysFS so that's >> good. At this point you will need to instrument etm_setup_aux [1] to >> understand what makes it unhappy. >> >> [1]. >> http://lxr.free-electrons.com/source/drivers/hwtracing/coresight >> / >> c >> o >> r >> e >> s >> ight-etm-perf.c >> >> On 20 April 2017 at 10:11, Etemadi, Mohammad >> mohammad.etemadi@intel.com wrote: >>> >>> I have 8008010000.etf in /sys/bus/coresight/devices >>> >>> Reza >>> >>> # ls /sys/bus/coresight/devices 8008010000.etf >>> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm >>> 8008d40000.etm 8008e40000.etm 8008f40000.etm # >>> >>> -----Original Message----- >>> From: Etemadi, Mohammad >>> Sent: Thursday, April 20, 2017 10:50 AM >>> To: 'Mathieu Poirier' mathieu.poirier@linaro.org >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: RE: Perf-opencsd-4.9 >>> >>> Thanks Mathieu >>> >>> Now, I am hitting another failure. It seems mmap has failed. >>> Does it get the physical address from the name? >>> >>> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname >>> failed to mmap with 12 (Cannot allocate memory) >>> >>> Regards, Reza >>> >>> >>> -----Original Message----- >>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>> Sent: Thursday, April 20, 2017 10:03 AM >>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>> Cc: Mike Leach mike.leach@linaro.org; >>> coresight@lists.linaro.org >>> Subject: Re: Perf-opencsd-4.9 >>> >>> On 19 April 2017 at 12:42, Etemadi, Mohammad >>> mohammad.etemadi@intel.com wrote: >>>> Thanks Mathieu and Mike >>>> >>>> I rebuild the Linux image with name changes in the device tree and >>>> still see the same error. >>>> >>>> Regards, Reza >>>> >>>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread >>>> uname >>>> event syntax error: 'cs_etm/8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>>> >>> >>> This is not the proper syntax. >>> >>>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ >>>> --per-thread uname >>>> event syntax error: 'cs_etm/@8008010000.cluster0etf/' >>>> ___ BPF support is not compiled >>> >>> This is the proper syntax and there is nothing wrong with this >>> command line. As such I did a few test on my side and was able to reproduce >>> the problem. Anything that starts with a 'c' after the '.' will trigger >>> this problem. >>> >>> I don't know if the BPF related error message is real - it could be >>> a false positive. But we definitely have a feature interaction problem. >>> On the flip side "cluster0etf" won't be accepted upstream so at one >>> point you will need to change it to simply "etf", and that works. >>> >>>> >>>> >>>> -----Original Message----- >>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>> Sent: Wednesday, April 19, 2017 10:58 AM >>>> To: Mike Leach mike.leach@linaro.org >>>> Cc: Etemadi, Mohammad mohammad.etemadi@intel.com; >>>> coresight@lists.linaro.org >>>> Subject: Re: Perf-opencsd-4.9 >>>> >>>> On 19 April 2017 at 09:43, Mike Leach mike.leach@linaro.org >>>> wrote: >>>>> Is it also possible that a leading @ is required? >>>>> i.e. >>>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ >>>>> --per-thread uname >>>> >>>> Definitely yes - that would be the first thing to try. >>>> >>>>> >>>>> Mike >>>>> >>>>> On 19 April 2017 at 16:07, Mathieu Poirier >>>>> mathieu.poirier@linaro.org wrote: >>>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad >>>>>> mohammad.etemadi@intel.com wrote: >>>>>>> >>>>>>> Hello Mathieu >>>>>>> >>>>>>> BPF exists when I am building "perf" but I am getting an error >>>>>>> indicating "BPF support is not compiled" >>>>>>> When I am running it on the target. Am I missing a package? >>>>>>> >>>>>>> When building "perf": >>>>>>> .. bpf: [ on ] >>>>>>> >>>>>>> When running "perf" on the target. >>>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ >>>>>>> --per-thread uname >>>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/' >>>>>>> ___ BPF support is not compiled >>>>>> >>>>>> The perf command line parser is insanely complex. Here I >>>>>> suppose the extra "-etf" is causing the bison parser to hit >>>>>> a BPF rule, hence the error message. If you drop the "-etf" >>>>>> in the DT I'm pretty sure the message will go away. You can >>>>>> also enhance the flex/bison parser but to me the "address.name" >>>>>> convention is enough. >>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Etemadi, Mohammad >>>>>>> Sent: Monday, April 17, 2017 4:03 PM >>>>>>> To: Mathieu Poirier mathieu.poirier@linaro.org >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: RE: Perf-opencsd-4.9 >>>>>>> >>>>>>> >>>>>>> Thanks Mathieu >>>>>>> >>>>>>> Regards, Reza >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>> Sent: Monday, April 17, 2017 2:51 PM >>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>> Cc: coresight@lists.linaro.org >>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>> >>>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad >>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>> Thanks Mathieu >>>>>>>> >>>>>>>> I fixed the power domain issue. Now, it can discover ETMs. >>>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1 >>>>>>>> Rather than creating cpu1, it tries to recreate cpu0. >>>>>>>> >>>>>>>> How does it map ETMs to CPUs? >>>>>>> >>>>>>> The mapping between tracer and CPU is done in the device tree - >>>>>>> as such there seems to be a problem with the DT specification. If the code >>>>>>> can't find a CPU for a tracer it will default to '0', which seems to be >>>>>>> happening here. Look at the dragonboard example [1]. >>>>>>> >>>>>>> Instrumenting the code to understand what is going on is always >>>>>>> a good idea. >>>>>>> >>>>>>> Mathieu >>>>>>> >>>>>>> [1]. >>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux. >>>>>>> g >>>>>>> i >>>>>>> t >>>>>>> / >>>>>>> t >>>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1 >>>>>>> 1 >>>>>>> - >>>>>>> r >>>>>>> c >>>>>>> 7 >>>>>>> # >>>>>>> n >>>>>>> 1 >>>>>>> 1 >>>>>>> 14 >>>>>>> >>>>>>>> >>>>>>>> Regards, Reza >>>>>>>> >>>>>>>> entering ETM4_probe >>>>>>>> >>>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized >>>>>>>> >>>>>>>> Leaving ETMv4 Probe >>>>>>>> >>>>>>>> entering ETMv4_probe >>>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0' >>>>>>>> ------------[ cut here ]------------ >>>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31 >>>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>> Sent: Monday, April 17, 2017 9:23 AM >>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>> Cc: coresight@lists.linaro.org >>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>> >>>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad >>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>> Hello Mathieu >>>>>>>>> >>>>>>>>> We have changed the device tree to reflect all CoreSight >>>>>>>>> components on our platform. >>>>>>>>> Still I do not see any device listed in the following >>>>>>>>> directory >>>>>>>>> >>>>>>>>> /sys/bus/coresight/devices >>>>>>>>> >>>>>>>>> Am I missing something in Linux 4.9 configuration? I have >>>>>>>>> enabled CONFIG_CORESIGHT in .config. >>>>>>>> >>>>>>>> When the system boots AMBA probes for devices connected >>>>>>>> (and >>>>>>>> discoverable) on the bus. The first thing to do is make >>>>>>>> sure the CS devices show up at enumeration time. I >>>>>>>> suggest instrumenting function >>>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. >>>>>>>> If not then it is probably a matter of enabling the debug power domain. >>>>>>>> While debugging I also suggest to boot with the "nohlt" option >>>>>>>> or disable CPUidle completely. That way tracers (usually located in the >>>>>>>> CPU/cluster power domain) are guaranteed to be enabled. >>>>>>>> >>>>>>>> If CS devices show up then you will need to instrument the >>>>>>>> _probe() function of each CS driver to see what makes them unhappy. >>>>>>>> >>>>>>>> Mathieu >>>>>>>> >>>>>>>> [1]. >>>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3 >>>>>>>> 4 >>>>>>>> 4 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, Reza >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org] >>>>>>>>> Sent: Friday, March 31, 2017 3:37 PM >>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>> >>>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad >>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>> Thanks Mathieu >>>>>>>>>> >>>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to >>>>>>>>>> both opencsd and tools/perf makefiles. >>>>>>>>>> If I see more problems, I may have to switch building them >>>>>>>>>> on the target. >>>>>>>>> >>>>>>>>> Ok, let me know. >>>>>>>>> >>>>>>>>> I will be travelling for the next two weeks. As such I may >>>>>>>>> be slow to respond, if at all. While I am away you can always send your >>>>>>>>> questions to "coresight@lists.linaro.org". There is a lot of very >>>>>>>>> knowledgeable people on there that can help you. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, Reza >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mathieu Poirier >>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM >>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>> >>>>>>>>>> For my own sanity I tried doing the same thing and >>>>>>>>>> everything works as advertised. The only difference is that I'm not cross >>>>>>>>>> compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and >>>>>>>>>> the cross compilation isn't part of the equation. >>>>>>>>>> >>>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 >>>>>>>>>> (b50067a52cf3). On the openCSD side I am using the master branch >>>>>>>>>> (054c07caa2eb). >>>>>>>>>> >>>>>>>>>> Mathieu >>>>>>>>>> >>>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad >>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>> Thanks Mathieu >>>>>>>>>>> >>>>>>>>>>> I am using master branch >>>>>>>>>>> >>>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git >>>>>>>>>>> my-opencsd >>>>>>>>>>> git checkout -b master origin/master >>>>>>>>>>> >>>>>>>>>>> Is this a correct branch for opencsd? >>>>>>>>>>> >>>>>>>>>>> I am cross compiling both opencsd and tools/perf. >>>>>>>>>>> >>>>>>>>>>> Regards, Reza >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM >>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>> >>>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad >>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>> Hello Mathieu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have taken all the patches. When building tools/perf we >>>>>>>>>>>> get the following compilation errors. >>>>>>>>>>>> Do you have some ideas if we are missing a patch? >>>>>>>>>>> >>>>>>>>>>> The problem comes from mismatches between the kernel and >>>>>>>>>>> the openCSD version. We try to keep them working for older versions but >>>>>>>>>>> that can't be guaranteed all the time. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> CC util/parse-events.o >>>>>>>>>>>> CC util/parse-events-flex.o >>>>>>>>>>>> CC util/pmu.o >>>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? >>>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] >>>>>>>>>>>> static const char * const cs_etm_global_header_fmts[] = { >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Right, you are building branch 4.9 with a gcc version that >>>>>>>>>>> is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1]. >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3 >>>>>>>>>>> 4 >>>>>>>>>>> 4 >>>>>>>>>>> 5 >>>>>>>>>>> b >>>>>>>>>>> 8 >>>>>>>>>>> 7 >>>>>>>>>>> c >>>>>>>>>>> e >>>>>>>>>>> a >>>>>>>>>>> a4e >>>>>>>>>>> f >>>>>>>>>>> 0 >>>>>>>>>>> 0 >>>>>>>>>>> 15c5d25c4b3 >>>>>>>>>>> >>>>>>>>>>>> CC util/pmu-flex.o >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function >>>>>>>>>>>> ?cs_etm_decoder__gen_trace_elem_printer?: >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each >>>>>>>>>>>> undeclared identifier is reported only once for each >>>>>>>>>>>> function it appears in >>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: >>>>>>>>>>>> ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function) >>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM: >>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> >>>>>>>>>>> Those two are related to the openCSD version. What version >>>>>>>>>>> of the library are you working with? In any case if you stick with the >>>>>>>>>>> latest revision (0.5.4) or TIP you should be fine. The commits that added >>>>>>>>>>> these two is here [2]. >>>>>>>>>>> >>>>>>>>>>> [2]. >>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac >>>>>>>>>>> c >>>>>>>>>>> 3 >>>>>>>>>>> d >>>>>>>>>>> 0 >>>>>>>>>>> a >>>>>>>>>>> f >>>>>>>>>>> d >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> b24 >>>>>>>>>>> 2 >>>>>>>>>>> b >>>>>>>>>>> d >>>>>>>>>>> 9f7328e7104 >>>>>>>>>>> >>>>>>>>>>>> mv: cannot stat >>>>>>>>>>>> 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': >>>>>>>>>>>> No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> Regards, Reza >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM >>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>> >>>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad >>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have links to these patches? >>>>>>>>>>>> >>>>>>>>>>>> It's all there: >>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4. >>>>>>>>>>>> 9 >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM >>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>> >>>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad >>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>> Thanks Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is an in-house board and we had to do some BSP >>>>>>>>>>>>>> changes to boot Linux 4.9 on our board. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok. >>>>>>>>>>>>> >>>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 >>>>>>>>>>>>>> correctly with our source tree. >>>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf >>>>>>>>>>>>>> directory? >>>>>>>>>>>>> >>>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, >>>>>>>>>>>>> that will make life easier on you. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, Reza >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Mathieu Poirier >>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org] >>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM >>>>>>>>>>>>>> To: Etemadi, Mohammad mohammad.etemadi@intel.com >>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad >>>>>>>>>>>>>> mohammad.etemadi@intel.com wrote: >>>>>>>>>>>>>>> Hello Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I >>>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, that should be an interesting project. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for >>>>>>>>>>>>>>> perf-opencsd-4.9. >>>>>>>>>>>>>>> Looks like this tree consists of base >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to >>>>>>>>>>>>>>> support CoreSight and instruction trace using Perf >>>>>>>>>>>>>>> tool. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All the kernel side of the solution is already upstream. >>>>>>>>>>>>>> What is on gitHub it part of the user space perf tools that haven't been >>>>>>>>>>>>>> upstreamed yet (we are working on it) and the openCSD decoding library. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For adding trace functionality, Is it possible to >>>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - >>>>>>>>>>>>>> can you rephrase of give me more details? All the patches on top of >>>>>>>>>>>>>> mainline (on gitHub) should apply cleanly to your tree. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also which files must be changed to reflect specific >>>>>>>>>>>>>>> SOC CoreSight topology? >>>>>>>>>>>>>> >>>>>>>>>>>>>> CoreSight is different on every platform, which is why >>>>>>>>>>>>>> all the platform specific stuff has been pushed to the device tree. Is this >>>>>>>>>>>>>> a commercial board or something in-house? >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the former case support for vexpress[1], juno >>>>>>>>>>>>>> (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also >>>>>>>>>>>>>> something for HiKey that I could share with you if need be. If this is an >>>>>>>>>>>>>> in-house project you will need to make up your own device tree base on the >>>>>>>>>>>>>> topology you are working with. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm always interested by what people are doing with >>>>>>>>>>>>>> CoreSight. Get back to me if you have more questions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Mathieu >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> v >>>>>>>>>>>>>> e >>>>>>>>>>>>>> x >>>>>>>>>>>>>> p >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> ss- >>>>>>>>>>>>>> v >>>>>>>>>>>>>> 2 >>>>>>>>>>>>>> p >>>>>>>>>>>>>> - >>>>>>>>>>>>>> c >>>>>>>>>>>>>> a >>>>>>>>>>>>>> 15_a7.dts [2]. >>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot >>>>>>>>>>>>>> / >>>>>>>>>>>>>> d >>>>>>>>>>>>>> t >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> a >>>>>>>>>>>>>> r >>>>>>>>>>>>>> m >>>>>>>>>>>>>> / >>>>>>>>>>>>>> jun >>>>>>>>>>>>>> o >>>>>>>>>>>>>> - >>>>>>>>>>>>>> b >>>>>>>>>>>>>> a >>>>>>>>>>>>>> s >>>>>>>>>>>>>> e >>>>>>>>>>>>>> .dtsi [3]. >>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv >>>>>>>>>>>>>> a >>>>>>>>>>>>>> l >>>>>>>>>>>>>> d >>>>>>>>>>>>>> s >>>>>>>>>>>>>> / >>>>>>>>>>>>>> l >>>>>>>>>>>>>> i >>>>>>>>>>>>>> n >>>>>>>>>>>>>> u >>>>>>>>>>>>>> x.g >>>>>>>>>>>>>> i >>>>>>>>>>>>>> t >>>>>>>>>>>>>> / >>>>>>>>>>>>>> t >>>>>>>>>>>>>> r >>>>>>>>>>>>>> e >>>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4. >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> 1 >>>>>>>>>>>>>> - >>>>>>>>>>>>>> r >>>>>>>>>>>>>> c >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, Reza >>>>>> _______________________________________________ >>>>>> CoreSight mailing list >>>>>> CoreSight@lists.linaro.org >>>>>> https://lists.linaro.org/mailman/listinfo/coresight >>>>> >>>>> >>>>> >>>>> -- >>>>> Mike Leach >>>>> Principal Engineer, ARM Ltd. >>>>> Blackburn Design Centre. UK