Thanks Mathieu
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like “time: option in the perf script.
Have you tried cycle accurate mode and time stamps?
Regards, Reza
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server
perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
Sent: Monday, June 12, 2017 9:36 AM
To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
Subject: Re: Perf-opencsd-4.9
Good day,
On 9 June 2017 at 10:18, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
Hello Mathieu
Currently I am using CLI commands like "perf record" and "perf script" to configure and enable the trace. I am wondering if there is a set of APIs that we can do the same at
run-time inside the application.
There are currently no plans to work on something like that.
On the "snapshot mode" topic, these are the steps I have done. If I do not use -S option,
It works as expected. I used control-c to terminate the perf and stop the trace.
Application test_server is in a while loop and it does not terminate by itself.
I remember testing snapshot mode while working on the perf API - I'd have to investigate but currently don't have the bandwidth.
Mathieu
#perf record -S -e cs_etm/@8008010000.etf/ --per-thread ./test_server
^C[ perf record: Woken up 2 times to write data ]
Warning:
AUX data lost 1 times out of 1!
[ perf record: Captured and wrote 0.001 MB perf.data ]
#perf script -f --ns -F pid,comm,sym,symoff,ip,cpu
0x368 [0x70]: failed to process type: 10
#
Also looks like timing information is missing from the decoded trace. Here are the commands I have used. It does not like time option in the perf script.
perf record -e cs_etm/timestamp,@8008010000.etf/ --per-thread ./test_server
perf script -f --ns –F time,pid,comm,sym,symoff,ip,cpu
Regards, Reza
-----Original Message-----
From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
Sent: Friday, June 09, 2017 10:19 AM
To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
Subject: Re: Perf-opencsd-4.9
On 9 June 2017 at 08:21, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
> Thanks Mathieu
>
> Do you also know if there is a plan to support Perf APIs for
> configuring, starting/stopping, storage and decoding the trace? Or it will be only supported through CLI?
I'm not completely sure of what you are referring to - in order to avoid confusion could you please expand on the enhancement you are referring to?
>
> Regards, Reza
>
> -----Original Message-----
> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
> Sent: Friday, June 09, 2017 9:00 AM
> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
> Subject: Re: Perf-opencsd-4.9
>
> On 8 June 2017 at 23:42, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>> Hello Mathieu
>>
>> Does "perf" support snapshot mode? I have tried -S option and it does not work.
>> I am using signal to stop the trace but it does not work. How should
>> I stop the trace "snapshot" mode?
>
> I'm not sure it working anymore - it is part of the crums here and there that needs fixing.
>
>>
>> Also "perf record" with -C does not work for me as well. I like to
>> enable the trace only On one core but it does not work for me.
>>
>> I tried the following
>>
>> perf record -C 2 -e cs_etm/@8008010000.etf/ --per-thread taskset -c
>> 2 uname
>
> That doesn't work right now. I am working on it but need to have a
> chat with Peter Z. at Linux Plumbers in
>
> September to see how we're going to address the problem.
>
> Mathieu
>
>>
>>
>> Regards, Reza
>>
>>
>> -----Original Message-----
>> From: Etemadi, Mohammad
>> Sent: Wednesday, May 03, 2017 10:20 AM
>> To: Mathieu Poirier <mathieu.poirier@linaro.org>
>> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
>> Subject: RE: Perf-opencsd-4.9
>>
>> Thanks Mathieu.
>>
>> I also have another question about the usage of "perf"
>>
>> We are interested in a usecase where trace is running on all the cores and we want to stop the trace on all the cores when an event of interest occurs. This is basically "snapshot mode". Is this supported with perf-opencsd-xx? What "perf record" command should we use?
>>
>> Regards, Reza
>>
>> -----Original Message-----
>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>> Sent: Wednesday, May 03, 2017 10:02 AM
>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
>> Subject: Re: Perf-opencsd-4.9
>>
>> On 3 May 2017 at 08:58, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>> Thanks Mathieu
>>>
>>> Do you have a patch for Mike's fix?
>>> I can apply only that one and see if it helps.
>>
>> https://github.com/Linaro/OpenCSD/commit/f78c49583688676522effbea6ab5
>> 4
>> f71bf6907d4
>>
>>>
>>> Regards, Reza
>>>
>>> -----Original Message-----
>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>> Sent: Wednesday, May 03, 2017 9:56 AM
>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
>>> Subject: Re: Perf-opencsd-4.9
>>>
>>> On 3 May 2017 at 08:48, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>> Hello Mathieu
>>>>
>>>> We are using perf-opencsd-4.9.
>>>
>>> Right, that is too old and you will need to upgrade.
>>>
>>>> Currently we are using Linux 4.9 on our target. I think the latest
>>>> is
>>>> perf-opencsd-4.11 Can we move to perf-opencsd-4.11 while using Linux kernel 4.9? Is there a dependency?
>>>
>>> I wouldn't do it - there is always a serious amount of churn in the perf user space tools and it's always better to keep them in sync. If you really want to stay on 4.9 then you'll need to backport the patches in 4.10 and 4.11 to 4.9. It shouldn't be a big deal but understand that after that I can't help you anymore (because the kernel is unreproducible on my side).
>>>
>>>>
>>>> "perf record" shows different X and Y each time I run it for the same application (uname)
>>>> AUX data lost X times out of Y!
>>>
>>> Yes that is normal. It depends on how the application being tested has been scheduled/preempted by the scheduler.
>>>
>>>>
>>>> Is this expected? The values are quite different for each run.
>>>>
>>>> As I mentioned earlier, in some runs, "perf report" does not come back at all.
>>>
>>> Right, let's hope that Mike's fix will help you.
>>>
>>>>
>>>> Regards, Reza
>>>>
>>>> -----Original Message-----
>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>> Sent: Wednesday, May 03, 2017 9:29 AM
>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
>>>> Subject: Re: Perf-opencsd-4.9
>>>>
>>>> What branch are you working with? Mike has seen (and fixed) a problem like that a few weeks ago so make sure to use the latest code. It could also be a totally new problem. In such a case it would be interesting if you could instrument the perf user space code and see where things go wrong.
>>>>
>>>> Mathieu
>>>>
>>>> On 2 May 2017 at 09:20, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>> Thanks Mathieu
>>>>>
>>>>> Sometimes also I am seeing that "perf report --stdio" does not return at all and perf takes lots of
>>>>> CPU cycles. It looks "perf" is trying to process perf.data but it cannot progress.
>>>>>
>>>>> Regards, Reza
>>>>>
>>>>> -----Original Message-----
>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>> Sent: Tuesday, May 02, 2017 9:56 AM
>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>> Cc: Mike Leach <mike.leach@linaro.org>; coresight@lists.linaro.org
>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>
>>>>> It means that out of the 16 trace snippets that were generated, a wrap-around of the internal buffer occurred 6 times. Try using an ETR or reduce the amount of traces you are generating with filters.
>>>>>
>>>>> On 1 May 2017 at 15:18, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>> Hello Mathieu
>>>>>>
>>>>>> There is a warning "AUX data lost 6 times out of 16!" what does this warning imply?
>>>>>>
>>>>>> Regards, Reza
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>> Sent: Friday, April 21, 2017 9:21 AM
>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>> Cc: Mike Leach <mike.leach@linaro.org>;
>>>>>> coresight@lists.linaro.org
>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>
>>>>>> On 20 April 2017 at 13:02, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>> Thanks Mathieu
>>>>>>>
>>>>>>> The issue was related to the number of CPUs. We have 4 clusters each with 4 cores (total 16 cores).
>>>>>>> I have only defined 4 ETMs (only first cluster).
>>>>>>> After setting maxcpus=4 in bootargs, I do not see mmap failure anymore and I am getting the following logs.
>>>>>>> He still somehow thinks there are 16 cores...
>>>>>>
>>>>>> The error message comes from the perf user space tool looking for all the processors known to the system [1]. It is a little chatty but doesn't have an impact on trace collection.
>>>>>>
>>>>>> [1]. /sys/bus/event_source/devices/cs_etm/
>>>>>>
>>>>>>>
>>>>>>> Regards, Reza
>>>>>>>
>>>>>>> ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname
>>>>>>> cs_etm_get_ro: error reading: cpu4/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu4/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu5/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu5/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu6/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu6/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu7/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu7/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu8/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu8/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu9/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu9/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu10/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu10/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu11/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu11/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu12/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu12/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu13/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu13/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu14/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu14/mgmt/etmidr
>>>>>>> cs_etm_get_ro: error reading: cpu15/mgmt/etmccer
>>>>>>> cs_etm_get_ro: error reading: cpu15/mgmt/etmidr Linux [ perf record:
>>>>>>> Woken up 7 times to write data ]
>>>>>>> Warning:
>>>>>>> AUX data lost 6 times out of 16!
>>>>>>>
>>>>>>> [ perf record: Captured and wrote 0.238 MB perf.data ] #
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>> Sent: Thursday, April 20, 2017 11:19 AM
>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>;
>>>>>>> coresight@lists.linaro.org
>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>
>>>>>>> The sink in the perf command line matches what is in sysFS so that's good. At this point you will need to instrument etm_setup_aux [1] to understand what makes it unhappy.
>>>>>>>
>>>>>>> [1].
>>>>>>> http://lxr.free-electrons.com/source/drivers/hwtracing/coresight
>>>>>>> /
>>>>>>> c
>>>>>>> o
>>>>>>> r
>>>>>>> e
>>>>>>> s
>>>>>>> ight-etm-perf.c
>>>>>>>
>>>>>>> On 20 April 2017 at 10:11, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>
>>>>>>>> I have 8008010000.etf in /sys/bus/coresight/devices
>>>>>>>>
>>>>>>>> Reza
>>>>>>>>
>>>>>>>> # ls /sys/bus/coresight/devices 8008010000.etf
>>>>>>>> 80080f0000.mainfunnel 8008810000.funnel 8008c40000.etm
>>>>>>>> 8008d40000.etm 8008e40000.etm 8008f40000.etm #
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Etemadi, Mohammad
>>>>>>>> Sent: Thursday, April 20, 2017 10:50 AM
>>>>>>>> To: 'Mathieu Poirier' <mathieu.poirier@linaro.org>
>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>;
>>>>>>>> coresight@lists.linaro.org
>>>>>>>> Subject: RE: Perf-opencsd-4.9
>>>>>>>>
>>>>>>>> Thanks Mathieu
>>>>>>>>
>>>>>>>> Now, I am hitting another failure. It seems mmap has failed.
>>>>>>>> Does it get the physical address from the name?
>>>>>>>>
>>>>>>>> # ./perf record -e cs_etm/@8008010000.etf/ --per-thread uname
>>>>>>>> failed to mmap with 12 (Cannot allocate memory)
>>>>>>>>
>>>>>>>> Regards, Reza
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>>> Sent: Thursday, April 20, 2017 10:03 AM
>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>;
>>>>>>>> coresight@lists.linaro.org
>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>
>>>>>>>> On 19 April 2017 at 12:42, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>> Thanks Mathieu and Mike
>>>>>>>>>
>>>>>>>>> I rebuild the Linux image with name changes in the device tree and still see the same error.
>>>>>>>>>
>>>>>>>>> Regards, Reza
>>>>>>>>>
>>>>>>>>> # ./perf record -e cs_etm/8008010000.cluster0etf/ --per-thread uname
>>>>>>>>> event syntax error: 'cs_etm/8008010000.cluster0etf/'
>>>>>>>>> \___ BPF support is not compiled
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is not the proper syntax.
>>>>>>>>
>>>>>>>>> # ./perf record -e cs_etm/@8008010000.cluster0etf/ --per-thread uname
>>>>>>>>> event syntax error: 'cs_etm/@8008010000.cluster0etf/'
>>>>>>>>> \___ BPF support is not compiled
>>>>>>>>
>>>>>>>> This is the proper syntax and there is nothing wrong with this command line. As such I did a few test on my side and was able to reproduce the problem. Anything that starts with a 'c' after the '.' will trigger this problem.
>>>>>>>>
>>>>>>>> I don't know if the BPF related error message is real - it could be a false positive. But we definitely have a feature interaction problem.
>>>>>>>> On the flip side "cluster0etf" won't be accepted upstream so at one point you will need to change it to simply "etf", and that works.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>>>> Sent: Wednesday, April 19, 2017 10:58 AM
>>>>>>>>> To: Mike Leach <mike.leach@linaro.org>
>>>>>>>>> Cc: Etemadi, Mohammad <mohammad.etemadi@intel.com>;
>>>>>>>>> coresight@lists.linaro.org
>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>
>>>>>>>>> On 19 April 2017 at 09:43, Mike Leach <mike.leach@linaro.org> wrote:
>>>>>>>>>> Is it also possible that a leading @ is required?
>>>>>>>>>> i.e.
>>>>>>>>>> ./perf record -e cs_etm/@8008010000.cluster0-etf/ --per-thread uname
>>>>>>>>>
>>>>>>>>> Definitely yes - that would be the first thing to try.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> On 19 April 2017 at 16:07, Mathieu Poirier <mathieu.poirier@linaro.org> wrote:
>>>>>>>>>>> On 18 April 2017 at 14:12, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Mathieu
>>>>>>>>>>>>
>>>>>>>>>>>> BPF exists when I am building "perf" but I am getting an error indicating "BPF support is not compiled"
>>>>>>>>>>>> When I am running it on the target. Am I missing a package?
>>>>>>>>>>>>
>>>>>>>>>>>> When building "perf":
>>>>>>>>>>>> .. bpf: [ on ]
>>>>>>>>>>>>
>>>>>>>>>>>> When running "perf" on the target.
>>>>>>>>>>>> ./perf record -e cs_etm/8008010000.cluster0-etf/ --per-thread uname
>>>>>>>>>>>> event syntax error: 'cs_etm/8008010000.cluster0-etf/'
>>>>>>>>>>>> \___ BPF support is not compiled
>>>>>>>>>>>
>>>>>>>>>>> The perf command line parser is insanely complex. Here I
>>>>>>>>>>> suppose the extra "-etf" is causing the bison parser to hit
>>>>>>>>>>> a BPF rule, hence the error message. If you drop the "-etf"
>>>>>>>>>>> in the DT I'm pretty sure the message will go away. You can
>>>>>>>>>>> also enhance the flex/bison parser but to me the "address.name" convention is enough.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Etemadi, Mohammad
>>>>>>>>>>>> Sent: Monday, April 17, 2017 4:03 PM
>>>>>>>>>>>> To: Mathieu Poirier <mathieu.poirier@linaro.org>
>>>>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>>>>> Subject: RE: Perf-opencsd-4.9
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Mathieu
>>>>>>>>>>>>
>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>> Sent: Monday, April 17, 2017 2:51 PM
>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>
>>>>>>>>>>>> On 17 April 2017 at 13:08, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>> Thanks Mathieu
>>>>>>>>>>>>>
>>>>>>>>>>>>> I fixed the power domain issue. Now, it can discover ETMs.
>>>>>>>>>>>>> For some reason, it cannot create /sys/devices/cs_etm/cpu1
>>>>>>>>>>>>> Rather than creating cpu1, it tries to recreate cpu0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How does it map ETMs to CPUs?
>>>>>>>>>>>>
>>>>>>>>>>>> The mapping between tracer and CPU is done in the device tree - as such there seems to be a problem with the DT specification. If the code can't find a CPU for a tracer it will default to '0', which seems to be happening here. Look at the dragonboard example [1].
>>>>>>>>>>>>
>>>>>>>>>>>> Instrumenting the code to understand what is going on is always a good idea.
>>>>>>>>>>>>
>>>>>>>>>>>> Mathieu
>>>>>>>>>>>>
>>>>>>>>>>>> [1].
>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.
>>>>>>>>>>>> g
>>>>>>>>>>>> i
>>>>>>>>>>>> t
>>>>>>>>>>>> /
>>>>>>>>>>>> t
>>>>>>>>>>>> ree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.1
>>>>>>>>>>>> 1
>>>>>>>>>>>> -
>>>>>>>>>>>> r
>>>>>>>>>>>> c
>>>>>>>>>>>> 7
>>>>>>>>>>>> #
>>>>>>>>>>>> n
>>>>>>>>>>>> 1
>>>>>>>>>>>> 1
>>>>>>>>>>>> 14
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>
>>>>>>>>>>>>> entering ETM4_probe
>>>>>>>>>>>>>
>>>>>>>>>>>>> coresight-etm4x 8008c40000.etm: ETM 4.0 initialized
>>>>>>>>>>>>>
>>>>>>>>>>>>> Leaving ETMv4 Probe
>>>>>>>>>>>>>
>>>>>>>>>>>>> entering ETMv4_probe
>>>>>>>>>>>>> sysfs: cannot create duplicate filename '/devices/cs_etm/cpu0'
>>>>>>>>>>>>> ------------[ cut here ]------------
>>>>>>>>>>>>> WARNING: CPU: 0 PID: 1 at fs/sysfs/dir.c:31
>>>>>>>>>>>>> sysfs_warn_dup+0x68/0x88 Modules linked in:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>> Sent: Monday, April 17, 2017 9:23 AM
>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 14 April 2017 at 16:35, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>> Hello Mathieu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have changed the device tree to reflect all CoreSight components on our platform.
>>>>>>>>>>>>>> Still I do not see any device listed in the following
>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /sys/bus/coresight/devices
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When the system boots AMBA probes for devices connected
>>>>>>>>>>>>> (and
>>>>>>>>>>>>> discoverable) on the bus. The first thing to do is make
>>>>>>>>>>>>> sure the CS devices show up at enumeration time. I
>>>>>>>>>>>>> suggest instrumenting function
>>>>>>>>>>>>> amba_device_try_add() [1] to see if CS devices are discovered. If not then it is probably a matter of enabling the debug power domain.
>>>>>>>>>>>>> While debugging I also suggest to boot with the "nohlt" option or disable CPUidle completely. That way tracers (usually located in the CPU/cluster power domain) are guaranteed to be enabled.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If CS devices show up then you will need to instrument the _probe() function of each CS driver to see what makes them unhappy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mathieu
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1].
>>>>>>>>>>>>> http://lxr.free-electrons.com/source/drivers/amba/bus.c#L3
>>>>>>>>>>>>> 4
>>>>>>>>>>>>> 4
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>> Sent: Friday, March 31, 2017 3:37 PM
>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 31 March 2017 at 14:32, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>> Thanks Mathieu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles.
>>>>>>>>>>>>>>> If I see more problems, I may have to switch building them on the target.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok, let me know.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight@lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>> From: Mathieu Poirier
>>>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>>> Sent: Friday, March 31, 2017 3:27 PM
>>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mathieu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 31 March 2017 at 13:52, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>>> Thanks Mathieu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am using master branch
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd
>>>>>>>>>>>>>>>> git checkout -b master origin/master
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is this a correct branch for opencsd?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am cross compiling both opencsd and tools/perf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>> From: Mathieu Poirier
>>>>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>>>> Sent: Friday, March 31, 2017 2:34 PM
>>>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 31 March 2017 at 12:58, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>>>> Hello Mathieu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have taken all the patches. When building tools/perf we get the following compilation errors.
>>>>>>>>>>>>>>>>> Do you have some ideas if we are missing a patch?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CC util/parse-events.o
>>>>>>>>>>>>>>>>> CC util/parse-events-flex.o
>>>>>>>>>>>>>>>>> CC util/pmu.o
>>>>>>>>>>>>>>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts?
>>>>>>>>>>>>>>>>> defined but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = {
>>>>>>>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3
>>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>>> 5
>>>>>>>>>>>>>>>> b
>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>> 7
>>>>>>>>>>>>>>>> c
>>>>>>>>>>>>>>>> e
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> a4e
>>>>>>>>>>>>>>>> f
>>>>>>>>>>>>>>>> 0
>>>>>>>>>>>>>>>> 0
>>>>>>>>>>>>>>>> 15c5d25c4b3
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CC util/pmu-flex.o
>>>>>>>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?:
>>>>>>>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function)
>>>>>>>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_SWTRACE:
>>>>>>>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each
>>>>>>>>>>>>>>>>> undeclared identifier is reported only once for each
>>>>>>>>>>>>>>>>> function it appears in
>>>>>>>>>>>>>>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function)
>>>>>>>>>>>>>>>>> case OCSD_GEN_TRC_ELEM_CUSTOM:
>>>>>>>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [2].
>>>>>>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04ac
>>>>>>>>>>>>>>>> c
>>>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>>> d
>>>>>>>>>>>>>>>> 0
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> f
>>>>>>>>>>>>>>>> d
>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>> b
>>>>>>>>>>>>>>>> b24
>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>> b
>>>>>>>>>>>>>>>> d
>>>>>>>>>>>>>>>> 9f7328e7104
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp':
>>>>>>>>>>>>>>>>> No such file or directory
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>> From: Mathieu Poirier
>>>>>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:15 AM
>>>>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do you have links to these patches?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's all there:
>>>>>>>>>>>>>>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.
>>>>>>>>>>>>>>>>> 9
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>> From: Mathieu Poirier
>>>>>>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 10:06 AM
>>>>>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>>>>>> Thanks Mathieu
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree.
>>>>>>>>>>>>>>>>>>> So, are you saying that I only need to replace tool/perf directory?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>> From: Mathieu Poirier
>>>>>>>>>>>>>>>>>>> [mailto:mathieu.poirier@linaro.org]
>>>>>>>>>>>>>>>>>>> Sent: Thursday, March 30, 2017 9:40 AM
>>>>>>>>>>>>>>>>>>> To: Etemadi, Mohammad <mohammad.etemadi@intel.com>
>>>>>>>>>>>>>>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad <mohammad.etemadi@intel.com> wrote:
>>>>>>>>>>>>>>>>>>>> Hello Matt
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi there,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I
>>>>>>>>>>>>>>>>>>>> like to try instruction tracing using perf-opencsd-4.9.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ok, that should be an interesting project.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9.
>>>>>>>>>>>>>>>>>>>> Looks like this tree consists of base
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Linux kernel 4.9 plus some some additions to
>>>>>>>>>>>>>>>>>>>> support CoreSight and instruction trace using Perf tool.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For adding trace functionality, Is it possible to
>>>>>>>>>>>>>>>>>>>> get a patch that I can apply to my base Linux 4.9?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Mathieu
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/d
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>>>>>>>>>> x
>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>>>>>>>>>> ss-
>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> c
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> 15_a7.dts [2].
>>>>>>>>>>>>>>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> d
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>> m
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> jun
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> b
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>>>>>>>>>> .dtsi [3].
>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torv
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> d
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> i
>>>>>>>>>>>>>>>>>>> n
>>>>>>>>>>>>>>>>>>> u
>>>>>>>>>>>>>>>>>>> x.g
>>>>>>>>>>>>>>>>>>> i
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>>>>>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.
>>>>>>>>>>>>>>>>>>> 1
>>>>>>>>>>>>>>>>>>> 1
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>> c
>>>>>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards, Reza
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> CoreSight mailing list
>>>>>>>>>>> CoreSight@lists.linaro.org
>>>>>>>>>>> https://lists.linaro.org/mailman/listinfo/coresight
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mike Leach
>>>>>>>>>> Principal Engineer, ARM Ltd.
>>>>>>>>>> Blackburn Design Centre. UK