On 26 July 2017 at 09:38, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
> Hello Mathieu, Mike
>
>
Good day,
>
> I have a few questions about perf and decoder road maps and bug fixes. Here
> is a list of issues that
>
> I have found out. I like to know if these are known problems and what is the
> process to get them resolved.
>
>
>
> 1) Snapshot mode (option –S) does not work. Ideally we need to run perf
> record with options –a and –S
>
> to trace in snapshot mode on all the cores
That shouldn't be hard to fix - I'll open a card for this.
>
>
>
> 2) Perf record with option –C also does not work. We want, in some cases,
> limit the trace to a subset of cores
I've been trying to fix this for a while but have failed completely.
To address the issue I don't see any other way than to change the perf
ABI between user and kernel space, something that has been rejected by
the community in no uncertain terms. I will sit down with Peter Z. at
Linux Plumbers in September to see how best to move forward.
This is very high on my list of priorities. It is also the first step
in supporting a multi-sink topology.
>
>
>
> 3) Time and cycle information is in the raw trace but perf decoder cannot
> decode it
That one is for Mike. It will also require modifications to the perf tools.
>
>
>
> Thanks for all your help.
Any time,
Mathieu
>
>
>
> Regards, Reza
On 25 July 2017 at 21:49, yoma sophian <sophian.yoma(a)gmail.com> wrote:
> hi Mathieu:
>>
>> Get (and recompile) the latest revision of the openCSD library - that
>> should do the trick.
> after using the latest revision of the opcdCSD library, compile is pass.
> But I have some question about trace collection and decode flow
> 1. for trace collection, shall we ask perf to save some other places?
> such as USB or somewhere alse
We welcome patches.
> 2. from HOWTO document, is perf.data the trace data from ETM?
The perf.data file contains the trace data from the ETM devices, the
metadata associated with the ETM configuration and miscellaneous perf
events recorded during the session.
> 3. Why we need .debug directory?
That is where all the binaries that were involved in the trace session
are collected. Those are needed by the decoding library for proper
handling of the trace data.
> thanks ur help in advance.
Title: CoreSight: Roadmap and Feedback
Duration: 45 minutes
Type of Session: Presentation + active discussion
Abstract (100 words):
The CoreSight suite of drivers and the openCSD library have come a
long way since they were first introduced. Up to now development has
been squarely aimed at providing a foundation for people to start
working from and has such concentrated on mainstream features. As the
solution matures and is being used by the community trends are
emerging regarding items to address and new features to add. This
session will highlight what Linaro is currently working on and the
roadmap envisioned for the next 12 months. From there the audience
will be encouraged to provide feedback on the plan and given the
opportunity to voice their opinion on what they think are important
features to work on.
Extended Abstract (500 words):
(I do not think and extended abstract is necessary for this topic)
Hello Mathieu
Is there a perf record option to select trace buffer size? For example something like "-m,16" to allocate 16 pages for the trace buffer?
Regards, Reza
hi all:
I use follow command for getting perf-opencsd-master.
git clone -b perf-opencsd-master
https://github.com/Linaro/OpenCSD.git perf-opencsd-master
And git commit sha is 27ecd6f2b547 ("coresight: etr: Add barrier
packet for synchronisation").
When I try to "make -C tools/perf", I get below compile error message:
util/cs-etm-decoder/cs-etm-decoder.c:238:7: error:
‘OCSD_GEN_TRC_ELEM_SWTRACE’ undeclared (first use in this function)
case OCSD_GEN_TRC_ELEM_SWTRACE:
^
util/cs-etm-decoder/cs-etm-decoder.c:238:7: note: each undeclared
identifier is reported only once for each function it appears in
util/cs-etm-decoder/cs-etm-decoder.c:239:7: error:
‘OCSD_GEN_TRC_ELEM_CUSTOM’ undeclared (first use in this function)
case OCSD_GEN_TRC_ELEM_CUSTOM:
Then I try to grep "OCSD_GEN_TRC_ELEM_SWTRACE" both in
perf-opencsd-master and opencsd/decoder directories, but get nothing.
Did I use the wrong git repository or missing any environment setting?
Thanks for you kind help in advance,
On 14 April 2017 at 16:35, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
> Hello Mathieu
>
> We have changed the device tree to reflect all CoreSight components on our platform.
> Still I do not see any device listed in the following directory
>
> /sys/bus/coresight/devices
>
> Am I missing something in Linux 4.9 configuration? I have enabled CONFIG_CORESIGHT in .config.
When the system boots AMBA probes for devices connected (and
discoverable) on the bus. The first thing to do is make sure the CS
devices show up at enumeration time. I suggest instrumenting function
amba_device_try_add() [1] to see if CS devices are discovered. If not
then it is probably a matter of enabling the debug power domain.
While debugging I also suggest to boot with the "nohlt" option or
disable CPUidle completely. That way tracers (usually located in the
CPU/cluster power domain) are guaranteed to be enabled.
If CS devices show up then you will need to instrument the _probe()
function of each CS driver to see what makes them unhappy.
Mathieu
[1]. http://lxr.free-electrons.com/source/drivers/amba/bus.c#L344
>
> Regards, Reza
>
>
> -----Original Message-----
> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
> Sent: Friday, March 31, 2017 3:37 PM
> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
> Subject: Re: Perf-opencsd-4.9
>
> On 31 March 2017 at 14:32, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>> Thanks Mathieu
>>
>> I am using Yocto tool chain. So, I have to add --sysroot to both opencsd and tools/perf makefiles.
>> If I see more problems, I may have to switch building them on the target.
>
> Ok, let me know.
>
> I will be travelling for the next two weeks. As such I may be slow to respond, if at all. While I am away you can always send your questions to "coresight(a)lists.linaro.org". There is a lot of very knowledgeable people on there that can help you.
>
>>
>> Regards, Reza
>>
>> -----Original Message-----
>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>> Sent: Friday, March 31, 2017 3:27 PM
>> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
>> Subject: Re: Perf-opencsd-4.9
>>
>> For my own sanity I tried doing the same thing and everything works as advertised. The only difference is that I'm not cross compiling and I have gcc 5.4.0. We know how to fix the gcc 6.2 problem and the cross compilation isn't part of the equation.
>>
>> For the perf tools I am working with branch perf-openscd-4.9 (b50067a52cf3). On the openCSD side I am using the master branch (054c07caa2eb).
>>
>> Mathieu
>>
>> On 31 March 2017 at 13:52, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>>> Thanks Mathieu
>>>
>>> I am using master branch
>>>
>>> git clone https://github.com/Linaro/OpenCSD.git my-opencsd
>>> git checkout -b master origin/master
>>>
>>> Is this a correct branch for opencsd?
>>>
>>> I am cross compiling both opencsd and tools/perf.
>>>
>>> Regards, Reza
>>>
>>> -----Original Message-----
>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>> Sent: Friday, March 31, 2017 2:34 PM
>>> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
>>> Subject: Re: Perf-opencsd-4.9
>>>
>>> On 31 March 2017 at 12:58, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>>>> Hello Mathieu
>>>>
>>>>
>>>> We have taken all the patches. When building tools/perf we get the following compilation errors.
>>>> Do you have some ideas if we are missing a patch?
>>>
>>> The problem comes from mismatches between the kernel and the openCSD version. We try to keep them working for older versions but that can't be guaranteed all the time.
>>>
>>>>
>>>> CC util/parse-events.o
>>>> CC util/parse-events-flex.o
>>>> CC util/pmu.o
>>>> util/cs-etm.c:1282:27: error: ?cs_etm_global_header_fmts? defined
>>>> but not used [-Werror=unused-const-variable=] static const char * const cs_etm_global_header_fmts[] = {
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Right, you are building branch 4.9 with a gcc version that is higher than 6.2. We have a patch for that on the 4.11-rc1 branch [1].
>>>
>>> [1].
>>> https://github.com/Linaro/OpenCSD/commit/2379238cd554a3445b87ceaa4ef0
>>> 0
>>> 15c5d25c4b3
>>>
>>>> CC util/pmu-flex.o
>>>> util/cs-etm-decoder/cs-etm-decoder.c: In function ?cs_etm_decoder__gen_trace_elem_printer?:
>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: error: ?OCSD_GEN_TRC_ELEM_SWTRACE? undeclared (first use in this function)
>>>> case OCSD_GEN_TRC_ELEM_SWTRACE:
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>> util/cs-etm-decoder/cs-etm-decoder.c:204:7: note: each undeclared
>>>> identifier is reported only once for each function it appears in
>>>> util/cs-etm-decoder/cs-etm-decoder.c:205:7: error: ?OCSD_GEN_TRC_ELEM_CUSTOM? undeclared (first use in this function)
>>>> case OCSD_GEN_TRC_ELEM_CUSTOM:
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Those two are related to the openCSD version. What version of the library are you working with? In any case if you stick with the latest revision (0.5.4) or TIP you should be fine. The commits that added these two is here [2].
>>>
>>> [2].
>>> https://github.com/Linaro/OpenCSD/commit/885cb935cf04acc3d0afd2bb242b
>>> d
>>> 9f7328e7104
>>>
>>>> mv: cannot stat 'util/cs-etm-decoder/.cs-etm-decoder.o.tmp': No such
>>>> file or directory
>>>>
>>>> Regards, Reza
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>> Sent: Thursday, March 30, 2017 10:15 AM
>>>> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
>>>> Subject: Re: Perf-opencsd-4.9
>>>>
>>>> On 30 March 2017 at 09:12, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>>>>> Thanks.
>>>>>
>>>>> Do you have links to these patches?
>>>>
>>>> It's all there:
>>>> https://github.com/Linaro/OpenCSD/commits/perf-opencsd-4.9
>>>>
>>>>>
>>>>> Regards, Reza
>>>>>
>>>>> -----Original Message-----
>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>> Sent: Thursday, March 30, 2017 10:06 AM
>>>>> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>
>>>>> On 30 March 2017 at 08:51, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>>>>>> Thanks Mathieu
>>>>>>
>>>>>> This is an in-house board and we had to do some BSP changes to boot Linux 4.9 on our board.
>>>>>
>>>>> Ok.
>>>>>
>>>>>> I just want to make sure that I merge perf-openCSD-4.9 correctly with our source tree.
>>>>>> So, are you saying that I only need to replace tool/perf directory?
>>>>>
>>>>> Simply apply the patches you find on perf-opencsd-4.9, that will make life easier on you.
>>>>>
>>>>>>
>>>>>> Regards, Reza
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mathieu Poirier [mailto:mathieu.poirier@linaro.org]
>>>>>> Sent: Thursday, March 30, 2017 9:40 AM
>>>>>> To: Etemadi, Mohammad <mohammad.etemadi(a)intel.com>
>>>>>> Subject: Re: Perf-opencsd-4.9
>>>>>>
>>>>>> On 29 March 2017 at 19:47, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
>>>>>>> Hello Matt
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We have Linux 4.9 running on our ARMv8 platform. I like to try
>>>>>>> instruction tracing using perf-opencsd-4.9.
>>>>>>
>>>>>> Ok, that should be an interesting project.
>>>>>>
>>>>>>>
>>>>>>> I noticed that there is a Linaro Linux source tree for perf-opencsd-4.9.
>>>>>>> Looks like this tree consists of base
>>>>>>>
>>>>>>> Linux kernel 4.9 plus some some additions to support CoreSight
>>>>>>> and instruction trace using Perf tool.
>>>>>>
>>>>>> All the kernel side of the solution is already upstream. What is on gitHub it part of the user space perf tools that haven't been upstreamed yet (we are working on it) and the openCSD decoding library.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> For adding trace functionality, Is it possible to get a patch
>>>>>>> that I can apply to my base Linux 4.9?
>>>>>>
>>>>>> I'm not sure of what you mean by "a patch I can add" - can you rephrase of give me more details? All the patches on top of mainline (on gitHub) should apply cleanly to your tree.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Also which files must be changed to reflect specific SOC CoreSight topology?
>>>>>>
>>>>>> CoreSight is different on every platform, which is why all the platform specific stuff has been pushed to the device tree. Is this a commercial board or something in-house?
>>>>>>
>>>>>> In the former case support for vexpress[1], juno (R0/1/2)[2] and the dragonboard[3] are already upstream. There is also something for HiKey that I could share with you if need be. If this is an in-house project you will need to make up your own device tree base on the topology you are working with.
>>>>>>
>>>>>> I'm always interested by what people are doing with CoreSight. Get back to me if you have more questions.
>>>>>>
>>>>>> Regards,
>>>>>> Mathieu
>>>>>>
>>>>>> [1].
>>>>>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/vexpress-v2
>>>>>> p
>>>>>> -
>>>>>> c
>>>>>> a
>>>>>> 15_a7.dts [2].
>>>>>> http://lxr.free-electrons.com/source/arch/arm64/boot/dts/arm/juno-
>>>>>> b
>>>>>> a
>>>>>> s
>>>>>> e
>>>>>> .dtsi [3].
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>>>> /
>>>>>> t
>>>>>> r
>>>>>> e
>>>>>> e/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.11-rc4
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards, Reza
Hello Mathieu
We also like to trace minor page fault in the kernel when a user thread hits a page fault.
I am using the following perf command to start the trace for a given thread.
perf record -e cs_etm/(a)8008046000.etf/ --per-thread --pid 2231&
Trace captured has only user space activity and I do not see any kernel trace.
Is this the correct perf command?
Regards, Reza
Hi Thierry,
I see you have also sent this mail to Mathieu who has answered some of the points and cc:ed the linaro coresight mailing list.
I'll give you my spin on a couple of things here.....
> -----Original Message-----
> From: Thierry Laviron
> Sent: 29 June 2017 15:45
> To: Mike Leach
> Subject: Using Coresight in SysFS mode on Juno board
>
> Hi Mike,
>
>
>
> I am currently trying to get trace data using the CoreSight system in SysFS
> mode on my Juno r2 board.
>
>
>
> I found some documentation on how to use it in the
> Documentation/trace/coresight.txt file of the perf-opencsd-4.11 branch of the
> OpenCSD repository.
>
>
>
> This document says that I can retrieve the trace data from /dev/ using dd, for
> example in my case that would be
>
> root@juno-debian:~# dd if=/dev/20070000.etr of=~/cstrace.bin
>
>
>
> However, I am assuming this produces a dump of the memory buffer as it was
> when I stopped trace collection,
>
> And that I do not have the full trace data generated (because it does not fit on
> the buffer).
>
> I would like to be able to capture a continuous stream of data from the ETR, but
> did not find how should I do that.
>
It is not possible to read trace while still collecting it - the process you are tracing must be stopped while trace is saved. Perf can achieve this as it is integrated into the kernel, but this is difficult to achieve from the sysfs interface.
As Mathieu says, you need to limit the amount of trace to the application you are tracing - but even so, the rate of trace collection can easily overflow buffers.
>
>
> I am writing a C program. Can I open a read access to the ETR buffer like this?
>
> open("/dev/20070000.etr", O_RDONLY);
>
>
>
> and then read its content, to write somewhere else? (e.g. to a file on the disc)?
>
>
>
> As a second step, I am also trying to filter the trace generated. I found some
> useful documentation in
>
> Documentation/ABI/testing/sysfs-bus-coresight-devices-etm4x
>
> However, while this is very useful to understand what are the purpose of the
> different files that appear in the
>
> /sys/bus/coresight/devices/<mmap>.etm/ folders, I am not sure of the format
> to put stuff in.
>
>
>
> For example, I want to use the Context ID comparator, so the ETM traces only
> the process I am interested in.
>
> I assume I need to write the PID of my process in ctxid_pid, probably write 0x1
> in ctxid_idx to activate it, and leave 0x0 in ctxid_mask
>
> according to the ETM v4.3 architecture specification.
>
> But I feel that I am missing something else, as it seems the ETM is not taking
> the filter into account.
>
i) you will need to have enabled PID=>context idr tracking in your kernel.
ii) you need to set up the ViewInst event resource selector to select a context ID event to start and stop the trace, in addition to setting the context ID comparators.
Additionally you will need some address range enabled as well - though by default the etm drivers set up the full address range under sysfs.
The hardware registers needed for all this are described in the ETM TRM, but at present I don't know of any docs that map the sysfs names onto the relevant HW registers.
Regards
Mike
>
>
> If there is more relevant documentation on this that I have not found, I would
> appreciate if you could point me to it.
>
> If not, and what I am trying to do will not work, I would welcome some advice
> on how to do it properly.
>
>
>
> Thanks in advance.
>
>
>
> Best regards,
>
>
>
> Thierry Laviron
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 11 July 2017 at 09:25, Etemadi, Mohammad <mohammad.etemadi(a)intel.com> wrote:
> Hello Mathew
>
>
>
> In our platform we have a few clusters each has its own funnel and ETF.
> There is no ETR. Each cluster has 4 ETMs.
>
> How can I enable the trace for all the clusters? Does the following commands
> only enables the trace in one cluster?
>
> How can I enable trace for all the clusters?
>
>
>
>
>
> perf record -e cs_etm/(a)8008010000.etf/ --per-thread uname
>
> perf record -e cs_etm/(a)8008030000.etf/ --per-thread uname
>
> perf record -e cs_etm/(a)8008050000.etf/ --per-thread uname
>
> perf record -e cs_etm/(a)8008070000.etf/ --per-thread uname
Hi Reza,
I'm adding the CoreSight mailing list, there is a lot of knowledgeable
people on there that can help you if I'm not around or don't know the
answer to your questions. I suggest you CC the list when you need
information.
>From the above description I deduce the platform doesn't have a common
sink for all the tracers available on the board - instead it has an
ETF for each cluster. When I wrote the CS drivers I could foresee
this kind of topology would show up one day but simply didn't have any
HW to test on. As such it was written with the assumption that all
tracers have a common sink. Yours is the second platform I'm being
told about where my initial assumptions don't hold anymore.
Unfortunately there is no way to enable traces for all clusters with
the current implementation. I have plans to work on it though... For
now you will need to use the taskset utility to confine an application
to specific processors. So something like:
# perf record -e cs_etm/(a)8008010000.etf/ --per-thread taskset 0x($MASK) uname
will do the trick.
Thanks,
Mathieu
>
>
>
>
>
> Regards, Reza
