Good morning Mike,

I am currently travelling and as such won't be able to look at your email until Monday the 13th.

Thanks, Mathieu

On Fri, 10 Aug 2018 at 05:24, Mike Leach mike.leach@linaro.org wrote:

Hello,

On 7 August 2018 at 14:43, Mike Bazov mike@perception-point.io wrote:

...

Greetings,

I'm trying to enable the Coresight ETMv4 trace from kernel mode. I saw there is no documentation on how to do this, except using the sysfs user mode interface and perf.

To overcome this i looked a little bit in the coresight.h header file, and came to these APIs:

...

extern int coresight_enable(struct coresight_device *csdev); extern void coresight_disable(struct coresight_device *csdev);

And the sysfs implementation uses these APIs when enabing/disabling the trace code, so i thought this could suit my needs. The next problem was actually getting the coresight devices data structures, which aren't exported and are actually provided internally to perf and sysfs. So i exported the coresight bus type:

...

struct bus_type coresight_bustype = { .name = "coresight", }; EXPORT_SYMBOL(coresight_bustype);

And enumerated the bus and just looked for the type CORESIGHT_DEV_TYPE_SOURCE, since the only source on my board is ETMv4, there isn't any conflicts so i should get only ETMv4s.

This worked, and indeed i collected a coresight device for every CPU, and enabled the trace successfully while being in non-atomic context.

The problem is, i must enable the trace in atomic context synchronously on the current thread's CPU(i can't issue a work-queue to enable the trace for me). So.. i got many BUG() errors because of non-atomic API usage, for example, the allocation of the TMC-ETR buffer:

...

dma_alloc_coherent(drvdata->dev, etr_buf->size, &flat_buf->daddr, GFP_KERNEL);

So my questions are:

1. Is there a more documented way of enabling coresight from kernel mode? i

believe i achieved this using cheats.

Not at present. The perf and sysfs methods are those supported for users to get at trace.

...

2. I see there is no exported kernel API to config the coresight trace

attributes(for example, filter EL0). I can only do so from sysfs.. am i missing something?

Additional kernel APIs are obviously feasible - assuming there is a need for them and resource to implement them. Evidently you have a use cases that satisfy the first condition.

We are planning updates to the system to handle complex configuration

• i.e. where muliple configurable filters / sequencers / counters need

setting from the kernel / sysfs - possibly in the form of a configuration file or some such method. This is in part to avoid having to massively overload the perf command line options to set up these configurations.

Once this is in play then this should make it easier for general configuration within the kernel.

...

3. Are there any plans to make the Coresight infrastructure atomic-context

friendly? If there are, is the development in progress? if not.. how would you suggest tackling the issues i've described in this message?

Again - no plans at present - though the lack of a more formal kernel API may have resulted in this not being considered to date. Currently enabling an ETM will result in enabling of all devices in the path to the sink being used - and with the ETR enabling results in the memory buffer for the ETR being allocated.

Assuming you are able to configure CoreSight outside an atomic context, one approach may be to allow preparatory work to happen outside the atomic-context, and leave the final enable of devices inside the atomic context. However this approach would seem to require some significant restructuring of existing code.

Matthieu Poirier - who maintains the CS drivers in the kernel and is on this list, should have further insights when he returns from vacation.

Regards

Mike

...

Thank you!

---

CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

On Mon, 13 Aug 2018 at 09:13, Mike Bazov mike@perception-point.io wrote:

Hello,

...

Good morning Mike, I am currently travelling and as such won't be able to look at your email until Monday the 13th.

Sure thing.

Mike, thanks for the detailed answer.

Yes, everything in Mikes' answer is accurate.

I'm currently in a need of a well defined kernel API for enabling a source and a sink. Generally, i need a similar functionality that sysfs provides, only in a form of kernel-mode API. I need to enable a source from everywhere in the kernel(to keep it simple, non-atomic context), and disable it. Also, enable a sink and keep it enabled even if a source is no longer enabled. Mike, so i understand this kind of feature isn't planned. I'm thinking about doing it myself... are Coresight patches welcomed? Is there an "easier" way of achieving what i want?

Patches are always welcomed and I don't think there is an "easy" way to get out of this one. What you want to do will probably end up being fairly complex. I would start by closely understanding how operation of the CS infrastructure is done from the perf interface - you should be find just sticking to the kernel part. There reservation of a "path" and memory for the sink is done in preparatory steps where it is permitted to sleep (non-atomic). After that components can be enabled from an atomic context, i.e when the process of interest is installed on a processor. Currently things are woven with the perf_aux_output_[begin|end]() interface but that could easily be decoupled.

On the aspect of trace collection, did you envision using the entries in devFS? If that is the case a mechanism to correlate tracer configuration and trace data will need to be developed, just like what we did for perf.

Taking a step back, tracers can also be found on X86 and MIPs (if I'm not mistaking) architectures. As such the new kernel API would have to be usable by those as well, which complicates the task even further.

So all that being said I think it is feasible, but be prepared to invest a significant amount of time and effort.

Regards, Mathieu

Thanks, Mike.

On Fri, Aug 10, 2018 at 4:43 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:

...

Good morning Mike,

I am currently travelling and as such won't be able to look at your email until Monday the 13th.

Thanks, Mathieu

On Fri, 10 Aug 2018 at 05:24, Mike Leach mike.leach@linaro.org wrote:

...

Hello,

On 7 August 2018 at 14:43, Mike Bazov mike@perception-point.io wrote:

...

Greetings,

I'm trying to enable the Coresight ETMv4 trace from kernel mode. I saw there is no documentation on how to do this, except using the sysfs user mode interface and perf.

To overcome this i looked a little bit in the coresight.h header file, and came to these APIs:

...

extern int coresight_enable(struct coresight_device *csdev); extern void coresight_disable(struct coresight_device *csdev);

And the sysfs implementation uses these APIs when enabing/disabling the trace code, so i thought this could suit my needs. The next problem was actually getting the coresight devices data structures, which aren't exported and are actually provided internally to perf and sysfs. So i exported the coresight bus type:

...

struct bus_type coresight_bustype = { .name = "coresight", }; EXPORT_SYMBOL(coresight_bustype);

And enumerated the bus and just looked for the type CORESIGHT_DEV_TYPE_SOURCE, since the only source on my board is ETMv4, there isn't any conflicts so i should get only ETMv4s.

This worked, and indeed i collected a coresight device for every CPU, and enabled the trace successfully while being in non-atomic context.

The problem is, i must enable the trace in atomic context synchronously on the current thread's CPU(i can't issue a work-queue to enable the trace for me). So.. i got many BUG() errors because of non-atomic API usage, for example, the allocation of the TMC-ETR buffer:

...

dma_alloc_coherent(drvdata->dev, etr_buf->size, &flat_buf->daddr, GFP_KERNEL);

So my questions are:

1. Is there a more documented way of enabling coresight from kernel mode? i

believe i achieved this using cheats.

Not at present. The perf and sysfs methods are those supported for users to get at trace.

...

2. I see there is no exported kernel API to config the coresight trace

attributes(for example, filter EL0). I can only do so from sysfs.. am i missing something?

Additional kernel APIs are obviously feasible - assuming there is a need for them and resource to implement them. Evidently you have a use cases that satisfy the first condition.

We are planning updates to the system to handle complex configuration

• i.e. where muliple configurable filters / sequencers / counters need

setting from the kernel / sysfs - possibly in the form of a configuration file or some such method. This is in part to avoid having to massively overload the perf command line options to set up these configurations.

Once this is in play then this should make it easier for general configuration within the kernel.

...

3. Are there any plans to make the Coresight infrastructure atomic-context

friendly? If there are, is the development in progress? if not.. how would you suggest tackling the issues i've described in this message?

Again - no plans at present - though the lack of a more formal kernel API may have resulted in this not being considered to date. Currently enabling an ETM will result in enabling of all devices in the path to the sink being used - and with the ETR enabling results in the memory buffer for the ETR being allocated.

Assuming you are able to configure CoreSight outside an atomic context, one approach may be to allow preparatory work to happen outside the atomic-context, and leave the final enable of devices inside the atomic context. However this approach would seem to require some significant restructuring of existing code.

Matthieu Poirier - who maintains the CS drivers in the kernel and is on this list, should have further insights when he returns from vacation.

Regards

Mike

...

Thank you!

---

CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

Hi Mike,

Interesting idea. Could you explain how you deal with the situation where paths for different sources need different configuration of the downstream CoreSight? The situation I’m thinking of is (ignoring funnels):

ETM#1 \ ETF -> ETR / ETM#2

A path with source ETM#1 to sink ETF can’t be active at the same time as a path from source ETM#2 to sink ETR, because the former will need the ETF to be in buffer mode while the latter will need the ETF to be in FIFO mode.

I’d expect you could build these two incompatible paths, but not simultaneously enable them? So coresight_enable_path would check that any other paths using the same ETF were using it in the same mode, and if it was idle, it would switch it into the right mode.

Also how is the trace source id handled? As we have only about 120 possible trace source ids and we have chips with 128 cores funnelling into one sink, we can’t have a fixed allocation of trace sources to trace source ids (i.e. we can’t fix it in the device tree or anything like that). So we need to be able to dynamically allocate trace source ids. Could that be done in coresight_enable_path? So all enabled paths would have distinct trace source ids.

Al

From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of Mike Bazov Sent: 14 August 2018 14:01 To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: Re: Enabling Coresight in atomic context.

Hello,

Patches are always welcomed and I don't think there is an "easy" way to get out of this one. What you want to do will probably end up being fairly complex. I would start by closely understanding how operation of the CS infrastructure is done from the perf interface you should be find just sticking to the kernel part. There reservation of a "path" and memory for the sink is done in preparatory steps where it is permitted to sleep (non-atomic). After that components can be enabled from an atomic context, i.e when the process of interest is installed on a processor. Currently things are woven with the perf_aux_output_[begin|end]() interface but that could easily be decoupled.

On the aspect of trace collection, did you envision using the entries in devFS? If that is the case a mechanism to correlate tracer configuration and trace data will need to be developed, just like what we did for perf.

Taking a step back, tracers can also be found on X86 and MIPs (if I'm not mistaking) architectures. As such the new kernel API would have to be usable by those as well, which complicates the task even further. So all that being said I think it is feasible, but be prepared to invest a significant amount of time and effort.

The "generic" tracing kernel API is a different thing. In it's Coresight implementation it will use the kernel API I need. After taking a few days to understand how the infrastructure works, to make the API as flexible as it can be, I thought about this: Just like there's a perf implementation and a sysfs implementation, the "api" implementation(coresight-api) will be introduced, which will also be a new mode(CS_MODE_API).

I propose these APIs(some of them exist, but need to be exported and changed a little):

* coresight_build_path(struct coresight_device *source, struct coresight_device *sink): Create a coresight path from the provided source and sink,.

* coresight_enable_path(struct coresight_path *path): Enable a Coresight path except the source. This will also glue a source to a specific path. You cannot assign a different path to this source until the path is destroyed.

* coresight_disable_path(struct coresight_path *path) Disable the path to the sink, including the sink.(if there is more than 1 path to the same sink, does not disable the sink until a refcount reaches 0).

* coresight_destroy_path(struct coresight_path *path): Frees the path, releases the source from that path. The source device can be assigned to a different path.

* coresight_enable_source(struct coresight_device *source); Enables the source. This will actually make the source device play the actual trace data in to the sink(i.e. etm4_enable_hw(), or, increase a refcount if the source is already playing). Uses the path assigned in "coresight_enable_sink()".

* coresight_disable_source(struct coresight_device *source); Disables the source. This will stop the source from playing trace data(or, if the refcount > 0, decrease the refcount). Uses the path assigned in "coresight_enable_sink()".

* coresight_read_sink(struct coresight_device *sink, void *buf, size_t size); Read trace data from the sink(advance the read pointer).

* coresight_setup_sink_buffer(struct coresight_device *sink, void *pages, int nr_pages); Allocate a sink buffer(similar to the perf functionality)

The sysfs and api modes will use different buffers to avoid collision.

I realize most of the API is actually making the internal coresight implementation "public", but I really think this is necessary. Building a path to a specific sink is something a user would want to do, as well as disabling and enabling the path whenever he wishes(this is something I actually need).

In order to use this API, the user needs a method of getting the actual (struct coresight_device *). There will be a list of coresight devices exported in the "coresight.h" header, which can be iterated using a macro "foreach_coresight_device()". The user will be able to extract a specific sink and source for his needs.

I think this API is powerful, and will give the user full Coresight functionality. From diving into the code, this seems very possible, and will not require major infrastructure changes. I will appreciate your thoughts, tips, and hints.

Thanks, Mike. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

On Tue, 14 Aug 2018 at 08:10, Mike Bazov mike@perception-point.io wrote:

Hello,

...

Interesting idea. Could you explain how you deal with the situation where

...

paths for different sources need different configuration of the downstream

...

CoreSight? The situation I’m thinking of is (ignoring funnels):

...

...

ETM#1

...

\

...

   ETF -> ETR

...

/

...

ETM#2

...

...

A path with source ETM#1 to sink ETF can’t be active at the same time as a

...

path from source ETM#2 to sink ETR, because the former will need the ETF

...

to be in buffer mode while the latter will need the ETF to be in FIFO mode.

...

...

I’d expect you could build these two incompatible paths, but not

...

simultaneously enable them? So coresight_enable_path would check

...

that any other paths using the same ETF were using it in the same mode,

...

and if it was idle, it would switch it into the right mode.

I haven't thought about that scenario, but you are right. Also the solution you've provided seems great.

When enabling a path, you can only enable it if it isn't already enabled, or if it is enabled with the same configuration.

Depending on the trace scenario there is a couple more things to keep in mind:

1) CPU-wide or per-thread mode: In CPU-wide mode sources can use the same sink simultaneously. In per-thread mode only a single source can use the sink.

2) The session: In CPU-wide mode we don't want sources from different trace session to use the same sink.

...

Also how is the trace source id handled? As we have only about 120 possible

...

trace source ids and we have chips with 128 cores funnelling into one sink,

...

we can’t have a fixed allocation of trace sources to trace source ids (i.e. we

...

can’t fix it in the device tree or anything like that). So we need to be able to

...

dynamically allocate trace source ids. Could that be done in

...

coresight_enable_path? So all enabled paths would have distinct

...

trace source ids.

Seems like a great idea for the API.

One thing i can't really understand is, why haven't these problems occur in sysfs/perf mode? seems like they aren't really

specific to the API I proposed.

Simply because CS is complex, quirky, in the process of maturing and the team has very limited resources. We fix things based on the use case we currently work on.

Thanks,

Mike.

On Tue, Aug 14, 2018 at 4:31 PM, Al Grant Al.Grant@arm.com wrote:

...

Hi Mike,

Interesting idea. Could you explain how you deal with the situation where

paths for different sources need different configuration of the downstream

CoreSight? The situation I’m thinking of is (ignoring funnels):

ETM#1

  \

    ETF -> ETR

  /

ETM#2

A path with source ETM#1 to sink ETF can’t be active at the same time as a

path from source ETM#2 to sink ETR, because the former will need the ETF

to be in buffer mode while the latter will need the ETF to be in FIFO mode.

I’d expect you could build these two incompatible paths, but not

simultaneously enable them? So coresight_enable_path would check

that any other paths using the same ETF were using it in the same mode,

and if it was idle, it would switch it into the right mode.

Also how is the trace source id handled? As we have only about 120 possible

trace source ids and we have chips with 128 cores funnelling into one sink,

we can’t have a fixed allocation of trace sources to trace source ids (i.e. we

can’t fix it in the device tree or anything like that). So we need to be able to

dynamically allocate trace source ids. Could that be done in

coresight_enable_path? So all enabled paths would have distinct

trace source ids.

Al

From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of Mike Bazov Sent: 14 August 2018 14:01 To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: Re: Enabling Coresight in atomic context.

Hello,

...

Patches are always welcomed and I don't think there is an "easy" way

...

to get out of this one. What you want to do will probably end up

...

being fairly complex. I would start by closely understanding how

...

operation of the CS infrastructure is done from the perf interface

...

you should be find just sticking to the kernel part. There

...

reservation of a "path" and memory for the sink is done in preparatory

...

steps where it is permitted to sleep (non-atomic). After that

...

components can be enabled from an atomic context, i.e when the process

...

of interest is installed on a processor. Currently things are woven

...

with the perf_aux_output_[begin|end]() interface but that could easily

...

be decoupled.

...

On the aspect of trace collection, did you envision using the entries

...

in devFS? If that is the case a mechanism to correlate tracer

...

configuration and trace data will need to be developed, just like what

...

we did for perf.

...

Taking a step back, tracers can also be found on X86 and MIPs (if I'm

...

not mistaking) architectures. As such the new kernel API would have

...

to be usable by those as well, which complicates the task even

...

further.

...

So all that being said I think it is feasible, but be prepared to

...

invest a significant amount of time and effort.

The "generic" tracing kernel API is a different thing. In it's Coresight implementation it will use the kernel API I need.

After taking a few days to understand how the infrastructure works, to make the API as flexible as it can be, I thought about this:

Just like there's a perf implementation and a sysfs implementation, the "api" implementation(coresight-api) will be introduced, which will also be

a new mode(CS_MODE_API).

I propose these APIs(some of them exist, but need to be exported and changed a little):

• coresight_build_path(struct coresight_device *source, struct coresight_device *sink):

  Create a coresight path from the provided source and sink,.

• coresight_enable_path(struct coresight_path *path): Enable a Coresight path except the source. This will

  also glue a source to a specific path. You cannot assign a different path to this source until the path is destroyed.

• coresight_disable_path(struct coresight_path *path)

  Disable the path to the sink, including the sink.(if there is more than 1 path to the same sink, does not disable the sink until a refcount reaches 0).

• coresight_destroy_path(struct coresight_path *path):

  Frees the path, releases the source from that path. The source device can be assigned to a different path.

• coresight_enable_source(struct coresight_device *source);

  Enables the source. This will actually make the source device play the actual trace data in to the sink(i.e. etm4_enable_hw(), or, increase a refcount if

  the source is already playing). Uses the path assigned in "coresight_enable_sink()".

• coresight_disable_source(struct coresight_device *source);

  Disables the source. This will stop the source from playing trace data(or, if the refcount > 0, decrease the refcount).

  Uses the path assigned in "coresight_enable_sink()".

• coresight_read_sink(struct coresight_device *sink, void *buf, size_t size);

  Read trace data from the sink(advance the read pointer).

• coresight_setup_sink_buffer(struct coresight_device *sink, void *pages, int nr_pages);

  Allocate a sink buffer(similar to the perf functionality)

The sysfs and api modes will use different buffers to avoid collision.

I realize most of the API is actually making the internal coresight implementation "public", but I really think this is necessary. Building a path to a specific sink

is something a user would want to do, as well as disabling and enabling the path whenever he wishes(this is something I actually need).

In order to use this API, the user needs a method of getting the actual (struct coresight_device *). There will be a list of coresight devices

exported in the "coresight.h" header, which can be iterated using a macro "foreach_coresight_device()". The user will be able to extract a specific

sink and source for his needs.

I think this API is powerful, and will give the user full Coresight functionality. From diving into the code, this seems very possible,

and will not require major infrastructure changes.

I will appreciate your thoughts, tips, and hints.

Thanks, Mike.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

On Thu, 16 Aug 2018 at 02:10, Mike Bazov mike@perception-point.io wrote:

...

Depending on the trace scenario there is a couple more things to keep in mind:

1. CPU-wide or per-thread mode: In CPU-wide mode sources can use the

same sink simultaneously. In per-thread mode only a single source can use the sink. 2) The session: In CPU-wide mode we don't want sources from different trace session to use the same sink.

CPU-wide/per-thread are different names for sysfs/perf?(respectively).

They are not.

When tracing via sysFS and keeping the default configuration, everything that is happening on a processor is logged. That is called "CPU-wide". Any process that get scheduled out of the processor won't be traced. On perf one can execute:

# perf record -e cs_etm/@20070000.etr/ --per-thread my_application (example 1)

# perf record -e cs_etm/@20070000.etr/ -C 2,3,4 my_application (example 2)

For example 1, perf will switch on the tracer associated to the CPU where my_application has been installed for execution. If the process gets scheduled on a different CPU perf will do the right thing and follow it around. That is called "per-thread".

In example 2 everything that is happening on CPU 2,3,4 will be traced for as long as my_application is executing, regardless of where my_application is executing. That is also a CPU-wide trace scenario.

When you say "we don't want sources from different trace session to use the same sink", you mean we don't want to mix a sink with perf/sysfs modes simultaneously on the same sink?

First, we never want perf and sysFS to use the same sink.

Second, say we have the following two commands being executed one after the other:

# perf record -e cs_etm/@20070000.etr/ -C 0,1 my_application # perf record -e cs_etm/@20070000.etr/ -C 2,3 another_application

We can't allow the second session to use the same sink as the first one.

...

Look at what we've done in coresight-etm-perf.c. There you'll find code that uses the internal CS implementation to serve the perf interface. I suggest you do the same thing but to serve internal kernel clients (which I suspect will ultimately be some sort of module). The hope here is to expose as little as possible of the internal CS core code.

I think i'll introduce a module called "coresight-api.c", that implements the API. I'll just send the final proof-of-concept for the mailing list's review, i'm working on it.

...

As for the above proposition I can't comment much without having a better understanding of what you're trying to do. I suggest to put a proof of concept together and send it to the list for review - that will help us understand what you want to do.

Will do.

...

While doing so I suggest you keep an eye on the coresight next tree[1]. There is a lot of things currently happening in the subsystem and the goal posts are moving regularly.

Thanks.

Some more technical questions regarding the code(things i encountered as part of the development process), i'd appreciate some guidance:

1. I noticed coresight_enable()/coresight_disable() are actually exported for other modules to use, and declared in "coresight.h". Is there any reason to export these functions of only sysfs uses them? Like i said in the first post, i used these APIs, but i had a really hard time finding the coresight device i need from the existing code without exporting the coresight device bus symbol.. so they aren't really practical. Why not put them in "coresight-priv.h"?

That is a relic from the original implementation where individual drivers could be compiled as modules. That functionality was removed because it quickly became difficult to account for all the failure conditions when modules are removed arbitrarily. The symbols don't need to be exported and at first glance the function can probably be move to coresight-priv.h

2. I noticed the reference counting for enabling/disabling coresight sources is actually only respected in "coresight.c", and not in the perf implementation. When enabling a source in perf the reference count is increased(source_ops(source)->enable() is called directely). Any reason for doing so?

The first interface the CS framework worked with was sysFS. After that the perf API became available. Confronted with the vastness and complexity of setting up a complete end-to-end solution (from kernel driver to user space decoding) my goal quickly became to "just make it work". The logic was that once we have something out, other people (like you) would join in to help.

As such you will find places like that where things aren't exactly how they should be - heck, I find them in my original code all the time. You should test it but once again I think you are correct - coresight_enable_source() should be called from etm_event_start().

A non-related question: are there any restrictions on the mailing clients for this mailing list? i'm using gmail, and i'm not sure if it's acceptable here(i know some mailing lists disallow it).

No restriction related to gmail - that's what Linaro is using. Using plain text mode is highly appreciated though.

Thanks! Mike.

On Wed, Aug 15, 2018 at 7:10 PM, Mathieu Poirier mathieu.poirier@linaro.org wrote:

...

On Tue, 14 Aug 2018 at 08:10, Mike Bazov mike@perception-point.io wrote:

...

Hello,

...

Interesting idea. Could you explain how you deal with the situation where

...

paths for different sources need different configuration of the downstream

...

CoreSight? The situation I’m thinking of is (ignoring funnels):

...

...

ETM#1

...

\

...

   ETF -> ETR

...

/

...

ETM#2

...

...

A path with source ETM#1 to sink ETF can’t be active at the same time as a

...

path from source ETM#2 to sink ETR, because the former will need the ETF

...

to be in buffer mode while the latter will need the ETF to be in FIFO mode.

...

...

I’d expect you could build these two incompatible paths, but not

...

simultaneously enable them? So coresight_enable_path would check

...

that any other paths using the same ETF were using it in the same mode,

...

and if it was idle, it would switch it into the right mode.

I haven't thought about that scenario, but you are right. Also the solution you've provided seems great.

When enabling a path, you can only enable it if it isn't already enabled, or if it is enabled with the same configuration.

Depending on the trace scenario there is a couple more things to keep in mind:

1. CPU-wide or per-thread mode: In CPU-wide mode sources can use the

same sink simultaneously. In per-thread mode only a single source can use the sink.

2. The session: In CPU-wide mode we don't want sources from different

trace session to use the same sink.

...

...

Also how is the trace source id handled? As we have only about 120 possible

...

trace source ids and we have chips with 128 cores funnelling into one sink,

...

we can’t have a fixed allocation of trace sources to trace source ids (i.e. we

...

can’t fix it in the device tree or anything like that). So we need to be able to

...

dynamically allocate trace source ids. Could that be done in

...

coresight_enable_path? So all enabled paths would have distinct

...

trace source ids.

Seems like a great idea for the API.

One thing i can't really understand is, why haven't these problems occur in sysfs/perf mode? seems like they aren't really

specific to the API I proposed.

Simply because CS is complex, quirky, in the process of maturing and the team has very limited resources. We fix things based on the use case we currently work on.

...

Thanks,

Mike.

On Tue, Aug 14, 2018 at 4:31 PM, Al Grant Al.Grant@arm.com wrote:

...

Hi Mike,

Interesting idea. Could you explain how you deal with the situation where

paths for different sources need different configuration of the downstream

CoreSight? The situation I’m thinking of is (ignoring funnels):

ETM#1

  \

    ETF -> ETR

  /

ETM#2

A path with source ETM#1 to sink ETF can’t be active at the same time as a

path from source ETM#2 to sink ETR, because the former will need the ETF

to be in buffer mode while the latter will need the ETF to be in FIFO mode.

I’d expect you could build these two incompatible paths, but not

simultaneously enable them? So coresight_enable_path would check

that any other paths using the same ETF were using it in the same mode,

and if it was idle, it would switch it into the right mode.

Also how is the trace source id handled? As we have only about 120 possible

trace source ids and we have chips with 128 cores funnelling into one sink,

we can’t have a fixed allocation of trace sources to trace source ids (i.e. we

can’t fix it in the device tree or anything like that). So we need to be able to

dynamically allocate trace source ids. Could that be done in

coresight_enable_path? So all enabled paths would have distinct

trace source ids.

Al

From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of Mike Bazov Sent: 14 August 2018 14:01 To: Mathieu Poirier mathieu.poirier@linaro.org Cc: coresight@lists.linaro.org Subject: Re: Enabling Coresight in atomic context.

Hello,

...

Patches are always welcomed and I don't think there is an "easy" way

...

to get out of this one. What you want to do will probably end up

...

being fairly complex. I would start by closely understanding how

...

operation of the CS infrastructure is done from the perf interface

...

you should be find just sticking to the kernel part. There

...

reservation of a "path" and memory for the sink is done in preparatory

...

steps where it is permitted to sleep (non-atomic). After that

...

components can be enabled from an atomic context, i.e when the process

...

of interest is installed on a processor. Currently things are woven

...

with the perf_aux_output_[begin|end]() interface but that could easily

...

be decoupled.

...

On the aspect of trace collection, did you envision using the entries

...

in devFS? If that is the case a mechanism to correlate tracer

...

configuration and trace data will need to be developed, just like what

...

we did for perf.

...

Taking a step back, tracers can also be found on X86 and MIPs (if I'm

...

not mistaking) architectures. As such the new kernel API would have

...

to be usable by those as well, which complicates the task even

...

further.

...

So all that being said I think it is feasible, but be prepared to

...

invest a significant amount of time and effort.

The "generic" tracing kernel API is a different thing. In it's Coresight implementation it will use the kernel API I need.

After taking a few days to understand how the infrastructure works, to make the API as flexible as it can be, I thought about this:

Just like there's a perf implementation and a sysfs implementation, the "api" implementation(coresight-api) will be introduced, which will also be

a new mode(CS_MODE_API).

I propose these APIs(some of them exist, but need to be exported and changed a little):

• coresight_build_path(struct coresight_device *source, struct coresight_device *sink):

  Create a coresight path from the provided source and sink,.

• coresight_enable_path(struct coresight_path *path): Enable a Coresight path except the source. This will

  also glue a source to a specific path. You cannot assign a different path to this source until the path is destroyed.

• coresight_disable_path(struct coresight_path *path)

  Disable the path to the sink, including the sink.(if there is more than 1 path to the same sink, does not disable the sink until a refcount reaches 0).

• coresight_destroy_path(struct coresight_path *path):

  Frees the path, releases the source from that path. The source device can be assigned to a different path.

• coresight_enable_source(struct coresight_device *source);

  Enables the source. This will actually make the source device play the actual trace data in to the sink(i.e. etm4_enable_hw(), or, increase a refcount if

  the source is already playing). Uses the path assigned in "coresight_enable_sink()".

• coresight_disable_source(struct coresight_device *source);

  Disables the source. This will stop the source from playing trace data(or, if the refcount > 0, decrease the refcount).

  Uses the path assigned in "coresight_enable_sink()".

• coresight_read_sink(struct coresight_device *sink, void *buf, size_t size);

  Read trace data from the sink(advance the read pointer).

• coresight_setup_sink_buffer(struct coresight_device *sink, void *pages, int nr_pages);

  Allocate a sink buffer(similar to the perf functionality)

The sysfs and api modes will use different buffers to avoid collision.

I realize most of the API is actually making the internal coresight implementation "public", but I really think this is necessary. Building a path to a specific sink

is something a user would want to do, as well as disabling and enabling the path whenever he wishes(this is something I actually need).

In order to use this API, the user needs a method of getting the actual (struct coresight_device *). There will be a list of coresight devices

exported in the "coresight.h" header, which can be iterated using a macro "foreach_coresight_device()". The user will be able to extract a specific

sink and source for his needs.

I think this API is powerful, and will give the user full Coresight functionality. From diving into the code, this seems very possible,

and will not require major infrastructure changes.

I will appreciate your thoughts, tips, and hints.

Thanks, Mike.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Greetings,

I have attached a small POC(patch file) i wrote for the kernel ETM API. Basically, i created a new mode called "CS_MODE_API", and made a new module called "coresight-etm-api.c" that exposes these APIs:

- coresight_etm_create_session(cpu): Create a coresight ETM session. In implementation terms, this simply means creating a path from an ETM source to a sink, and enable the whole patch except the source.

- coresight_etm_destroy_session(cpu): Destroy a coresight ETM session. In implementation terms, this simply means disabling a coresight path(except the source), and freeing it.

- coresight_etm_play(session, config): Start playing trace data. In implementation terms, this simply means enabling a source.

- coresight_etm_pause(session) Stop playing trace data. In implementation terms, this simply means disabling a source.

- coresight_etm_alloc_buffer(size) Allocate a trace buffer data. Only supported in TMC-ETR.

- coresight_etm_read(session) Read trace data from the session's sink.

To make this change as small as possible(hopefully), the etm-api implementation takes an activated sink(that is activated via sysfs) as the sink for the path. I didn't want to add a method of activating a sink from the API, just yet.

I don't deal with the "different TMC configuration" issues. If 2 kernel clients request the TMC for different configurations, i wouldn't know about it just as the sysfs/perf modes don't know about it.

To allow kernel-clients to read trace data "on-the-fly", and to avoid the "prepare()->read()->unprepare()" way, i have implemented a double-buffer in the CS_MODE_API. A read buffer and a write buffer. Whenever the read buffer is empty, a buffer swap occurs.

I'd appreciate an input, review, tips, hints, improvements. Please tell me if something is lacking in my explanation. This is highly experimental(it does work though). Notice this patch file can be applied on the latest coresight "next" branch TIP (as of the writing of this message).

Thanks, Mike.

On Fri, Aug 17, 2018 at 12:15 AM, Mike Bazov mike@perception-point.io wrote:

Greetings,

When tracing via sysFS and keeping the default configuration,

...

everything that is happening on a processor is logged. That is called "CPU-wide". Any process that get scheduled out of the processor won't be traced. On perf one can execute: # perf record -e cs_etm/@20070000.etr/ --per-thread my_application (example 1) # perf record -e cs_etm/@20070000.etr/ -C 2,3,4 my_application (example 2) For example 1, perf will switch on the tracer associated to the CPU where my_application has been installed for execution. If the process gets scheduled on a different CPU perf will do the right thing and follow it around. That is called "per-thread". In example 2 everything that is happening on CPU 2,3,4 will be traced for as long as my_application is executing, regardless of where my_application is executing. That is also a CPU-wide trace scenario.

I was more under the impression that CPU-wide records everything that the CPU executes (this is achieved by using sysfs like you described), regardless of the term "thread". I don't really understand why CPU-wide is the right term for example 2. Both of the examples record "per-thread", except of the CPU mask. Example 1 doesn't mask any CPUs, where example 2 masks all CPUs except 2, 3, 4, It still doesn't record "CPU-wide", only "per-thread", but on non-masked CPUs(if it weren't "per-thread", it wouldn't care about scheduling a thread and disable/enable accordingly). I'm a little confused, It really seems like sysfs==cpu-wide and perf==per-thread. Perhaps chagning the modes to "CS_MODE_PER_THREAD", "CS_MODE_CPU_WIDE" and make the sysfs and perf implementation use these modes is something that solves the puzzle for me.

...

As such you will find places like that where things aren't exactly how

they should be - heck, I find them in my original code all the time.

...

You should test it but once again I think you are correct - coresight_enable_source() should be called from etm_event_start().

After a second look, actually i think calling coresight_enable_source() will be problematic. Using the sysfs implementation(coresight.c) from perf is problematic, since it maintains a reference count per-device. If there's a sysfs session running on a tracer, and perf uses the __same__ path to the sink and the same source, using coresight_enable_path() will result in simply increasing the reference count without returning any errors.. So calling source_ops(source)->enable() directly actually will result in an error, which is the expected behavior(?)

Also, just out of curiosity, what happens when perf is requested to record a multithreaded process? the "per-thread" mechanism doesn't seem to fit here, because of the "single tracer to single-sink" rule.

Good day Mike,

On Mon, 20 Aug 2018 at 03:21, Mike Bazov mike@perception-point.io wrote:

Greetings,

I have attached a small POC(patch file) i wrote for the kernel ETM API. Basically, i created a new mode called "CS_MODE_API", and made a new module called "coresight-etm-api.c" that exposes these APIs:

• coresight_etm_create_session(cpu): Create a coresight ETM session. In implementation terms, this simply means creating a path from an ETM source to a sink, and enable the whole patch except the source.

• coresight_etm_destroy_session(cpu): Destroy a coresight ETM session. In implementation terms, this simply means disabling a coresight path(except the source), and freeing it.

• coresight_etm_play(session, config): Start playing trace data. In implementation terms, this simply means enabling a source.

• coresight_etm_pause(session) Stop playing trace data. In implementation terms, this simply means disabling a source.

• coresight_etm_alloc_buffer(size) Allocate a trace buffer data. Only supported in TMC-ETR.

• coresight_etm_read(session) Read trace data from the session's sink.

To make this change as small as possible(hopefully), the etm-api implementation takes an activated sink(that is activated via sysfs) as the sink for the path. I didn't want to add a method of activating a sink from the API, just yet.

I don't deal with the "different TMC configuration" issues. If 2 kernel clients request the TMC for different configurations, i wouldn't know about it just as the sysfs/perf modes don't know about it.

In the code we go to great length to make sure sysFS and perf don't impact each other. Places that don't should be considered a bug and fixed.

To allow kernel-clients to read trace data "on-the-fly", and to avoid the "prepare()->read()->unprepare()" way, i have implemented a double-buffer in the CS_MODE_API. A read buffer and a write buffer. Whenever the read buffer is empty, a buffer swap occurs.

I'd appreciate an input, review, tips, hints, improvements. Please tell me if something is lacking in my explanation. This is highly experimental(it does work though). Notice this patch file can be applied on the latest coresight "next" branch TIP (as of the writing of this message).

I haven't looked at the code in your patch yet. Before I do so please split it in smaller chunks and remove the Change-Id.

Regards, Mathieu

Thanks, Mike.

On Fri, Aug 17, 2018 at 12:15 AM, Mike Bazov mike@perception-point.io wrote:

...

Greetings,

...

When tracing via sysFS and keeping the default configuration, everything that is happening on a processor is logged. That is called "CPU-wide". Any process that get scheduled out of the processor won't be traced. On perf one can execute: # perf record -e cs_etm/@20070000.etr/ --per-thread my_application (example 1) # perf record -e cs_etm/@20070000.etr/ -C 2,3,4 my_application (example 2) For example 1, perf will switch on the tracer associated to the CPU where my_application has been installed for execution. If the process gets scheduled on a different CPU perf will do the right thing and follow it around. That is called "per-thread". In example 2 everything that is happening on CPU 2,3,4 will be traced for as long as my_application is executing, regardless of where my_application is executing. That is also a CPU-wide trace scenario.

I was more under the impression that CPU-wide records everything that the CPU executes (this is achieved by using sysfs like you described), regardless of the term "thread". I don't really understand why CPU-wide is the right term for example 2. Both of the examples record "per-thread", except of the CPU mask. Example 1 doesn't mask any CPUs, where example 2 masks all CPUs except 2, 3, 4, It still doesn't record "CPU-wide", only "per-thread", but on non-masked CPUs(if it weren't "per-thread", it wouldn't care about scheduling a thread and disable/enable accordingly). I'm a little confused, It really seems like sysfs==cpu-wide and perf==per-thread. Perhaps chagning the modes to "CS_MODE_PER_THREAD", "CS_MODE_CPU_WIDE" and make the sysfs and perf implementation use these modes is something that solves the puzzle for me.

...

As such you will find places like that where things aren't exactly how

they should be - heck, I find them in my original code all the time. You should test it but once again I think you are correct - coresight_enable_source() should be called from etm_event_start().

After a second look, actually i think calling coresight_enable_source() will be problematic. Using the sysfs implementation(coresight.c) from perf is problematic, since it maintains a reference count per-device. If there's a sysfs session running on a tracer, and perf uses the __same__ path to the sink and the same source, using coresight_enable_path() will result in simply increasing the reference count without returning any errors.. So calling source_ops(source)->enable() directly actually will result in an error, which is the expected behavior(?)

Also, just out of curiosity, what happens when perf is requested to record a multithreaded process? the "per-thread" mechanism doesn't seem to fit here, because of the "single tracer to single-sink" rule.

Hi Mike,

On 16/08/18 22:15, Mike Bazov wrote:

Greetings,

When tracing via sysFS and keeping the default configuration,

...

everything that is happening on a processor is logged. That is called "CPU-wide". Any process that get scheduled out of the processor won't be traced. On perf one can execute: # perf record -e cs_etm/@20070000.etr/ --per-thread my_application (example 1) # perf record -e cs_etm/@20070000.etr/ -C 2,3,4 my_application (example 2) For example 1, perf will switch on the tracer associated to the CPU where my_application has been installed for execution. If the process gets scheduled on a different CPU perf will do the right thing and follow it around. That is called "per-thread". In example 2 everything that is happening on CPU 2,3,4 will be traced for as long as my_application is executing, regardless of where my_application is executing. That is also a CPU-wide trace scenario.

I was more under the impression that CPU-wide records everything that the CPU executes (this is achieved by using sysfs like you described), regardless of the term "thread". I don't really understand why CPU-wide is the right term for example 2. Both of the examples record "per-thread", except of the CPU mask. Example 1 doesn't mask any CPUs, where example 2 masks all CPUs except 2, 3, 4, It still doesn't record "CPU-wide", only "per-thread", but on non-masked CPUs(if it weren't "per-thread", it wouldn't care about scheduling a thread and disable/enable accordingly). I'm a little confused, It really seems like sysfs==cpu-wide and perf==per-thread. Perhaps chagning the modes to "CS_MODE_PER_THREAD", "CS_MODE_CPU_WIDE" and make the sysfs and perf implementation use these modes is something that solves the puzzle for me.

...

As such you will find places like that where things aren't exactly how

they should be - heck, I find them in my original code all the time.

...

You should test it but once again I think you are correct - coresight_enable_source() should be called from etm_event_start().

After a second look, actually i think calling coresight_enable_source() will be problematic. Using the sysfs implementation(coresight.c) from perf is problematic, since it maintains a reference count per-device. If there's a sysfs session running on a tracer, and perf uses the __same__ path to the sink and the same source, using coresight_enable_path() will result in simply increasing the reference count without returning any errors.. So calling source_ops(source)->enable() directly actually will result in an error, which is the expected behavior(?)

I had a patch to switch the perf API to use corsight_enable_source(), but dropped it to keep the perf changes smaller. I can resurrect it and post the same.

Btw, have you explored using the "perf_event" API from the kernel ?

e.g, You could create an event using perf_event_kernel_counter(). May be it is easier to switch to the perf event API (and enhance it there) rather than adding another mode to the mix. I will do some digging.

Kind regards Suzuki

Hi Mike,

On Mon, 3 Sep 2018 at 05:51, Mike Bazov mike@perception-point.io wrote:

Greetings,

I have attached 4 patches that implement the API i needed. I haven't send it officially as a patch to the mailing list, since this is a POC and merely a request for review and "approval" of the approach i have taken here. The patches are rebased on the latest TIP of coresight tree "next" branch(correct to the time of sending this message).

Thanks for working on this. Please resend using a cover-letter, "git send-email" and a clean run through checkpatch.pl. I would simply use the "RFC" tag instead of "PATCH" when submitting proof of concepts like that.

Mathieu

On Tue, 14 Aug 2018 at 07:01, Mike Bazov mike@perception-point.io wrote:

Hello,

...

Patches are always welcomed and I don't think there is an "easy" way to get out of this one. What you want to do will probably end up being fairly complex. I would start by closely understanding how operation of the CS infrastructure is done from the perf interface you should be find just sticking to the kernel part. There reservation of a "path" and memory for the sink is done in preparatory steps where it is permitted to sleep (non-atomic). After that components can be enabled from an atomic context, i.e when the process of interest is installed on a processor. Currently things are woven with the perf_aux_output_[begin|end]() interface but that could easily be decoupled.

...

On the aspect of trace collection, did you envision using the entries in devFS? If that is the case a mechanism to correlate tracer configuration and trace data will need to be developed, just like what we did for perf.

...

Taking a step back, tracers can also be found on X86 and MIPs (if I'm not mistaking) architectures. As such the new kernel API would have to be usable by those as well, which complicates the task even further. So all that being said I think it is feasible, but be prepared to invest a significant amount of time and effort.

The "generic" tracing kernel API is a different thing. In it's Coresight implementation it will use the kernel API I need. After taking a few days to understand how the infrastructure works, to make the API as flexible as it can be, I thought about this: Just like there's a perf implementation and a sysfs implementation, the "api" implementation(coresight-api) will be introduced, which will also be a new mode(CS_MODE_API).

I propose these APIs(some of them exist, but need to be exported and changed a little):

• coresight_build_path(struct coresight_device *source, struct coresight_device *sink): Create a coresight path from the provided source and sink,.

• coresight_enable_path(struct coresight_path *path): Enable a Coresight path except the source. This will also glue a source to a specific path. You cannot assign a different path to this source until the path is destroyed.

• coresight_disable_path(struct coresight_path *path) Disable the path to the sink, including the sink.(if there is more than 1 path to the same sink, does not disable the sink until a refcount reaches 0).

• coresight_destroy_path(struct coresight_path *path): Frees the path, releases the source from that path. The source device can be assigned to a different path.

• coresight_enable_source(struct coresight_device *source); Enables the source. This will actually make the source device play the actual trace data in to the sink(i.e. etm4_enable_hw(), or, increase a refcount if the source is already playing). Uses the path assigned in "coresight_enable_sink()".

• coresight_disable_source(struct coresight_device *source); Disables the source. This will stop the source from playing trace data(or, if the refcount > 0, decrease the refcount). Uses the path assigned in "coresight_enable_sink()".

• coresight_read_sink(struct coresight_device *sink, void *buf, size_t size); Read trace data from the sink(advance the read pointer).

• coresight_setup_sink_buffer(struct coresight_device *sink, void *pages, int nr_pages); Allocate a sink buffer(similar to the perf functionality)

The sysfs and api modes will use different buffers to avoid collision.

I realize most of the API is actually making the internal coresight implementation "public", but I really think this is necessary. Building a path to a specific sink is something a user would want to do, as well as disabling and enabling the path whenever he wishes(this is something I actually need).

In order to use this API, the user needs a method of getting the actual (struct coresight_device *). There will be a list of coresight devices exported in the "coresight.h" header, which can be iterated using a macro "foreach_coresight_device()". The user will be able to extract a specific sink and source for his needs.

I think this API is powerful, and will give the user full Coresight functionality. From diving into the code, this seems very possible, and will not require major infrastructure changes. I will appreciate your thoughts, tips, and hints.

Look at what we've done in coresight-etm-perf.c. There you'll find code that uses the internal CS implementation to serve the perf interface. I suggest you do the same thing but to serve internal kernel clients (which I suspect will ultimately be some sort of module). The hope here is to expose as little as possible of the internal CS core code.

As for the above proposition I can't comment much without having a better understanding of what you're trying to do. I suggest to put a proof of concept together and send it to the list for review - that will help us understand what you want to do. While doing so I suggest you keep an eye on the coresight next tree[1]. There is a lot of things currently happening in the subsystem and the goal posts are moving regularly.

Mathieu

[1]. https://git.linaro.org/kernel/coresight.git/log/?h=next

Thanks, Mike.