Hi Mathieu,

Thank you for your suggestions!

I discussed a possible design solution with Suzuki, and he suggested the N:N topology would follow a different approach since, in that case, the hardware capability of triggering interrupts is implemented and each buffer is independent with each other.

The trace data is lost because of hardware/memory limitations and due to the system not reading frequently enough the trace data buffer. This idea would provide an improvement for the second scenario, where the trace buffer is read independently w.r.t. task scheduling. At the moment it's indeed possible, taking into account the same workload, to produce a final trace in size of orders of magnitude bigger just by artificially increasing the system load. It would be great to have the ETM subsystem provide a less variable perf.data size.

Strobing is very beneficial but it works on a different abstraction layer: regardless of having strobing on or off, if the trace buffer is only read a couple of times because of an idle system, the final perf.data size will be the same.

I was thinking about defining a rcu_work to be submitted into a workqueue for getting the trace to be read while at the same time it's generated, but unfortunately Suzuki thinks an RCU solution would no be suitable in this use case, and I'm open to any other suggestions.

Please let me know if you are aware of any crucial problem which could prevent a trace consumer and producer to execute at the same time.

Thanks, Andrea ________________________________ From: Mathieu Poirier mathieu.poirier@linaro.org Sent: 10 August 2020 20:33 To: Andrea Brunato Andrea.Brunato@arm.com Cc: nd nd@arm.com; coresight@lists.linaro.org coresight@lists.linaro.org; Suzuki Poulose Suzuki.Poulose@arm.com Subject: Re: [RFC] Making trace reads independent to task scheduling

Hi Andrea,

On Fri, Aug 07, 2020 at 10:05:22AM +0000, Andrea Brunato wrote:

Good morning,

I noticed the trace data is read from the buffer only when the target task is scheduled in/out, leading to high variability of the trace data size. This can be an issue because trace data gets very different as in size (and thus as in its coverage) under different system load.

System load but also what the thread itself is doing. An IO intensive thread will block and be swapped out more often.

Is getting trace reads independent w.r.t. task scheduling an idea which has already been considered?

Several times yes, mostly to deal with sink buffer overflow. The issue is more serious in components with limited internal memory.

One possibility I’m considering is using an independent timer which triggers periodically trace reads from the sink buffer to achieve less variability. A couple of buffers (struct etr_buf) could also be used: one for gathering the trace data coming from the sink and another whose content to be copied over to the perf aux buffer. When the timer triggers, the buffers are switched such that it's always possible to copy the trace data to the perf aux buffer (in one buffer) while gathering the trace data coming from all the ETMs (in the other one).

The problem can't be fixed only by a timer, there are other things to consider. And even then we will always have cases where data is lost, making it very hard for users to understand how to work with a CS system.

The best way to handle this is for the sink to generate an interrupt when it has reached a watermark level. And even then careful consideration would have to be taken to avoid scheduling a task that uses the sink, which wouldn't be trivial. Failure to do that would lead to data loss, the same way we have today.

I have been thinking about this solution with a N:1 source:sink topology in mind, but I'm not sure how this would fit in a N:N topology, where every sink has its own buffer.

Right, that's just another example of the "other things to consider" I alluded to above.

What do you think about it? Are you aware of any limitations that should be taken into account?

If we think this could work, the next step would probably be to prototype something that works on my N:1 topology board

In the end using a timer is a SW workaround to put a band-aid on a feature the HW doesn't provide. I would much rather concentrate on supporting new IP blocks that generate interrups or something like Mike's strobing feature that, when properly configured, would likely yield the same kind of results.

That being said you are free to spend time on it - just be warned it is a long way to go with many pitfalls to avoid.

Regards, Mathieu

Thanks, Andrea