Thanks for both of your suggestions! I will try return stack and stall control later.

Another problem I found recently is about performance degradation when using ETM:

When running a program with ETM enabled on it using perf interface, ETM records data and sends to ETR, and ETR moves the data to memory. And kernel copies

the data from ETR memory to a per-cpu buffer.

So I think the performance cost is: ETR uses memory bus to move data, and the kernel uses both cpu and memory bus to copy data.

In my experiments, the kernel moving data can take a lot of cpu-cycles, which slows down the running program.

When testing a single thread busy loop program, the overhead in kernel is about 11% cpu-cycles (comparing to all cpu-cycles running the program).

When testing a two threads busy looping program, the overhead in kernel is about 74% cpu-cycles.

In both cases, the overhead in user space is very small, about 0.2% cpu-cycles.

I am surprised at the performance difference between single thread and multi thread programs. I haven't spent much time on this.

I will check the accuracy and reason for the difference.

Any suggestions or comments are welcomed.

On Thu, Sep 19, 2019 at 5:04 AM Mike Leach <mike.leach@linaro.org> wrote:

HI Yabin,

A couple of additional suggestions:-
1) If available use the return stack - this can remove a number of
address packets from the trace stream.
2) Check that both SYSSTALL and STALLCTL are 1 on IDR3 on your device
- otherwise using stall will not work.
3) Ensure that the coresight system is correctly described in the
device tree in respect of possible replicators and TPIU devices on the
system. TPIU will ordinarily be disabled, but the bus to the TPIU may
go through a downsizer - which can produce backpressure beyond any
replicator. The coresight infrastucture should disable the TPIU branch
on a correctly described system

Regards

Mike

On Wed, 18 Sep 2019 at 16:35, Mathieu Poirier
<mathieu.poirier@linaro.org> wrote:
>
> Hi Yabin,
>
> On Tue, 17 Sep 2019 at 15:53, Yabin Cui <yabinc@google.com> wrote:
> >
> > Hi guys,
> >
> > I am trying to reduce ETM data lose. There seems to have two types of data lose:
> > 1. caused by instruction trace buffer overflow, which generates an Overflow packet after recovering.
> > 2. caused by ETR buffer overflow, which generates an PERF_RECORD_AUX with PERF_AUX_FLAG_TRUNCATED.
> >
> > In practice, the second one is unlikely to happen when I set ETR buffer size to 4M, and can be improved by setting higher buffer size.
> > The first one happens much more frequently, can happen about 21K times when generating 5.3M etm data.
> > So I want to know if there is any way to reduce that.
> >
> > I found in the ETM arch manual that the overflow seems controlled by TRCSTALLCTLR, a stall control register.
> > But TRCIDR3.NOOVERFLOW is not supported on my experiment device, which seems using qualcomm sdm845.
> > TRCSTALLCTLR.ISTALL seems can be set by writing to mode file in /sys. But I didn't find any way to set
> > TRCSTALLCTLR.LEVEL in linux kernel. So I wonder if it is ever used.
> >
> > Do you guys have any suggestions?
>
> In order to get a timely response to your questions I advise to CC the
> coresight mailing list (which I have included). There is a lot of
> knowledgeable people there that can also help, especially with
> architecture specific configuration.
>
> TRCSTALLCTLR.LEVEL is currently not accessible to users because we
> simply never needed to use the feature. If using the ISTALL and LEVEL
> parameters help with your use case send a patch that exposes the
> entire TRCSTALLCTLR register (rather than just the LEVEL field) and
> I'll be happy to fold it in.
>
> Thanks,
> Mathieu
>
> >
> > Best,
> > Yabin
> >
> _______________________________________________
> CoreSight mailing list
> CoreSight@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/coresight

--
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK