On Tue, May 18, 2021 at 10:00:40PM +0800, Leo Yan wrote:
Hi Denis,
On Fri, May 14, 2021 at 02:02:25AM -0700, Denis Nikitin wrote:
[...]
Hi Mattieu and Leo,
I did some evaluation of the snapshot mode.
Thanks a lot for the evaluation and share back the result.
Performance overhead is indeed higher than with ETR polling patch. Here are some numbers for comparison (measured on browser Speedometer2 benchmark): Runtime overhead of ETM tracing with ETR poll period 100ms is less than 0.5%. Snapshot mode gives 2.1%. With 10ms period I see 4.6% with ETR polling and 22% in snapshot mode.
It's not expected that the snapshot mode causes so big overload. In my head, two factors might cause the overload:
- The perf interaction between the user space and kernel space;
- The data copying from the ETR's buffer to the AUX ring buffer.
Check one thing: what's the buffer size for ETR polling mode and for snapshot mode in your experiments?
If I remember correctly, by default the snapshot mode uses 4MB for ETR buffer, if copying 4MB per 10ms, then it's likely to cause big overload. So at the first glance, the overhead difference might be caused by the by the different buffer size between ETR poll mode and snapshot mode.
We could probably utilize the ETM strobing feature and reduce frequency of data collection but I see a problem when I'm using both. Within a minute of profiling the ETM generates a reasonable profile size (with strobing autofdo,preset=9 with period 0x1000 it is up to 20MB). But then the size grows unproportionally. With a 4 minute run I got a 6.3GB profile.
Just check, as Mathieu has suggested, have you applied the patch [1] on your local code base for fixing the data copying for snapshot mode?
After applied this patch, one possibility for unproportional issue is perf tool itself introduces many activities in snapshot mode (especially for 10ms period), so the perf tool contributes much extra trace data.
Another potential issue is: after setting strobing mode, the snapshot mode will disable the complete paths for tracers and ETR, so if the strobing configuration is lost after re-enable tracers, then it might cause the huge trace data in the later phase. For this case, we definitely should fix it.
I don't see such a problem with the ETR polling patch.
Leo, could you please take a look at this problem?
Sure. For easier reproducing the issue, could you share me the detailed commands (and source code)?
P.s. I saw Mike suggests to continue the ETR polling development, this is not conflict with snapshot mode. At my side, I will investigate the snapshot mode, but don't want to disturb the process for ETR polling mode, so when the ETR polling patch series is get ready, please go ahead for upstreaming the patch series.
Just to clarify my position - I would definitely like to see a solution that extends or re-use the current snapshot mode rather than introducing a new mechanism to collect data.
Thanks, Leo
[1]. https://lists.linaro.org/pipermail/coresight/2021-April/006254.html