Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or what we can do to reduce this kind of data loss?
Any response will be highly appreciated.
Thanks, Qi
Hi Qi
On 30/03/2021 09:18, liuqi (BA) wrote:
Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or what
we can do to reduce this kind of data loss?
It depends on what configuration you have. On legacy coresight sinks (read TMC-ETR), without an overflow interrupt we can't do much, except for increasing the buffer size allocated for the trace collection. For perf you could use,
perf record -m ,<memory-for-AUX-buf> -e ...
Also, it is not clear what "data is of interest". There is strobbing support for ETM from Mike, which is under review. You may use it to avoid the continuous collection of trace and use a periodic collection and thus avoiding the overflows. Again, that may not guarantee you have the data of interest captured.
The solution would be to wire up the cross triggers to trigger a sink stop upon the even of interest. This doesn't exist today. But it could be built on top of the CTI and the complex configuration series.
Kind regards Suzuki
Any response will be highly appreciated.
Thanks, Qi
On 30/03/2021 09:18, liuqi (BA) wrote:
Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or what
we can do to reduce this kind of data loss?
It depends on what configuration you have. On legacy coresight sinks (read TMC-ETR), without an overflow interrupt we can't do much, except for increasing the buffer size allocated for the trace collection. For perf you could use,
perf record -m ,<memory-for-AUX-buf> -e ...
Also, it is not clear what "data is of interest". There is strobbing support for ETM from Mike, which is under review. You may use it to avoid the continuous collection of trace and use a periodic collection and thus avoiding the overflows. Again, that may not guarantee you have the data of interest captured.
You could also look at address filters, there are two kinds:
An address range filter will capture trace within a given range, i.e. only code between the lower and higher address will be traced.
A start/stop filter will start or stop the trace when an address is reached - for example you could stop the trace when you hit a kernel error handler.
More details here:
https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
Al
The solution would be to wire up the cross triggers to trigger a sink stop upon the even of interest. This doesn't exist today. But it could be built on top of the CTI and the complex configuration series.
Kind regards Suzuki
Any response will be highly appreciated.
Thanks, Qi
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
HiAs far as I remember data is retrieved from the sink and put into the perf aux buffer at thread removal from the scheduler. And data can be lost here if the time slice of the scheduler is too big compared to the size of the sink. A second reason for the data loss is the size of the aux buffer, and this can be configured when mmaping the aux buffer. Kind RegardsZied Guermazi Gesendet von Yahoo Mail auf Android
Am Di., März 30, 2021 at 6:26 PM schrieb Al GrantAl.Grant@arm.com: > On 30/03/2021 09:18, liuqi (BA) wrote:
Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or what
we can do to reduce this kind of data loss?
It depends on what configuration you have. On legacy coresight sinks (read TMC-ETR), without an overflow interrupt we can't do much, except for increasing the buffer size allocated for the trace collection. For perf you could use,
perf record -m ,<memory-for-AUX-buf> -e ...
Also, it is not clear what "data is of interest". There is strobbing support for ETM from Mike, which is under review. You may use it to avoid the continuous collection of trace and use a periodic collection and thus avoiding the overflows. Again, that may not guarantee you have the data of interest captured.
You could also look at address filters, there are two kinds:
An address range filter will capture trace within a given range, i.e. only code between the lower and higher address will be traced.
A start/stop filter will start or stop the trace when an address is reached - for example you could stop the trace when you hit a kernel error handler.
More details here:
https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
Al
The solution would be to wire up the cross triggers to trigger a sink stop upon the even of interest. This doesn't exist today. But it could be built on top of the CTI and the complex configuration series.
Kind regards Suzuki
Any response will be highly appreciated.
Thanks, Qi
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
_______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On 2021/3/31 0:51, zied guermazi wrote:
Hi As far as I remember data is retrieved from the sink and put into the perf aux buffer at thread removal from the scheduler. And data can be lost here if the time slice of the scheduler is too big compared to the size of the sink. A second reason for the data loss is the size of the aux buffer, and this can be configured when mmaping the aux buffer. Kind Regards Zied Guermazi
Hi Zied,
Thanks for your reply. I've tried perf record -m, and it really help.
Qi
Gesendet von Yahoo Mail auf Android https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature
Am Di., März 30, 2021 at 6:26 PM schrieb Al Grant <Al.Grant@arm.com>: > On 30/03/2021 09:18, liuqi (BA) wrote: > > Hi Mathieu & Suzuki, > > > > Here is a question about the amount of data traced by ETM device. > > > > In high-CPU-utilization scene, volume of ETM trace data may be very > > large, but limited buffer in sink device which is used to store trace > > data can not hold all of these data. It seems that data loss in sink > > device is inevitable. > > > So how could we ensure the data we are interested in isn't lost, or > > > what > > we can do to reduce this kind of data loss? > > It depends on what configuration you have. On legacy coresight sinks (read > TMC-ETR), without an overflow interrupt we can't do much, except for > increasing the buffer size allocated for the trace collection. > For perf you could use, > > perf record -m ,<memory-for-AUX-buf> -e ... > > Also, it is not clear what "data is of interest". There is strobbing support for ETM > from Mike, which is under review. You may use it to avoid the continuous > collection of trace and use a periodic collection and thus avoiding the overflows. > Again, that may not guarantee you have the data of interest captured. You could also look at address filters, there are two kinds: An address range filter will capture trace within a given range, i.e. only code between the lower and higher address will be traced. A start/stop filter will start or stop the trace when an address is reached - for example you could stop the trace when you hit a kernel error handler. More details here: https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md Al > > The solution would be to wire up the cross triggers to trigger a sink stop upon > the even of interest. This doesn't exist today. But it could be built on top of the > CTI and the complex configuration series. > > Kind regards > Suzuki > > > > > Any response will be highly appreciated. > > > > Thanks, > > Qi > > > > _______________________________________________ > CoreSight mailing list > CoreSight@lists.linaro.org <mailto:CoreSight@lists.linaro.org> > https://lists.linaro.org/mailman/listinfo/coresight _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org <mailto:CoreSight@lists.linaro.org> https://lists.linaro.org/mailman/listinfo/coresight
On 2021/3/31 0:25, Al Grant wrote:
On 30/03/2021 09:18, liuqi (BA) wrote:
Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or what
we can do to reduce this kind of data loss?
It depends on what configuration you have. On legacy coresight sinks (read TMC-ETR), without an overflow interrupt we can't do much, except for increasing the buffer size allocated for the trace collection. For perf you could use,
perf record -m ,<memory-for-AUX-buf> -e ...
Also, it is not clear what "data is of interest". There is strobbing support for ETM from Mike, which is under review. You may use it to avoid the continuous collection of trace and use a periodic collection and thus avoiding the overflows. Again, that may not guarantee you have the data of interest captured.
You could also look at address filters, there are two kinds:
An address range filter will capture trace within a given range, i.e. only code between the lower and higher address will be traced.
A start/stop filter will start or stop the trace when an address is reached - for example you could stop the trace when you hit a kernel error handler.
More details here:
https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
Al
Hi Al
Thanks for your reply
Qi
The solution would be to wire up the cross triggers to trigger a sink stop upon the even of interest. This doesn't exist today. But it could be built on top of the CTI and the complex configuration series.
Kind regards Suzuki
Any response will be highly appreciated.
Thanks, Qi
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On 2021/3/30 19:54, Suzuki K Poulose wrote:
Hi Qi
On 30/03/2021 09:18, liuqi (BA) wrote:
Hi Mathieu & Suzuki,
Here is a question about the amount of data traced by ETM device.
In high-CPU-utilization scene, volume of ETM trace data may be very large, but limited buffer in sink device which is used to store trace data can not hold all of these data. It seems that data loss in sink device is inevitable.
So how could we ensure the data we are interested in isn't lost, or
what we can do to reduce this kind of data loss?
It depends on what configuration you have. On legacy coresight sinks (read TMC-ETR), without an overflow interrupt we can't do much, except for increasing the buffer size allocated for the trace collection. For perf you could use,
perf record -m ,<memory-for-AUX-buf> -e ...
Also, it is not clear what "data is of interest". There is strobbing support for ETM from Mike, which is under review. You may use it to avoid the continuous collection of trace and use a periodic collection and thus avoiding the overflows. Again, that may not guarantee you have the data of interest captured.
The solution would be to wire up the cross triggers to trigger a sink stop upon the even of interest. This doesn't exist today. But it could be built on top of the CTI and the complex configuration series.
Kind regards Suzuki
Hi Suzuki,
Thanks for your answer, I've increased the buffer size and use perf record -m, it seems better.
But I meet another question when I use perf report to show the trace data. When I use:
perf report -D -i record.data > log
I could see the trace data in log, but when I only use perf report, it doesn't work:
[root@localhost lq]# ~/perf report -i record.data Error: The record.data data has no samples! # To display the perf.data header info, please use --header/--header-only options. [root@localhost lq]# ~/perf -v perf version 5.8.0-rc4
I find perf report only works when I use -k to figure vmlinux pathname, but I guess this is not the right way for ETM.
Any response will be highly appreciated. Qi
Any response will be highly appreciated.
Thanks, Qi
.
On Tue, Apr 06, 2021 at 11:25:36AM +0800, liuqi (BA) wrote:
[...]
Hi Suzuki,
Thanks for your answer, I've increased the buffer size and use perf record -m, it seems better.
But I meet another question when I use perf report to show the trace data. When I use:
perf report -D -i record.data > log
I could see the trace data in log, but when I only use perf report, it doesn't work:
[root@localhost lq]# ~/perf report -i record.data Error: The record.data data has no samples! # To display the perf.data header info, please use --header/--header-only options. [root@localhost lq]# ~/perf -v perf version 5.8.0-rc4
I find perf report only works when I use -k to figure vmlinux pathname, but I guess this is not the right way for ETM.
It's quite likely you hit the known issue that Perf tool misses to load Kernel and userspace DSOs when parse CoreSight trace data. If the tool fails to find the map (between PC value and DSOs), then it skips to generate samples for CS trace data.
Note, actually there have two issues, one for kernel DSO and another is for userspace DSOs. For the kernel symbols loading, you could specify the option '-k' with kernel vmlinux path, or you could build the kernel with configuration "CONFIG_PROC_KCORE=y", so perf tool can load kernel DSO successfully with any one of these two ways.
For the userspace DSOs loading, this is a known issue; James Clark has a patch set for trying to fix the similiar issue [1]; or you could do a quick try for the patch [2], please note for patch [2], it's not the right fixing (we should move forward based on James' work), just in case if you want to do quickly try and locate if you are running into the same issue.
Thanks, Leo
[1] https://lists.linaro.org/pipermail/coresight/2021-February/005816.html [2] https://lore.kernel.org/patchwork/patch/1340992/