Hi,
I've encountered a perf.data file whose PERF_RECORD_AUXTRACE_INFO has an additional ETM register per CPU (i.e. the metadata for each CPU is 8 bytes longer than normal). Some patch somewhere must be adding this register for reasons unknown.
Upstream perf won't even print this file, it reports failure to process record type 70 and errors out. The metadata reading code in util/cs-etm.c expects exactly CS_ETMV4_PRIV_MAX registers, no more. I.e. it won't go skipping past extra ones to find the next "magic" number.
It's unfortunate that "perf report -D" errors out, when the reason for using -D may be to investigate why perf in general isn't working. I want to fix this at least.
Now firstly, has anyone got any ideas why there might be an additional register, does anyone know of a patch that does this?
And secondly, could/should we make the metadata format more robust? Cases of more or fewer registers than expected can, to some extent, be handled by looking for the magic number, but that assumes that registers don't legitimately have values equal to the magic number.
The version number could help us evolve upstream to a new format but it isn't really an answer to robust interchange in general. Should we at least have the metadata describe its own length (i.e. number of registers) so that perf tools can read it, even if they might not be able to fully understand it.
Al
Hi Al,
On Tue, Nov 24, 2020 at 04:53:04PM +0000, Al Grant wrote:
Hi,
I've encountered a perf.data file whose PERF_RECORD_AUXTRACE_INFO has an additional ETM register per CPU (i.e. the metadata for each CPU is 8 bytes longer than normal). Some patch somewhere must be adding this register for reasons unknown.
So firstly check one thing: is this perf.data file generated by your local machine (and you could see the corresponding code)? Or if this perf.data file is shared by others so you have no chance to check the perf source code?
If it's the first case, you could check the file arch/arm/util/cs-etm.c.
Upstream perf won't even print this file, it reports failure to process record type 70 and errors out. The metadata reading code in util/cs-etm.c expects exactly CS_ETMV4_PRIV_MAX registers, no more. I.e. it won't go skipping past extra ones to find the next "magic" number.
It's unfortunate that "perf report -D" errors out, when the reason for using -D may be to investigate why perf in general isn't working. I want to fix this at least.
Yeah, I think some minor hacking can workaround the incompatible issue.
Now firstly, has anyone got any ideas why there might be an additional register, does anyone know of a patch that does this?
You could see that ETMv4 metadata was introduced by the commit a818c563ae166 ("perf tools: Add coresight etm PMU record capabilities") since Sept, 2016. So for the mainline kernel, the metadata has not been changed for a long while.
If this perf.data is generated by any branch rather than Linux mainline kernel, I suspect if someone uses a self maintained patch and adds an extra entry into metadata, we cannot know which register's value is saved into the metadata. So I think it's better to inquiry the person who has the source code for the perf tool and check if made any change for metadata.
And secondly, could/should we make the metadata format more robust? Cases of more or fewer registers than expected can, to some extent, be handled by looking for the magic number, but that assumes that registers don't legitimately have values equal to the magic number.
The version number could help us evolve upstream to a new format but it isn't really an answer to robust interchange in general. Should we at least have the metadata describe its own length (i.e. number of registers) so that perf tools can read it, even if they might not be able to fully understand it.
Agreed that a self described format is better for metadata structure; I'd like to leave this question to Mathieu who might have thoughts for this.
Thanks, Leo
Hi Al,
On Wed, 25 Nov 2020 at 00:54, Leo Yan leo.yan@linaro.org wrote:
Hi Al,
On Tue, Nov 24, 2020 at 04:53:04PM +0000, Al Grant wrote:
Hi,
I've encountered a perf.data file whose PERF_RECORD_AUXTRACE_INFO has an additional ETM register per CPU (i.e. the metadata for each CPU is 8 bytes longer than normal). Some patch somewhere must be adding this register for reasons unknown.
So firstly check one thing: is this perf.data file generated by your local machine (and you could see the corresponding code)? Or if this perf.data file is shared by others so you have no chance to check the perf source code?
If it's the first case, you could check the file arch/arm/util/cs-etm.c.
Upstream perf won't even print this file, it reports failure to process record type 70 and errors out. The metadata reading code in util/cs-etm.c expects exactly CS_ETMV4_PRIV_MAX registers, no more. I.e. it won't go skipping past extra ones to find the next "magic" number.
It's unfortunate that "perf report -D" errors out, when the reason for using -D may be to investigate why perf in general isn't working. I want to fix this at least.
Yeah, I think some minor hacking can workaround the incompatible issue.
Now firstly, has anyone got any ideas why there might be an additional register, does anyone know of a patch that does this?
You could see that ETMv4 metadata was introduced by the commit a818c563ae166 ("perf tools: Add coresight etm PMU record capabilities") since Sept, 2016. So for the mainline kernel, the metadata has not been changed for a long while.
If this perf.data is generated by any branch rather than Linux mainline kernel, I suspect if someone uses a self maintained patch and adds an extra entry into metadata, we cannot know which register's value is saved into the metadata. So I think it's better to inquiry the person who has the source code for the perf tool and check if made any change for metadata.
And secondly, could/should we make the metadata format more robust? Cases of more or fewer registers than expected can, to some extent, be handled by looking for the magic number, but that assumes that registers don't legitimately have values equal to the magic number.
The version number could help us evolve upstream to a new format but it isn't really an answer to robust interchange in general. Should we at least have the metadata describe its own length (i.e. number of registers) so that perf tools can read it, even if they might not be able to fully understand it.
What do you want to happen here? As Leo says - you really must match the versions of perf reading a file to the kernel / perf that wrote the file. As I understand it, there are never guarantees of backward compatibility as perf develops alongside the kernel.
Even if we did adjust the format so that the dump could display registers it did not recognise as "unknown = <value>", what happens next?
We cannot go on to decode the trace as these parameters have a significant bearing on packet decode and interpretation. We cannot simply assume that the ones we are expecting still have valid values and are sufficient for decode.
Regards
Mike
p.s. You may already know this, but I have recently discovered that:
- perf record -D --header
reports some interesting stuff regarding the versions and command line of perf used to generate the file, with what appear to be git commit hashes. May make it easier to track down the origin of your perf.data file => # ======== # captured on : Tue Aug 18 20:32:19 2020 # header version : 1 # data offset : 440 # data size : 470872 # feat offset : 471312 # hostname : linaro-developer # os release : 5.9.0-rc1cplx-v2cplx-v2-00005-ga922ba33d09f # perf version : 5.9.rc1.gff1304be0a05 # arch : aarch64 # nrcpus online : 4 # nrcpus avail : 4 # cpuid : 0x00000000410fd030 # total memory : 899432 kB # cmdline : /home/linaro/perf-tests/perf-5.9-rc1 record -e cs_etm/@tmc_etr0/k uname -a
Agreed that a self described format is better for metadata structure; I'd like to leave this question to Mathieu who might have thoughts for this.
Thanks, Leo _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
-
Al Grant -
Leo Yan -
Mike Leach