Hi all,
On Tue, Nov 10, 2020 at 11:14:16AM -0700, Mathieu Poirier wrote:
On Tue, Nov 10, 2020 at 02:52:43PM +0800, Leo Yan wrote:
Hi all,
[ + CoreSight Mailing List ]
On the mainlne kernel with the latest commit 407ab579637c "Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm", the perf cs-etm testing fails.
I found the perf cs-etm testing itself has a typo issue and a regression introduced by Perf tool in the recent mainline kernel, so I sent out the two patches for fixing the related issues [1].
Even after applying these two fixings for the testing script, I still observe the etm testing failure with snapshot mode. I manully checked the perf data and confirmed the perf data contains the raw trace data for snapshot (so 'perf script -D' can work well), but it fails to generate any branch samples, so the command 'perf script' will not output any samples.
Seems to me, this issue is more likely related with driver or Perf tool's common code change but not the testing itself;
If the trace data is available then it is likely the kernel drivers worked properly. I would concentrate on the perf tools - with the amount of code churn in that subsystem many regressions have been introduced before.
to use 'git bisect' to narrow down the snapshot mode failure, if you have any idea for fixing this issue, please let me know. Otherwise, I will dig into this issue and keep posted.
I'm afraid a bisect is your only option.
Firstly, sorry for this is a long replying, in this email I will try to explain what's the problem, but I am also seeking your suggestions for how to fix the issue.
Let me give out a quick way for reproducing the issue:
# sh test_cs_etm_snapshot.sh => it is pasted at the end of email # perf script => it cannot output any samples for snapshot mode
Here have a question: why tool cannot output samples for both kernel and user space programs?
### Failure for parsing Kernel symbols
For output the kernel's trace data, perf needs to read out the symbols from the kernel's DSO. As far as I know, there have three ways to import the kernel's symbols:
- By default, perf tool to read out the runtime kernel symbols from /proc/kallsyms and /proc/kcore; to support both of these two proc nodes, we need to enable kernel configurations: CONFIG_PROC_KCORE=y CONFIG_KALLSYMS=y - Copy vmlinux in the 'pwd' folder when run 'perf' command; - Specify the vmlinux path with option '-k'.
For my case, since I missed to enable configs for KCORE and doesn't copy or specify vmlinux, thus the kernel symbols are failed to parsed and cannot synthesize the samples [1]. After enabled KCORE option, then tool outputs the kernel symbols properly. Since there have no any warning log to remind the failure for finding DSO or symbols, and I encounered the similiar issues and usually it took me serveral hours or even more time to locate the issue. So share the info at here and also I think it's better to commit patch to improve the verbose log.
### Failure for parsing user space symbols
After the kernel symbols parsing is fixed, the test case can pass. But the difficult part is for the userspace symbol's parsing. Sometime, we even don't notice that it fails to synthesize samples for the userspace trace data.
The rationale is: perf tool imports the userspace's DSOs when handling events PERF_RECORD_MMAP2; in theory, we should expect the events PERF_RECORD_MMAP2 should arrive ahead event PERF_RECORD_AUX, so that perf tool can load the userspace DSOs and get ready the symbol tree before parsing any address in the userspace.
But I found the events PERF_RECORD_MMAP2 and PERF_RECORD_AUX are interleaved with each other in the perf.data file and it's possible to process all AUX trace data but without user space program's DSO, thus we cannot see any samples from userspace. This is why I sent RFC patch to 'force' the PERF_RECORD_MMAP2 events to be processed prior to PERF_RECORD_AUX [2]; but the change in the RFC patch is more likely a workaround rather than a real fixing. The reason is if there have PERF_RECORD_AUX event is coming and even the DSO is not prepared, it should skip a very small amount trace data and exit from the perf cs-etm decoding loop, and perf tool can continue to import DSOs based on the sequential PERF_RECORD_MMAP2 events, afterwards the perf tool can decode CoreSight trace data and generate samples.
This is a common issue for all AUX trace cases (like cs-etm, SPE, Intel-pt, etc), after looking into the code for Intel-PT, I think Intel-pt uses the timestamp to correlate different events. The key point is it pushes the first Intel-PT's sample with timestamp in the auxtrace heap [3], if any new event's timestamp is prior to Intel-PT's trace timestamp, Intel-pt's decoding loop will bail out [4] and process other events ahead. Simply to say, the events are handled in the order of the timestamp.
I think this approach can be applied on Arm SPE, since Arm SPE uses arch timer's counter as timestamp, Arm SPE trace and other events can be correlated with each other with the same timeline axis.
For CoreSight, I'm not sure how to proceed; the Arch timer counter is not mandatory for connecting CoreSight, but if some platform has used Arch timer counter for CoreSight (Like Juno board), should we move forward for this?
Otherwise, as Adrian Hunter suggested: "If you are processing data based on PERF_RECORD_AUX timestamp then you need to pay attention to the offset. PERF_RECORD_AUX gives you aux_offset/aux_size and auxtrace_buffer (which may contain data from several PERF_RECORD_AUX) gives you offset/size.", so should we extract the aux_offset and aux_size for PERF_RECORD_AUX event and every time only decode the trace data no more than aux_size for every PERF_RECORD_AUX event? This means we need to add additional checking for buffer length in the functions cs_etm__process_queues() and cs_etm__run_decoder() to avoid processing any trace data not belonging to the event.
How about you think for this? Thanks for reading it!
Leo
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tool... [2] https://lore.kernel.org/patchwork/patch/1340992/ [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tool... [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tool...
---8<---
test_cs_etm_snapshot.sh:
#!/bin/sh
arm_cs_etm_snapshot_test() { echo "Recording trace with snapshot mode"
perf record -e cs_etm/@tmc_etr0/ -S \ -- dd if=/dev/zero of=/dev/null &
PERFPID=$!
# Wait for perf program sleep 2
# Send signal to snapshot trace data kill -USR2 $PERFPID
# Stop perf program kill $PERFPID wait $PERFPID }
arm_cs_etm_snapshot_test exit 0