This test commonly fails on Arm Juno because the instruction interval
is large enough to miss generating any samples for Perf in system-wide
mode.
Fix this by lowering the interval until a comfortable number of Perf
instructions are generated. The test is still quick to run because only
a small amount of trace is gathered.
Before:
sudo ./perf test coresight -vvv
...
Recording trace with system wide mode
Looking at perf.data file for dumping branch samples:
Looking at perf.data file for reporting branch samples:
Looking at perf.data file for instruction samples:
CoreSight system wide testing: FAIL
...
After:
sudo ./perf test coresight -vvv
...
Recording trace with system wide mode
Looking at perf.data file for dumping branch samples:
Looking at perf.data file for reporting branch samples:
Looking at perf.data file for instruction samples:
CoreSight system wide testing: PASS
...
Signed-off-by: James Clark <james.clark(a)arm.com>
---
tools/perf/tests/shell/test_arm_coresight.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/tests/shell/test_arm_coresight.sh b/tools/perf/tests/shell/test_arm_coresight.sh
index e4cb4f1806ff..daad786cf48d 100755
--- a/tools/perf/tests/shell/test_arm_coresight.sh
+++ b/tools/perf/tests/shell/test_arm_coresight.sh
@@ -70,7 +70,7 @@ perf_report_instruction_samples() {
# 68.12% touch libc-2.27.so [.] _dl_addr
# 5.80% touch libc-2.27.so [.] getenv
# 4.35% touch ld-2.27.so [.] _dl_fixup
- perf report --itrace=i1000i --stdio -i ${perfdata} 2>&1 | \
+ perf report --itrace=i20i --stdio -i ${perfdata} 2>&1 | \
egrep " +[0-9]+\.[0-9]+% +$1" > /dev/null 2>&1
}
--
2.28.0
On 13/10/2022 14:45, Christian Borntraeger wrote:
> Probably not related to this patch set, but
> tools/perf/Documentation/perf-arm-coresight.txt
>
> results in
> cd tools/perf/Documentation/
> make man
> ASCIIDOC perf-arm-coresight.xml
> asciidoc: ERROR: perf-arm-coresight.txt: line 2: malformed manpage title
> asciidoc: ERROR: perf-arm-coresight.txt: line 3: name section expected
> asciidoc: FAILED: perf-arm-coresight.txt: line 3: section title expected
>
> in linux-next.
>
I think this is the same as what has been reported here:
https://lore.kernel.org/linux-perf-users/20220909152803.2317006-1-carsten.h…
Carsten, are you able to take a look at this?
Thanks
James
The current method for allocating trace source ID values to sources is
to use a fixed algorithm for CPU based sources of (cpu_num * 2 + 0x10).
The STM is allocated ID 0x1.
This fixed algorithm is used in both the CoreSight driver code, and by
perf when writing the trace metadata in the AUXTRACE_INFO record.
The method needs replacing as currently:-
1. It is inefficient in using available IDs.
2. Does not scale to larger systems with many cores and the algorithm
has no limits so will generate invalid trace IDs for cpu number > 44.
Additionally requirements to allocate additional system IDs on some
systems have been seen.
This patch set introduces an API that allows the allocation of trace IDs
in a dynamic manner.
Architecturally reserved IDs are never allocated, and the system is
limited to allocating only valid IDs.
Each of the current trace sources ETM3.x, ETM4.x and STM is updated to use
the new API.
For the ETMx.x devices IDs are allocated on certain events
a) When using sysfs, an ID will be allocated on hardware enable, or a read of
sysfs TRCTRACEID register and freed when the sysfs reset is written.
b) When using perf, ID is allocated on during setup AUX event, and freed on
event free. IDs are communicated using the AUX_OUTPUT_HW_ID packet.
The ID allocator is notified when perf sessions start and stop
so CPU based IDs are kept constant throughout any perf session.
Note: This patchset breaks some backward compatibility for perf record and
perf report.
The version of the AUXTRACE_INFO has been updated to reflect the fact that
the trace source IDs are generated differently. This will
mean older versions of perf report cannot decode the newer file.
Applies to coresight/next [4d45bc82df66]
Tested on DB410c
Changes since v2:
1) Improved backward compatibility: (requested by James)
Using the new version of perf on an old kernel will generate a usable file
legacy metadata values are set by the new perf and will be used if mew
ID packets are not present in the file.
Using an older version of perf / simpleperf on an updated kernel may still
work. The trace ID allocator has been updated to use the legacy ID values
where possible, so generated file and used trace IDs will match up to the
point where the legacy algorithm is broken anyway.
2) Various changes to the ID allocator and ID packet format.
(suggested by Suzuki)
3) per CPU ID info in allocator now stored as atomic type to allow a passive read
without taking the allocator spinlock. perf flow now allocates and releases ID
values in setup_aux / free_event. Device enable and event enable use the passive
read to set the allocated values. This simplifies the locking mechanisms on the
perf run and fixes issues that arose with locking dependencies.
Changes since v1:
(after feedback & discussion with Mathieu & Suzuki).
1) API has changed. The global trace ID map is managed internally, so it
is no longer passed in to the API functions.
2) perf record does not use sysfs to find the trace IDs. These are now
output as AUX_OUTPUT_HW_ID events. The drivers, perf record, and perf report
have been updated accordingly to generate and handle these events.
Mike Leach (13):
coresight: trace-id: Add API to dynamically assign Trace ID values
coresight: Remove obsolete Trace ID unniqueness checks
coresight: stm: Update STM driver to use Trace ID API
coresight: etm4x: Update ETM4 driver to use Trace ID API
coresight: etm3x: Update ETM3 driver to use Trace ID API
coresight: etmX.X: stm: Remove trace_id() callback
coresight: perf: traceid: Add perf notifiers for Trace ID
perf: cs-etm: Move mapping of Trace ID and cpu into helper function
perf: cs-etm: Update record event to use new Trace ID protocol
kernel: events: Export perf_report_aux_output_id()
perf: cs-etm: Handle PERF_RECORD_AUX_OUTPUT_HW_ID packet
coresight: events: PERF_RECORD_AUX_OUTPUT_HW_ID used for Trace ID
coresight: trace-id: Add debug & test macros to Trace ID allocation
drivers/hwtracing/coresight/Makefile | 2 +-
drivers/hwtracing/coresight/coresight-core.c | 49 +--
.../hwtracing/coresight/coresight-etm-perf.c | 23 ++
drivers/hwtracing/coresight/coresight-etm.h | 3 +-
.../coresight/coresight-etm3x-core.c | 92 +++--
.../coresight/coresight-etm3x-sysfs.c | 27 +-
.../coresight/coresight-etm4x-core.c | 79 ++++-
.../coresight/coresight-etm4x-sysfs.c | 27 +-
drivers/hwtracing/coresight/coresight-etm4x.h | 3 +
drivers/hwtracing/coresight/coresight-stm.c | 49 +--
.../hwtracing/coresight/coresight-trace-id.c | 266 ++++++++++++++
.../hwtracing/coresight/coresight-trace-id.h | 78 +++++
include/linux/coresight-pmu.h | 35 +-
include/linux/coresight.h | 3 -
kernel/events/core.c | 1 +
tools/include/linux/coresight-pmu.h | 48 ++-
tools/perf/arch/arm/util/cs-etm.c | 21 +-
.../perf/util/cs-etm-decoder/cs-etm-decoder.c | 7 +
tools/perf/util/cs-etm.c | 331 +++++++++++++++---
tools/perf/util/cs-etm.h | 14 +-
20 files changed, 933 insertions(+), 225 deletions(-)
create mode 100644 drivers/hwtracing/coresight/coresight-trace-id.c
create mode 100644 drivers/hwtracing/coresight/coresight-trace-id.h
--
2.17.1
On 21/07/2022 14:03, Sudeep Holla wrote:
> With lockdeps enabled, we get the following warning:
>
> ======================================================
> WARNING: possible circular locking dependency detected
> ------------------------------------------------------
> kworker/u12:1/53 is trying to acquire lock:
> ffff80000adce220 (coresight_mutex){+.+.}-{4:4}, at: coresight_set_assoc_ectdev_mutex+0x3c/0x5c
> but task is already holding lock:
> ffff80000add1f60 (ect_mutex){+.+.}-{4:4}, at: cti_probe+0x318/0x394
>
> which lock already depends on the new lock.
> the existing dependency chain (in reverse order) is:
>
> -> #1 (ect_mutex){+.+.}-{4:4}:
> __mutex_lock_common+0xd8/0xe60
> mutex_lock_nested+0x44/0x50
> cti_add_assoc_to_csdev+0x4c/0x184
> coresight_register+0x2f0/0x314
> tmc_probe+0x33c/0x414
>
> -> #0 (coresight_mutex){+.+.}-{4:4}:
> __lock_acquire+0x1a20/0x32d0
> lock_acquire+0x160/0x308
> __mutex_lock_common+0xd8/0xe60
> mutex_lock_nested+0x44/0x50
> coresight_set_assoc_ectdev_mutex+0x3c/0x5c
> cti_update_conn_xrefs+0x6c/0xf8
> cti_probe+0x33c/0x394
>
> other info that might help us debug this:
> Possible unsafe locking scenario:
> CPU0 CPU1
> ---- ----
> lock(ect_mutex);
> lock(coresight_mutex);
> lock(ect_mutex);
> lock(coresight_mutex);
> *** DEADLOCK ***
>
> 4 locks held by kworker/u12:1/53:
> #0: ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1fc/0x63c
> #1: (deferred_probe_work){+.+.}-{0:0}, at: process_one_work+0x228/0x63c
> #2: (&dev->mutex){....}-{4:4}, at: __device_attach+0x48/0x1a8
> #3: (ect_mutex){+.+.}-{4:4}, at: cti_probe+0x318/0x394
>
> To fix the same, call cti_add_assoc_to_csdev without the holding
> coresight_mutex and confine the locking while setting the associated
> ect / cti device using coresight_set_assoc_ectdev_mutex().
>
> Cc: Mathieu Poirier <mathieu.poirier(a)linaro.org>
> Cc: Suzuki K Poulose <suzuki.poulose(a)arm.com>
> Cc: Mike Leach <mike.leach(a)linaro.org>
> Cc: Leo Yan <leo.yan(a)linaro.org>
> Signed-off-by: Sudeep Holla <sudeep.holla(a)arm.com>
Thanks for the fix. I will push this out at rc1 as fixes.
Thanks
Suzuki
I'm still leaving out CONFIG_CORESIGHT_SOURCE_ETM4X because it depends
on the outcome of the investigation into CONFIG_PID_IN_CONTEXTIDR, but
I think we should enable these ones for now and start getting some of
the benefits sooner.
Changes since v1:
* Remove CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y which shouldn't be
enabled by default
-----
As suggested by Catalin here's the change to add Coresight to defconfig.
Unfortunately I don't think we should add CONFIG_CORESIGHT_SOURCE_ETM4X
which builds a few files until [1] is merged because of the overhead
of CONFIG_PID_IN_CONTEXTIDR.
[1]: https://lore.kernel.org/lkml/20211021134530.206216-1-leo.yan@linaro.org/T/
applies to arm64/for-next/core (e99db032d186)
James Clark (1):
arm64: defconfig: Add Coresight as module
arch/arm64/configs/defconfig | 8 ++++++++
1 file changed, 8 insertions(+)
--
2.28.0
On 22/09/2022 11:52, Catalin Marinas wrote:
> On Thu, Sep 22, 2022 at 10:34:45AM +0100, James Clark wrote:
>> On 21/09/2022 16:08, Catalin Marinas wrote:
>>> 2. Always on CONFIG_PID_IN_CONTEXTIDR (we might as well remove the
>>> Kconfig entry). This would write the root pid namespace value
>>> (task_pid_nr()).
>>
>> If we're not worried about the overhead after all, this would be the
>> easiest solution. And then SPE or Coresight already decide whether they
>> want to use the value or not, so no further changes are needed.
>>
>> From Leo's patch there is a table that shows a 1% overhead with it
>> enabled permanently, and I've heard a figure like that mentioned before.
>> So I could also resurrect that patch to use static keys? Although it's a
>> bit more complicated, that would be my preference. And then we can have
>> that mode always on.
>
> I don't think we should bother with static keys, just always enable it
> but try to reduce/group the ISBs from all the functions called on the
> __switch_to() path. We may actually get a speed-up.
>
Ok thanks I will take a look at this
On 21/09/2022 16:08, Catalin Marinas wrote:
> On Wed, Sep 21, 2022 at 03:05:34PM +0100, James Clark wrote:
>> As suggested by Catalin here's the change to add Coresight to defconfig.
>>
>> Unfortunately I don't think we should add CONFIG_CORESIGHT_SOURCE_ETM4X
>> which builds a few files until [1] is merged because of the overhead
>> of CONFIG_PID_IN_CONTEXTIDR.
>>
>> [1]: https://lore.kernel.org/lkml/20211021134530.206216-1-leo.yan@linaro.org/T/
>
> I thought the overhead wasn't the problem, it's mostly negligible. We
> can probably save a few more cycles on the __switch_to() path by
> replacing several isb()s in those functions with a single one just
> before cpu_switch_to().
>
> IIRC the issue is that unless a process runs in the root pid namespace,
> the actual pid written to contextidr is meaningless.
This is true, and Leo has recently disabled it in that scenario in
aab473867fed.
>
> Now that you reminded me of that thread, I see three options (sorry, not
> entirely related to the defconfig updates):
>
> 1. Remove CONFIG_PID_IN_CONTEXTIDR and corresponding code completely,
> find other events to correlate the task with the trace.
Unfortunately when tracing per core we would need kernel timestamps in
the trace to correlate to the switch records. At the moment Coresight is
using a different clock source so it's not possible and we're still
using the context ID to correlate samples.
With FEAT_TRF in v8.4 it will be possible to do this and we've started
working towards that here: 0f00b223ea22. But we'd still have to support
older hardware too, so CONFIG_PID_IN_CONTEXTIDR can't be removed completely.
For SPE it's not required because we already have the right timestamps
in the samples and we've added support for no context IDs in the decoder
here: 27d113cfe892
>
> 2. Always on CONFIG_PID_IN_CONTEXTIDR (we might as well remove the
> Kconfig entry). This would write the root pid namespace value
> (task_pid_nr()).
If we're not worried about the overhead after all, this would be the
easiest solution. And then SPE or Coresight already decide whether they
want to use the value or not, so no further changes are needed.
From Leo's patch there is a table that shows a 1% overhead with it
enabled permanently, and I've heard a figure like that mentioned before.
So I could also resurrect that patch to use static keys? Although it's a
bit more complicated, that would be my preference. And then we can have
that mode always on.
>
> 3. Similar to (2) but instead write task_pid_nr_ns(). An alternative
> here is to write -1 if the task is not in the root pid namespace.
>
> Strong preference for (1).
>
On 21/09/2022 17:46, Mathieu Poirier wrote:
> On Wed, Sep 21, 2022 at 04:26:59PM +0100, Mark Brown wrote:
>> On Wed, Sep 21, 2022 at 03:05:35PM +0100, James Clark wrote:
>>
>>> +CONFIG_CORESIGHT_CTI=m
>>> +CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
>>
>
> I agree - integration registers should not be enabled by default.
>
>> Do we want this turned on by default? According to the
>> description it's a bit dangerous and it's exposed via sysfs
>> rather than debugfs.
>
>
Should I disable just CONFIG_CORESIGHT_CTI_INTEGRATION_REGS or
CONFIG_CORESIGHT_CTI as well? There are other writable registers exposed
via sysfs outside of these two options, so I wanted to check if it's
just the integration registers that are the issue.
