TRBE currently stops trace collection at each buffer limit, so interrupt latency can cause trace loss. This series adds trigger-based maintenance and WRAP mode to reduce these gaps, while protecting unconsumed data and retaining limit-based handling where required by errata.
The perf changes preserve decoder state across contiguous indexed AUX fragments, with resets when continuity cannot be established. Thread stacks still flush at each data block boundary. For normal recording, a quarter-buffer watermark wakes userspace earlier.
The 11 patches cover:
Patches 01-02: Next-buffer computation cleanup and refactoring. Patches 03-07: Trigger support, headroom, save/restore, and KUnit tests. Patches 08-11: Decoder continuity and earlier userspace wakeups.
This series is based on coresight-next and requires "coresight: trbe: Refactor buffer and IRQ handling" [1] to be applied first.
This series has been tested on the Orion6 board with TRBE and the Juno-r2 board with ETR using the following commands:
perf record -e cs_etm// -a -- bash -c 'for i in {1..5}; do timeout 0.1 yes > /dev/null; sleep 0.1; done' perf script --itrace=i1000iTZ
[1] https://lore.kernel.org/linux-arm-kernel/20261016-trbe_enable_disable_irq_ha...
Signed-off-by: Leo Yan leo.yan@arm.com --- Changes in v2: - Split the enable/disable, perf-handle lifecycle, and IRQ handling fixes into the separate prerequisite series, leaving 11 patches here. - Restrict KUnit buffer and padding stubs to the synthetic test handle, preserving normal buffer lookup and AUX accounting for real perf IRQs. - Centralize decoder-reset decisions in cs_etm__get_trace(), while retaining thread-stack flushing at every data block boundary. - Reset per-thread decoders for unknown or changed CPUs, preserve CPU metadata on all split pieces, and retain resets through empty fragments. - Reset the decoder on logical AUX offset discontinuities, including clipped snapshots and missing AUX records, while preserving continuity across internal buffer splits. - Link to v1: https://lore.kernel.org/r/20251201-trbe_buffer_refactor_v1-1-v1-0-7da32b076b...
--- Leo Yan (11): coresight: trbe: Remove misleading comment coresight: trbe: Refactor next buffer computation coresight: trbe: Track trigger count support per CPU coresight: trbe: Use trigger mode for buffer maintenance coresight: trbe: Leave headroom for stop-on-trigger mode coresight: trbe: Save and restore trigger counter coresight: trbe: Add KUnit tests for compute normal buffer perf cs-etm: Reset decoder on COLLISION flag perf cs-etm: Reset per-thread decoder on CPU migration perf cs-etm: Reset decoder on AUX offset discontinuity perf cs-etm: Set watermark for AUX trace
drivers/hwtracing/coresight/Kconfig | 15 + .../coresight/coresight-trbe-kunit-tests.c | 644 +++++++++++++++++++++ drivers/hwtracing/coresight/coresight-trbe.c | 342 ++++++++--- drivers/hwtracing/coresight/coresight-trbe.h | 18 + tools/perf/arch/arm/util/cs-etm.c | 15 + tools/perf/util/auxtrace.h | 2 + tools/perf/util/cs-etm.c | 92 ++- 7 files changed, 1029 insertions(+), 99 deletions(-) --- base-commit: 343cfcbb3ecfdb80aa576a428cfa3b65ad783842 change-id: 20260617-trbe_buffer_refactor_v2-8813e33fde8b prerequisite-message-id: 20261016-trbe_enable_disable_irq_handle_refactor-v2-0-3f57cf3199b1@arm.com prerequisite-patch-id: 295f75af9986fc6193b0f3ec0f144096c1194efd prerequisite-patch-id: 60541bcf6bd25acf076fca2dd2e62ea382541fd5 prerequisite-patch-id: 0a6dbd3a69611c9bdca8ad61ea943d6577251fb8 prerequisite-patch-id: 0f4dc971d1375cd5672a63cd148b6e7f6dfdd9b3 prerequisite-patch-id: b42114d2185bbd15c9095191417aad34137cc5e9 prerequisite-patch-id: fdcdce0d19c611de475c6e9737ef544a9fb19153 prerequisite-patch-id: 948b6aa1254adfc9f89a5aba3ef45e41d891debe prerequisite-patch-id: ba1cc65a72bcfef9e22fea14f52eeaeeb2c551d3 prerequisite-patch-id: d0efcfbe7d1f7886aaa5c34655c14a979bf55863 prerequisite-patch-id: af045abf5718e35f16a5b98ec7c04213cf24d426 prerequisite-patch-id: d0af1d9f5551622858815d148de5dc3c94cdf564 prerequisite-patch-id: 9d02a95ecafda17debf728ef1098b98030019a10 prerequisite-patch-id: c6251f03223c3bd4bcf43abc292907dd93cad527 prerequisite-patch-id: 690af6c1769b93d8712aa73b322bb4e5f0acdc66 prerequisite-patch-id: 21c3d8dacd550b977ae549a1865e43523ed64f98 prerequisite-patch-id: 0b5666068c0fed73e6906473b38861556881e5b0 prerequisite-patch-id: f5c97c9acc3e691b3612999eae97c3366acf3d97 prerequisite-patch-id: 55846a3b80ca3196062d06522af768ac83751d97 prerequisite-patch-id: 5f5be1d1c925a9c0d945eba516a9075153045292 prerequisite-patch-id: 8e39271308702e557abc4e4e79fc96d0157844a7 prerequisite-patch-id: b99199a58c4a87408c4c2b0b39ee4861f5cd9d80 prerequisite-patch-id: 1bdbfe0417a4a98984b2fbe8d664738657de3032 prerequisite-patch-id: aeab33675e29539d239c8ffdf4bbd4cf861972c6
Best regards,
Since the rounded-up wakeup address is always higher than the head, the limit cannot be less than the head caused by wakeup capping.
The described scenario is never valid caused by wakeup capping, remove the comment to avoid confusion.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 29a1882b1f3c2eb7166f71eeef37badece67039c..ada4794c51f3eb9142cd62f4209affe308142d1b 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -505,10 +505,10 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) limit = min(limit, round_up(wakeup, PAGE_SIZE));
/* - * There are two situation when this can happen i.e limit is before + * There is a situation when this can happen i.e limit is before * the head and hence TRBE cannot be configured. * - * 1) head < tail (aligned down with PAGE_SIZE) and also they are both + * head < tail (aligned down with PAGE_SIZE) and also they are both * within the same PAGE size range. * * PAGE_SIZE @@ -519,18 +519,6 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| * +------------|------|--------|-------+ * trbe_base trbe_base + nr_pages - * - * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both - * head and wakeup are within same PAGE size range. - * - * PAGE_SIZE - * |----------------------| - * - * limit head wakeup tail - * +----|------|-------|--------|-------+ - * |$$$$$$$$$$$|=======|========|$$$$$$$| - * +----|------|-------|--------|-------+ - * trbe_base trbe_base + nr_pages */ if (limit > head) return limit;
compute_trbe_buffer_limit() only returned the next TRBE limit pointer, leaving __arm_trbe_enable() to derive the remaining buffer state and detect the -ENOSPC case.
Replace it with trbe_compute_next(), which computes the next TRBE buffer state as one operation. Let the snapshot and normal paths set buf->trbe_limit directly and return 0 or an errno, then update trbe_write and trbe_hw_base after a successful limit calculation. The normal path now reports -ENOSPC directly when no writable area is available.
This refactoring is for future extensions, such as calculating additional values (e.g., the trigger count). No functional changes are intended.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 44 +++++++++++++++++----------- 1 file changed, 27 insertions(+), 17 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index ada4794c51f3eb9142cd62f4209affe308142d1b..09ed9bcf4a70479169094697dcc3d445629cfaae 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -349,7 +349,7 @@ static void trbe_pad_buf(struct perf_output_handle *handle, int len) perf_aux_output_skip(handle, len); }
-static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) +static int trbe_snapshot_offset(struct perf_output_handle *handle) { struct trbe_buf *buf = etm_perf_sink_config(handle);
@@ -358,7 +358,8 @@ static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) * the decoder to reset in case of an overflow or corruption. * So we can use the entire buffer for the snapshot mode. */ - return buf->nr_pages * PAGE_SIZE; + buf->trbe_limit = buf->trbe_base + buf->nr_pages * PAGE_SIZE; + return 0; }
static u64 trbe_min_trace_buf_size(struct perf_output_handle *handle) @@ -527,7 +528,7 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) return 0; }
-static unsigned long trbe_normal_offset(struct perf_output_handle *handle) +static int trbe_normal_offset(struct perf_output_handle *handle) { struct trbe_buf *buf = etm_perf_sink_config(handle); u64 limit = __trbe_normal_offset(handle); @@ -546,19 +547,34 @@ static unsigned long trbe_normal_offset(struct perf_output_handle *handle) limit = __trbe_normal_offset(handle); head = PERF_IDX2OFF(handle->head, buf); } - return limit; + + if (!limit) + return -ENOSPC; + + buf->trbe_limit = buf->trbe_base + limit; + return 0; }
-static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) +static int trbe_compute_next(struct perf_output_handle *handle) { struct trbe_buf *buf = etm_perf_sink_config(handle); - unsigned long offset; + int ret; + + perf_aux_output_flag(handle, PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW);
if (buf->snapshot) - offset = trbe_snapshot_offset(handle); + ret = trbe_snapshot_offset(handle); else - offset = trbe_normal_offset(handle); - return buf->trbe_base + offset; + ret = trbe_normal_offset(handle); + + if (ret) + return ret; + + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); + + /* Set the base of the TRBE to the buffer base */ + buf->trbe_hw_base = buf->trbe_base; + return 0; }
static void clr_trbe_status(void) @@ -1022,15 +1038,9 @@ static int __arm_trbe_enable(struct trbe_buf *buf, { int ret = 0;
- perf_aux_output_flag(handle, PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW); - buf->trbe_limit = compute_trbe_buffer_limit(handle); - buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); - if (buf->trbe_limit == buf->trbe_base) { - ret = -ENOSPC; + ret = trbe_compute_next(handle); + if (ret) goto err; - } - /* Set the base of the TRBE to the buffer base */ - buf->trbe_hw_base = buf->trbe_base;
ret = trbe_apply_work_around_before_enable(buf); if (ret)
Some TRBE errata are not compatible with using trigger count mode. In particular, CPUs affected by the overwrite-in-fill-mode or out-of-range workarounds must continue to use the existing fill-mode handling.
Record whether trigger count mode is supported in each CPU's probe. This prepares the enable path to avoid programming TRBTRG_EL1 on affected CPUs.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 09ed9bcf4a70479169094697dcc3d445629cfaae..92cd9c28cb4f6d9da7d41aa9eeedf4295a810006 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -140,6 +140,7 @@ struct trbe_save_state { * @trbe_align - Software alignment used for the TRBPTR_EL1. * @cpu - CPU this TRBE belongs to. * @mode - Mode of current operation. (perf/disabled) + * @trig_is_supported - Trigger count support flag. * @buf - Buffer configuration retained for the enabled session. * @handle - Perf output handle from successful enable until disable. * @drvdata - TRBE specific drvdata @@ -151,6 +152,7 @@ struct trbe_cpudata { u64 trbe_align; int cpu; enum cs_mode mode; + bool trig_is_supported; struct trbe_buf *buf; struct perf_output_handle *handle; struct trbe_drvdata *drvdata; @@ -222,6 +224,15 @@ static bool trbe_is_broken(struct trbe_cpudata *cpudata) return trbe_has_erratum(cpudata, TRBE_IS_BROKEN); }
+static bool trbe_trigger_count_supported(struct trbe_cpudata *cpudata) +{ + if (trbe_may_overwrite_in_fill_mode(cpudata) || + trbe_may_write_out_of_range(cpudata)) + return false; + + return true; +} + static int trbe_alloc_node(struct perf_event *event) { if (event->cpu == -1) @@ -1609,6 +1620,7 @@ static void arm_trbe_probe_cpu(void *info) else cpudata->trbe_align = cpudata->trbe_hw_align;
+ cpudata->trig_is_supported = trbe_trigger_count_supported(cpudata); cpudata->trbe_flag = get_trbe_flag_update(trbidr); cpudata->cpu = cpu; cpudata->drvdata = drvdata;
TRBE currently uses FILL mode for AUX record. When the write pointer reaches LIMIT, trace collection stops and a maintenance interrupt is raised. However, due to IRQ latency, trace data may be lost during the window in which tracing is halted but the program continues to run.
Use TRBTRG_EL1 to request maintenance without always stopping trace.
1) If the perf wakeup point is before the programmed LIMIT, program a trigger event at the wakeup point while keeping FILL mode so LIMIT remains the hard stop.
head watermark tail +----+---------------+---------+-------+ |$$$$| | |$$$$$$$| +----+---------------+---------+-------+ base `---- count ----' limit base + nr_pages
$$$ : Filled trace data
2) If the writable range extends past the end of the buffer, use WRAP mode with Stop-on-trigger so tracing continues past the limit but stops at the trigger before overwriting unconsumed data.
Do not program a trigger count that exceed the 32-bit TRBTRG_EL1.TRG field.
watermark tail head +--------+-----------+---------+-------+ | | |$$$$$$$$$| | +--------+-----------+---------+-------+ base base + nr_pages limit
`-------> >-- counter ---------'
The available AUX space can be unaligned even when the head and tail are aligned, because perf reserves one byte to distinguish a full ring from an empty one. Round the wrap-protection count down to the TRBIDR_EL1.Align granule before programming TRBTRG_EL1. Use the LIMIT fallback if the rounded count no longer extends beyond the limit.
Keep the LIMIT-based wakeup path when trigger mode is bypassed. Add a TRIG fault action so trigger events can be handled separately from WRAP events. Keep access not allowed stops ahead of WRAP and TRG classification. The driver sets TRG when arming the counter, so a set TRG bit alone does not establish that the counter has expired.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 193 ++++++++++++++++++++------- drivers/hwtracing/coresight/coresight-trbe.h | 18 +++ 2 files changed, 166 insertions(+), 45 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 92cd9c28cb4f6d9da7d41aa9eeedf4295a810006..95bbdbaca81bd59ac97b272216489c2f66393fa7 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -49,6 +49,7 @@ #define TRBE_TRACE_MIN_BUF_SIZE 64
enum trbe_fault_action { + TRBE_FAULT_ACT_TRIG, TRBE_FAULT_ACT_WRAP, TRBE_FAULT_ACT_SPURIOUS, TRBE_FAULT_ACT_FATAL, @@ -68,6 +69,7 @@ struct trbe_buf { unsigned long trbe_hw_base; unsigned long trbe_limit; unsigned long trbe_write; + unsigned long trbe_trg_count; int nr_pages; void **pages; bool snapshot; @@ -169,6 +171,13 @@ struct trbe_drvdata { struct platform_device *pdev; };
+/* Compute the buffer next state */ +struct trbe_buf_next { + u64 head; + u64 limit; + u32 count; +}; + static void trbe_check_errata(struct trbe_cpudata *cpudata) { int i; @@ -291,6 +300,7 @@ static void trbe_reset_local(struct trbe_cpudata *cpudata) write_sysreg_s(0, SYS_TRBLIMITR_EL1); isb(); trbe_drain_buffer(); + write_sysreg_s(0, SYS_TRBTRG_EL1); write_sysreg_s(0, SYS_TRBPTR_EL1); write_sysreg_s(0, SYS_TRBBASER_EL1); write_sysreg_s(0, SYS_TRBSR_EL1); @@ -392,6 +402,77 @@ static u64 trbe_min_trace_buf_size(struct perf_output_handle *handle) return size; }
+/* + * Wakeup may be arbitrarily far into the future. If it's not in the + * current generation, either we'll wrap before hitting it, or it's + * in the past and has been handled already. + * + * Return true when the wakeup point belongs to the current writable + * generation and is located between the current head but before tail. + * + * head wakeup tail + * +----|---------------|-------|-------+ + * |$$$$|###############|%%%%%%%|$$$$$$$| + * +----|---------------|-------|-------+ + * trbe_base limit trbe_base + nr_pages + */ +static bool trbe_wakeup_before_tail(struct perf_output_handle *handle, + u64 head, u64 wakeup) +{ + return ((handle->wakeup - handle->head) < handle->size && + head <= wakeup); +} + +static u32 trbe_normal_trigger_count(struct perf_output_handle *handle, + u64 head, u64 wakeup, u64 limit) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_cpudata *cpudata = buf->cpudata; + const u64 bufsize = buf->nr_pages * PAGE_SIZE; + u64 count; + + if (!cpudata->trig_is_supported) + return 0; + + /* + * Set a trigger at the watermark if it can be reached before the + * programmed limit. + */ + if (trbe_wakeup_before_tail(handle, head, wakeup)) { + count = round_up(wakeup - head, cpudata->trbe_hw_align); + if (count && head + count < limit) + goto out; + } + + if (limit != bufsize) + return 0; + + /* + * Otherwise, when LIMIT is at the end of the buffer and the writable + * region wraps, use a trigger count to stop before unconsumed data + * can be overwritten after wrap. + * + * wakeup tail head + * +----|---------------|-------|-------+ + * |####|###############|$$$$$$$|#######| + * +----|---------------|-------|-------+ + * trbe_base trbe_base + nr_pages + * limit + * + * |>>>>>>>> + * >>>>>>>>>>>>>>>>>>>>>| + * ` Trigger counter covers the writable region, rounded down + * to the hardware alignment. + */ + count = round_down(handle->size, cpudata->trbe_hw_align); + if (count <= limit - head) + return 0; + +out: + /* Ensure the count fits in the 32-bit TRBTRG_EL1.TRG field */ + return count > U32_MAX ? 0 : count; +} + /* * TRBE Limit Calculation * @@ -402,13 +483,15 @@ static u64 trbe_min_trace_buf_size(struct perf_output_handle *handle) * %%%% - Free area, disabled, trace will not be written * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped */ -static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle, + struct trbe_buf_next *next) { struct trbe_buf *buf = etm_perf_sink_config(handle); struct trbe_cpudata *cpudata = buf->cpudata; const u64 bufsize = buf->nr_pages * PAGE_SIZE; u64 limit = bufsize; u64 head, tail, wakeup; + u32 count = 0;
head = PERF_IDX2OFF(handle->head, buf);
@@ -453,10 +536,12 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) wakeup = PERF_IDX2OFF(handle->wakeup, buf);
/* - * Lets calculate the buffer area which TRBE could write into. There - * are three possible scenarios here. Limit needs to be aligned with - * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the - * unconsumed data. + * Calculate the buffer limit. There are three possible scenarios. + * The limit needs to be aligned with PAGE_SIZE per the TRBE + * requirement. Always avoid clobbering the unconsumed data. + * + * How tracing proceeds after the limit is reached (e.g. wrapping + * or stopping) depends on the subsequent buffer management. * * 1) head < tail * @@ -466,9 +551,8 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) * +----|-----------------------|-------+ * trbe_base limit trbe_base + nr_pages * - * TRBE could write into [head..tail] area. Unless the tail is right at - * the end of the buffer, neither an wrap around nor an IRQ is expected - * while being enabled. + * TRBE could write into [head..tail] area. Programming the limit at + * 'tail' prevents from overwriting into unconsumed data. * * 2) head == tail * @@ -500,20 +584,20 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) limit = round_down(tail, PAGE_SIZE);
/* - * Wakeup may be arbitrarily far into the future. If it's not in the - * current generation, either we'll wrap before hitting it, or it's - * in the past and has been handled already. + * The trigger count is used either to generate an early maintenance + * interrupt at the watermark or to guard unconsumed data before + * wrapping. + */ + count = trbe_normal_trigger_count(handle, head, wakeup, limit); + + /* + * A zero count value indicates that no trigger should be programmed, + * so notification relies on the programmed limit. * * If there's a wakeup before we wrap, arrange to be woken up by the * page boundary following it. Keep the tail boundary if that's lower. - * - * head wakeup tail - * +----|---------------|-------|-------+ - * |$$$$|###############|%%%%%%%|$$$$$$$| - * +----|---------------|-------|-------+ - * trbe_base limit trbe_base + nr_pages */ - if ((handle->wakeup - handle->head) < handle->size && head <= wakeup) + if (!count && trbe_wakeup_before_tail(handle, head, wakeup)) limit = min(limit, round_up(wakeup, PAGE_SIZE));
/* @@ -532,8 +616,12 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) * +------------|------|--------|-------+ * trbe_base trbe_base + nr_pages */ - if (limit > head) + if (limit > head) { + next->head = head; + next->limit = limit; + next->count = count; return limit; + }
trbe_pad_buf(handle, handle->size); return 0; @@ -542,8 +630,10 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) static int trbe_normal_offset(struct perf_output_handle *handle) { struct trbe_buf *buf = etm_perf_sink_config(handle); - u64 limit = __trbe_normal_offset(handle); - u64 head = PERF_IDX2OFF(handle->head, buf); + struct trbe_buf_next next = { }; + u64 limit; + + limit = __trbe_normal_offset(handle, &next);
/* * If the head is too close to the limit and we don't @@ -553,16 +643,16 @@ static int trbe_normal_offset(struct perf_output_handle *handle) * We might have to do this more than once to make sure * we have enough required space. */ - while (limit && ((limit - head) < trbe_min_trace_buf_size(handle))) { - trbe_pad_buf(handle, limit - head); - limit = __trbe_normal_offset(handle); - head = PERF_IDX2OFF(handle->head, buf); + while (limit && next.limit - next.head < trbe_min_trace_buf_size(handle)) { + trbe_pad_buf(handle, next.limit - next.head); + limit = __trbe_normal_offset(handle, &next); }
if (!limit) return -ENOSPC;
- buf->trbe_limit = buf->trbe_base + limit; + buf->trbe_limit = buf->trbe_base + next.limit; + buf->trbe_trg_count = next.count; return 0; }
@@ -607,6 +697,7 @@ static void set_trbe_limit_pointer_enabled(struct trbe_buf *buf) { u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); unsigned long addr = buf->trbe_limit; + u64 to_limit = buf->trbe_limit - buf->trbe_write;
WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_EL1_LIMIT_SHIFT))); WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); @@ -616,24 +707,32 @@ static void set_trbe_limit_pointer_enabled(struct trbe_buf *buf) trblimitr &= ~TRBLIMITR_EL1_TM_MASK; trblimitr &= ~TRBLIMITR_EL1_LIMIT_MASK;
- /* - * Fill trace buffer mode is used here while configuring the - * TRBE for trace capture. In this particular mode, the trace - * collection is stopped and a maintenance interrupt is raised - * when the current write pointer wraps. This pause in trace - * collection gives the software an opportunity to capture the - * trace data in the interrupt handler, before reconfiguring - * the TRBE. - */ - trblimitr |= (TRBLIMITR_EL1_FM_FILL << TRBLIMITR_EL1_FM_SHIFT) & - TRBLIMITR_EL1_FM_MASK; + if (!buf->trbe_trg_count || buf->trbe_trg_count == to_limit) { + /* + * No separate trigger is needed. Use FILL mode so tracing stops + * and raises a maintenance interrupt when the write pointer + * reaches the limit. + */ + trblimitr |= TRBE_LIMIT_MODE(TRBLIMITR_EL1_FM_FILL, + TRBLIMITR_EL1_TM_IGNR); + } else if (buf->trbe_trg_count < to_limit) { + /* + * The trigger lies before the limit. Use FILL mode to preserve + * the limit as the hard stop, and enable trigger IRQs for + * early maintenance, such as watermark handling. + */ + trblimitr |= TRBE_LIMIT_MODE(TRBLIMITR_EL1_FM_FILL, + TRBLIMITR_EL1_TM_IRQ); + } else if (buf->trbe_trg_count > to_limit) { + /* + * The trigger lies beyond the limit. Use WRAP mode so tracing + * can continue past the limit, and stop on the trigger before + * it can overwrite unconsumed trace data. + */ + trblimitr |= TRBE_LIMIT_MODE(TRBLIMITR_EL1_FM_WRAP, + TRBLIMITR_EL1_TM_STOP); + }
- /* - * Trigger mode is not used here while configuring the TRBE for - * the trace capture. Hence just keep this in the ignore mode. - */ - trblimitr |= (TRBLIMITR_EL1_TM_IGNR << TRBLIMITR_EL1_TM_SHIFT) & - TRBLIMITR_EL1_TM_MASK; trblimitr |= (addr & PAGE_MASK); set_trbe_enabled(buf->cpudata, trblimitr); } @@ -645,6 +744,7 @@ static void trbe_enable_hw(struct trbe_buf *buf) WARN_ON(buf->trbe_write >= buf->trbe_limit); set_trbe_base_pointer(buf->trbe_hw_base); set_trbe_write_pointer(buf->trbe_write); + set_trbe_trigger_count(buf->trbe_trg_count);
/* * Synchronize all the register updates @@ -692,8 +792,7 @@ static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *hand case TRBSR_EL1_BSC_FILLED: break; case TRBSR_EL1_BSC_TRIGGER_EVENT: - err_str = "Unsupported trigger event"; - goto out_fatal; + break; case TRBSR_EL1_BSC_MANUAL_STOP: err_str = "Unsupported manual stop"; goto out_fatal; @@ -716,6 +815,9 @@ static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *hand if (is_trbe_wrap(trbsr)) return TRBE_FAULT_ACT_WRAP;
+ if (is_trbe_trg(trbsr)) + return TRBE_FAULT_ACT_TRIG; + return TRBE_FAULT_ACT_SPURIOUS;
out_fatal: @@ -1370,6 +1472,7 @@ static irqreturn_t arm_trbe_irq_handler(int irq, void *dev)
act = trbe_get_fault_act(handle, status); switch (act) { + case TRBE_FAULT_ACT_TRIG: case TRBE_FAULT_ACT_WRAP: truncated = !!trbe_handle_overflow(cpudata, act, !is_trbe_running(status)); diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h index 8ba9ef4056e4dc3968816c6a7cbf2859f2bbed36..fe84af94525f8c2dc40ac9c472bec5c1117a40b9 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.h +++ b/drivers/hwtracing/coresight/coresight-trbe.h @@ -19,6 +19,10 @@
#include "coresight-etm-perf.h"
+#define TRBE_LIMIT_MODE(fm, tm) \ + (FIELD_PREP(TRBLIMITR_EL1_FM_MASK, (fm)) | \ + FIELD_PREP(TRBLIMITR_EL1_TM_MASK, (tm))) + static inline bool is_trbe_available(void) { u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); @@ -104,6 +108,20 @@ static inline void set_trbe_write_pointer(unsigned long addr) write_sysreg_s(addr, SYS_TRBPTR_EL1); }
+static inline void set_trbe_trigger_count(unsigned long count) +{ + u64 trbsr; + + write_sysreg_s(count, SYS_TRBTRG_EL1); + + /* TRBSR_EL1.TRG has been cleared in clr_trbe_status() */ + if (!count) + return; + + trbsr = read_sysreg_s(SYS_TRBSR_EL1); + write_sysreg_s(trbsr | TRBSR_EL1_TRG, SYS_TRBSR_EL1); +} + static inline unsigned long get_trbe_limit_pointer(void) { u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1);
Arm ARM (ARM DDI 0487 M.c), rule R_XRRWP says:
| If a Trigger Event is generated when collection is not stopped and | the trigger mode is Stop on trigger, then all of the following occur: | The Trace Buffer Unit initiates a trace unit flush of the trace unit. | ...
A Stop-on-trigger event may initiates a trace flush. The TRBE might therefore continue writing trace data after the trigger event is generated. If the trigger is programmed exactly at the boundary of the free space, that flush can overwrite unconsumed trace data.
Reserve 4KiB of headroom when calculating the wrap-protection trigger count. If the available space is not large enough to provide that headroom, returns 0 for counter so effectively do not use it.
Subtract the headroom before rounding the count down to the hardware alignment, retaining the check that the resulting trigger lies beyond LIMIT. Rounding down preserves at least the requested headroom.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 95bbdbaca81bd59ac97b272216489c2f66393fa7..8187e264ff3c86d445b96c4935a8dbdf5ad9e77c 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -171,6 +171,21 @@ struct trbe_drvdata { struct platform_device *pdev; };
+/* + * Arm ARM (ARM DDI 0487 M.B), section D6.5.3 Trigger event, R_XRRWP states: + * + * "If a Trigger Event is generated when collection is not stopped and the + * trigger mode is Stop on trigger, then all of the following occur: + * - The Trace Buffer Unit initiates a trace unit flush of the trace unit. + * - The TRB_TRIG event is generated." + * + * The TRBE might therefore continue writing trace data after the trigger event + * is generated. When Stop-on-trigger is used to protect unconsumed data after + * buffer wrap, reserve 4KiB of headroom in the trigger count so it is safe to + * complete flush without overwriting that data. + */ +#define TRBE_TRIGGER_STOP_HEADROOM SZ_4K + /* Compute the buffer next state */ struct trbe_buf_next { u64 head; @@ -460,11 +475,15 @@ static u32 trbe_normal_trigger_count(struct perf_output_handle *handle, * limit * * |>>>>>>>> - * >>>>>>>>>>>>>>>>>>>>>| - * ` Trigger counter covers the writable region, rounded down - * to the hardware alignment. + * >>>>>>>>>>>>>>>>>|---| + * ` headroom + * + * Trigger counter covers the writable region, but leave headroom + * to avoid overwrite risk caused by trace unit flush after + * trigger event is generated. */ - count = round_down(handle->size, cpudata->trbe_hw_align); + count = round_down(handle->size - TRBE_TRIGGER_STOP_HEADROOM, + cpudata->trbe_hw_align); if (count <= limit - head) return 0;
Add TRBTRG_EL1 to the saved TRBE state and restore it along with the other TRBE registers.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/coresight-trbe.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 8187e264ff3c86d445b96c4935a8dbdf5ad9e77c..391e5013647477ae145d777d406c63355bc38278 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -127,12 +127,14 @@ static int trbe_errata_cpucaps[] = { * @trbbaser - Trace Buffer Base Register value * @trbptr - Trace Buffer Write Pointer Register value * @trbsr - Trace Buffer Status Register value + * @trbtrg - Trace Buffer Trigger Counter Register */ struct trbe_save_state { u64 trblimitr; u64 trbbaser; u64 trbptr; u64 trbsr; + u64 trbtrg; };
/* @@ -1537,6 +1539,7 @@ static int arm_trbe_save(struct coresight_device *csdev) state->trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); state->trbptr = read_sysreg_s(SYS_TRBPTR_EL1); state->trbsr = read_sysreg_s(SYS_TRBSR_EL1); + state->trbtrg = read_sysreg_s(SYS_TRBTRG_EL1); return 0; }
@@ -1548,6 +1551,7 @@ static void arm_trbe_restore(struct coresight_device *csdev) write_sysreg_s(state->trbbaser, SYS_TRBBASER_EL1); write_sysreg_s(state->trbptr, SYS_TRBPTR_EL1); write_sysreg_s(state->trbsr, SYS_TRBSR_EL1); + write_sysreg_s(state->trbtrg, SYS_TRBTRG_EL1);
if (!(state->trblimitr & TRBLIMITR_EL1_E)) { write_sysreg_s(state->trblimitr, SYS_TRBLIMITR_EL1);
For normal buffer mode, the TRBE computes both the programmed limit and the optional trigger count. This logic has several boundary cases around head/tail wrapping, wakeup placement, alignment padding, trigger-count headroom, and erratum-specific handling.
Add KUnit coverage for trbe_normal_offset(). The tests call the helper and verify the returned trbe_limit and trbe_trg_count fields.
Compile the tests into coresight-trbe.c so they can exercise private TRBE helpers without exposing data structures. Use the fixture trigger-support flag to cover both supported and unsupported trigger-count configurations.
Route all TRBE handle-to-buffer lookups through trbe_handle_buf(), which uses the perf sink configuration for genuine handles.
Use a test-scoped static stub to provide the synthetic buffer only for the handle under test and simulate padding for that handle. Genuine perf handles keep normal buffer lookup, padding and AUX accounting, including when an IRQ interrupts the test task. Compile the stub support only when the TRBE tests are enabled.
Signed-off-by: Leo Yan leo.yan@arm.com --- drivers/hwtracing/coresight/Kconfig | 15 + .../coresight/coresight-trbe-kunit-tests.c | 644 +++++++++++++++++++++ drivers/hwtracing/coresight/coresight-trbe.c | 58 +- 3 files changed, 705 insertions(+), 12 deletions(-)
diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig index 6a4239ebb582e95f0ebe8e9c8738a726f27f60a1..845c0691f0eba9f0cc9be22cdc6fc27ff82d437e 100644 --- a/drivers/hwtracing/coresight/Kconfig +++ b/drivers/hwtracing/coresight/Kconfig @@ -214,6 +214,21 @@ config CORESIGHT_TRBE To compile this driver as a module, choose M here: the module will be called coresight-trbe.
+config CORESIGHT_TRBE_KUNIT_TESTS + bool "Enable Coresight TRBE unit tests" if !KUNIT_ALL_TESTS + depends on KUNIT + depends on CORESIGHT_TRBE + depends on KUNIT=y if CORESIGHT_TRBE=y + default KUNIT_ALL_TESTS + help + Enable KUnit coverage for the Coresight TRBE buffer offset + calculations. The tests are compiled into the TRBE driver and + exercise private helpers through the in-source KUnit pattern. + This is only useful for development and not intended for production. + + These tests are not expected to run concurrently with perf tracing + sessions. + config ULTRASOC_SMB tristate "Ultrasoc system memory buffer drivers" depends on ACPI || COMPILE_TEST diff --git a/drivers/hwtracing/coresight/coresight-trbe-kunit-tests.c b/drivers/hwtracing/coresight/coresight-trbe-kunit-tests.c new file mode 100644 index 0000000000000000000000000000000000000000..a594881a10c4aa31258c14bd1240fb16e9619935 --- /dev/null +++ b/drivers/hwtracing/coresight/coresight-trbe-kunit-tests.c @@ -0,0 +1,644 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <kunit/test.h> +#include <linux/limits.h> + +struct trbe_test_context { + struct perf_output_handle *handle; + struct trbe_buf *buf; +}; + +static struct trbe_buf *trbe_test_handle_buf_stub(struct perf_output_handle *handle) +{ + struct trbe_test_context *ctx = kunit_get_current_test()->priv; + + /* An IRQ on the test task can still use a genuine perf handle. */ + return handle == ctx->handle ? ctx->buf : NULL; +} + +static void trbe_test_init(struct perf_output_handle *handle, + struct trbe_buf *buf, + struct trbe_cpudata *cpudata, + int nr_pages, u64 trbe_align, + u64 trbe_hw_align, + bool trig_is_supported) +{ + *handle = (struct perf_output_handle) { }; + *buf = (struct trbe_buf) { }; + *cpudata = (struct trbe_cpudata) { + .trbe_align = trbe_align, + .trbe_hw_align = trbe_hw_align, + .trig_is_supported = trig_is_supported, + }; + + buf->nr_pages = nr_pages; + buf->cpudata = cpudata; +} + +static void __trbe_test_assert_normal_offset(struct kunit *test, + struct perf_output_handle *handle, + struct trbe_buf *buf, + int expected_ret, + unsigned long expected_limit, + unsigned long expected_count, + unsigned int line) +{ + struct trbe_test_context ctx = { + .handle = handle, + .buf = buf, + }; + void *priv = test->priv; + unsigned long limit; + unsigned long count; + int ret; + + buf->trbe_limit = 0; + buf->trbe_trg_count = 0; + + test->priv = &ctx; + kunit_activate_static_stub(test, trbe_test_handle_buf, + trbe_test_handle_buf_stub); + ret = trbe_normal_offset(handle); + kunit_deactivate_static_stub(test, trbe_test_handle_buf); + test->priv = priv; + + limit = buf->trbe_limit; + count = buf->trbe_trg_count; + + KUNIT_ASSERT_EQ_MSG(test, ret, expected_ret, + "line %u: ret=%d expected=%d", + line, ret, expected_ret); + KUNIT_ASSERT_EQ_MSG(test, limit, expected_limit, + "line %u: limit=%#lx expected=%#lx", + line, limit, expected_limit); + KUNIT_ASSERT_EQ_MSG(test, count, expected_count, + "line %u: count=%#lx expected=%#lx", + line, count, expected_count); +} + +#define trbe_test_assert_normal_offset(...) \ + __trbe_test_assert_normal_offset(__VA_ARGS__, __LINE__) + +static void test_normal_offset_limit(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 1, false); + + /* + * ### : Free space, $$$ : Filled space + * + * |################|################| + * `head `wakeup + * `tail `limit + */ + handle.head = 0; + handle.size = SZ_1M; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M / 2, 0); + + /* + * |################|################| + * `head `wakeup `tail + * `limit + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M / 2, 0); + + /* + * |#################################| + * `head `tail + * `wakeup `limit + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); + + /* + * |#################################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = SZ_1M; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |$$$$$$$$$$$$$$$$|########|#######| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M * 3 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, 0); + + /* + * |$$$$$$$$|$$$$$$$|################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M * 1 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |$$$$$$$$$$$$$$$$|################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M - 1; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |#########|$$$$$$$$$$|########|###| + * `tail `head `wakeup + * `limit + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = handle.head + SZ_1M / 8; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 7 / 8, 0); + + /* + * |####|####|$$$$$$$$$$|############| + * `tail `head + * `wakeup + * `limit + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 8; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, 0); + + /* + * |#######|########|$$$$$$$$$$$$$$$$| + * `head `wakeup `>tail + * `limit + */ + handle.head = SZ_1M; + handle.wakeup = SZ_1M + SZ_1M / 8; + handle.size = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M / 8, 0); + + /* + * |#######|$$$$$$$$$$$$$$$$$|#######| + * `tail `head + * `wakeup + * `limit + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, 0); + + /* + * |#######|$$$$$$$$|$$$$$$$$|#######| + * `tail `wakeup `head + * `limit + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, 0); + + /* + * |$$$$$$$|########|########|$$$$$$$| + * `head `wakeup `tail + * `limit + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M / 2, 0); + + /* + * |$$$$$$$|#################|$$$$$$$| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M * 3 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, 0); + + /* + * |$$$$$$$|#################|$$$$$$$| + * `wakeup `head `tail + * `limit + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, 0); + + /* + * |$$$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$| + * `head + * `tail + */ + handle.head = SZ_1M / 4; + handle.size = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); + + /* + * |$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$|#$| + * `head + * `tail + */ + handle.head = SZ_1M - SZ_1K * 2; + handle.size = SZ_1K; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); +} + +static void test_normal_offset_limit_and_counter(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 1, true); + + /* + * ### : Free space, $$$ : Filled space + * + * |################|################| + * `head `wakeup `limit + * `tail + * `----- count ----' + */ + handle.head = 0; + handle.size = SZ_1M; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M, SZ_1M / 2); + + /* + * |################|################| + * `head `wakeup `tail + * `limit + * `----- count ----' + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, SZ_1M / 2); + + /* + * |#################################| + * `head `tail + * `wakeup `limit + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); + + /* + * |#################################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = 0; + handle.size = SZ_1M - 1; + handle.wakeup = SZ_1M; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |$$$$$$$$$$$$$$$$|########|#######| + * `head `tail + * `wakeup + * `limit + * [ count ] + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M * 3 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, SZ_1M / 4); + + /* + * |$$$$$$$$|$$$$$$$|################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M * 1 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |$$$$$$$$$$$$$$$$|################| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 2; + handle.size = SZ_1M / 2 - 1; + handle.wakeup = SZ_1M - 1; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M - PAGE_SIZE, 0); + + /* + * |#########|$$$$$$$$$$|########|###| + * `tail `head `wakeup + * `limit + * [ count ] + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = handle.head + SZ_1M / 8; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M, SZ_1M / 8); + + /* + * |####|####|$$$$$$$$$$|############| + * `tail `head + * `wakeup + * `limit + * [ count >>> + * >>> ] + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 8; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, + SZ_1M / 2 - TRBE_TRIGGER_STOP_HEADROOM); + + /* + * |#######|########|$$$$$$$$$$$$$$$$| + * `head `wakeup `>tail + * `limit + * [ count ] + */ + handle.head = SZ_1M; + handle.wakeup = SZ_1M + SZ_1M / 8; + handle.size = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M / 2, SZ_1M / 8); + + /* + * |#######|$$$$$$$$$$$$$$$$$|#######| + * `tail `head + * `wakeup + * `limit + * [ count > + * >>> ] + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, + SZ_1M / 2 - TRBE_TRIGGER_STOP_HEADROOM); + + /* + * |#######|$$$$$$$$|$$$$$$$$|#######| + * `tail `wakeup `head + * `limit + * [ count > + * >>> ] + */ + handle.head = SZ_1M * 3 / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M + SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, + SZ_1M / 2 - TRBE_TRIGGER_STOP_HEADROOM); + + /* + * |$$$$$$$|########|########|$$$$$$$| + * `head `wakeup `tail + * `limit + * [ count ] + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, SZ_1M / 4); + + /* + * |$$$$$$$|#################|$$$$$$$| + * `head `tail + * `wakeup + * `limit + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = SZ_1M * 3 / 4; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, 0); + + /* + * |$$$$$$$|#################|$$$$$$$| + * `wakeup `head `tail + * `limit + */ + handle.head = SZ_1M / 4; + handle.size = SZ_1M / 2; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M * 3 / 4, 0); +} + +static void test_normal_offset_alignment_padding(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 1, false); + + handle.head = 128; + handle.size = 2 * PAGE_SIZE; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + 2 * PAGE_SIZE, 0); + + handle.head = SZ_1M - PAGE_SIZE / 2; + handle.size = PAGE_SIZE / 4; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); +} + +static void test_normal_offset_small_range_padding(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 1, false); + + handle.head = 2 * PAGE_SIZE; + handle.size = PAGE_SIZE / 4; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); +} + +static void test_normal_offset_trigger_count_alignment(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 256, true); + + handle.head = 0; + handle.size = SZ_1M; + handle.wakeup = PAGE_SIZE + 1; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, SZ_1M, + PAGE_SIZE + 256); + + handle.head = 0; + handle.size = 2 * PAGE_SIZE; + handle.wakeup = 2 * PAGE_SIZE - 1; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + 2 * PAGE_SIZE, 0); +} + +static void test_normal_offset_trigger_count_over_u32(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + unsigned long wakeup = (unsigned long)U32_MAX + 1; + int nr_pages = (wakeup + PAGE_SIZE) / PAGE_SIZE; + + trbe_test_init(&handle, &buf, &cpudata, nr_pages, PAGE_SIZE, 1, true); + + handle.head = 0; + handle.size = (unsigned long)nr_pages * PAGE_SIZE; + handle.wakeup = wakeup; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, wakeup, 0); +} + +static void test_normal_offset_trigger_count_handle_wrap(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, + PAGE_SIZE, 1, true); + + handle.head = ULONG_MAX - PAGE_SIZE + 1; + handle.size = 2 * PAGE_SIZE; + handle.wakeup = handle.head + PAGE_SIZE / 2; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + SZ_1M, PAGE_SIZE / 2); +} + +static void test_normal_offset_min_trace_size(struct kunit *test) +{ + struct perf_output_handle handle; + struct trbe_buf buf; + struct trbe_cpudata cpudata; + unsigned long head = 2 * PAGE_SIZE - 32; + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, 1, 1, false); + + handle.head = head; + handle.size = PAGE_SIZE + 96; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, 0, + 3 * PAGE_SIZE, 0); + + trbe_test_init(&handle, &buf, &cpudata, SZ_1M / PAGE_SIZE, 1, 1, false); + set_bit(TRBE_WORKAROUND_WRITE_OUT_OF_RANGE, cpudata.errata); + + handle.head = head; + handle.size = PAGE_SIZE + 96; + handle.wakeup = 0; + + trbe_test_assert_normal_offset(test, &handle, &buf, -ENOSPC, 0, 0); +} + +static struct kunit_case coresight_trbe_testcases[] = { + KUNIT_CASE(test_normal_offset_limit), + KUNIT_CASE(test_normal_offset_limit_and_counter), + KUNIT_CASE(test_normal_offset_alignment_padding), + KUNIT_CASE(test_normal_offset_small_range_padding), + KUNIT_CASE(test_normal_offset_trigger_count_alignment), + KUNIT_CASE(test_normal_offset_trigger_count_over_u32), + KUNIT_CASE(test_normal_offset_trigger_count_handle_wrap), + KUNIT_CASE(test_normal_offset_min_trace_size), + {} +}; + +static struct kunit_suite coresight_trbe_test_suite = { + .name = "coresight_trbe_test_suite", + .test_cases = coresight_trbe_testcases, +}; + +kunit_test_suites(&coresight_trbe_test_suite); diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 391e5013647477ae145d777d406c63355bc38278..7cea3542ad0797435c9deac3b5017da9f30b884c 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -20,6 +20,9 @@ #include <linux/iopoll.h> #include <linux/kvm_host.h> #include <linux/vmalloc.h> +#ifdef CONFIG_CORESIGHT_TRBE_KUNIT_TESTS +#include <kunit/static_stub.h> +#endif
#include "coresight-etm4x.h" #include "coresight-self-hosted-trace.h" @@ -377,10 +380,37 @@ static void __trbe_pad_buf(struct trbe_buf *buf, u64 offset, int len) memset((void *)buf->trbe_base + offset, ETE_IGNORE_PACKET, len); }
+static struct trbe_buf *trbe_test_handle_buf(struct perf_output_handle *handle) +{ +#ifdef CONFIG_CORESIGHT_TRBE_KUNIT_TESTS + KUNIT_STATIC_STUB_REDIRECT(trbe_test_handle_buf, handle); +#endif + return NULL; +} + +static struct trbe_buf *trbe_handle_buf(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = trbe_test_handle_buf(handle); + + if (buf) + return buf; + + return etm_perf_sink_config(handle); +} + static void trbe_pad_buf(struct perf_output_handle *handle, int len) { - struct trbe_buf *buf = etm_perf_sink_config(handle); - u64 head = PERF_IDX2OFF(handle->head, buf); + struct trbe_buf *buf; + u64 head; + + if (trbe_test_handle_buf(handle)) { + handle->head += len; + handle->size -= len; + return; + } + + buf = trbe_handle_buf(handle); + head = PERF_IDX2OFF(handle->head, buf);
__trbe_pad_buf(buf, head, len); if (!buf->snapshot) @@ -389,7 +419,7 @@ static void trbe_pad_buf(struct perf_output_handle *handle, int len)
static int trbe_snapshot_offset(struct perf_output_handle *handle) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle);
/* * The ETE trace has alignment synchronization packets allowing @@ -403,7 +433,7 @@ static int trbe_snapshot_offset(struct perf_output_handle *handle) static u64 trbe_min_trace_buf_size(struct perf_output_handle *handle) { u64 size = TRBE_TRACE_MIN_BUF_SIZE; - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); struct trbe_cpudata *cpudata = buf->cpudata;
/* @@ -443,7 +473,7 @@ static bool trbe_wakeup_before_tail(struct perf_output_handle *handle, static u32 trbe_normal_trigger_count(struct perf_output_handle *handle, u64 head, u64 wakeup, u64 limit) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); struct trbe_cpudata *cpudata = buf->cpudata; const u64 bufsize = buf->nr_pages * PAGE_SIZE; u64 count; @@ -507,7 +537,7 @@ static u32 trbe_normal_trigger_count(struct perf_output_handle *handle, static unsigned long __trbe_normal_offset(struct perf_output_handle *handle, struct trbe_buf_next *next) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); struct trbe_cpudata *cpudata = buf->cpudata; const u64 bufsize = buf->nr_pages * PAGE_SIZE; u64 limit = bufsize; @@ -650,7 +680,7 @@ static unsigned long __trbe_normal_offset(struct perf_output_handle *handle,
static int trbe_normal_offset(struct perf_output_handle *handle) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); struct trbe_buf_next next = { }; u64 limit;
@@ -679,7 +709,7 @@ static int trbe_normal_offset(struct perf_output_handle *handle)
static int trbe_compute_next(struct perf_output_handle *handle) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); int ret;
perf_aux_output_flag(handle, PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW); @@ -1193,7 +1223,7 @@ static int arm_trbe_enable(struct coresight_device *csdev, enum cs_mode mode, struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); struct perf_output_handle *handle = path->handle; - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); int ret;
WARN_ON(cpudata->cpu != smp_processor_id()); @@ -1333,7 +1363,7 @@ static int trbe_handle_overflow(struct trbe_cpudata *cpudata, { struct perf_output_handle *handle = cpudata->handle; struct perf_event *event = handle->event; - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); unsigned long size; struct etm_event_data *event_data; u64 trcprgctlr = 0; @@ -1381,7 +1411,7 @@ static int trbe_handle_overflow(struct trbe_cpudata *cpudata, * truncate the trace to stop the event. A subsequent normal restart * installs the buffer and its CPU association through sink enable. */ - buf = etm_perf_sink_config(handle); + buf = trbe_handle_buf(handle); if (!buf || buf->cpudata != cpudata) { ret = -EINVAL; goto truncate_event; @@ -1408,7 +1438,7 @@ static int trbe_handle_overflow(struct trbe_cpudata *cpudata,
static bool is_perf_trbe(struct perf_output_handle *handle) { - struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_buf *buf = trbe_handle_buf(handle); struct trbe_cpudata *cpudata; struct trbe_drvdata *drvdata; int cpu = smp_processor_id(); @@ -1988,3 +2018,7 @@ module_exit(arm_trbe_exit); MODULE_AUTHOR("Anshuman Khandual anshuman.khandual@arm.com"); MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); MODULE_LICENSE("GPL v2"); + +#ifdef CONFIG_CORESIGHT_TRBE_KUNIT_TESTS +#include "coresight-trbe-kunit-tests.c" +#endif
Resetting the decoder at every AUX fragment boundary can discard pending decode state even when the trace stream is continuous.
Store AUX flags when queuing indexed fragments. Preserve decoder state across their boundaries and request an explicit reset for the next block when the current fragment reports PERF_AUX_FLAG_COLLISION. Source restarts with in-band synchronization are handled by the decoder. Keep resetting between blocks queued without AUX records, where the flags are unavailable.
Keep setting and clearing the queue reset flag in cs_etm__get_trace(), where the preceding fragment is available. Clear a request after its nonempty block has been consumed and derive collision resets from that fragment. Preserve deferred requests through empty buffers. cs_etm__get_data_block() only performs the requested reset before processing the fetched block.
Keep flushing retained context-ID thread stacks at every data block boundary. AUX records do not distinguish buffer maintenance from normal event disable/re-enable. In-band restart packets synchronize the decoder, but only the current thread's stack is flushed when processing a discontinuity. Another thread may have changed its stack while tracing was disabled, and its first instruction sample after restart can otherwise contain stale return addresses before branch bookkeeping clears the stack.
Flush those stacks before decoding the new block. This conservatively limits callchains and branch history to each chunk while preserving decoder continuity. Improving stack continuity can be handled separately.
Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/auxtrace.h | 2 ++ tools/perf/util/cs-etm.c | 46 +++++++++++++++++++++++++++++++++------------- 2 files changed, 35 insertions(+), 13 deletions(-)
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h index 6947f3f284c029286837e4e1dd9f535a5f0b6334..6ea212802e418ebd6536e3ea5456bab7530eeb26 100644 --- a/tools/perf/util/auxtrace.h +++ b/tools/perf/util/auxtrace.h @@ -243,6 +243,7 @@ struct auxtrace { * @buffer_nr: used to number each buffer * @use_size: implementation actually only uses this number of bytes * @use_data: implementation actually only uses data starting at this address + * @aux_flags: AUX event flags */ struct auxtrace_buffer { struct list_head list; @@ -261,6 +262,7 @@ struct auxtrace_buffer { u64 buffer_nr; size_t use_size; void *use_data; + u64 aux_flags; };
/** diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 114b3cd2da495e54e78df8fbd707898312f95079..6f95730fe3b94ae3f534eee505132a2efbc4ff96 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -136,6 +136,9 @@ struct cs_etm_queue { */ struct intlist *own_traceid_list; u32 sink_id; + + /* Reset decoder before processing the fetched data block. */ + bool reset_decoder; };
static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm); @@ -1436,6 +1439,13 @@ cs_etm__get_trace(struct cs_etm_queue *etmq) struct auxtrace_buffer *old_buffer = aux_buffer; struct auxtrace_queue *queue;
+ /* + * Clear the reset request for the consumed block. Empty buffers have + * no decoded bytes, so preserve requests deferred through them. + */ + if (etmq->buf_used) + etmq->reset_decoder = false; + queue = &etmq->etm->queues.queue_array[etmq->queue_nr];
aux_buffer = auxtrace_buffer__next(queue, aux_buffer); @@ -1467,6 +1477,15 @@ cs_etm__get_trace(struct cs_etm_queue *etmq) etmq->buf_used = 0; etmq->buf_len = aux_buffer->size; etmq->buf = aux_buffer->data; + + /* + * A collision requires a reset after its fragment. Without AUX flags, + * reset between all blocks. + */ + if (old_buffer && (!etmq->etm->data_queued || + (old_buffer->aux_flags & PERF_AUX_FLAG_COLLISION))) + etmq->reset_decoder = true; + return 0; }
@@ -2162,17 +2181,15 @@ static int cs_etm__get_data_block(struct cs_etm_queue *etmq) if (!etmq->buf_len) return 0;
- /* - * We cannot assume consecutive blocks in the data file - * are contiguous, reset the decoder to force re-sync. - */ - ret = cs_etm_decoder__reset(etmq->decoder); - if (ret) - return ret; + if (etmq->reset_decoder) { + ret = cs_etm_decoder__reset(etmq->decoder); + if (ret) + return ret; + }
/* - * Since the decoder is reset, this causes a global trace - * discontinuity. Flush all thread stacks. + * AUX records do not distinguish buffer maintenance from a source + * restart. Flush all thread stacks before sampling the new block. */ cs_etm__flush_all_stack(etmq);
@@ -3154,9 +3171,8 @@ static u64 *cs_etm__create_meta_blk(u64 *buff_in, int *buff_in_offset, * file_offset. * * Normally, whole auxtrace buffers would be added to the queue. But we - * want to reset the decoder for every PERF_RECORD_AUX event, and the decoder - * is reset across each buffer, so splitting the buffers up in advance has - * the same effect. + * want to check every PERF_RECORD_AUX event to decide if need to reset + * decoder across each buffer, so splitting the buffers up in advance. */ static int cs_etm__queue_aux_fragment(struct perf_session *session, off_t file_offset, size_t sz, struct perf_record_aux *aux_event, struct perf_sample *sample) @@ -3168,6 +3184,7 @@ static int cs_etm__queue_aux_fragment(struct perf_session *session, off_t file_o union perf_event auxtrace_fragment; __u64 aux_offset, aux_size; enum cs_etm_format format; + struct auxtrace_buffer *buffer = NULL;
struct cs_etm_auxtrace *etm = container_of(session->auxtrace, struct cs_etm_auxtrace, @@ -3251,10 +3268,13 @@ static int cs_etm__queue_aux_fragment(struct perf_session *session, off_t file_o pr_debug3("CS ETM: Queue buffer size: %#"PRI_lx64" offset: %#"PRI_lx64 " tid: %d cpu: %d\n", aux_size, aux_offset, sample->tid, sample->cpu); err = auxtrace_queues__add_event(&etm->queues, session, &auxtrace_fragment, - file_offset, NULL); + file_offset, &buffer); if (err) return err;
+ if (buffer) + buffer->aux_flags = aux_event->flags; + format = (aux_event->flags & PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW) ? UNFORMATTED : FORMATTED; if (etmq->format != UNSET && format != etmq->format) {
In per-thread mode all AUX fragments for a thread are decoded by a single decoder, even if the task migrates between CPUs. If independent fragments come from different CPUs, the trace stream may not be contiguous for that decoder context.
Retain the CPU from each AUX sample in its queued fragment. Compare the preceding and fetched buffer CPUs in cs_etm__get_trace() and request a reset when either CPU is unknown or their CPU IDs differ. Indexed files can omit PERF_SAMPLE_CPU and hardware-ID records while remaining decodable through TID matching and the legacy metadata mapping. An unknown CPU therefore cannot establish trace continuity.
On 32-bit hosts, queuing a large AUX fragment can split it into several buffers while returning only the final piece. Save the queue tail before queuing and tag every newly added piece with the AUX sample CPU. Do this after queuing to preserve CPU filtering based on the original AUXTRACE record. Per-CPU fragments already match the sample CPU, so use the same assignment in both modes.
Skip CPU comparisons for internally split continuation buffers, which belong to the same AUX fragment. Keep AUX flags on the final piece so a collision applies after the whole fragment.
Check CPU metadata even for empty fragments. Preserve any reset request until the next nonempty block is decoded.
Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 6f95730fe3b94ae3f534eee505132a2efbc4ff96..8f04d7ceddd9d1f7ff22af234b19400efe27ba2b 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -1486,6 +1486,16 @@ cs_etm__get_trace(struct cs_etm_queue *etmq) (old_buffer->aux_flags & PERF_AUX_FLAG_COLLISION))) etmq->reset_decoder = true;
+ /* + * Unknown or different CPUs cannot establish per-thread continuity. + * Internally split buffers still belong to the same AUX fragment. + */ + if (etmq->etm->per_thread_decoding && old_buffer && + !aux_buffer->consecutive && + (old_buffer->cpu.cpu == -1 || aux_buffer->cpu.cpu == -1 || + old_buffer->cpu.cpu != aux_buffer->cpu.cpu)) + etmq->reset_decoder = true; + return 0; }
@@ -3251,10 +3261,15 @@ static int cs_etm__queue_aux_fragment(struct perf_session *session, off_t file_o if (aux_offset >= auxtrace_event->offset && aux_offset + aux_size <= auxtrace_event->offset + auxtrace_event->size) { struct cs_etm_queue *etmq = cs_etm__get_queue(etm, auxtrace_event->cpu); + struct auxtrace_buffer *pos; + struct list_head *head;
if (!etmq) return -EINVAL;
+ head = &etm->queues.queue_array[etmq->queue_nr].head; + pos = list_last_entry_or_null(head, struct auxtrace_buffer, list); + /* * If this AUX event was inside this buffer somewhere, create a new auxtrace event * based on the sizes of the aux event, and queue that fragment. @@ -3272,9 +3287,22 @@ static int cs_etm__queue_aux_fragment(struct perf_session *session, off_t file_o if (err) return err;
- if (buffer) + if (buffer) { buffer->aux_flags = aux_event->flags;
+ /* + * Preserve queue-time CPU filtering and tag every split + * piece added after the previous tail, up to the returned + * buffer, for migration detection. + */ + pos = list_prepare_entry(pos, head, list); + list_for_each_entry_continue(pos, head, list) { + pos->cpu.cpu = sample->cpu; + if (pos == buffer) + break; + } + } + format = (aux_event->flags & PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW) ? UNFORMATTED : FORMATTED; if (etmq->format != UNSET && format != etmq->format) {
Indexed AUX fragments do not necessarily form a contiguous trace stream. Snapshot reads can retain only the end of the AUX buffer and advance past the omitted prefix. AUX metadata records can also be lost when the normal perf ring buffer is full, leaving the corresponding trace unqueued. Neither case guarantees that the next queued fragment carries a collision flag or retains the source's restart synchronization packets.
Track the expected next logical AUX offset for each decoder queue in cs_etm__get_trace(). Request a decoder reset before processing a fragment whose offset differs from the expected end. This prevents decoding bytes after a gap using stale packet and execution state. The comparison also covers overlapping fragments and backward offsets.
Perf can internally split large buffers into consecutive pieces while retaining the original AUX offset. Preserve continuity across those pieces and accumulate their sizes when tracking the expected end. Keep the decoder byte index separate from the logical AUX offset. Track initialization separately, as zero is a valid expected offset after the counter wraps.
Zero-length fragments still carry logical offsets. Check them for gaps and retain any reset request until nonempty data is decoded.
Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 8f04d7ceddd9d1f7ff22af234b19400efe27ba2b..37ae7ff5149433ed27cb3c9b880e8422d742a74d 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -123,6 +123,9 @@ struct cs_etm_queue { u8 pending_timestamp_chan_id; enum cs_etm_format format; u64 offset; + /* Expected logical AUX offset after the last fetched buffer. */ + u64 next_aux_offset; + bool have_aux_offset; const unsigned char *buf; size_t buf_len, buf_used; /* Conversion between traceID and index in traceid_queues array */ @@ -1496,6 +1499,21 @@ cs_etm__get_trace(struct cs_etm_queue *etmq) old_buffer->cpu.cpu != aux_buffer->cpu.cpu)) etmq->reset_decoder = true;
+ /* + * Snapshot clipping or missing AUX records can leave a gap between + * fragments. Check logical AUX offsets, as etmq->offset only counts + * bytes submitted to the decoder. Internally split buffers retain + * the original offset and mark subsequent pieces as consecutive. + */ + if (etmq->have_aux_offset && !aux_buffer->consecutive && + aux_buffer->offset != etmq->next_aux_offset) + etmq->reset_decoder = true; + + if (!etmq->have_aux_offset || !aux_buffer->consecutive) + etmq->next_aux_offset = aux_buffer->offset; + etmq->next_aux_offset += aux_buffer->size; + etmq->have_aux_offset = true; + return 0; }
The kernel defaults to a watermark of half the AUX buffer. With small buffers, the reader can be notified too late to drain the trace before the available space is exhausted.
Use a quarter-buffer watermark for normal recording to wake the reader earlier and leave more space for trace generated while it handles the notification.
If the computed watermark is at most one page, set it to zero so the kernel selects its default. TRBE's write-out-of-range workaround requires PAGE_SIZE + 64 bytes per trace span and disables trigger counts. A one-page watermark can leave every span too small to enable tracing. The fallback retains a two-page watermark for a four-page AUX mapping.
Leave snapshot and AUX sampling behavior unchanged.
Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/arch/arm/util/cs-etm.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/tools/perf/arch/arm/util/cs-etm.c b/tools/perf/arch/arm/util/cs-etm.c index d2861d66a6612ea213066258672ba41769e578cb..2b29cd5fde4b8cc18e04d02308635d7b0c13ccf6 100644 --- a/tools/perf/arch/arm/util/cs-etm.c +++ b/tools/perf/arch/arm/util/cs-etm.c @@ -438,6 +438,21 @@ static int cs_etm_recording_options(struct auxtrace_record *itr, pr_debug2("%s snapshot size: %zu\n", CORESIGHT_ETM_PMU_NAME, opts->auxtrace_snapshot_size);
+ if (!opts->auxtrace_snapshot_mode && !opts->auxtrace_sample_mode) { + size_t aw = opts->auxtrace_mmap_pages * (size_t)page_size / 4; + u32 aux_watermark = aw > UINT_MAX ? UINT_MAX : aw; + + /* + * TRBE workarounds can require more than one page per trace + * span. If the watermark is too small, let the kernel select + * its default. + */ + if (aux_watermark <= page_size) + aux_watermark = 0; + + cs_etm_evsel->core.attr.aux_watermark = aux_watermark; + } + /* * To obtain the auxtrace buffer file descriptor, the auxtrace * event must come first.
-
Leo Yan