Add detailed documentation for Coresight panic kdump, which contains the idea for why need Coresight panic kdump and introduce the implementation of Coresight panic kdump framework; the last section gives an example for how to use it.

Suggested-by: Mathieu Poirier mathieu.poirier@linaro.org Signed-off-by: Leo Yan leo.yan@linaro.org --- .../trace/coresight/coresight-panic-kdump.txt | 99 ++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 100 insertions(+) create mode 100644 Documentation/trace/coresight/coresight-panic-kdump.txt

diff --git a/Documentation/trace/coresight/coresight-panic-kdump.txt b/Documentation/trace/coresight/coresight-panic-kdump.txt new file mode 100644 index 0000000..3fca77c --- /dev/null +++ b/Documentation/trace/coresight/coresight-panic-kdump.txt @@ -0,0 +1,99 @@ + Coresight Panic Kdump + ===================== + + Author: Leo Yan leo.yan@linaro.org + Date: September 11th, 2018 + +Introduction +------------ + +Coresight trace data is useful for postmortem debugging, the trace data can be +from different tracers: it can be from Embedded Trace Macrocell (ETM) or Program +Flow Trace Macrocell (PTM), which are mainly for program flow related trace; it +also can be from System Trace Macrocell (STM) for system level debugging info +generated by software. + +Coresight has different sinks for trace data to be routed, e.g. Embedded Trace +Buffer (ETB) is one kind of sink which provides on-chip storage of trace data, +usually uses SRAM as the buffer with several KBs size; if the SoC designs to +support 'Local ETF' (ARM DDI 0461B, chapter 1.2.7), every CPU has one local ETB +buffer so the per CPU trace data can avoid being overwritten by each other. +Trace Memory Controller (TMC) is another kind sink designed as a successor to +the CoreSight ETB to capture trace into DRAM. + +After Linux kernel panic has occurred, the trace data keeps the last execution +flow before issue occurs; thus we could consider the trace data is useful for +postmortem debugging, especially after saving trace data into DRAM and rely on +kdump to preserve them into vmcore file; at the end, we can retrieve trace data +from vmcore file and "offline" to analyze the execution flow. + + +Implementation +-------------- + +To support Coresight kdump functionality, when Coresight device acts as a PMU +device and works with perf framework, perf provides a ring buffer for storing +trace data and the ring buffer is accessed by user space with mmap() for high +efficiency data transferring; when the panic happens, Coresight kdump callback +function reuses the Coresight PMU event stop function to stop tracer and flush +trace data from hardware buffer (e.g. ETB RAM) to perf ring buffer, thus we can +save the trace data around the panic happening as possible. The perf ring +buffer is kept in DRAM and finally they can be stored into vmcore file by the +dump-capture kernel of kdump. + +When wrote this doc, the purpose is to keep Coresight kdump as simple as +possible in kernel and leave the complexity in crash tool to extract related +info. To delivery the correct tracer configuration info for decoder, ETM +driver saves related configuration info in the driver data structure and config +data structure when enable the hardware component. In the crash tool, by +accessing global symbol "csdev_src", it can get to know the handles for device +and driver data structures; thus it can extract tracer configurations from them. + +The perf ring buffer can be retrieved by accessing global symbol "ctx_handle" +which points to per CPU structure 'perf_output_handle', this variable can be +used to read ring buffer handler from 'perf_output_handle->rb' item. The ring +buffer handler points to ring buffer structure which contains buffer related +management information; at the end we can easily dump trace data from crash tool +according to information provided by the ring buffer structure. + + +Usage +----- + +Before use CoreSight kdump functionality, we need to build Linux kernel with +enabling 'CONFIG_CORESIGHT_KDUMP' configuration. + +After system booting up, firstly need to prepare dump-capture kernel, this can +refer to document [1] chapter 'Load the Dump-capture Kernel' for detailed steps. +For enabling the Coresight tracer with perf tool, please refer to document [2] +chapter 'How to use the tracer modules' for detailed steps. + +After kernel panic occurs, Coresight kdump callback function will be invoked +for every CPU to save trace data and kdump launches the dump-capture kernel, we +can utilize the dump-capture kernel to save kernel dump file, this can refer +to document [1] chapter 'Write Out the Dump File'. Thus all Coresight trace +data will be stored into kernel dump file. + +After get kernel dump file, we can use 'crash' tool + extension program +arm_cs_dump.so to extract trace data and decode trace data with OpenCSD [3]. + +Below is an example for post analysis for etmv4 tracer with crash extension +program; in the output folder, if every CPU has own perf ring buffer, the +extension program generates trace data file cstrace.X.bin for each CPU X. If +all CPUs share the same one ring buffer, it only generates file cstrace.0.bin. + + $ ./crash vmcore vmlinux + crash> extend arm_cs_dump.so + crash> arm_cs_dump -o etm_out + + $ ls etm_out + etm_out/ + ├── cstrace.0.bin + └── cstrace.0.log + +Finally the decoding info is displayed on the screen and also is saved into log +file cstrace.X.log (X is CPU number). + +[1] Documentation/kdump/kdump.txt +[2] Documentation/trace/coresight/coresight.txt +[3] https://github.com/Linaro/OpenCSD diff --git a/MAINTAINERS b/MAINTAINERS index 815cb90..031bdfd 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1371,6 +1371,7 @@ S: Maintained F: drivers/hwtracing/coresight/* F: Documentation/trace/coresight/coresight.txt F: Documentation/trace/coresight/coresight-cpu-debug.txt +F: Documentation/trace/coresight/coresight-panic-kdump.txt F: Documentation/devicetree/bindings/arm/coresight.txt F: Documentation/devicetree/bindings/arm/coresight-cpu-debug.txt F: Documentation/ABI/testing/sysfs-bus-coresight-devices-*

At every time the driver begins to write ring buffer, it will update the page index 'buf->cur' and offset value 'buf->offset'; but after update the ring buffer these two values will not be set. So these two values are mainly indicated the start point to fill ring buffer, but we have no chance to know where is the last writing point. This results in the post-mortem analysis tool (e.g. crash extension program) to read incomplete trace data from the dump.

This patch is to update CoreSight private structure to track the latest buffer state in 'buf->cur' and 'buf->offset', at the end this can delivery correct info of ring buffer for offline tool.

Signed-off-by: Leo Yan leo.yan@linaro.org --- drivers/hwtracing/coresight/coresight-tmc-etf.c | 8 ++++++++ 1 file changed, 8 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c index 6265370..4ea1cd5 100644 --- a/drivers/hwtracing/coresight/coresight-tmc-etf.c +++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c @@ -487,6 +487,14 @@ static unsigned long tmc_update_etf_buffer(struct coresight_device *csdev, to_read = to_read + barrier_sz; }

+ /* + * Though tmc_set_etf_buffer() will update the page index and offset + * for next data transferring, here update them to serve kernel panic + * emergency thus kdump can know the buffer last writing point. + */ + buf->cur = cur; + buf->offset = offset; + CS_LOCK(drvdata->base);

return to_read;

This patch invokes function coresight_emergency_stop() to stop Arm CoreSight before kdump is started. This feature is needed to stop CoreSight tracers and dump trace data from sink to buffer, the buffer can be either the aux buffer in perf mode or small size buffer within driver data structure for SysFS mode.

The related buffer can be saved in the vmcore file by kdump, users can retrieve the trace data from buffer in the vmcore and use it for post mortem analysis and reverse the program flow before panic.

Signed-off-by: Leo Yan leo.yan@linaro.org --- arch/arm64/kernel/smp.c | 5 +++++ 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 25fcd22..823d49a 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -41,6 +41,7 @@ #include <linux/of.h> #include <linux/irq_work.h> #include <linux/kexec.h> +#include <linux/coresight.h>

#include <asm/alternative.h> #include <asm/atomic.h> @@ -830,6 +831,8 @@ static void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs) #ifdef CONFIG_KEXEC_CORE crash_save_cpu(regs, cpu);

+ coresight_emergency_stop(); + atomic_dec(&waiting_for_crash_ipi);

local_irq_disable(); @@ -973,6 +976,8 @@ void crash_smp_send_stop(void)

cpus_stopped = 1;

+ coresight_emergency_stop(); + if (num_online_cpus() == 1) { sdei_mask_local_cpu(); return;