Add detailed documentation for Coresight panic kdump, which contains the idea for why need Coresight panic kdump and introduce the implementation of Coresight panic kdump framework; the last section gives an example for how to use it.
Suggested-by: Mathieu Poirier mathieu.poirier@linaro.org Signed-off-by: Leo Yan leo.yan@linaro.org --- .../trace/coresight/coresight-panic-kdump.txt | 99 ++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 100 insertions(+) create mode 100644 Documentation/trace/coresight/coresight-panic-kdump.txt
diff --git a/Documentation/trace/coresight/coresight-panic-kdump.txt b/Documentation/trace/coresight/coresight-panic-kdump.txt new file mode 100644 index 0000000..3fca77c --- /dev/null +++ b/Documentation/trace/coresight/coresight-panic-kdump.txt @@ -0,0 +1,99 @@ + Coresight Panic Kdump + ===================== + + Author: Leo Yan leo.yan@linaro.org + Date: September 11th, 2018 + +Introduction +------------ + +Coresight trace data is useful for postmortem debugging, the trace data can be +from different tracers: it can be from Embedded Trace Macrocell (ETM) or Program +Flow Trace Macrocell (PTM), which are mainly for program flow related trace; it +also can be from System Trace Macrocell (STM) for system level debugging info +generated by software. + +Coresight has different sinks for trace data to be routed, e.g. Embedded Trace +Buffer (ETB) is one kind of sink which provides on-chip storage of trace data, +usually uses SRAM as the buffer with several KBs size; if the SoC designs to +support 'Local ETF' (ARM DDI 0461B, chapter 1.2.7), every CPU has one local ETB +buffer so the per CPU trace data can avoid being overwritten by each other. +Trace Memory Controller (TMC) is another kind sink designed as a successor to +the CoreSight ETB to capture trace into DRAM. + +After Linux kernel panic has occurred, the trace data keeps the last execution +flow before issue occurs; thus we could consider the trace data is useful for +postmortem debugging, especially after saving trace data into DRAM and rely on +kdump to preserve them into vmcore file; at the end, we can retrieve trace data +from vmcore file and "offline" to analyze the execution flow. + + +Implementation +-------------- + +To support Coresight kdump functionality, when Coresight device acts as a PMU +device and works with perf framework, perf provides a ring buffer for storing +trace data and the ring buffer is accessed by user space with mmap() for high +efficiency data transferring; when the panic happens, Coresight kdump callback +function reuses the Coresight PMU event stop function to stop tracer and flush +trace data from hardware buffer (e.g. ETB RAM) to perf ring buffer, thus we can +save the trace data around the panic happening as possible. The perf ring +buffer is kept in DRAM and finally they can be stored into vmcore file by the +dump-capture kernel of kdump. + +When wrote this doc, the purpose is to keep Coresight kdump as simple as +possible in kernel and leave the complexity in crash tool to extract related +info. To delivery the correct tracer configuration info for decoder, ETM +driver saves related configuration info in the driver data structure and config +data structure when enable the hardware component. In the crash tool, by +accessing global symbol "csdev_src", it can get to know the handles for device +and driver data structures; thus it can extract tracer configurations from them. + +The perf ring buffer can be retrieved by accessing global symbol "ctx_handle" +which points to per CPU structure 'perf_output_handle', this variable can be +used to read ring buffer handler from 'perf_output_handle->rb' item. The ring +buffer handler points to ring buffer structure which contains buffer related +management information; at the end we can easily dump trace data from crash tool +according to information provided by the ring buffer structure. + + +Usage +----- + +Before use CoreSight kdump functionality, we need to build Linux kernel with +enabling 'CONFIG_CORESIGHT_KDUMP' configuration. + +After system booting up, firstly need to prepare dump-capture kernel, this can +refer to document [1] chapter 'Load the Dump-capture Kernel' for detailed steps. +For enabling the Coresight tracer with perf tool, please refer to document [2] +chapter 'How to use the tracer modules' for detailed steps. + +After kernel panic occurs, Coresight kdump callback function will be invoked +for every CPU to save trace data and kdump launches the dump-capture kernel, we +can utilize the dump-capture kernel to save kernel dump file, this can refer +to document [1] chapter 'Write Out the Dump File'. Thus all Coresight trace +data will be stored into kernel dump file. + +After get kernel dump file, we can use 'crash' tool + extension program +arm_cs_dump.so to extract trace data and decode trace data with OpenCSD [3]. + +Below is an example for post analysis for etmv4 tracer with crash extension +program; in the output folder, if every CPU has own perf ring buffer, the +extension program generates trace data file cstrace.X.bin for each CPU X. If +all CPUs share the same one ring buffer, it only generates file cstrace.0.bin. + + $ ./crash vmcore vmlinux + crash> extend arm_cs_dump.so + crash> arm_cs_dump -o etm_out + + $ ls etm_out + etm_out/ + ├── cstrace.0.bin + └── cstrace.0.log + +Finally the decoding info is displayed on the screen and also is saved into log +file cstrace.X.log (X is CPU number). + +[1] Documentation/kdump/kdump.txt +[2] Documentation/trace/coresight/coresight.txt +[3] https://github.com/Linaro/OpenCSD diff --git a/MAINTAINERS b/MAINTAINERS index 815cb90..031bdfd 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1371,6 +1371,7 @@ S: Maintained F: drivers/hwtracing/coresight/* F: Documentation/trace/coresight/coresight.txt F: Documentation/trace/coresight/coresight-cpu-debug.txt +F: Documentation/trace/coresight/coresight-panic-kdump.txt F: Documentation/devicetree/bindings/arm/coresight.txt F: Documentation/devicetree/bindings/arm/coresight-cpu-debug.txt F: Documentation/ABI/testing/sysfs-bus-coresight-devices-*