On 2024-10-03 at 19:13:10, Suzuki K Poulose (suzuki.poulose@arm.com) wrote:
Hi Linu
On 16/09/2024 11:34, Linu Cherian wrote:
Add documentation on using coresight during panic and watchdog.
Thank you so much for the documentation, this will be quite useful !
Some minor comments below.
Signed-off-by: Linu Cherian lcherian@marvell.com
Changelog from v9: This patch has been newly introduced.
Documentation/trace/coresight/panic.rst | 356 ++++++++++++++++++++++++ 1 file changed, 356 insertions(+) create mode 100644 Documentation/trace/coresight/panic.rst
diff --git a/Documentation/trace/coresight/panic.rst b/Documentation/trace/coresight/panic.rst new file mode 100644 index 000000000000..3b53d91cace8 --- /dev/null +++ b/Documentation/trace/coresight/panic.rst @@ -0,0 +1,356 @@ +=================================================== +Using Coresight for Kernel panic and Watchdog reset +===================================================
+Introduction +------------ +This documentation is about using Linux coresight trace support to +debug kernel panic and watchdog reset scenarios.
+Coresight trace during Kernel panic +----------------------------------- +From the coresight driver point of view, addressing the kernel panic +situation has four main requirements.
+a. Support for allocation of trace buffer pages from reserved memory area.
- Platform can advertise this using a new device tree property added to
- relevant coresight nodes.
+b. Support for stopping coresight blocks at the time of panic
+c. Saving required metadata in the specified format
+d. Support for reading trace data captured at the time of panic
+Allocation of trace buffer pages from reserved RAM +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A new optional device tree property "memory-region" is added to the +ETR/ETF device nodes, that would give the base address and size of trace
ETR/ETF => CoreSight TMC
Ack.
+buffer.
+Static allocation of trace buffers would ensure that both IOMMU enabled +and disabled cases are handled. Also, platforms that support persistent +RAM will allow users to read trace data in the subsequent boot without +booting the crashdump kernel.
+Note: +For ETR sink devices, this reserved region will be used for both trace +capture and trace data retrieval. +For ETF sink devices, internal SRAM would be used for trace capture, +and they would be synced to reserved region for retrieval.
+Disabling coresight blocks at the time of panic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In order to avoid the situation of losing relevant trace data after a +kernel panic, it would be desirable to stop the coresight blocks at the +time of panic.
+This can be achieved by configuring the comparator, CTI and sink +devices as below::
Trigger on panic- Comparator --->External out --->CTI -->External In---->ETR/ETF stop
+Saving metadata at the time of kernel panic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Coresight metadata involves all additional data that are required for a +successful trace decode in addition to the trace data. This involves +ETR/ETF, ETE register snapshot etc.
s/ETE/ETB ?
Ack.
+A new optional device property "memory-region" is added to +the ETR/ETF/ETE device nodes for this.
same here ^^
Ack.
+Reading trace data captured at the time of panic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Trace data captured at the time of panic, can be read from rebooted kernel +or from crashdump kernel using a special device file /dev/crash_tmc_xxx. +This device file is created only when there is a valid crashdata available.
+General flow of trace capture and decode incase of kernel panic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1. Enable source and sink on all the cores using the sysfs interface.
- ETR sinks should have trace buffers allocated from reserved memory,
- by selecting "resrv" buffer mode from sysfs.
+2. Run relevant tests.
+3. On a kernel panic, all coresight blocks are disabled, necessary
- metadata is synced by kernel panic handler.
- System would eventually reboot or boot a crashdump kernel.
+4. For platforms that supports crashdump kernel, raw trace data can be
- dumped using the coresight sysfs interface from the crashdump kernel
- itself. Persistent RAM is not a requirement in this case.
+5. For platforms that supports persistent RAM, trace data can be dumped
- using the coresight sysfs interface in the subsequent Linux boot.
- Crashdump kernel is not a requirement in this case. Persistent RAM
- ensures that trace data is intact across reboot.
+Coresight trace during Watchdog reset +------------------------------------- +The main difference between addressing the watchdog reset and kernel panic +case are below,
+a. Saving coresight metadata need to be taken care by the
- SCP(system control processor) firmware in the specified format,
- instead of kernel.
+b. Reserved memory region given by firmware for trace buffer and metadata
- has to be in persistent RAM.
- Note: This is a requirement for watchdog reset case but optional
- in kernel panic case.
+Watchdog reset can be supported only on platforms that meet the above +two requirements.
+Sample commands for testing a Kernel panic case with ETR sink +-------------------------------------------------------------
+1. Boot Linux kernel with "crash_kexec_post_notifiers" added to the kernel
- bootargs. This is mandatory if the user would like to read the tracedata
- from the crashdump kernel.
+2. Enable the preloaded ETM configuration
- #echo 1 > /sys/kernel/config/cs-syscfg/configurations/panicstop/enable
+3. Configure CTI using sysfs interface::
- #./cti_setup.sh
- #cat cti_setup.sh
- cd /sys/bus/coresight/devices/
- ap_cti_config () {
#ETM trig out[0] trigger to Channel 0echo 0 4 > channels/trigin_attach- }
- etf_cti_config () {
#ETF Flush in trigger from Channel 0echo 0 1 > channels/trigout_attachecho 1 > channels/trig_filter_enable- }
- etr_cti_config () {
#ETR Flush in from Channel 0echo 0 1 > channels/trigout_attachecho 1 > channels/trig_filter_enable- }
- ctidevs=`find . -name "cti*"`
- for i in $ctidevs
- do
cd $iconnection=`find . -name "ete*"`minor nit: this could be ete or etm
Ack.
if [ ! -z "$connection" ]thenecho "AP CTI config for $i"ap_cti_configficonnection=`find . -name "tmc_etf*"`if [ ! -z "$connection" ]thenecho "ETF CTI config for $i"etf_cti_configficonnection=`find . -name "tmc_etr*"`if [ ! -z "$connection" ]thenecho "ETR CTI config for $i"etr_cti_configficd ..- done
+Note: CTI connections are SOC specific and hence the above script is +added just for reference.
+4. Choose reserved buffer mode for ETR buffer
- #echo "resrv" > /sys/bus/coresight/devices/tmc_etr0/buf_mode_preferred
+5. Enable stop on flush trigger configuration
- #echo 1 > /sys/bus/coresight/devices/tmc_etr0/stop_on_flush
+6. Start Coresight tracing on cores 1 and 2 using sysfs interface
+7. Run some application on core 1
- #taskset -c 1 dd if=/dev/urandom of=/dev/null &
+8. Invoke kernel panic on core 2
- #echo 1 > /proc/sys/kernel/panic
- #taskset -c 2 echo c > /proc/sysrq-trigger
+9. From rebooted kernel or crashdump kernel, read crashdata
- #dd if=/dev/crash_tmc_etr0 of=/trace/cstrace.bin
+10. Run opencsd decoder tools/scripts to generate the instruction trace.
+Sample instruction trace dump +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Core1 dump::
- A etm4_enable_hw: ffff800008ae1dd4
- CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
- I etm4_enable_hw: ffff800008ae1dd4:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1dd8:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1ddc:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de0:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de4:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de8:
- d503233f paciasp
- I etm4_enable_hw: ffff800008ae1dec:
- a9be7bfd stp x29, x30, [sp, #-32]!
- I etm4_enable_hw: ffff800008ae1df0:
- 910003fd mov x29, sp
- I etm4_enable_hw: ffff800008ae1df4:
- a90153f3 stp x19, x20, [sp, #16]
- I etm4_enable_hw: ffff800008ae1df8:
- 2a0003f4 mov w20, w0
- I etm4_enable_hw: ffff800008ae1dfc:
- 900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
- I etm4_enable_hw: ffff800008ae1e00:
- 910f4273 add x19, x19, #0x3d0
- I etm4_enable_hw: ffff800008ae1e04:
- f8747a60 ldr x0, [x19, x20, lsl #3]
- E etm4_enable_hw: ffff800008ae1e08:
- b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
- I 149.039572921 etm4_enable_hw: ffff800008ae1e30:
- a94153f3 ldp x19, x20, [sp, #16]
- I 149.039572921 etm4_enable_hw: ffff800008ae1e34:
- 52800000 mov w0, #0x0 // #0
- I 149.039572921 etm4_enable_hw: ffff800008ae1e38:
- a8c27bfd ldp x29, x30, [sp], #32
- ..snip
149.052324811 chacha_block_generic: ffff800008642d80:- 9100a3e0 add x0,
- I 149.052324811 chacha_block_generic: ffff800008642d84:
- b86178a2 ldr w2, [x5, x1, lsl #2]
- I 149.052324811 chacha_block_generic: ffff800008642d88:
- 8b010803 add x3, x0, x1, lsl #2
- I 149.052324811 chacha_block_generic: ffff800008642d8c:
- b85fc063 ldur w3, [x3, #-4]
- I 149.052324811 chacha_block_generic: ffff800008642d90:
- 0b030042 add w2, w2, w3
- I 149.052324811 chacha_block_generic: ffff800008642d94:
- b8217882 str w2, [x4, x1, lsl #2]
- I 149.052324811 chacha_block_generic: ffff800008642d98:
- 91000421 add x1, x1, #0x1
- I 149.052324811 chacha_block_generic: ffff800008642d9c:
- f100443f cmp x1, #0x11
+Core 2 dump::
- A etm4_enable_hw: ffff800008ae1dd4
- CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
- I etm4_enable_hw: ffff800008ae1dd4:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1dd8:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1ddc:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de0:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de4:
- d503201f nop
- I etm4_enable_hw: ffff800008ae1de8:
- d503233f paciasp
- I etm4_enable_hw: ffff800008ae1dec:
- a9be7bfd stp x29, x30, [sp, #-32]!
- I etm4_enable_hw: ffff800008ae1df0:
- 910003fd mov x29, sp
- I etm4_enable_hw: ffff800008ae1df4:
- a90153f3 stp x19, x20, [sp, #16]
- I etm4_enable_hw: ffff800008ae1df8:
- 2a0003f4 mov w20, w0
- I etm4_enable_hw: ffff800008ae1dfc:
- 900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
- I etm4_enable_hw: ffff800008ae1e00:
- 910f4273 add x19, x19, #0x3d0
- I etm4_enable_hw: ffff800008ae1e04:
- f8747a60 ldr x0, [x19, x20, lsl #3]
- E etm4_enable_hw: ffff800008ae1e08:
- b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
- I 149.046243445 etm4_enable_hw: ffff800008ae1e30:
- a94153f3 ldp x19, x20, [sp, #16]
- I 149.046243445 etm4_enable_hw: ffff800008ae1e34:
- 52800000 mov w0, #0x0 // #0
- I 149.046243445 etm4_enable_hw: ffff800008ae1e38:
- a8c27bfd ldp x29, x30, [sp], #32
- I 149.046243445 etm4_enable_hw: ffff800008ae1e3c:
- d50323bf autiasp
- E 149.046243445 etm4_enable_hw: ffff800008ae1e40:
- d65f03c0 ret
- A ete_sysreg_write: ffff800008adfa18
- ..snip
- I 149.05422547 panic: ffff800008096300:
- a90363f7 stp x23, x24, [sp, #48]
- I 149.05422547 panic: ffff800008096304:
- 6b00003f cmp w1, w0
- I 149.05422547 panic: ffff800008096308:
- 3a411804 ccmn w0, #0x1, #0x4, ne // ne = any
- N 149.05422547 panic: ffff80000809630c:
- 540001e0 b.eq ffff800008096348 <panic+0xe0> // b.none
- I 149.05422547 panic: ffff800008096310:
- f90023f9 str x25, [sp, #64]
- E 149.05422547 panic: ffff800008096314:
- 97fe44ef bl ffff8000080276d0 <panic_smp_self_stop>
- A panic: ffff80000809634c
- I 149.05422547 panic: ffff80000809634c:
- 910102d5 add x21, x22, #0x40
- I 149.05422547 panic: ffff800008096350:
- 52800020 mov w0, #0x1 // #1
- E 149.05422547 panic: ffff800008096354:
- 94166b8b bl ffff800008631180 <bust_spinlocks>
- N 149.054225518 bust_spinlocks: ffff800008631180:
- 340000c0 cbz w0, ffff800008631198 <bust_spinlocks+0x18>
- I 149.054225518 bust_spinlocks: ffff800008631184:
- f000a321 adrp x1, ffff800009a98000 <pbufs.0+0xbb8>
- I 149.054225518 bust_spinlocks: ffff800008631188:
- b9405c20 ldr w0, [x1, #92]
- I 149.054225518 bust_spinlocks: ffff80000863118c:
- 11000400 add w0, w0, #0x1
- I 149.054225518 bust_spinlocks: ffff800008631190:
- b9005c20 str w0, [x1, #92]
- E 149.054225518 bust_spinlocks: ffff800008631194:
- d65f03c0 ret
- A panic: ffff800008096358
+Perf based testing +------------------
+Starting perf session +~~~~~~~~~~~~~~~~~~~~~ +ETF: +perf record -e cs_etm/panicstop,@tmc_etf1/ -C 1 +perf record -e cs_etm/panicstop,@tmc_etf2/ -C 2
+ETR: +perf record -e cs_etm/panicstop,@tmc_etr0/ -C 1,2
+Reading trace data after panic +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Same sysfs based method explained above can be used to retrieve and +decode the trace data after the reboot on kernel panic.
Linu Cherian