Re: Regression with CTI driver on Arm Juno-r2 board

Hi Leo,

On Sat, 13 Aug 2022 at 12:51, Leo Yan leo.yan@linaro.org wrote:

Hi Mike & all,

[ + CoreSight Mailing List ]

At my side, I found the perf cs-etm regression. When I execute cs-etm test with below commands:

# perf test list ... 87: Check Arm CoreSight trace data recording and synthesized samples ... $ perf test -v 87

Then it's stuck and has no response. In the console, sometimes I can see the log:

[ 94.248480] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

Apparently, there have lockup and introduces CPU stuck. But I didn't narrow down furthermore if is a hardware lockup or software lockup.

After git bisect, it's narrowed down the regression is related to the commit:

e7676a00bc52b994960f89b118b0d548e4542372 is the first bad commit commit e7676a00bc52b994960f89b118b0d548e4542372 Author: Mike Leach mike.leach@linaro.org Date: Wed Apr 13 22:49:25 2022 +0100

arm64: dts: juno: add CTI entries to device tree

Add Coresight Cross Trigger Interface(CTI) entries to the device tree
for all the Juno variants.

Link: https://lore.kernel.org/r/20220413214925.30359-1-mike.leach@linaro.org
Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Signed-off-by: Mike Leach <mike.leach@linaro.org>
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>

I built Linux kernel with below commands:

$KERNEL=/kernel/path $OUT=/build/out/path

pushd $KERNEL make defconfig O=$OUT ./scripts/config --file $OUT/.config -e CONFIG_BPF_SYSCALL ./scripts/config --file $OUT/.config -e CONFIG_BPF_JIT_ALWAYS_ON ./scripts/config --file $OUT/.config -e CONFIG_TRACEPOINTS ./scripts/config --file $OUT/.config -e CONFIG_KPROBES ./scripts/config --file $OUT/.config -e CONFIG_UPROBES ./scripts/config --file $OUT/.config -e CONFIG_KRETPROBES ./scripts/config --file $OUT/.config -e CONFIG_PROC_KCORE ./scripts/config --file $OUT/.config -e CONFIG_NOP_TRACER ./scripts/config --file $OUT/.config -e CONFIG_TRACER_MAX_TRACE ./scripts/config --file $OUT/.config -e CONFIG_RING_BUFFER ./scripts/config --file $OUT/.config -e CONFIG_EVENT_TRACING ./scripts/config --file $OUT/.config -e CONFIG_CONTEXT_SWITCH_TRACER ./scripts/config --file $OUT/.config -e CONFIG_TRACING ./scripts/config --file $OUT/.config -e CONFIG_GENERIC_TRACER ./scripts/config --file $OUT/.config -e CONFIG_FTRACE ./scripts/config --file $OUT/.config -e CONFIG_FUNCTION_TRACER ./scripts/config --file $OUT/.config -e CONFIG_FUNCTION_GRAPH_TRACER ./scripts/config --file $OUT/.config -e CONFIG_SCHED_TRACER ./scripts/config --file $OUT/.config -e CONFIG_FTRACE_SYSCALLS ./scripts/config --file $OUT/.config -e CONFIG_TRACER_SNAPSHOT ./scripts/config --file $OUT/.config -e CONFIG_KPROBE_EVENTS ./scripts/config --file $OUT/.config -e CONFIG_UPROBE_EVENTS ./scripts/config --file $OUT/.config -e CONFIG_BPF_EVENTS ./scripts/config --file $OUT/.config -e CONFIG_DYNAMIC_EVENTS ./scripts/config --file $OUT/.config -e CONFIG_PROBE_EVENTS ./scripts/config --file $OUT/.config -e CONFIG_DYNAMIC_FTRACE ./scripts/config --file $OUT/.config -e CONFIG_PID_IN_CONTEXTIDR ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_LINKS_AND_SINKS ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_LINK_AND_SINK_TMC ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_CATU ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_SINK_TPIU ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_SINK_ETBV10 ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_SOURCE_ETM4X ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_STM ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_CPU_DEBUG ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_CTI ./scripts/config --file $OUT/.config -e CONFIG_CORESIGHT_CTI_INTEGRATION_REGS ./scripts/config --file $OUT/.config -e CONFIG_DEBUG_INFO ./scripts/config --file $OUT/.config -e CONFIG_DEBUG_INFO_DWARF5 ./scripts/config --file $OUT/.config -e CONFIG_DEBUG_INFO_BTF ./scripts/config --file $OUT/.config -d CONFIG_DEBUG_INFO_REDUCED

yes "" | make oldconfig O=$OUT

make -j `nproc` Image dtbs O=$OUT popd

When I have some free time, will come back to dig more for it. If have any suggestion, please let me know, I am glad to try and test at my side.

Thanks, Leo

I note that you do seem to be able to run Carsten's tests on the same system. The CTI devices will introduce new sysfs interconnections between the various devices on the system. Bear in mind that these are bi-directional - could it be that the arm_cs_iterate_devices() routine is getting stuck?

Regards

Mike