CoreSight May 2020

coresight@lists.linaro.org

20 participants
34 discussions

by Andrea Brunato

Good morning, Is tracing a multi-threaded program a supported use case for perf cs-etm? If yes, are there any flags that should be specified with perf? Thanks, Andrea IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

3 years, 8 months

[PATCH v5 0/9] perf cs-etm: Support thread stack and callchain

by Leo Yan

This patch series adds support for thread stack and callchain; this patch set depends on the instruction sample fix patch set [1]. This patch set get more complex, so before divide into small groups, I'd like to use this patch set version to include all relevant patches, hope this can give whole context for related code change. Briefly, this patch can be divided into three parts, which also can be reviewed separately for every part: Patches 01, 02 are used to fix samples for one corner case is for accessing the branch's target address and trigger an exception. Essentially, an extra branch sample is added to reflect this mediate branch between the previous branch and exception entry. Patches 03, 04, 05, 06 are coming from patch v4, which are used to support thread stack and callchain. Patches 07, 08, 09 are used to fixup for exception entry and exit. This is mainly used to fix two cases, one part is to fixup the thread stack and callchain for the case when access branch target address and trigger exception; another part is to fixup the thread stack for instruction emulation (and other single step cases). This patch set has been tested on Juno-r2 after applied on perf/core branch with latest commit 85fc95d75970 ("perf maps: Add missing unlock to maps__insert() error case"), and this patch set is also applied on top of instruction sample fix patch set [1]. Test for option '-F,+callindent': # perf script -F,+callindent main 3258 1 branches: main ffffad684d20 __libc_start_main+0xe0 (/usr/lib/aarch64-linux-gnu/libc-2.28.so) main 3258 1 branches: lib_loop_test@plt aaaae2c4d78c main+0x18 (/root/coresight_test/main) main 3258 1 branches: _dl_fixup ffffad811b4c _dl_runtime_resolve+0x40 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 1 branches: _dl_lookup_symbol_x ffffad80c078 _dl_fixup+0xb8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 1 branches: do_lookup_x ffffad80849c _dl_lookup_symbol_x+0x104 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 1 branches: check_match ffffad807bf0 do_lookup_x+0x238 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 1 branches: strcmp ffffad807888 check_match+0x70 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 1 branches: lib_loop_test@plt aaaae2c4d78c main+0x18 (/root/coresight_test/main) main 3258 1 branches: lib_loop_test@plt aaaae2c4d78c main+0x18 (/root/coresight_test/main) main 3258 1 branches: lib_loop_test@plt aaaae2c4d78c main+0x18 (/root/coresight_test/main) main 3258 1 branches: lib_loop_test@plt aaaae2c4d78c main+0x18 (/root/coresight_test/main) [...] Test for option '--itrace=g': # perf script --itrace=g16l64i100 main 3258 100 instructions: ffffad816a80 memcpy+0x70 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad809468 _dl_new_object+0xa8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad801840 dl_main+0x778 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad81384c _dl_sysdep_start+0x36c (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800884 _dl_start_final+0xac (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800b00 _dl_start+0x200 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800048 _start+0x8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 100 instructions: ffffad80952c _dl_new_object+0x16c (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad801840 dl_main+0x778 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad81384c _dl_sysdep_start+0x36c (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800884 _dl_start_final+0xac (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800b00 _dl_start+0x200 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800048 _start+0x8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 100 instructions: ffffad8018dc dl_main+0x814 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad81384c _dl_sysdep_start+0x36c (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800884 _dl_start_final+0xac (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800b00 _dl_start+0x200 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800048 _start+0x8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) main 3258 100 instructions: ffff8000100878d0 el0_sync_handler+0x168 ([kernel.kallsyms]) ffff800010082d00 el0_sync+0x140 ([kernel.kallsyms]) ffffad801910 dl_main+0x848 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad81384c _dl_sysdep_start+0x36c (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800884 _dl_start_final+0xac (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800b00 _dl_start+0x200 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) ffffad800048 _start+0x8 (/usr/lib/aarch64-linux-gnu/ld-2.28.so) [...] Changes from v4: * Addressed Mike's suggestion for performance improvement for function cs_etm__instr_addr() for quick calculation for non T32; * Removed the patch 'perf cs-etm: Synchronize instruction sample with the thread stack' (Mike); * Fixed the issue for exception is taken for branch target address accessing, for the branch sample and stack thread handling, the related patches are 01, 02, 07; * Fixed the stack thread handling for instruction emulation and single step with patches 08, 09. Changes from v3: * Split out separate patch set for instruction samples fixing. * Rebased on latest perf/core branch. Changes from v2: * Added patch 01 to fix the unsigned variable comparison to zero (Suzuki). * Refined commit logs. Changes from v1: * Added comments for task thread handling (Mathieu). * Split patch 02 into two patches, one is for support thread stack and another is for callchain support (Mathieu). * Added a new patch to support branch filter. [1] https://lkml.org/lkml/2020/2/18/1406 Leo Yan (9): perf cs-etm: Defer to assign exception sample flag perf cs-etm: Reflect branch prior to exception perf cs-etm: Refactor instruction size handling perf cs-etm: Support thread stack perf cs-etm: Support branch filter perf cs-etm: Support callchain for instruction sample perf cs-etm: Fixup exception entry for thread stack perf thread: Add helper to get top return address perf cs-etm: Fixup exception exit for thread stack .../perf/util/cs-etm-decoder/cs-etm-decoder.c | 1 + tools/perf/util/cs-etm.c | 290 ++++++++++++++++-- tools/perf/util/thread-stack.c | 10 + tools/perf/util/thread-stack.h | 1 + 4 files changed, 268 insertions(+), 34 deletions(-) -- 2.17.1

3 years, 8 months

Context id tracing with kernel at EL2

by Al Grant

Hi, this is an incomplete patch for an issue with EL2 kernels, and I'm looking for feedback on how to complete it. The background is that to support tracing multiple address spaces we get ETM to embed the context id in the trace, and we build with CONFIG_PID_IN_CONTEXTIDR to get the scheduler to put the thread id in CONTEXTIDR_EL1. This is a known technique, it's what context id tracing is designed for. The problem is when the kernel is running not at EL1 (OS level) but at EL2 (hypervisor level), which is now becoming common. With HCR_EL2.E2H set, the kernel's writes to CONTEXTIDR_EL1 actually change a different physical register, CONTEXTIDR_EL2. However, ETM still traces CONTEXTIDR_EL1. So the context ids in the trace are zero, and trace cannot be reconstructed. ETM 4.1 has an option VMIDOPT to cause CONTEXTIDR_EL2 to be output in trace, in the VMID field replacing the value of VTTBR.VMID. So we can use that, but the trace follower, collecting events from OpenCSD, needs to be aware it needs to check the VMID field not the CID field. OpenCSD doesn't need to change but perf does. TRCCONFIGR is already in the metadata, so perf consumers can check it to see what's going on. The patch below does the kernel and userspace side but is not complete. The problem is that userspace perf creates the metadata copy of TRCCONFIGR based on its request (and fills in the other id registers by reading sysfs), but the detection of EL2/E2H happens in the kernel which adjusts TRCCONFIGR, and it's this config which is needed for decode. I see three ways round this: - have userspace test to see if the kernel is EL2 (somehow) and adjust the metadata to mirror what the kernel is doing - have the kernel pass the adjusted TRCCONFIGR back so perf can put it in the metadata - have the perf decoder get the thread id from whichever of VMID and CONTEXTID is available in a PE_CONTEXT element Obviously, the last is simplest, but it's a bodge, and means that OpenCSD will see VMIDs when its TRCCONFIGR says it won't. It's kind of cleanest to get the real TRCCONFIGR somehow, but how do we do that? Al diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c index a128b5063f46..96488a0cfdcf 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x.c +++ b/drivers/hwtracing/coresight/coresight-etm4x.c @@ -353,8 +353,32 @@ static int etm4_parse_event_config(struct etmv4_drvdata *drvdata, } if (attr->config & BIT(ETM_OPT_CTXTID)) - /* bit[6], Context ID tracing bit */ - config->cfg |= BIT(ETM4_CFG_BIT_CTXTID); + { + /* + * Enable context-id tracing. The assumption is that this + * will work with CONFIG_PID_IN_CONTEXTIDR to trace process + * id changes and support decode of multiple processes. + * But ETM's context id trace traces physical CONTEXTIDR_EL1, + * while the logical CONTEXTIDR_EL1 that is written to on + * process switch is either physical CONTEXTIDR_EL1 or + * CONTEXTIDR_EL2 depending on HCR_EL2.E2H. On principle + * we should continue to use logical CONTEXTIDR_EL1. + * In order to trace physical CONTEXTIDR_EL2, we need to + * enable VMID tracing and use the VMIDOPT flag to trace + * CONTEXTIDR_EL2 rather than VTTBR.VMID in the VMID field. + * Trace decoders will need to inspect TRCCONFIGR and use + * either the CID or the VMID field from the trace packet. + */ + if (!(is_kernel_in_hyp_mode() && + (read_sysreg(hcr_el2) & BIT(34)) != 0)) { + /* bit[6], Context ID tracing bit */ + config->cfg |= BIT(ETM4_CFG_BIT_CTXTID); + } else { + /* bits[7,15], trace CONTEXTID_EL2 in VMID field */ + config->cfg |= (BIT(ETM4_CFG_BIT_VMID) | + BIT(ETM4_CFG_BIT_VMIDOPT)); + } + } /* return stack - enable if selected and supported */ if ((attr->config & BIT(ETM_OPT_RETSTK)) && drvdata->retstack) diff --git a/include/linux/coresight-pmu.h b/include/linux/coresight-pmu.h index b0e35eec6499..c2f47b25daab 100644 --- a/include/linux/coresight-pmu.h +++ b/include/linux/coresight-pmu.h @@ -19,8 +19,10 @@ /* ETMv4 CONFIGR programming bits for the ETM OPTs */ #define ETM4_CFG_BIT_CYCACC 4 #define ETM4_CFG_BIT_CTXTID 6 +#define ETM4_CFG_BIT_VMID 7 #define ETM4_CFG_BIT_TS 11 #define ETM4_CFG_BIT_RETSTK 12 +#define ETM4_CFG_BIT_VMIDOPT 15 static inline int coresight_get_trace_id(int cpu) { diff --git a/tools/include/linux/coresight-pmu.h b/tools/include/linux/coresight-pmu.h index b0e35eec6499..c2f47b25daab 100644 --- a/tools/include/linux/coresight-pmu.h +++ b/tools/include/linux/coresight-pmu.h @@ -19,8 +19,10 @@ /* ETMv4 CONFIGR programming bits for the ETM OPTs */ #define ETM4_CFG_BIT_CYCACC 4 #define ETM4_CFG_BIT_CTXTID 6 +#define ETM4_CFG_BIT_VMID 7 #define ETM4_CFG_BIT_TS 11 #define ETM4_CFG_BIT_RETSTK 12 +#define ETM4_CFG_BIT_VMIDOPT 15 static inline int coresight_get_trace_id(int cpu) { diff --git a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c index cd92a99eb89d..a54cad778841 100644 --- a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c +++ b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c @@ -35,6 +35,7 @@ struct cs_etm_decoder { dcd_tree_handle_t dcd_tree; cs_etm_mem_cb_type mem_access; ocsd_datapath_resp_t prev_return; + uint32 thread_id_in_vmid:1; }; static u32 @@ -496,17 +497,24 @@ cs_etm_decoder__buffer_exception_ret(struct cs_etm_packet_queue *queue, static ocsd_datapath_resp_t cs_etm_decoder__set_tid(struct cs_etm_queue *etmq, + struct cs_etm_decoder *decoder, struct cs_etm_packet_queue *packet_queue, const ocsd_generic_trace_elem *elem, const uint8_t trace_chan_id) { pid_t tid; - /* Ignore PE_CONTEXT packets that don't have a valid contextID */ - if (!elem->context.ctxt_id_valid) - return OCSD_RESP_CONT; + if (!decoder->thread_id_in_vmid) { + /* Ignore PE_CONTEXT packets that don't have a valid contextID */ + if (!elem->context.ctxt_id_valid) + return OCSD_RESP_CONT; + tid = elem->context.context_id; + } else { + if (!elem->context.vmid_valid) + return OCSD_RESP_CONT; + tid = elem->context.vmid; + } - tid = elem->context.context_id; if (cs_etm__etmq_set_tid(etmq, tid, trace_chan_id)) return OCSD_RESP_FATAL_SYS_ERR; @@ -561,7 +569,7 @@ static ocsd_datapath_resp_t cs_etm_decoder__gen_trace_elem_printer( trace_chan_id); break; case OCSD_GEN_TRC_ELEM_PE_CONTEXT: - resp = cs_etm_decoder__set_tid(etmq, packet_queue, + resp = cs_etm_decoder__set_tid(etmq, decoder, packet_queue, elem, trace_chan_id); break; case OCSD_GEN_TRC_ELEM_ADDR_NACC: @@ -595,11 +603,15 @@ static int cs_etm_decoder__create_etm_packet_decoder( OCSD_BUILTIN_DCD_ETMV3 : OCSD_BUILTIN_DCD_PTM; trace_config = &config_etmv3; + decoder->thread_id_in_vmid = 0; break; case CS_ETM_PROTO_ETMV4i: cs_etm_decoder__gen_etmv4_config(t_params, &trace_config_etmv4); decoder_name = OCSD_BUILTIN_DCD_ETMV4I; trace_config = &trace_config_etmv4; + /* If VMID and VMIDOPT are set, thread id is in VMID not CID */ + decoder->thread_id_in_vmid = + ((trace_config_etmv4.reg.configr & 0x8080) == 0x8080); break; default: return -1; IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

3 years, 9 months

[PATCH 0/2] arm64: dts: qcom: sc7180: Add support for ETMv4 PM and skipping power up.

by Sai Prakash Ranjan

Patch 1 is a resend of https://lore.kernel.org/patchwork/patch/1230367/ since that patch seems to have added the property to replicator node instead of etm7 node either when the patch was applied to the tree. Patch 2 adds support to skip powering up of ETM to workaround the hardware errata where CPU watchdog counter is stopped when power up bit is set. Depends on https://lore.kernel.org/patchwork/cover/1242100/ changes. Sai Prakash Ranjan (2): arm64: dts: qcom: sc7180: Support ETMv4 power management arm64: dts: qcom: sc7180: Add support to skip powering up of ETM arch/arm64/boot/dts/qcom/sc7180.dtsi | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

4 years, 1 month

[PATCH v4 0/1] perf: cs_etm: Allow no sink on command line.

by Mike Leach

Currently starting CoreSight tracing using perf requires a sink selection on the command line:- sudo ./perf record -e cs_etm/@tmc_etr0/ --per-thread uname -a Not providing the @<sink> here results in an error and no trace produced. After this set (alongside the CoreSight change set [1]) the infrastructure will be able to select a default sink:- sudo ./perf record -e cs_etm// --per-thread uname -a This matches with the default operation provided with perf and intelpt. This patch removes the check that a sink value is provided on the command line with a NULL value passed to the CoreSight infrastructure if omitted. Note: If this set is applied to a system without [1], then the effect is benign as the existing CoreSight infrastructure will detect the error and refuse to trace. Applies to Linux coresight/next branch [1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/734854.html Changes since v3: 1) Removed RFC designation and distributed to wider audience. 2) Split set into perf user runtime (this set), and CoreSight driver code. Mike Leach (1): perf: cs-etm: Allow no CoreSight sink to be specified on command line tools/perf/arch/arm/util/cs-etm.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- 2.17.1

4 years, 1 month

[PATCH v4 0/5] Update CoreSight infrastructure to select a default sink.

by Mike Leach

This patchset provides a proposed infrastructure to allow for the automatic selection of a sink during CoreSight tracing operations. Currently starting tracing using perf requires a sink selection on the command line:- sudo ./perf record -e cs_etm/@tmc_etr0/ --per-thread uname -a After this set (and the follow-up perf change set) the infrastructure will be able to select a default sink:- sudo ./perf record -e cs_etm// --per-thread uname -a This matches with the default operation provided with perf and intelpt. Where no sink is specified at the start of a trace session, the CoreSight system will walk the connection graph from the source ETM, to find a suitable sink using the first encountered highest priority device. The CoreSight infrastructure is updated to define sink sub_types to differentiate between sinks with built in buffers (ETB / ETF) - BUFFER type, and those that use system memory (ETR) - SYSMEM - types. SYSMEM types are considered higher priority. When two sinks are found of equal priority, then the closest sink to the source in terms of connection nodes is chosen. The automatic sink selection will also operate if an ETM is enabled using sysfs commands, and no sink is currently enabled. Applies to Linux coresight/next branch Changes since v3: 1) Removed RFC designation and distributed to wider audience. 2) Split set into CoreSight driver code (this set), and perf user runtime set. 3) Minor cosmetic changes. Mike Leach (5): coresight: Fix comment in main header file. coresight: Add default sink selection to CoreSight base coresight: tmc: Update sink types for default selection. coresight: etm: perf: Add default sink selection to etm perf coresight: sysfs: Allow select default sink on source enable. .../hwtracing/coresight/coresight-etm-perf.c | 17 +- drivers/hwtracing/coresight/coresight-priv.h | 2 + drivers/hwtracing/coresight/coresight-tmc.c | 3 +- drivers/hwtracing/coresight/coresight.c | 147 +++++++++++++++++- include/linux/coresight.h | 6 +- 5 files changed, 168 insertions(+), 7 deletions(-) -- 2.17.1

4 years, 1 month

perf script for trace disassembly

by Al Grant

Hi, two years ago Leo posted a patch set including a "perf script" script for trace disassembly (arm-cs-trace-disasm.py): https://lists.linaro.org/pipermail/coresight/2018-May/001325.html I can't find this upstream - did it change name, or was there a reason it didn't make it? This kind of thing is becoming more interesting as we are looking to give people better ways of processing large volumes of trace without using "perf inject" as an intermediate step. Al IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

4 years, 1 month

Re: Random Softlockups when SMMU translation is enabled for TMC ETR

by Sai Prakash Ranjan

Hi guys, On 2020-05-28 14:16, Sai Prakash Ranjan wrote: > Hi folks, > > I was trying to enable SMMU translation for coresight TMC ETR > component via "iommus" property on SC7180 SoC. > And on enabling the trace, I observe random softlockups everytime. For > enabling trace, I just select ETR as > the sink and ETM as the source via sysfs and immediately start seeing > lockups. I have looked from the hardware > point of view and there is no errata as such and it works fine on > downstream kernel which is very old - 4.14 kernel. > This is reproducible even with KASAN disabled as well. Also I have > dumped other CPU stacks as well during softlockup > via adding dump_stack in ipi_cpu_stop and and no interesting logs > there as other cpus are idle. > As an update, this turned out to be due to firmware misconfiguration. Thanks, Sai -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

4 years, 1 month

[PATCHv3 0/2] coresight: etm4x: Add support to skip trace unit power up

by Sai Prakash Ranjan

This series adds support to skip powering up of trace unit on systems with an errata which stops CPU watchdog counter when power up bit is set (TRCPDCR.PU = 1). Setting this bit is not required on Qualcomm Technologies Inc. chipsets where this errata exists since the ETMs are in the same power domain as their respective CPU cores. Changes since v2: * Fix checkpatch warning for 80 characters. Changes since v1: * Use qcom,skip-power-up for naming as suggested by Mathieu. * Have check for DT property in initialization as per Mathieu. * Reword description for property as per Mathieu. * Split up dt-binding patch as checkpatch complains. Tingwei Zhang (2): coresight: etm4x: Add support to skip trace unit power up dt-bindings: arm: coresight: Add support to skip trace unit power up .../devicetree/bindings/arm/coresight.txt | 7 +++++ drivers/hwtracing/coresight/coresight-etm4x.c | 27 ++++++++++++------- drivers/hwtracing/coresight/coresight-etm4x.h | 3 +++ 3 files changed, 27 insertions(+), 10 deletions(-) -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

4 years, 2 months

Random Softlockups when SMMU translation is enabled for TMC ETR

by Sai Prakash Ranjan

Hi folks, I was trying to enable SMMU translation for coresight TMC ETR component via "iommus" property on SC7180 SoC. And on enabling the trace, I observe random softlockups everytime. For enabling trace, I just select ETR as the sink and ETM as the source via sysfs and immediately start seeing lockups. I have looked from the hardware point of view and there is no errata as such and it works fine on downstream kernel which is very old - 4.14 kernel. This is reproducible even with KASAN disabled as well. Also I have dumped other CPU stacks as well during softlockup via adding dump_stack in ipi_cpu_stop and and no interesting logs there as other cpus are idle. Logs for softlockup, [ 69.725539] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [ksoftirqd/0:9] [ 69.772709] irq event stamp: 144 [ 69.776047] hardirqs last enabled at (143): [<ffffffd05a0fb908>] run_ksoftirqd+0x34/0x70 [ 69.784452] hardirqs last disabled at (144): [<ffffffd05ab7bbd8>] __schedule+0x158/0x940 [ 69.792768] softirqs last enabled at (142): [<ffffffd05a0819dc>] __do_softirq+0x4bc/0x51c [ 69.801255] softirqs last disabled at (69): [<ffffffd05a0fb904>] run_ksoftirqd+0x30/0x70 [ 69.824147] pstate: 60400009 (nZCv daif +PAN -UAO) [ 69.829082] pc : skb_release_data+0x60/0x1a4 [ 69.833475] lr : skb_release_data+0x50/0x1a4 [ 69.837864] sp : ffffff8147cab770 [ 69.841279] x29: ffffff8147cab770 x28: ffffff8133277d00 [ 69.846744] x27: 0000000000000034 x26: ffffff814441a30e [ 69.852204] x25: ffffff811f0c0000 x24: 0000000000000000 [ 69.857666] x23: ffffff81292cc000 x22: ffffff814441a380 [ 69.863133] x21: 0000000000000001 x20: ffffff814441a3a0 [ 69.868595] x19: ffffff81055a7540 x18: ffffff8114f80280 [ 69.874062] x17: 00000000000005a4 x16: ffffff8114f80278 [ 69.879526] x15: 0000000000000000 x14: 0000000000000001 [ 69.884994] x13: 00000000caf7cc01 x12: 00000000b6bc5499 [ 69.890453] x11: 0000000000000000 x10: dfffffd000000001 [ 69.895920] x9 : 0000000000000001 x8 : 0000000000000000 [ 69.901377] x7 : 0000000000000000 x6 : 0000000000000000 [ 69.906839] x5 : 0000000000000000 x4 : 0000000000000000 [ 69.912307] x3 : ffffffd05a99ea98 x2 : 0000000000000001 [ 69.917771] x1 : 0000000000000004 x0 : 0000000000000001 [ 69.923231] Call trace: [ 69.925765] skb_release_data+0x60/0x1a4 [ 69.929801] skb_release_all+0x2c/0x38 [ 69.933669] consume_skb+0x48/0x6c [ 69.937177] packet_rcv+0x5c/0x3f0 [ 69.940685] __netif_receive_skb_one_core+0x50/0x60 [ 69.945704] __netif_receive_skb+0x4c/0x8c [ 69.949926] netif_receive_skb_internal+0xb8/0x198 [ 69.954856] napi_gro_receive+0x19c/0x360 [ 69.959030] r8152_poll+0x518/0x7b4 [r8152] [ 69.963336] net_rx_action+0x114/0x32c [ 69.967198] __do_softirq+0x258/0x51c [ 69.970968] irq_exit+0xd8/0xf8 [ 69.974215] __handle_domain_irq+0x8c/0xc4 [ 69.978430] gic_handle_irq+0x100/0x1c4 [ 69.982378] el1_irq+0xbc/0x180 [ 69.985619] smpboot_thread_fn+0x84/0x2bc [ 69.989753] kthread+0x128/0x138 [ 69.993081] ret_from_fork+0x10/0x18 [ 69.996769] Kernel panic - not syncing: softlockup: hung tasks [ 70.017342] Call trace: [ 70.019867] dump_backtrace+0x0/0x188 [ 70.023640] show_stack+0x20/0x2c [ 70.027061] dump_stack+0xdc/0x144 [ 70.030573] panic+0x174/0x37c [ 70.033724] softlockup_fn+0x0/0x60 [ 70.037324] __hrtimer_run_queues+0x264/0x498 [ 70.041808] hrtimer_interrupt+0xf0/0x22c [ 70.045946] arch_timer_handler_phys+0x40/0x50 [ 70.050524] handle_percpu_devid_irq+0x8c/0x158 [ 70.055190] __handle_domain_irq+0x84/0xc4 [ 70.059412] gic_handle_irq+0x100/0x1c4 [ 70.063359] el1_irq+0xbc/0x180 [ 70.066595] skb_release_data+0x60/0x1a4 [ 70.070638] skb_release_all+0x2c/0x38 [ 70.074498] consume_skb+0x48/0x6c [ 70.078007] packet_rcv+0x5c/0x3f0 [ 70.081518] __netif_receive_skb_one_core+0x50/0x60 [ 70.086533] __netif_receive_skb+0x4c/0x8c [ 70.090746] netif_receive_skb_internal+0xb8/0x198 [ 70.095673] napi_gro_receive+0x19c/0x360 [ 70.099825] r8152_poll+0x518/0x7b4 [r8152] [ 70.104129] net_rx_action+0x114/0x32c [ 70.107984] __do_softirq+0x258/0x51c [ 70.111760] irq_exit+0xd8/0xf8 [ 70.115003] __handle_domain_irq+0x8c/0xc4 [ 70.119215] gic_handle_irq+0x100/0x1c4 [ 70.123168] el1_irq+0xbc/0x180 [ 70.126410] smpboot_thread_fn+0x84/0x2bc [ 70.130540] kthread+0x128/0x138 [ 70.133862] ret_from_fork+0x10/0x18 [ 70.137570] SMP: stopping secondary CPUs [ 70.355132] Kernel Offset: 0x4a000000 from 0xffffffd010000000 [ 70.361047] PHYS_OFFSET: 0xfffffffac0000000 [ 70.365357] CPU features: 0x0006,2a80aa18 [ 70.369489] Memory Limit: none One more such softlockup, [ 122.634764] watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [sugov:0:1729] [ 122.683618] irq event stamp: 0 [ 122.686813] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 122.693310] hardirqs last disabled at (0): [<ffffffd254aec380>] copy_process+0x3e4/0xca8 [ 122.701655] softirqs last enabled at (0): [<ffffffd254aec398>] copy_process+0x3fc/0xca8 [ 122.710003] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 122.731030] pstate: 20400009 (nzCv daif +PAN -UAO) [ 122.736001] pc : preempt_schedule_irq+0x48/0x84 [ 122.740697] lr : preempt_schedule_irq+0x44/0x84 [ 122.745389] sp : ffffffc015c13a80 [ 122.748828] x29: ffffffc015c13a80 x28: ffffff815285af80 [ 122.754330] x27: ffffff815285af80 x26: ffffffc010004000 [ 122.759833] x25: ffffffc010000000 x24: 0000000000000000 [ 122.765332] x23: 0000000080c00009 x22: ffffffd2555492e8 [ 122.770826] x21: 0000000000000060 x20: 00000000000000e0 [ 122.776321] x19: ffffff815285af80 x18: 0000000000000005 [ 122.781825] x17: 0000000000000000 x16: 0000000000000001 [ 122.787319] x15: 0000000000000010 x14: 0000000000000010 [ 122.792822] x13: 0000000000000000 x12: ffffffd254a9bb14 [ 122.798320] x11: 632d4b6b9e7ff300 x10: 632d4b6b9e7ff300 [ 122.803820] x9 : ffffffd25554750c x8 : 0000000000000000 [ 122.809318] x7 : ffffffd254ba17b4 x6 : 0000000000000000 [ 122.814815] x5 : 0000000000010000 x4 : 0000000000000080 [ 122.820312] x3 : ffffffc015c13a00 x2 : ffffffd254a93c04 [ 122.825809] x1 : ffffffc015c13a00 x0 : ffffffd255548028 [ 122.831309] Call trace: [ 122.833876] preempt_schedule_irq+0x48/0x84 [ 122.838217] arm64_preempt_schedule_irq+0x4c/0x70 [ 122.843092] el1_irq+0xd4/0x180 [ 122.846364] __mutex_unlock_slowpath+0x260/0x290 [ 122.851150] mutex_unlock+0x24/0x30 [ 122.854780] dev_pm_opp_xlate_required_opp+0xd8/0x114 [ 122.860010] qcom_cpufreq_hw_target_index+0xf4/0x1a4 [ 122.865160] __cpufreq_driver_target+0x1e8/0x2f4 [ 122.869947] sugov_work+0x58/0x70 [ 122.873397] kthread_worker_fn+0x10c/0x1f8 [ 122.877652] kthread+0x11c/0x12c [ 122.881014] ret_from_fork+0x10/0x18 [ 122.884733] Kernel panic - not syncing: softlockup: hung tasks [ 122.905309] Call trace: [ 122.907872] dump_backtrace+0x0/0x188 [ 122.911678] show_stack+0x20/0x2c [ 122.915130] dump_stack+0xc8/0x124 [ 122.918667] panic+0x168/0x388 [ 122.921857] softlockup_fn+0x0/0x60 [ 122.925485] __hrtimer_run_queues+0x250/0x47c [ 122.930003] hrtimer_interrupt+0xf0/0x22c [ 122.934171] arch_timer_handler_phys+0x40/0x50 [ 122.938782] handle_percpu_devid_irq+0x8c/0x158 [ 122.943485] __handle_domain_irq+0x84/0xc4 [ 122.947737] gic_handle_irq+0xd0/0x178 [ 122.951625] el1_irq+0xbc/0x180 [ 122.954903] preempt_schedule_irq+0x48/0x84 [ 122.959239] arm64_preempt_schedule_irq+0x4c/0x70 [ 122.964110] el1_irq+0xd4/0x180 [ 122.967387] __mutex_unlock_slowpath+0x260/0x290 [ 122.972169] mutex_unlock+0x24/0x30 [ 122.975791] dev_pm_opp_xlate_required_opp+0xd8/0x114 [ 122.981015] qcom_cpufreq_hw_target_index+0xf4/0x1a4 [ 122.986150] __cpufreq_driver_target+0x1e8/0x2f4 [ 122.990933] sugov_work+0x58/0x70 [ 122.994378] kthread_worker_fn+0x10c/0x1f8 [ 122.998629] kthread+0x11c/0x12c [ 123.001982] ret_from_fork+0x10/0x18 [ 123.005741] SMP: stopping secondary CPUs [ 123.223202] Kernel Offset: 0x1244a00000 from 0xffffffc010000000 [ 123.229319] PHYS_OFFSET: 0xffffffdb80000000 [ 123.233655] CPU features: 0x0006,2a80aa18 [ 123.237809] Memory Limit: none Thanks, Sai -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

4 years, 2 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CoreSight May 2020