On 06/07/2023 13:47, Pandey, Radhey Shyam wrote:
-----Original Message----- From: Pandey, Radhey Shyam Sent: Thursday, July 6, 2023 5:40 PM To: James Clark james.clark@arm.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com; Sarangi, Anirudha anirudha.sarangi@amd.com Subject: RE: Linux coresight arm64 : Incorrect data in cstrace.bin
-----Original Message----- From: James Clark james.clark@arm.com Sent: Wednesday, July 5, 2023 5:44 PM To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com; Sarangi, Anirudha anirudha.sarangi@amd.com Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
On 05/07/2023 12:48, Pandey, Radhey Shyam wrote:
-----Original Message----- From: James Clark james.clark@arm.com Sent: Wednesday, July 5, 2023 4:29 PM To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
On 05/07/2023 10:56, Pandey, Radhey Shyam wrote:
> -----Original Message----- > From: Suzuki K Poulose suzuki.poulose@arm.com > Sent: Wednesday, July 5, 2023 1:57 PM > To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com; > mathieu.poirier@linaro.org; mike.leach@linaro.org; > leo.yan@linaro.org; alexander.shishkin@linux.intel.com > Cc: coresight@lists.linaro.org; > linux-arm-kernel@lists.infradead.org > Subject: Re: Linux coresight arm64 : Incorrect data in > cstrace.bin > > Hi Radhe Shyam > > On 05/07/2023 05:38, Pandey, Radhey Shyam wrote: >> Hi, >> >> I am using linux 6.1 kernel coresight framework to capture ETM
trace.
>> Enabled coresight driver and added coresight component device >> node in DTS. With that, we could probe all coresight components. >> >> But when we do capture using sysfs and then read the trace.bin >> using ptm2human we see invalid trace data. Trace content >> changes on each capture. Any pointers to help narrow down the
issue?
>> >> Tried: sysfs capture and perf (with OpenCSD enabled) >> >> Development board: VCK190 : >> https://docs.xilinx.com/r/en-US/ug1366-vck190-eval-bd >> https://www.xilinx.com/support/documents/architecture- > manuals/am011-ve >> rsal-acap-trm.pdf >> >> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i amba >> [ 0.301054] Serial: AMBA PL011 UART driver >> [ 0.306226] amba f0c20000.funnel: Fixing up cyclic dependency
with
> f0d70000.etm >> [ 0.313624] amba f0c20000.funnel: Fixing up cyclic dependency
with
> f0d30000.etm >> [ 0.321110] amba f0c30000.etf: Fixing up cyclic dependency with > f0c20000.funnel >> [ 1.470247] Serial: AMBA driver >> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i coresight >> [ 1.865006] cs_system_cfg: CoreSight Configuration manager
initialised
>> [ 1.876858] coresight etm0: CPU0: etm v4.0 initialized >> [ 1.885666] coresight etm1: CPU1: etm v4.0 initialized >> [ 1.894435] coresight-cpu-debug f0d00000.debug1: Coresight
debug-
CPU0
> initialized >> [ 1.902072] coresight-cpu-debug f0d40000.debug1: Coresight
debug-
CPU1
> initialized >> >> xilinx-vck190-20231:/home/petalinux# ls
/sys/bus/coresight/devices/
>> etm0 etm1 funnel0 tmc_etf0 >> >> cd /sys/bus/coresight/devices/ echo 1 > tmc_etf0/enable_sink >> echo 1 > etm0/enable_source echo 0 > etm0/enable_source echo 0 > >> tmc_etf0/enable_sink cd /root/ dd if=/dev/tmc_etf0 >> of=cstrace_28Jun.bin >> >> ./ptm2human/ptm2human -e -i cstrace.bin -d > > Please note that ptm2human is for PTM trace decoding and > ETMv4 uses a different format and thus is not compatible. > >> >> I also tried OpenCSD integration with PERF. > > Have you made sure the perf is "linked" to the opencsd ?
Earlier I statically linked but after adding CORESIGHT=1.
linux-xlnx$ make ARCH=arm64 NO_LIBELF=1 NO_JVMTI=1 VF=1
CORESIGHT=1 -C
tools/perf/ make: Entering directory 'linux-xlnx/tools/perf' BUILD: Doing 'make -j24' parallel build
Makefile.config:520: *** Error: No libopencsd library found or the version
is not up-to-date.
Please install recent libopencsd to build with CORESIGHT=1. Stop.
echo $CSINCLUDES /scratch/development/coresight/my-opencsd/decoder/include radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls /scratch/development/coresight/my-opencsd/decoder/include common i_dec interfaces mem_acc opencsd opencsd.h pkt_printers
echo $CSLIBS /scratch/development/coresight/my-opencsd/decoder/lib/builddir radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls
/scratch/development/coresight/my-opencsd/decoder/lib/builddir
libopencsd.a libopencsd_c_api.so libopencsd_c_api.so.1.4.0
libopencsd.so.1
libopencsd_c_api.a libopencsd_c_api.so.1 libopencsd.so
libopencsd.so.1.4.0
Anything I am missing to fix this opencsd lib not found?
There is a minimum version of 1.1.1 but it looks like you are using higher than that so that part should be fine.
Personally I "make install" OpenCSD to the system path. I had a play around with $CSLIBS it and maybe there is also some stickyness to the feature detection so a make clean might help to make sure the errors you are seeing are real.
Lastly you can print the output of why the feature test compilation failed which might help. I see some linking issues in there if I delete my system installed version of OpenCSD, so there might be a bug with just using CSINCLUDES and CSLIBS:
$ cat tools/build/feature/test-libopencsd.make.output
/usr/bin/ld: warning: libopencsd.so.1, needed by opencsd- local/decoder/lib/builddir//libopencsd_c_api.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `vtable for StmTrcPacket' /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `DecodeTree::getDecoderStats(unsigned char, _ocsd_decode_stats**)' /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
Can you try the make install and see if that works? If that works but the other way doesn't I can try looking into why its not and make a
fix.
I tried make install but still it errors out. Looking at make.output I am seeing multiple undefined references.
NOTE I have to still set CSINCLUDES and CSLIBS as we are cross-compiling Perf on a x86 host machine.
This should still work if you make install them to the arch specific folder for example /usr/lib/aarch64-linux-gnu/. If do 'apt install
libopencsd-dev:arm64'
I would get the arm ones in there.
Then if I cross build like this I get openCSD linked:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/perf CORESIGHT=1
So for now you should be able to work around the issue in this way.
I tried copying opencsd libraries to /usr/lib/aarch64-linux-gnu but still I see undefined reference. Is it because I am using cross-compiler from server (toolchain path set to $PATH variable)?
/proj/<snip>/Vitis/2023.2/gnu/aarch64/lin/aarch64-linux/bin/aarch64-linux- gnu-gcc I assume CSLIBS and CSINCLUDES should be sufficient to cross compile perf with coresight support?
FYI, I tried with static linking and could proceed to perf compilation.
Static compilation works because the link line is slightly different (it includes -lopencsd rather than just -lopencsd_c_api:
ifeq ($(findstring -static,${LDFLAGS}),-static) OPENCSDLIBS += -lopencsd -lstdc++
I think that this is related to the issue and there have been some recent changes around here. So yes I'll look into the fix for the non static and $CSLIBS mode.
make LDFLAGS=-static NO_LIBELF=1 NO_JVMTI=1 VF=1 CORESIGHT=1 -C tools/perf/
<snip> ... libbfd-liberty-z: [ OFF ] ... libopencsd: [ on ] ... libunwind-x86: [ OFF ]
But it ended with compilation failure. Seems like OpenCSD version mismatch. I will try to downgrade opencsd library and see if it fixes.
util/cs-etm-decoder/cs-etm-decoder.c: In function 'cs_etm_decoder__gen_trace_elem_printer': util/cs-etm-decoder/cs-etm-decoder.c:567:9: error: enumeration value 'OCSD_GEN_TRC_ELEM_INSTRUMENTATION' not handled in switch [-Werror=switch-enum] 567 | switch (elem->elem_type) {
This should be fixed here [1]. Are you building the latest version of Perf? Downgrading OpenCSD would work to fix this, but so would building the latest version of Perf.
It's also possible that you are pointing to an old OpenCSD header but new libraries. The OpenCSD version number is pulled out of /usr/include/opencsd/ocsd_if_version.h so it's possible to get out of sync with the .so file.
cat tools/build/feature/test-libopencsd.make.output <snip library paths>
aarch64-xilinx-linux-ld.real: warning: libopencsd.so.1, needed by libopencsd_c_api.so, not found (try using -rpath or -rpath-link) aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`vtable for StmTrcPacket'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::getDecoderStats(unsigned char, _ocsd_decode_stats**)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::logMappedRanges()'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`OcsdLibDcdRegister::deregisterAllDecoders()'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`PtmTrcPacket::toString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::s_error_logger'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::addBinFileRegionMemAcc(_ocsd_file_mem_region const*,
int,
_ocsd_mem_space_acc_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`EtmV4ITrcPacket::~EtmV4ITrcPacket()'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`OcsdLibDcdRegister::getDecoderRegister()'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`typeinfo for TraceComponent'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`EtmV3TrcPacket::~EtmV3TrcPacket()'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`OcsdTraceElement::toString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::addGenElemPrinter(TrcGenericElementPrinter**)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::DestroyDecodeTree(DecodeTree*)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`ocsdError::getErrorString[abi:cxx11](ocsdError const&)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::setGenTraceElemOutI(ITrcGenElemIn*)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`ocsdError::ocsdError(ocsdError const*)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`ocsdError::ocsdError(_ocsd_err_severity_t, _ocsd_err_t, unsigned int, unsigned char, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`EtmV4ITrcPacket::toString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`DecodeTree::createDecoder(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, CSConfig const*)'
aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined reference to
`StmTrcPacket::StmTrcPacket()'
Yes this looks like what I see as well. There is probably some issue with CSLIBS not being propagated to all parts of the build. I would recommend using the system library path for now and I will look into a fix
next week.
Unless you are happy to look into it and send a fix that would be great too.
James
Thanks, Radhey
Thanks James
On 07/07/2023 09:49, James Clark wrote:
On 06/07/2023 13:47, Pandey, Radhey Shyam wrote:
-----Original Message----- From: Pandey, Radhey Shyam Sent: Thursday, July 6, 2023 5:40 PM To: James Clark james.clark@arm.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com; Sarangi, Anirudha anirudha.sarangi@amd.com Subject: RE: Linux coresight arm64 : Incorrect data in cstrace.bin
-----Original Message----- From: James Clark james.clark@arm.com Sent: Wednesday, July 5, 2023 5:44 PM To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com; Sarangi, Anirudha anirudha.sarangi@amd.com Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
On 05/07/2023 12:48, Pandey, Radhey Shyam wrote:
-----Original Message----- From: James Clark james.clark@arm.com Sent: Wednesday, July 5, 2023 4:29 PM To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K Poulose suzuki.poulose@arm.com; mathieu.poirier@linaro.org; mike.leach@linaro.org; leo.yan@linaro.org; alexander.shishkin@linux.intel.com Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
On 05/07/2023 10:56, Pandey, Radhey Shyam wrote: >> -----Original Message----- >> From: Suzuki K Poulose suzuki.poulose@arm.com >> Sent: Wednesday, July 5, 2023 1:57 PM >> To: Pandey, Radhey Shyam radhey.shyam.pandey@amd.com; >> mathieu.poirier@linaro.org; mike.leach@linaro.org; >> leo.yan@linaro.org; alexander.shishkin@linux.intel.com >> Cc: coresight@lists.linaro.org; >> linux-arm-kernel@lists.infradead.org >> Subject: Re: Linux coresight arm64 : Incorrect data in >> cstrace.bin >> >> Hi Radhe Shyam >> >> On 05/07/2023 05:38, Pandey, Radhey Shyam wrote: >>> Hi, >>> >>> I am using linux 6.1 kernel coresight framework to capture ETM
trace.
>>> Enabled coresight driver and added coresight component device >>> node in DTS. With that, we could probe all coresight components. >>> >>> But when we do capture using sysfs and then read the trace.bin >>> using ptm2human we see invalid trace data. Trace content >>> changes on each capture. Any pointers to help narrow down the
issue?
>>> >>> Tried: sysfs capture and perf (with OpenCSD enabled) >>> >>> Development board: VCK190 : >>> https://docs.xilinx.com/r/en-US/ug1366-vck190-eval-bd >>> https://www.xilinx.com/support/documents/architecture- >> manuals/am011-ve >>> rsal-acap-trm.pdf >>> >>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i amba >>> [ 0.301054] Serial: AMBA PL011 UART driver >>> [ 0.306226] amba f0c20000.funnel: Fixing up cyclic dependency
with
>> f0d70000.etm >>> [ 0.313624] amba f0c20000.funnel: Fixing up cyclic dependency
with
>> f0d30000.etm >>> [ 0.321110] amba f0c30000.etf: Fixing up cyclic dependency with >> f0c20000.funnel >>> [ 1.470247] Serial: AMBA driver >>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i coresight >>> [ 1.865006] cs_system_cfg: CoreSight Configuration manager
initialised
>>> [ 1.876858] coresight etm0: CPU0: etm v4.0 initialized >>> [ 1.885666] coresight etm1: CPU1: etm v4.0 initialized >>> [ 1.894435] coresight-cpu-debug f0d00000.debug1: Coresight
debug-
CPU0 >> initialized >>> [ 1.902072] coresight-cpu-debug f0d40000.debug1: Coresight
debug-
CPU1 >> initialized >>> >>> xilinx-vck190-20231:/home/petalinux# ls
/sys/bus/coresight/devices/
>>> etm0 etm1 funnel0 tmc_etf0 >>> >>> cd /sys/bus/coresight/devices/ echo 1 > tmc_etf0/enable_sink >>> echo 1 > etm0/enable_source echo 0 > etm0/enable_source echo 0 > >>> tmc_etf0/enable_sink cd /root/ dd if=/dev/tmc_etf0 >>> of=cstrace_28Jun.bin >>> >>> ./ptm2human/ptm2human -e -i cstrace.bin -d >> >> Please note that ptm2human is for PTM trace decoding and >> ETMv4 uses a different format and thus is not compatible. >> >>> >>> I also tried OpenCSD integration with PERF. >> >> Have you made sure the perf is "linked" to the opencsd ? > > Earlier I statically linked but after adding CORESIGHT=1. > > > linux-xlnx$ make ARCH=arm64 NO_LIBELF=1 NO_JVMTI=1 VF=1 CORESIGHT=1 -C > tools/perf/ > make: Entering directory 'linux-xlnx/tools/perf' > BUILD: Doing 'make -j24' parallel build > > Makefile.config:520: *** Error: No libopencsd library found or the > version is not up-to-date. > Please install recent libopencsd to build with CORESIGHT=1. Stop. > > > echo $CSINCLUDES > /scratch/development/coresight/my-opencsd/decoder/include > radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls > /scratch/development/coresight/my-opencsd/decoder/include > common i_dec interfaces mem_acc opencsd opencsd.h > pkt_printers > > echo $CSLIBS > /scratch/development/coresight/my-opencsd/decoder/lib/builddir > radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls /scratch/development/coresight/my-opencsd/decoder/lib/builddir > libopencsd.a libopencsd_c_api.so libopencsd_c_api.so.1.4.0 libopencsd.so.1 > libopencsd_c_api.a libopencsd_c_api.so.1 libopencsd.so libopencsd.so.1.4.0 > > Anything I am missing to fix this opencsd lib not found?
There is a minimum version of 1.1.1 but it looks like you are using higher than that so that part should be fine.
Personally I "make install" OpenCSD to the system path. I had a play around with $CSLIBS it and maybe there is also some stickyness to the feature detection so a make clean might help to make sure the errors you are seeing are real.
Lastly you can print the output of why the feature test compilation failed which might help. I see some linking issues in there if I delete my system installed version of OpenCSD, so there might be a bug with just using CSINCLUDES and CSLIBS:
$ cat tools/build/feature/test-libopencsd.make.output
/usr/bin/ld: warning: libopencsd.so.1, needed by opencsd- local/decoder/lib/builddir//libopencsd_c_api.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `vtable for StmTrcPacket' /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `DecodeTree::getDecoderStats(unsigned char, _ocsd_decode_stats**)' /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so: undefined reference to `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
Can you try the make install and see if that works? If that works but the other way doesn't I can try looking into why its not and make a
fix.
I tried make install but still it errors out. Looking at make.output I am seeing multiple undefined references.
NOTE I have to still set CSINCLUDES and CSLIBS as we are cross-compiling Perf on a x86 host machine.
This should still work if you make install them to the arch specific folder for example /usr/lib/aarch64-linux-gnu/. If do 'apt install
libopencsd-dev:arm64'
I would get the arm ones in there.
Then if I cross build like this I get openCSD linked:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/perf CORESIGHT=1
So for now you should be able to work around the issue in this way.
I tried copying opencsd libraries to /usr/lib/aarch64-linux-gnu but still I see undefined reference. Is it because I am using cross-compiler from server (toolchain path set to $PATH variable)?
/proj/<snip>/Vitis/2023.2/gnu/aarch64/lin/aarch64-linux/bin/aarch64-linux- gnu-gcc I assume CSLIBS and CSINCLUDES should be sufficient to cross compile perf with coresight support?
FYI, I tried with static linking and could proceed to perf compilation.
Static compilation works because the link line is slightly different (it includes -lopencsd rather than just -lopencsd_c_api:
ifeq ($(findstring -static,${LDFLAGS}),-static) OPENCSDLIBS += -lopencsd -lstdc++
I think that this is related to the issue and there have been some recent changes around here. So yes I'll look into the fix for the non static and $CSLIBS mode.
There was an issue, thanks for the report. The fix should be here: https://lore.kernel.org/all/20230707154546.456720-1-james.clark@arm.com/T/#u
Please give it a go and reply with the outcome on that thread.
-
James Clark