Hi Mike and Mathieu,
-----Original Message----- From: Mathieu Poirier mathieu.poirier@linaro.org Sent: Friday, March 22, 2019 6:09 PM To: Mike Leach mike.leach@linaro.org Cc: Wojciech Żmuda wzmuda@n7space.com; Michal Mosdorf mmosdorf@n7space.com; coresight@lists.linaro.org; Michał Kurowski mkurowski@n7space.com Subject: Re: Empty trace in TMC-ETR on Xilinx Zynq US+ MPSoC
On Fri, 22 Mar 2019 at 09:52, Mike Leach mike.leach@linaro.org wrote:
Hi,
On Thu, 21 Mar 2019 at 18:21, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Leo,
I tried to mimic this behaviour on my platform by adding similar reference to the only SMMU node defined in xilinx/zynqmp.dtsi. In my case it's iommus = <&smmu 0xc5>; since there is no dedicated SMMU for ETR (and I don't see it in TRM) and 0xc5 is stream ID calculated from the CoreSight master ID (TRM Chapter 16, Table 16-11). I can see in dmesg that SMMU is enabled and ETR is added to iommu group 0, but this
does not change the behaviour. I'd appreciate any suggestions if this direction seem worth further debugging.
To be honest, I don't have experience for SMMU; but based on Hikey and DB410c, both doesn't connect ETR with SMMU and I can run perf on both them (please note, I did this for 1~2 monthes ago).
That's a good clue. I'm leaving the SMMU hypothesis for now then.
Another interesting observation is that I'm actually unable to access anything below the 4k ETF in the topology I sketched. I can't use ETF2 nor STM via sysfs. I wonder if there is some ATB configuration that may
be worth checking as well?
I personally think the straightforward method is to use sysfs mode to verify the path from sources to sink and you could dump trace raw data; e.g. for Hikey I use below commands:
echo 1 > /sys/bus/coresight/devices/f6404000.etf/enable_sink echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source dd if=/dev/f6404000.etr of=/tmp/etr_raw_data
Using this way, you could firstly confirm if can capture raw data; the purpose for doing this is to verify if clock/power have been configured properly on your platform. If this doesn't work (as you said ETF2 or STM cannot be used via sysfs), I think you should firstly debug for sysfs mode and the next step is to use perf tool.
I followed this way and here is the result.
I enabled sinks and sources: root@zynq:/sys/bus/coresight/devices# echo 1 > fe970000.etr/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fe950000.etf2/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fe940000.etf1/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fec40000.etm0/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fed40000.etm1/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fee40000.etm2/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fef40000.etm3/enable_source
OK - please try enabling _one_ sink only. I believe that the CoreSight infrastructure will enable a path from source to enabled sink when you enable the first source. The issue with enabling ETF1 as a sink is that it will not forward any data to the other enabled sinks further down the line.
In theory, if you enable ETF2, then enable an ETM source, then the coresight drivers will enable ETM1 as a hardware FIFO link, _not_ as a sink so that any data passes through.
Mike is correct - the framework will use the first enabled sink that is encoutered on a path. As such if an ETF is encoutered on a path but hasn't been enabled it will be configured as a FIFO.
Ok, that's something I was not aware of. With this approach I'm getting slightly better results - I can access ETF2 via dd now:
root@zynq:/sys/bus/coresight/devices# echo 1 > fe950000.etf2/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fec40000.etm0/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fed40000.etm1/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fee40000.etm2/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fef40000.etm3/enable_source root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe950000.etf2 of=/root/trace.bin 0+1 records in 0+1 records out 16 bytes copied, 0.00126131 s, 12.7 kB/s
The trace buffer is only 16-bytes long and filled with zeros (except that one byte):
root@zynq:/sys/bus/coresight/devices# hexdump /root/trace.bin 0000000 0001 0000 0000 0000 0000 0000 0000 0000 0000010
Interestingly, internal buffer of ETF2 is set to 8kB (RSZ register): root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe950004 32 0x00000800
On the bright side, I can see that the components on the way are actually enabled, Including funnels I saw disabled yesterday (which seems perfectly clear now): root@zynq:/sys/bus/coresight/devices# cat fe940000.etf1/mgmt/ctl 0x1 root@zynq:/sys/bus/coresight/devices# cat fe950000.etf2/mgmt/ctl 0x1 root@zynq:/sys/bus/coresight/devices# cat fe930000.funnel2/funnel_ctrl 0x702 root@zynq:/sys/bus/coresight/devices# cat fe920000.funnel1/funnel_ctrl 0x70f
The same happens when I try ETR instead of ETF2 (after power off/power on cycle).
I've also did an experiment with checking registers of ETR while it was selected as sink in perf. I examined RRP and RWP registers while perf was tracing and I could observed these pointers changing. It looks like the component is working - it's just the trace that's empty.
So, framework-wise everything seems fine. It smells like memory access issue maybe?
Thank you for joining the discussion. I'll appreciate any hints that may bump into your heads.
Best regards, Wojciech
Mathieu.
I recommend testing with each sink enabled separately to see if you get trace. Remember to disable everything you enabled between tests.
Regards
Mike
Then, while trying to consume trace from sinks, only ETF1 (being the topmost ETF with 4k buffer, closest to A53 cores) works:
root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe970000.etr of=/root/trace.bin dd: failed to open '/dev/fe970000.etr': Invalid argument root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe950000.etf2 of=/root/trace.bin dd: failed to open '/dev/fe950000.etf2': Invalid argument root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe940000.etf1 of=/root/trace.bin 8+0 records in 8+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00134656 s, 3.0 MB/s
Trace.bin is not filled with zeros. I didn't bother decoding it, but on the first glance it looks like a valid trace. This is the same
topology that works for me with perf.
Further examination showed me an interesting thing. I checked sysfs-mapped registers of ETFs, ETR and funnels:
root@zynq:/sys/bus/coresight/devices# cat fe970000.etr/mgmt/ctl 0x0 root@zynq:/sys/bus/coresight/devices# cat fe950000.etf2/mgmt/ctl 0x0 root@zynq:/sys/bus/coresight/devices# cat fe940000.etf1/mgmt/ctl 0x1 root@zynq:/sys/bus/coresight/devices# cat fe920000.funnel1/funnel_ctrl 0x701 root@zynq:/sys/bus/coresight/devices# cat fe930000.funnel2/funnel_ctrl 0x300
If I understand this correctly, ETF2 (8kB), ETR and funnel2 (which routes trace to ETF2 and ETR) are disabled, while ETF1 and funnel1 (which routes trace to ETF1) are enabled. Interestingly, I can't enable funnel2, neither with sysfs or direct memory access (I have kernel with
CONFIG_STRICT_DEVMEM=n, tried that from u-boot as well):
root@zynq:/sys/bus/coresight/devices# echo 0x301 > fe930000.funnel2/funnel_ctrl -bash: fe930000.funnel2/funnel_ctrl: Permission denied root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x00000300 root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x301 root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x00000300
My current hypothesis is that I'm not able to get trace, because these
devices are disabled.
I have no idea at the moment what may cause it. If my understanding of TRM is correct, All these devices, working and not-working ones,
are in the same full-power domain.
Also, I suspect, if they were powered off, I would get 0x00s or 0xffs
trying to read them.
Leo, have you ever witnessed something similar on your boards?
I tried everything above with 'nohlt' added to kernel arguments. I don't think this was relevant though, because I was using kernel with
CPU_IDLE=n before.
BTW, could you explain what's the issue for you cannot use ETF2 via
sysfs?
This is caused by clocks, or dt bindings?
I said it a bit imprecisely before. I meant that I'm not able to use ETF2 in perf, since it gives me empty trace, just like ETR does. I have troubles getting STM to work via sysfs, however I suspect I may
be making some mistakes regarding setting up policies. I'd leave that for later anyway.
Leo, what do you think of releasing my changes to Zynq Ultrascale+ DTS
on this list as RFC?
Perhaps it would encourage community to help with bringing up CoreSight on this platform, since some of the topology is already
working.
Thank you and best regards, Wojciech _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight