On Fri, 22 Mar 2019 at 09:52, Mike Leach mike.leach@linaro.org wrote:
Hi,
On Thu, 21 Mar 2019 at 18:21, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Leo,
I tried to mimic this behaviour on my platform by adding similar reference to the only SMMU node defined in xilinx/zynqmp.dtsi. In my case it's iommus = <&smmu 0xc5>; since there is no dedicated SMMU for ETR (and I don't see it in TRM) and 0xc5 is stream ID calculated from the CoreSight master ID (TRM Chapter 16, Table 16-11). I can see in dmesg that SMMU is enabled and ETR is added to iommu group 0, but this
does not change the behaviour. I'd appreciate any suggestions if this direction seem worth further debugging.
To be honest, I don't have experience for SMMU; but based on Hikey and DB410c, both doesn't connect ETR with SMMU and I can run perf on both them (please note, I did this for 1~2 monthes ago).
That's a good clue. I'm leaving the SMMU hypothesis for now then.
Another interesting observation is that I'm actually unable to access anything below the 4k ETF in the topology I sketched. I can't use ETF2 nor STM via sysfs. I wonder if there is some ATB configuration that may
be worth checking as well?
I personally think the straightforward method is to use sysfs mode to verify the path from sources to sink and you could dump trace raw data; e.g. for Hikey I use below commands:
echo 1 > /sys/bus/coresight/devices/f6404000.etf/enable_sink echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source dd if=/dev/f6404000.etr of=/tmp/etr_raw_data
Using this way, you could firstly confirm if can capture raw data; the purpose for doing this is to verify if clock/power have been configured properly on your platform. If this doesn't work (as you said ETF2 or STM cannot be used via sysfs), I think you should firstly debug for sysfs mode and the next step is to use perf tool.
I followed this way and here is the result.
I enabled sinks and sources: root@zynq:/sys/bus/coresight/devices# echo 1 > fe970000.etr/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fe950000.etf2/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fe940000.etf1/enable_sink root@zynq:/sys/bus/coresight/devices# echo 1 > fec40000.etm0/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fed40000.etm1/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fee40000.etm2/enable_source root@zynq:/sys/bus/coresight/devices# echo 1 > fef40000.etm3/enable_source
OK - please try enabling _one_ sink only. I believe that the CoreSight infrastructure will enable a path from source to enabled sink when you enable the first source. The issue with enabling ETF1 as a sink is that it will not forward any data to the other enabled sinks further down the line.
In theory, if you enable ETF2, then enable an ETM source, then the coresight drivers will enable ETM1 as a hardware FIFO link, _not_ as a sink so that any data passes through.
Mike is correct - the framework will use the first enabled sink that is encoutered on a path. As such if an ETF is encoutered on a path but hasn't been enabled it will be configured as a FIFO.
Mathieu.
I recommend testing with each sink enabled separately to see if you get trace. Remember to disable everything you enabled between tests.
Regards
Mike
Then, while trying to consume trace from sinks, only ETF1 (being the topmost ETF with 4k buffer, closest to A53 cores) works:
root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe970000.etr of=/root/trace.bin dd: failed to open '/dev/fe970000.etr': Invalid argument root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe950000.etf2 of=/root/trace.bin dd: failed to open '/dev/fe950000.etf2': Invalid argument root@zynq:/sys/bus/coresight/devices# dd if=/dev/fe940000.etf1 of=/root/trace.bin 8+0 records in 8+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00134656 s, 3.0 MB/s
Trace.bin is not filled with zeros. I didn't bother decoding it, but on the first glance it looks like a valid trace. This is the same topology that works for me with perf.
Further examination showed me an interesting thing. I checked sysfs-mapped registers of ETFs, ETR and funnels:
root@zynq:/sys/bus/coresight/devices# cat fe970000.etr/mgmt/ctl 0x0 root@zynq:/sys/bus/coresight/devices# cat fe950000.etf2/mgmt/ctl 0x0 root@zynq:/sys/bus/coresight/devices# cat fe940000.etf1/mgmt/ctl 0x1 root@zynq:/sys/bus/coresight/devices# cat fe920000.funnel1/funnel_ctrl 0x701 root@zynq:/sys/bus/coresight/devices# cat fe930000.funnel2/funnel_ctrl 0x300
If I understand this correctly, ETF2 (8kB), ETR and funnel2 (which routes trace to ETF2 and ETR) are disabled, while ETF1 and funnel1 (which routes trace to ETF1) are enabled. Interestingly, I can't enable funnel2, neither with sysfs or direct memory access (I have kernel with CONFIG_STRICT_DEVMEM=n, tried that from u-boot as well):
root@zynq:/sys/bus/coresight/devices# echo 0x301 > fe930000.funnel2/funnel_ctrl -bash: fe930000.funnel2/funnel_ctrl: Permission denied root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x00000300 root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x301 root@zynq:/sys/bus/coresight/devices# busybox devmem 0xfe930000 32 0x00000300
My current hypothesis is that I'm not able to get trace, because these devices are disabled. I have no idea at the moment what may cause it. If my understanding of TRM is correct, All these devices, working and not-working ones, are in the same full-power domain. Also, I suspect, if they were powered off, I would get 0x00s or 0xffs trying to read them. Leo, have you ever witnessed something similar on your boards?
I tried everything above with 'nohlt' added to kernel arguments. I don't think this was relevant though, because I was using kernel with CPU_IDLE=n before.
BTW, could you explain what's the issue for you cannot use ETF2 via sysfs? This is caused by clocks, or dt bindings?
I said it a bit imprecisely before. I meant that I'm not able to use ETF2 in perf, since it gives me empty trace, just like ETR does. I have troubles getting STM to work via sysfs, however I suspect I may be making some mistakes regarding setting up policies. I'd leave that for later anyway.
Leo, what do you think of releasing my changes to Zynq Ultrascale+ DTS on this list as RFC? Perhaps it would encourage community to help with bringing up CoreSight on this platform, since some of the topology is already working.
Thank you and best regards, Wojciech _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight