Hi,
I'm trying to run Coresight on XIlinx Zynq Ultrascale+ with ZCU104 board. I have already made some progress, however I'm stuck on getting valid timestamp packets in the decoded trace.
I'm using mainline Linux forked at v5.0-rc4. So far, I manually added Coresight nodes to DTS basing on the Ultrascale+ datasheet, since neither mainline Linux nor the Xilinx fork seem to have it. Looks like this step went fine, since I can see drivers reporting in dmesg and related sysfs nodes being registered:
root@zynq:~# dmesg | grep -i coresight [ 3.745681] coresight-etm4x fec40000.etm0: CPU0: ETM v4.0 initialized [ 3.752318] coresight-etm4x fed40000.etm1: CPU1: ETM v4.0 initialized [ 3.758952] coresight-etm4x fee40000.etm2: CPU2: ETM v4.0 initialized [ 3.765587] coresight-etm4x fef40000.etm3: CPU3: ETM v4.0 initialized [ 3.772190] coresight-stm fe9c0000.stm: stm_register_device failed, probing deferred [ 3.780004] coresight-cpu-debug fec10000.debug0: Coresight debug-CPU0 initialized [ 3.787531] coresight-cpu-debug fed10000.debug1: Coresight debug-CPU1 initialized [ 3.795058] coresight-cpu-debug fee10000.debug2: Coresight debug-CPU2 initialized [ 3.802587] coresight-cpu-debug fef10000.debug3: Coresight debug-CPU3 initialized [ 3.917423] coresight-stm fe9c0000.stm: STM500 initialized
root@zynq:~# ls /sys/bus/coresight/devices/ fe920000.funnel1 fe940000.etf1 fe970000.etr fe9c0000.stm fed40000.etm1 fef40000.etm3 fe930000.funnel2 fe950000.etf2 fe980000.tpiu fec40000.etm0 fee40000.etm2 replicator
Overall trace acquisition and decoding with perf+OpenCSD also looks good, apart from timestamp packets. When I request timestamping, the value is constant 0x0:
root@zynq:~/cs_test# perf record -e cs_etm/timestamp,@fe940000.etf1/u --filter 'filter 0x764/0x2c@./sum' --per-thread ./sum Val: 20 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.002 MB perf.data ] root@zynq:~/cs_test# perf report --dump | grep TIMESTAMP Idx:231; ID:12; I_TIMESTAMP : Timestamp.; Updated val = 0x0 Idx:254; ID:12; I_TIMESTAMP : Timestamp.; Updated val = 0x0 Idx:279; ID:12; I_TIMESTAMP : Timestamp.; Updated val = 0x0
According to the datasheet, Ultrascale+ has a timestamp generator. Perf confirms it in the registers values dump:
root@zynq:~/cs_test# perf report --dump | grep 'TRCCONFIGR|TRCIDR0' TRCCONFIGR 800 TRCIDR0 28000ea1 TRCCONFIGR 800 TRCIDR0 28000ea1 TRCCONFIGR 800 TRCIDR0 28000ea1 TRCCONFIGR 800 TRCIDR0 28000ea1
These value match what ETMv4 architecture specs says: - bit 11 (TS) of TRCCONFIGR is 1, what indicates global timestamping is enabled - bits 24:28 (TSSIZE) of TRCIDR0 are 0b1000, what indicates support for 64-bit-long timestamps
In the kernel source code I can see that the ETMv4 driver writes the EVENT bits (0:7) of the TRCTSCTLR register. I didn't thoroughly analyze this code, but I take it as another hint that timestamping is actually supported.
At this point I wonder if anybody has witnessed timestamps working on this platform, whether under Linux or baremetal. I would appreciate even the tiniest suggestions where should I look next to get them working.
Thanks and best regards, Wojciech
Overall trace acquisition and decoding with perf+OpenCSD also looks good, apart from timestamp packets. When I request timestamping, the value is constant 0x0:
I don't know any details of this platform, but the ETM gets the timestamp from a global timestamp generator block. If that block isn't enabled, the timestamp will be written as zeroes. The problem isn't in the ETM itself - the framework needs to ensure the global timestamp generator is enabled when ETM is enabled (or anything else that nedes the timestamp, like STM).
If you're lucky, the timestamp block is mapped into the memory space in the same way as the ETMs are, and the CoreSight driver framework could enable it. If you're unlucky, the timestamp block is hidden away in a secure debug area and can only be enabled by vendor firmware or JTAG debug.
Does the datasheet say anything about the timestamp block, e.g. is it listed in the device memory space as seen by the application CPUs?
Al
Hi Al,
If you're lucky, the timestamp block is mapped into the memory space in the same way as the ETMs are, and the CoreSight driver framework could enable it. If you're unlucky, the timestamp block is hidden away in a secure debug area and can only be enabled by vendor firmware or JTAG debug.
Looks like I'm lucky indeed - the US+ datasheet mentions there's a TSGEN block in the memory map. Could you please elaborate on how the CS framework can enable it? I would suspect there are some coresight-timestamp-like bindings, but I can't see anything related. How can I tell the driver where the timestamp generator is mapped?
I appreciate your help! Regards, Wojciech
Looks like I'm lucky indeed - the US+ datasheet mentions there's a TSGEN block in the memory map. Could you please elaborate on how the CS framework can enable it? I would suspect there are some coresight-timestamp-like bindings, but I can't see anything related. How can I tell the driver where the timestamp generator is mapped?
I'm not sure you can, at any rate I can't see the code for it in the upstream source. But it's something the framework is meant to do. You'd have to modify the framework to read the timestamp generator address from the DTS and then enable/disable it under control of a reference count.
If you have the base physical address of the timestamp control registers (you may have two regions, one for control registers and one for read registers - you want the control registers), the enable bit is bit 0 of word 0. You can read it like this:
busybox devmem xxxxxx 32
where xxxxx is the base address. You might see either 0x00000000 or 0x00000002. If the bottom bit is set then the timestamp is already enabled. Otherwise enable it by writing back e.g.
busybox devmem xxxxxx 32 1
I'm only guessing that that's the issue here - it could be something different, but it's worth a try.
Al
On Tue, 19 Feb 2019 at 13:05, Al Grant Al.Grant@arm.com wrote:
Looks like I'm lucky indeed - the US+ datasheet mentions there's a TSGEN block in the memory map. Could you please elaborate on how the CS framework can enable it? I would suspect there are some coresight-timestamp-like bindings, but I can't see anything related. How can I tell the driver where the timestamp generator is mapped?
So far this isn't something we had to deal with - all the platforms we worked with had the timestamp generator enabled. I see in your initial email you're already using the keywork "timestamp" on the perf command line. That does the trick on Juno and the 410c. As Al pointed out it is likely you will have to add support for TSGEN block.
Mathieu
I'm not sure you can, at any rate I can't see the code for it in the upstream source. But it's something the framework is meant to do. You'd have to modify the framework to read the timestamp generator address from the DTS and then enable/disable it under control of a reference count.
If you have the base physical address of the timestamp control registers (you may have two regions, one for control registers and one for read registers - you want the control registers), the enable bit is bit 0 of word 0. You can read it like this:
busybox devmem xxxxxx 32where xxxxx is the base address. You might see either 0x00000000 or 0x00000002. If the bottom bit is set then the timestamp is already enabled. Otherwise enable it by writing back e.g.
busybox devmem xxxxxx 32 1I'm only guessing that that's the issue here - it could be something different, but it's worth a try.
Al _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Hi Mathieu, thanks for joining the thread.
So far this isn't something we had to deal with - all the platforms we worked with had the timestamp generator enabled. I see in your initial email you're already using the keywork "timestamp" on the perf command line. That does the trick on Juno and the 410c. As Al pointed out it is likely you will have to add support for TSGEN block.
I see. Actually, if I understand the datasheet correctly, timestamp generator should be enabled by default on US+ as well. Perhaps it's the firmware who turns it off? I remember a post on this mailing list, that a firmware update on Juno fixed a similar empty timestamps problem on that platform. Fortunately, Xilinx's forks of U-Boot and ATF are public, so I'll inspect them for such operations.
If that's the case, do you think it would be beneficial to add separate bindings for TSGEN? Perhaps just another reg pair and reg-name value to ETM node, similarly to what's already done for stimulus base in STM. ETM driver could look for reg-name 'timestamp', get the corresponding pair from reg, check if TSGEN is disabled and enable it if necessary. If I understand correctly, that part should be generic, as TSGEN is a standard CS component. It's just a random idea, but I'd appreciate your comment on that.
If you have the base physical address of the timestamp control registers (you may have two regions, one for control registers and one for read registers - you want the control registers), the enable bit is bit 0 of word 0. You can read it like this:
Looks like a nice and quick way to prove it. Thanks Al, I'll give it a try as soon as I have the board back on my desk.
Best regards, Wojciech
I see. Actually, if I understand the datasheet correctly, timestamp generator should be enabled by default on US+ as well. Perhaps it's the firmware who turns it off? I remember a post on this mailing list, that a firmware update on Juno fixed a similar empty timestamps problem on that platform. Fortunately, Xilinx's forks of U-Boot and ATF are public, so I'll inspect them for such operations.
If that's the case, do you think it would be beneficial to add separate bindings for TSGEN? Perhaps just another reg pair and reg-name value to ETM node, similarly to what's already done for stimulus base in STM. ETM driver could look for reg-name 'timestamp', get the corresponding pair from reg, check if TSGEN is disabled and enable it if necessary. If I understand correctly, that part should be generic, as TSGEN is a standard CS component. It's just a random idea, but I'd appreciate your comment on that.
It would be useful to have both.
Where the chip is managed by vendor firmware and the timestamp generator is shared with a system control processor (which in TrustZone terms is Secure) then the timestamp generator really does need to be directly controlled by the firmware. In that case we need an API (e.g. via the SMC API) to ask for self-hosted debug be to be enabled. That could do any number of things - turn on the power domain, turn off firewalling of debug logic etc. We were trying to get that into the SMC API - I will chase up where that has got to.
Where Linux is running "bare metal" with no firmware underneath, and not sharing debug logic with Secure management, then it might be expedient to have Linux directly access the timestamp generator.
Every chip is different and there are any number of ways of setting them up... we need to give ourselves some options.
Al
On Tue, 19 Feb 2019 at 14:42, Al Grant Al.Grant@arm.com wrote:
I see. Actually, if I understand the datasheet correctly, timestamp generator should be enabled by default on US+ as well. Perhaps it's the firmware who turns it off? I remember a post on this mailing list, that a firmware update on Juno fixed a similar empty timestamps problem on that platform. Fortunately, Xilinx's forks of U-Boot and ATF are public, so I'll inspect them for such operations.
If that's the case, do you think it would be beneficial to add separate bindings for TSGEN? Perhaps just another reg pair and reg-name value to ETM node, similarly to what's already done for stimulus base in STM. ETM driver could look for reg-name 'timestamp', get the corresponding pair from reg, check if TSGEN is disabled and enable it if necessary. If I understand correctly, that part should be generic, as TSGEN is a standard CS component. It's just a random idea, but I'd appreciate your comment on that.
It would be useful to have both.
Where the chip is managed by vendor firmware and the timestamp generator is shared with a system control processor (which in TrustZone terms is Secure) then the timestamp generator really does need to be directly controlled by the firmware. In that case we need an API (e.g. via the SMC API) to ask for self-hosted debug be to be enabled. That could do any number of things - turn on the power domain, turn off firewalling of debug logic etc. We were trying to get that into the SMC API - I will chase up where that has got to.
Right, there is a number of things to consider here hence suggesting to use the clock framework for this. That way a clock_get/put() would do all the things we want without smearing the CS subsystem.
Where Linux is running "bare metal" with no firmware underneath, and not sharing debug logic with Secure management, then it might be expedient to have Linux directly access the timestamp generator.
Every chip is different and there are any number of ways of setting them up... we need to give ourselves some options.
Al
Hi Al and Mathieu,
First, thank you Al for your suggestion to examine TSGEN memory. That was the right path. The EN bit of the CNTCR register was indeed 0 and setting it to 1 enabled timestamps:
Idx:82685; ID:16; I_TIMESTAMP : Timestamp.; Updated val = 0x143dba290 Idx:82978; ID:16; I_TIMESTAMP : Timestamp.; Updated val = 0x143dbdc28 Idx:83713; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x14565a9ff Idx:87068; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x14568896e Idx:87828; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x146f25d9f Idx:91159; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x146f536ac
For now, I configured U-Boot to do 'mw fe900000 1' before each boot, so I don't have to do it via 'busybox devmem' from Linux - the latter solution requires kernel built with DEVMEM=y and STRICT_DEVMEM=n.
On Tue, 19 Feb 2019 at 14:42, Al Grant Al.Grant@arm.com wrote:
I see. Actually, if I understand the datasheet correctly, timestamp generator should be enabled by default on US+ as well. Perhaps it's the firmware who turns it off? I remember a post on this mailing list, that a firmware update on Juno fixed a similar empty timestamps problem on that platform. Fortunately, Xilinx's forks of U-Boot and ATF are public, so I'll inspect them for such operations.
Update on this: there is no code related to TSGEN in U-Boot and ATF Xilinx forks. I guess this hardware block is just disabled by default.
If that's the case, do you think it would be beneficial to add separate bindings for TSGEN? Perhaps just another reg pair and reg-name value to ETM node, similarly to what's already done for stimulus base in STM. ETM driver could look for reg-name 'timestamp', get the corresponding pair from reg, check if TSGEN is disabled and enable it if necessary. If I understand correctly, that part should be generic, as TSGEN is a standard CS component. It's just a random idea, but I'd appreciate your comment on that.
It would be useful to have both.
Where the chip is managed by vendor firmware and the timestamp generator is shared with a system control processor (which in TrustZone terms is Secure) then the timestamp generator really does need to be directly controlled by the firmware. In that case we need an API (e.g. via the SMC API) to ask for self-hosted debug be to be enabled. That could do any number of things - turn on the power domain, turn off firewalling of debug logic etc. We were trying to get that into the SMC API - I will chase up where that has got to.
Right, there is a number of things to consider here hence suggesting to use the clock framework for this. That way a clock_get/put() would do all the things we want without smearing the CS subsystem.
I checked that and found an interesting thing. Ultrascale+ (a.k.a zynqmp) has firmware and clock drivers in drivers/firmware/xilinx and drivers/clk/zynqmp/clkc.c. However, these drivers bind to nothing. There is no node with xlnx,zynqmp-firmware compatible in zynqmp DTS files and clock nodes use fixed-clock compatible only instead of xlnx,zynqmp-clk. These bindings are even documented in Documentation/devicetree/bindings/firmware/xilinx/xlnx,zynqmp-firmware.txt. This looks like some unfinished work.
Nevertheless, these two drivers look pretty decent. It might be worth checking if they work with current Xilinx firmware release. I do not see any CoreSight/debug related things in the SMC API exposed by ATF though, so direct memory access is the only option to enable TSGEN on this platform, at least at the moment.
Thanks, Wojciech
First, thank you Al for your suggestion to examine TSGEN memory. That was the right path. The EN bit of the CNTCR register was indeed 0 and setting it to 1 enabled timestamps:
Idx:82685; ID:16; I_TIMESTAMP : Timestamp.; Updated val =0x143dba290 Idx:82978; ID:16; I_TIMESTAMP : Timestamp.; Updated val = 0x143dbdc28 Idx:83713; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x14565a9ff Idx:87068; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x14568896e Idx:87828; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x146f25d9f Idx:91159; ID:10; I_TIMESTAMP : Timestamp.; Updated val = 0x146f536ac
For now, I configured U-Boot to do 'mw fe900000 1' before each boot, so I don't have to do it via 'busybox devmem' from Linux - the latter solution requires kernel built with DEVMEM=y and STRICT_DEVMEM=n.
Thanks for following up. It's really useful to have that feedback. It would be interesting to know if the timestamp is correlated with the kernel timestamp. It occurred to me that just starting and stopping TSGEN dynamically, might not be a good idea, if the timestamp actually stops - if that was the case, then to get a trace timestamp that could be correlated with the continuously running system timestamp used by the kernel, we'd need to think a bit more about how we set up TSGEN.
Enabling the trace timestamp at boot and leaving it enabled is fine, as it at least ensures both timestamps are running all the time. For production devices where we care about having no power impact from trace except when actually tracing, we still need to think about enabling the timestamp dynamically.
Al
On Tue, 19 Feb 2019 at 14:42, Al Grant Al.Grant@arm.com wrote:
I see. Actually, if I understand the datasheet correctly, timestamp generator should be enabled by default on US+ as well. Perhaps it's the firmware who turns it off? I remember a post on this mailing list, that a firmware update on Juno fixed a similar empty timestamps problem on that platform. Fortunately, Xilinx's forks of
U-Boot and ATF are public, so I'll inspect them for such operations.
Update on this: there is no code related to TSGEN in U-Boot and ATF Xilinx forks. I guess this hardware block is just disabled by default.
If that's the case, do you think it would be beneficial to add separate bindings for TSGEN? Perhaps just another reg pair and reg-name value to ETM node, similarly to what's already done for stimulus
base in STM.
ETM driver could look for reg-name 'timestamp', get the corresponding pair from reg, check if TSGEN is disabled and enable it if
necessary.
If I understand correctly, that part should be generic, as TSGEN is a standard CS component. It's just a random idea, but I'd appreciate your
comment on that.
It would be useful to have both.
Where the chip is managed by vendor firmware and the timestamp generator is shared with a system control processor (which in TrustZone terms is Secure) then the timestamp generator really does need to be directly controlled by the firmware. In that case we need an API (e.g. via the SMC API) to ask for self-hosted debug be to be enabled. That could do any number of things - turn on the power domain, turn off firewalling of debug logic etc. We were trying to get that into the SMC API - I will chase up where that has got to.
Right, there is a number of things to consider here hence suggesting to use the clock framework for this. That way a clock_get/put() would do all the things we want without smearing the CS subsystem.
I checked that and found an interesting thing. Ultrascale+ (a.k.a zynqmp) has firmware and clock drivers in drivers/firmware/xilinx and drivers/clk/zynqmp/clkc.c. However, these drivers bind to nothing. There is no node with xlnx,zynqmp-firmware compatible in zynqmp DTS files and clock nodes use fixed-clock compatible only instead of xlnx,zynqmp-clk. These bindings are even documented in Documentation/devicetree/bindings/firmware/xilinx/xlnx,zynqmp-firmware.txt. This looks like some unfinished work.
Nevertheless, these two drivers look pretty decent. It might be worth checking if they work with current Xilinx firmware release. I do not see any CoreSight/debug related things in the SMC API exposed by ATF though, so direct memory access is the only option to enable TSGEN on this platform, at least at the moment.
Thanks, Wojciech
-
Al Grant -
Mathieu Poirier -
Wojciech Żmuda