Hi Kim,

After further consideration, the difference is explained by running on a different cluster. Mines using a core on the A53 cluster, yours on the A57/A72.
But still not an explanation for the hard crash you are seeing.

Mike


On 14 March 2017 at 13:10, Mike Leach <mike.leach@linaro.org> wrote:
Hi Kim,

On 14 March 2017 at 02:40, Kim Phillips <kim.phillips@arm.com> wrote:
> On Mon, 13 Mar 2017 09:29:14 -0600
> Mathieu Poirier <mathieu.poirier@linaro.org> wrote:
>
>> On 10 March 2017 at 18:12, Kim Phillips <kim.phillips@arm.com> wrote:
>> > BTW, these are in linux-next yet missing from Linus' ToT:
>>
>> "ToT" ?  I'm afraid I don't understand.  Can you be more specific?
>
> ToT == Top of Tree, i.e., the head of Linus' current master branch.
>
>> > $ sudo $PERF record -o perf.data -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 sort_O3
>> > Bubble sorting array of 30000 elements
>> > 17527 ms
>> > [ perf record: Woken up 2 times to write data ]
>> > [ perf record: Captured and wrote 2.799 MB perf.data ]
>> >
>> > Then the system locks up hard.  No response on any terminal including
>> > the serial console.
>>
>> Please try again with branch perf-opencsd-4.11-rc1.  Mike fixed a
>> problem with error code handling a couple of weeks ago.  With a little
>> bit of good will from the universe, this will be the same problem.
>
> $ ./perf --version
> perf version 4.11.rc1.ga5e89a5
> $ git log --oneline -2   #perf needed the one patch at the bottom of this email
> a5e89a55d8b8 perf tool: cs-etm: fix build error by removing unused cs_etm_global_header_fmts array
> c50837387458 perf: cs-etm: Add decoder reset to stub

No sure why you need the added patch here - I can build without it from the opencsd-perf-4.11-rc1 branch.

> $ uname -a   # kernel & perf commit IDs match except from that one commit
> Linux juno 4.11.0-rc1-gc508373 #1 SMP PREEMPT Mon Mar 13 14:31:40 CDT 2017 aarch64 GNU/Linux
> $ for i in /sys/bus/coresight/devices/*/enable_sink; do echo 1 | sudo tee $i; done
> 1
> 1
> 1
> 1

As mentioned previously - you do not need to enable any sinks when using perf.

> $ sudo taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
> failed to mmap with 12 (Cannot allocate memory)

That said I get this too if I do enable sinks. However as you say after the initial attempt the problem disappears.

> $ sudo taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
> Linux
> [ perf record: Woken up 2 times to write data ]
> [ perf record: Captured and wrote 0.015 MB perf.data ]
>
> So it took one more invocation to get past the failed mmap, then it
> hard-hung the board, as usual.  These messages were emitted on the
> serial console right before the lockup:
>
> [ 2528.475156] coresight-replicator-qcom 20120000.replicator: REPLICATOR enabled
> [ 2528.482232] coresight-funnel 20150000.funnel: FUNNEL inport 0 enabled
> [ 2528.488623] coresight-tmc 20010000.etf: TMC-ETF enabled
> [ 2528.493801] coresight-funnel 20040000.funnel: FUNNEL inport 0 enabled
> [ 2528.500195] coresight-funnel 220c0000.funnel: FUNNEL inport 1 enabled
> [ 2528.526213] coresight-funnel 220c0000.funnel: FUNNEL inport 1 disabled
> [ 2528.532681] coresight-funnel 20040000.funnel: FUNNEL inport 0 disabled
> [ 2528.539163] coresight-tmc 20010000.etf: TMC disabled
> [ 2528.544079] coresight-funnel 20150000.funnel: FUNNEL inport 0 disabled
> [ 2528.550542] coresight-replicator-qcom 20120000.replicator: REPLICATOR disabled
> [ 2528.557708] coresight-tmc 20070000.etr: TMC-ETR disabled
>


OK, I get no hard lockups here just a nice normal perf record run......

=====================================================
root@linaro-developer:/home/mleach/auto-fdo/example# ../../perf-tools/perf -v
perf version 4.11.rc1.gc50837
root@linaro-developer:/home/mleach/auto-fdo/example# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
taskset: failed to execute ./perf: No such file or directory
root@linaro-developer:/home/mleach/auto-fdo/example# taskset -c 2 ../../perf-tools/perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 2 times to write data ]

[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/auto-fdo/example#
root@linaro-developer:/home/mleach/auto-fdo/example# for i in /sys/bus/coresight/devices/*/enable_sink; do echo 1 | sudo tee $i; done
1
1
1
1
root@linaro-developer:/home/mleach/auto-fdo/example# taskset -c 2 ../../perf-tools/perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
failed to mmap with 12 (Cannot allocate memory)
root@linaro-developer:/home/mleach/auto-fdo/example# taskset -c 2 ../../perf-tools/perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/auto-fdo/example#
=====================================================

Same if I use sudo .... rather than root....

=====================================================
mleach@linaro-developer:~/auto-fdo/example$ sudo taskset -c 2 ../../perf-tools/perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname -a
Linux linaro-developer 4.11.0-rc1-gdd18d85-dirty #17 SMP PREEMPT Tue Mar 14 11:18:30 GMT 2017 aarch64 GNU/Linux
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.077 MB perf.data ]
mleach@linaro-developer:~/auto-fdo/example$
=====================================================

However, when I look at the CS component output from the serial console I am seeing a difference....

=====================================================
[ 1866.398817] coresight-replicator-qcom 20120000.replicator: REPLICATOR enabled
[ 1866.405886] coresight-funnel 20150000.funnel: FUNNEL inport 0 enabled
[ 1866.412264] coresight-tmc 20010000.etf: TMC-ETF enabled
[ 1866.417438] coresight-funnel 20040000.funnel: FUNNEL inport 1 enabled
[ 1866.423816] coresight-funnel 230c0000.funnel: FUNNEL inport 0 enabled
​​

[ 1866.435520] coresight-funnel 230c0000.funnel: FUNNEL inport 0 disabled
[ 1866.441990] coresight-funnel 20040000.funnel: FUNNEL inport 1 disabled
[ 1866.448456] coresight-tmc 20010000.etf: TMC disabled
[ 1866.453373] coresight-funnel 20150000.funnel: FUNNEL inport 0 disabled
[ 1866.459837] coresight-replicator-qcom 20120000.replicator: REPLICATOR disabled
=====================================================

​Check you are using the correct .dtb for your board revison. (juno.dtb, juno-r1.dtb, juno-r2.dtb).
Not sure how this would cause a hang, but it does explain the reduced amount of data you are capturing compared to my test runs.

Otherwise I have attached my .config in case there are key differences there.

Regards

Mike​

> So, perf-opencsd-4.11-rc1 doesn't work either / exhibits the same
> bad behaviour.  Even if I only echo 1 enable_sink
> file, .../20010000.etf/enable_sink, the machine still hangs.
>
> Thanks,
>
> Kim
>
>> > From a5e89a55d8b8d44caa9a9750304288776bb1573c Mon Sep 17 00:00:00 2001
>> > From: Kim Phillips <kim.phillips@arm.com>
>> > Date: Thu, 9 Mar 2017 21:39:51 -0600
>> > Subject: [PATCH] perf tool: cs-etm: fix build error by removing unused
>> >  cs_etm_global_header_fmts array
>> > MIME-Version: 1.0
>> > Content-Type: text/plain; charset=UTF-8
>> > Content-Transfer-Encoding: 8bit
>> >
>> > util/cs-etm.c:1466:27: error: ‘cs_etm_global_header_fmts’ defined but not used [-Werror=unused-const-variable=]
>> >  static const char * const cs_etm_global_header_fmts[] = {
>> >                            ^~~~~~~~~~~~~~~~~~~~~~~~~
>> > cc1: all warnings being treated as errors
>> > mv: cannot stat 'util/.cs-etm.o.tmp': No such file or directory
>> > /home/kim/git/OpenCSD/tools/build/Makefile.build:101: recipe for target 'util/cs-etm.o' failed
>> > make[4]: *** [util/cs-etm.o] Error 1
>> > make[4]: *** Waiting for unfinished jobs....
>> > /home/kim/git/OpenCSD/tools/build/Makefile.build:144: recipe for target 'util' failed
>> > make[3]: *** [util] Error 2
>> > Makefile.perf:526: recipe for target 'libperf-in.o' failed
>> > make[2]: *** [libperf-in.o] Error 2
>> > make[2]: *** Waiting for unfinished jobs....
>> > Makefile.perf:207: recipe for target 'sub-make' failed
>> > make[1]: *** [sub-make] Error 2
>> > Makefile:68: recipe for target 'all' failed
>> > make: *** [all] Error 2
>> > make: Leaving directory '/home/kim/git/OpenCSD/tools/perf'
>> >
>> > Signed-off-by: Kim Phillips <kim.phillips@arm.com>
>> > ---
>> >  tools/perf/util/cs-etm.c | 6 ------
>> >  1 file changed, 6 deletions(-)
>> >
>> > diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c
>> > index 70825e7..5ff3c3e 100644
>> > --- a/tools/perf/util/cs-etm.c
>> > +++ b/tools/perf/util/cs-etm.c
>> > @@ -1463,12 +1463,6 @@ static int cs_etm__process_auxtrace_event(struct perf_session *session,
>> >
>> >  }
>> >
>> > -static const char * const cs_etm_global_header_fmts[] = {
>> > -  [CS_HEADER_VERSION_0]    = "   Header version                 %"PRIx64"\n",
>> > -  [CS_PMU_TYPE_CPUS]       = "   PMU type/num cpus              %"PRIx64"\n",
>> > -  [CS_ETM_SNAPSHOT]        = "   Snapshot                       %"PRIx64"\n",
>> > -};
>> > -
>> >  static const char * const cs_etm_priv_fmts[] = {
>> >    [CS_ETM_MAGIC]           = "   Magic number                   %"PRIx64"\n",
>> >    [CS_ETM_CPU]             = "   CPU                            %"PRIx64"\n",
>> > --
>> > 2.10.2
> _______________________________________________
> CoreSight mailing list
> CoreSight@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/coresight



--
Mike Leach
Principal Engineer, ARM Ltd.
Blackburn Design Centre. UK



--
Mike Leach
Principal Engineer, ARM Ltd.
Blackburn Design Centre. UK