HI Kim

On 15 March 2017 at 05:38, Kim Phillips <kim.phillips@arm.com> wrote:
[re-adding cc list, assuming didn't hit reply-all?]

​Thanks - button fatigue!​
 

On Tue, 14 Mar 2017 19:12:08 +0000
Mike Leach <mike.leach@linaro.org> wrote:

> On 14 March 2017 at 16:10, Kim Phillips <kim.phillips@arm.com> wrote:
>
> > still results in:
> >
> > util/cs-etm.c:1466:27: error: ‘cs_etm_global_header_fmts’ defined but not
> > used [-Werror=unused-const-variable=]
> >  static const char * const cs_etm_global_header_fmts[] = {
> >                            ^~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > What toolchain and/or version are you using?
> >
> ​I'm using a linaro build of gcc 4.9.
>
> ​aarch64-linux-gnu-gcc (Linaro GCC 4.9-2015.05) 4.9.3 20150413 (prerelease)

Linaro appear to have removed that release from their repo:

http://releases.linaro.org/components/toolchain/binaries/

So I used this one - which AFAICT is the closest to your version - to
cross-build both the kernel with your config, and perf:

aarch64-linux-gnu-gcc (Linaro GCC 4.9-2016.02) 4.9.4 20151028 (prerelease)

Building perf didn't require the patch I sent due to an old gcc bug
that apparently finally got fixed recently:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=28901

I then ran the resulting kernel (which still had the extra patch, so
commit 0d15341 == c50837 + the patch), and perf:

root@juno:~# ./perf --version
perf version 4.11.rc1.gc50837
root@juno:~# strings -a perf | grep "GCC: ("
GCC: (Linaro GCC 4.9-2016.02) 4.9.4 20151028 (prerelease)
root@juno:~# dmesg | grep gcc
[    0.000000] Linux version 4.11.0-rc1-g0d15341 (kim@dupont) (gcc version 4.9.4 20151028 (prerelease) (Linaro GCC 4.9-2016.02) ) #3 SMP PREEMPT Tue Mar 14 22:41:34 CDT 2017
root@juno:~# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
[  870.355660] coresight-replicator-qcom 20120000.replicator: REPLICATOR enabled
[  870.362736] coresight-funnel 20150000.funnel: FUNNEL inport 0 enabled
[  870.369127] coresight-tmc 20010000.etf: TMC-ETF enabled
[  870.374304] coresight-funnel 20040000.funnel: FUNNEL inport 0 enabled
[  870.380698] coresight-funnel 220c0000.funnel: FUNNEL inport 1 enabled
[  870.387858] coresight-funnel 220c0000.funnel: FUNNEL inport 1 disabled
[  870.394325] coresight-funnel 20040000.funnel: FUNNEL inport 0 disabled
[  870.400806] coresight-tmc 20010000.etf: TMC disabled
[  870.405722] coresight-funnel 20150000.funnel: FUNNEL inport 0 disabled
[  870.412184] coresight-replicator-qcom 20120000.replicator: REPLICATOR disabled
[  870.419350] coresight-tmc 20070000.etr: TMC-ETR disabled
[  870.425083] coresight-replicator-qcom 20120000.replicator: REPLICATOR enabled
[  870.432153] coresight-funnel 20150000.funnel: FUNNEL inport 0 enabled
[  870.438542] coresight-tmc 20010000.etf: TMC-ETF enabled
[  870.443718] coresight-funnel 20040000.funnel: FUNNEL inport 0 enabled
[  870.450112] coresight-funnel 220c0000.funnel: FUNNEL inport 1 enabled
Linux
[  870.476156] coresight-funnel 220c0000.funnel: FUNNEL inport 1 disabled
[  870.482625] coresight-funnel 20040000.funnel: FUNNEL inport 0 disabled
[  870.489106] coresight-tmc 20010000.etf: TMC disabled
[  870.494023] coresight-funnel 20150000.funnel: FUNNEL inport 0 disabled
[  870.500485] coresight-replicator-qcom 20120000.replicator: REPLICATOR disabled
[  870.507651] coresight-tmc 20070000.etr: TMC-ETR disabled
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.015 MB perf.data ]

which appears to have worked the first time.

Then I tried a second time - literally up-arrow, enter - and got the
same exact hard hang as I get with the modern compilers the first time:

root@juno:~# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
[ 1965.355162] coresight-replicator-qcom 20120000.replicator: REPLICATOR enabled
[ 1965.362238] coresight-funnel 20150000.funnel: FUNNEL inport 0 enabled
[ 1965.368629] coresight-tmc 20010000.etf: TMC-ETF enabled
[ 1965.373807] coresight-funnel 20040000.funnel: FUNNEL inport 0 enabled
[ 1965.380201] coresight-funnel 220c0000.funnel: FUNNEL inport 1 enabled
Linux
[ 1965.405984] coresight-funnel 220c0000.funnel: FUNNEL inport 1 disabled
[ 1965.412453] coresight-funnel 20040000.funnel: FUNNEL inport 0 disabled
[ 1965.418934] coresight-tmc 20010000.etf: TMC disabled
[ 1965.423850] coresight-funnel 20150000.funnel: FUNNEL inport 0 disabled
[ 1965.430312] coresight-replicator-qcom 20120000.replicator: REPLICATOR disabled
[ 1965.437478] coresight-tmc 20070000.etr: TMC-ETR disabled
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote

which is along the same instability lines as the other time that
execution behaviour differed depending on whether it executed first or
not...

> > > > $ sudo taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u
> > --per-thread
> > > taskset -c 2 uname
> > > > failed to mmap with 12 (Cannot allocate memory)
> > >
> > > That said I get this too if I do enable sinks. However as you say after
> > the
> > > initial attempt the problem disappears.
> >
> > Right, at least we have one problem reproduced on both sides now.

...here.

> > It hung before completing that last sentence.
> >
> > Perhaps the bug is exasperated by toolchain and/or host and target
> > rootfs distribution differences?  My juno target runs debian, and I
> > recently upgraded to stretch in order to get a version of gcc that
> > would support autoFDO.
>
> ​My juno is has a debian-jessie-developer root-fs from ​
> http://releases.linaro.org/debian/images/developer-arm64/16.04/
> At present I cannot build autofdo - I still have some package issues, but
> for perf and the kernel I am x-compiling on my ubuntu VM anyway, so the
> installed compilers have no effect on the perf runs,.

OK, can you try a more modern toolchain please?  The one you're using
isn't available anymore, AutoFDO requires gcc 5 and higher, and you're
not seeing the build failures others see, but most importantly, it
should make it easier to reproduce the hard lockup: at least that's
the case on the Juno r2.


​OK - tried the linaro 5.3 gcc package - this is the latest recommended compiler from the ARM connected community page for using the Linaro deliverables with Juno:-

aarch64-linux-gnu-gcc (Linaro GCC 5.3-2016.05) 5.3.1 20160412
Copyright (C) 2015 Free Software Foundation, Inc.

Built kernel and perf.

Ran a couple of times to make sure - but no issues.
=========================================================
​root@linaro-developer:/home/mleach/perf-tools# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/perf-tools# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.076 MB perf.data ]
root@linaro-developer:/home/mleach/perf-tools# strings -a perf | grep "GCC: ("
GCC: (Linaro GCC 5.3-2016.05) 5.3.1 20160412
root@linaro-developer:/home/mleach/perf-tools#
=========================================================

​Next tried the 6.2 toolset from linaro.​

mleach@ubu-16-04-vm:~/work/opencsd-perf-4.9-10$ ${CROSS_COMPILE}gcc --version
aarch64-linux-gnu-gcc (Linaro GCC 6.2-2016.11) 6.2.1 20161016
Copyright (C) 2016 Free Software Foundation, Inc.

​Built kernel and perf - so build issue on perf, applied your patch. (Sorry, I missed the question a couple of e-mails ago - it can be submitted to this list).

Checked gcc version and ran perf a few times
==============================================================
​root@linaro-developer:/home/mleach/perf-tools# strings -a perf | grep "GCC: ("
GCC: (Linaro GCC 6.2-2016.11) 6.2.1 20161016
root@linaro-developer:/home/mleach/perf-tools#

root@linaro-developer:/home/mleach/perf-tools# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/perf-tools#
root@linaro-developer:/home/mleach/perf-tools# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/perf-tools# taskset -c 2 ./perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 uname
Linux
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.077 MB perf.data ]
root@linaro-developer:/home/mleach/perf-tools#
==============================================================

​Worked fine.​

​so....I'm back on the funnel / trace amount and core differences.
taskset -c 2 -> selects cpu 2 -> A53(0) on both juno-r1 and juno-r2. So why on your system is the funnel for the A72 cluster (cpu 0-1) being activated.
if I use taskset -c 0 (or 1) then I get the A57 cluster funnel activated as expected. (2-5 gets the A53 cluster funnel)

This seems to be indicative of some .dts issue on your system / r2. Nothing immediately obvious in the .dts that I can see. 


 
Thanks,

Kim

​Mike​


--
Mike Leach
Principal Engineer, ARM Ltd.
Blackburn Design Centre. UK