Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 + quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call JetPack, based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the --per-thread parameter I'm getting the: failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use mainline or older, you can trace only single-thread applications in per-thread mode. Skipping --per-thread flag is equivalent to specifying --all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2 patch set has just appeared on this list. You can try to cherry-pick their patches on your branch. Then, you will be able to use --cpu <n> and --all-cpus. I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using Linux 5.1-rc1 (cloned about a week ago) with CPU-wide tracing patch set by Linaro. CoreSight development goes rapidly, so it's a good idea to keep up with this list and Linux mainline changelog, because interesting changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores don't have CoreSight, that probably means they don't have ETMs connected, thus they will not generate any trace data. I don't know if Linux can schedule processes on them. If not, I see no harm - your program will be traced no matter what CPU kernel puts it on. If it can, the worst case would be empty trace. You can save yourself from the latter by scheduling your process with taskset to the main cores only, omitting Denvers. But again, it's just my educated guess and I'd wait for the second opinion of the wise people residing here.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
On Tue, 26 Mar 2019 at 10:54, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 + quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call JetPack, based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the --per-thread parameter I'm getting the: failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use mainline or older, you can trace only single-thread applications in per-thread mode. Skipping --per-thread flag is equivalent to specifying --all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2 patch set has just appeared on this list. You can try to cherry-pick their patches on your branch. Then, you will be able to use --cpu <n> and --all-cpus. I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using Linux 5.1-rc1 (cloned about a week ago) with CPU-wide tracing patch set by Linaro. CoreSight development goes rapidly, so it's a good idea to keep up with this list and Linux mainline changelog, because interesting changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores don't have CoreSight, that probably means they don't have ETMs connected, thus they will not generate any trace data. I don't know if Linux can schedule processes on them. If not, I see no harm - your program will be traced no matter what CPU kernel puts it on. If it can, the worst case would be empty trace. You can save yourself from the latter by scheduling your process with taskset to the main cores only, omitting Denvers. But again, it's just my educated guess and I'd wait for the second opinion of the wise people residing here.
Everything pointed out by Wojciech in the above reply is accurate.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Hi Mathieu,
I took your latest changes from 5.1-rc2-cpu-wide-v2 https://git.linaro.org/people/mathieu.poirier/coresight.git/log/?h=5.1-rc2-cpu-wide-v2 but this broke the functionality completely for me. Now when I try to issue:
perf record -e cs_etm/@8030000.etf/u -a uname I get this response:
failed to set sink "8030000.etf" on event cs_etm/@8030000.etf/u with 2 (No such file or directory).
The previous version of the perf still works fine. Any ideas?
On Tue, Mar 26, 2019 at 8:19 PM Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Tue, 26 Mar 2019 at 10:54, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 +
quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call JetPack,
based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the
--per-thread parameter I'm getting the:
failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use
mainline or older, you can trace only
single-thread applications in per-thread mode. Skipping --per-thread
flag is equivalent to specifying
--all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2 patch
set has just appeared on this list.
You can try to cherry-pick their patches on your branch. Then, you will
be able to use --cpu <n> and --all-cpus.
I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process
with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using Linux
5.1-rc1 (cloned about a week ago)
with CPU-wide tracing patch set by Linaro. CoreSight development goes
rapidly, so it's a good idea to keep
up with this list and Linux mainline changelog, because interesting
changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this
system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores
don't have CoreSight, that probably means
they don't have ETMs connected, thus they will not generate any trace
data. I don't know if Linux can schedule
processes on them. If not, I see no harm - your program will be traced
no matter what CPU kernel puts it on.
If it can, the worst case would be empty trace. You can save yourself
from the latter by scheduling your process
with taskset to the main cores only, omitting Denvers. But again, it's
just my educated guess and I'd wait for
the second opinion of the wise people residing here.
Everything pointed out by Wojciech in the above reply is accurate.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On Wed, 27 Mar 2019 at 09:18, Srdjan Stokic srdjan.stokic@auroralabs.com wrote:
Hi Mathieu,
I took your latest changes from 5.1-rc2-cpu-wide-v2 but this broke the functionality completely for me. Now when I try to issue:
perf record -e cs_etm/@8030000.etf/u -a uname I get this response:
failed to set sink "8030000.etf" on event cs_etm/@8030000.etf/u with 2 (No such file or directory).
The previous version of the perf still works fine. Any ideas?
That is interesting... Especially when v2 doesn't do anything other than moving code around. This error is in user space and the code path propably never reached the kernel. Based on the error message you're getting I'm guessing it can't find the right sink under "/sys/bus/event_source/devices/cs_etm/sinks/". If "8030000.etf" doesn't appear under that directory things won't work.
Otherwise I'm afraid you'll need to instrument the code to find out where the error happens.
Please keep me posted on what you find, Mathieu.
On Tue, Mar 26, 2019 at 8:19 PM Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Tue, 26 Mar 2019 at 10:54, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 + quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call JetPack, based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the --per-thread parameter I'm getting the: failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use mainline or older, you can trace only single-thread applications in per-thread mode. Skipping --per-thread flag is equivalent to specifying --all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2 patch set has just appeared on this list. You can try to cherry-pick their patches on your branch. Then, you will be able to use --cpu <n> and --all-cpus. I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using Linux 5.1-rc1 (cloned about a week ago) with CPU-wide tracing patch set by Linaro. CoreSight development goes rapidly, so it's a good idea to keep up with this list and Linux mainline changelog, because interesting changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores don't have CoreSight, that probably means they don't have ETMs connected, thus they will not generate any trace data. I don't know if Linux can schedule processes on them. If not, I see no harm - your program will be traced no matter what CPU kernel puts it on. If it can, the worst case would be empty trace. You can save yourself from the latter by scheduling your process with taskset to the main cores only, omitting Denvers. But again, it's just my educated guess and I'd wait for the second opinion of the wise people residing here.
Everything pointed out by Wojciech in the above reply is accurate.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
You are right. Under "/sys/bus/event_source/devices/cs_etm/" I don't have a folder named sinks. But I can properly see all coresight devices under "/sys/bus/coresight/devices". I realized that I didn't take all the necessary patches from the latest kernel, but it seems like patching 4.9 Kernel is far too complicated at this point. (I'm not very skilled at this, my background is embedded C++ dev, not Linux kernel dev.) I'm guessing we'll have to wait for nVidia to update their L4T custom kernel to 5.0 or newer in order to have everything working properly.
On Wed, Mar 27, 2019 at 4:34 PM Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Wed, 27 Mar 2019 at 09:18, Srdjan Stokic srdjan.stokic@auroralabs.com wrote:
Hi Mathieu,
I took your latest changes from 5.1-rc2-cpu-wide-v2 but this broke the
functionality completely for me. Now when I try to issue:
perf record -e cs_etm/@8030000.etf/u -a uname I get this response:
failed to set sink "8030000.etf" on event cs_etm/@8030000.etf/u with 2
(No such file or directory).
The previous version of the perf still works fine. Any ideas?
That is interesting... Especially when v2 doesn't do anything other than moving code around. This error is in user space and the code path propably never reached the kernel. Based on the error message you're getting I'm guessing it can't find the right sink under "/sys/bus/event_source/devices/cs_etm/sinks/". If "8030000.etf" doesn't appear under that directory things won't work.
Otherwise I'm afraid you'll need to instrument the code to find out where the error happens.
Please keep me posted on what you find, Mathieu.
On Tue, Mar 26, 2019 at 8:19 PM Mathieu Poirier <
mathieu.poirier@linaro.org> wrote:
On Tue, 26 Mar 2019 at 10:54, Wojciech Żmuda wzmuda@n7space.com
wrote:
Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 +
quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call
JetPack, based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the
--per-thread parameter I'm getting the:
failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use
mainline or older, you can trace only
single-thread applications in per-thread mode. Skipping --per-thread
flag is equivalent to specifying
--all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2
patch set has just appeared on this list.
You can try to cherry-pick their patches on your branch. Then, you
will be able to use --cpu <n> and --all-cpus.
I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process
with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using
Linux 5.1-rc1 (cloned about a week ago)
with CPU-wide tracing patch set by Linaro. CoreSight development goes
rapidly, so it's a good idea to keep
up with this list and Linux mainline changelog, because interesting
changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this
system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores
don't have CoreSight, that probably means
they don't have ETMs connected, thus they will not generate any trace
data. I don't know if Linux can schedule
processes on them. If not, I see no harm - your program will be
traced no matter what CPU kernel puts it on.
If it can, the worst case would be empty trace. You can save yourself
from the latter by scheduling your process
with taskset to the main cores only, omitting Denvers. But again,
it's just my educated guess and I'd wait for
the second opinion of the wise people residing here.
Everything pointed out by Wojciech in the above reply is accurate.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On Fri, 29 Mar 2019 at 03:51, Srdjan Stokic srdjan.stokic@auroralabs.com wrote:
You are right. Under "/sys/bus/event_source/devices/cs_etm/" I don't have a folder named sinks. But I can properly see all coresight devices under "/sys/bus/coresight/devices". I realized that I didn't take all the necessary patches from the latest kernel, but it seems like patching 4.9 Kernel is far too complicated at this point. (I'm not very skilled at this, my background is embedded C++ dev, not Linux kernel dev.) I'm guessing we'll have to wait for nVidia to update their L4T custom kernel to 5.0 or newer in order to have everything working properly.
Thank you for the upate. Yes, back porting to 4.9 would be very tedious.
Mathieu
On Wed, Mar 27, 2019 at 4:34 PM Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Wed, 27 Mar 2019 at 09:18, Srdjan Stokic srdjan.stokic@auroralabs.com wrote:
Hi Mathieu,
I took your latest changes from 5.1-rc2-cpu-wide-v2 but this broke the functionality completely for me. Now when I try to issue:
perf record -e cs_etm/@8030000.etf/u -a uname I get this response:
failed to set sink "8030000.etf" on event cs_etm/@8030000.etf/u with 2 (No such file or directory).
The previous version of the perf still works fine. Any ideas?
That is interesting... Especially when v2 doesn't do anything other than moving code around. This error is in user space and the code path propably never reached the kernel. Based on the error message you're getting I'm guessing it can't find the right sink under "/sys/bus/event_source/devices/cs_etm/sinks/". If "8030000.etf" doesn't appear under that directory things won't work.
Otherwise I'm afraid you'll need to instrument the code to find out where the error happens.
Please keep me posted on what you find, Mathieu.
On Tue, Mar 26, 2019 at 8:19 PM Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Tue, 26 Mar 2019 at 10:54, Wojciech Żmuda wzmuda@n7space.com wrote:
Hi Srdjan,
Hello,
I have a Jetson TX2 board which has dual-core NVIDIA Denver2 + quad-core ARM Cortex-A57.
nVidia recently released a new SDK package (which they call JetPack, based on Ubuntu and 4.9 Kernel). out of the box it comes with coresight disabled, so I had to recompile the kernel in order to enable it. Coresight kind of works, I'm getting some data out of it, but I'm trying to use Perf in order to do the tracing.
The issue I'm facing is that when I execute:
perf record -e cs_etm/@8030000.etf/u --per-thread uname
it works properly, and I'm getting great reports, but if I omit the --per-thread parameter I'm getting the: failed to mmap with 12 (Cannot allocate memory)
Mainline Linux does not yet contain CPU-wide tracing. If you use mainline or older, you can trace only single-thread applications in per-thread mode. Skipping --per-thread flag is equivalent to specifying --all-cpus flag and this mode is not yet supported in mainline.
Mathieu and friends are working on CPU-wide tracing and their v2 patch set has just appeared on this list. You can try to cherry-pick their patches on your branch. Then, you will be able to use --cpu <n> and --all-cpus. I did that (with v1) and I can confirm it works well.
mmap error also happens if I want to record already running process with --pid parameter.
I tried that yesterday and it worked for me. Please mind I'm using Linux 5.1-rc1 (cloned about a week ago) with CPU-wide tracing patch set by Linaro. CoreSight development goes rapidly, so it's a good idea to keep up with this list and Linux mainline changelog, because interesting changes may appear any day.
If I specify the number of the cpu: perf record -e cs_etm/@8030000.etf/u --cpu 0 uname
then recording works but when I try to do perf report it says: 0x228 [0x40]: failed to process type: 7
What might be interesting to mention is that the processors on this system are arranged in weird combination. cpu 0,3,4,5 are ARM cores which have coresight on them, and cpu 1,2 are Denver cores without coresight. I'm suspecting that this might create some issues.
I'm not an expert, but I don't think it would matter. If Denver cores don't have CoreSight, that probably means they don't have ETMs connected, thus they will not generate any trace data. I don't know if Linux can schedule processes on them. If not, I see no harm - your program will be traced no matter what CPU kernel puts it on. If it can, the worst case would be empty trace. You can save yourself from the latter by scheduling your process with taskset to the main cores only, omitting Denvers. But again, it's just my educated guess and I'd wait for the second opinion of the wise people residing here.
Everything pointed out by Wojciech in the above reply is accurate.
Regards, Wojciech
Do you have any ideas on how to proceed to diagnose the issue?
Thank you.
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
-
Mathieu Poirier -
Srdjan Stokic -
Wojciech Żmuda