Hi there,
I worked on a few patches to the help the userspace perf tools do better at loading debug symbols -- this is particularly of interest on Ubuntu and Linaro where separated debug symbols in .ddebs are typically not loaded even when present. Note that even with the patches, the contents of .ddebs are still not cached by perf along with the other profiled binaries, so you will still lose the symbols if you uninstall/upgrade your ddebs in the meantime. This should get fixed in the future.
Due to mailserver issues, Arnaldo kindly helped merge these patches for me. The patches are now merged to the linux-2.6-tip master branch.
You can find details of the commits here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/608775
[ Nico: WIll says he's going to pull these patches into his linaro branch, if it's more convenient for you to pull from there. ]
Covering notes explaining the changes follow.
Cheers ---Dave
From c1cb454cc818d367e6b679600bbe7c2fc6d75c2c Mon Sep 17 00:00:00 2001
From: Dave Martin dave.martin@linaro.org Date: Tue, 27 Jul 2010 16:44:36 +0100 Subject: [PATCH 0/2] tools/perf: Fix debug symbol lookup issues for some OSes
On Ubuntu (and some other OSes) debug symbols for most of the binaries are split out into separately-installable files under /usr/lib/debug or elsewhere.
perf already supports this setup, but as a side-effect of abbreviating the prefix of paths beneath the current directory to './' for display purposes, the paths constructed to search for the separate debug images are incorrect.
As a result, symbols may not be found even if they are available.
The simplest fix seems to be just to remove the cwd abbreviation feature; this makes all displayed paths absolute in e.g., perf report -v, but otherwise appears harmless.
Dave Martin (2): tools/perf: Don't abbreviate file paths relative to the cwd tools/perf: Remove unneeded code for tracking the cwd in perf sessions
tools/perf/util/event.c | 2 +- tools/perf/util/map.c | 22 +--------------------- tools/perf/util/map.h | 2 +- tools/perf/util/session.c | 22 +--------------------- 4 files changed, 4 insertions(+), 44 deletions(-)
From fa6c691b767bfcdf9a704aeab58ddc6f618b9be8 Mon Sep 17 00:00:00 2001
From: Dave Martin dave.martin@linaro.org Date: Fri, 30 Jul 2010 12:31:46 +0100 Subject: [PATCH 0/3] perf tools: Better image search for separated debug images
The perf tools' buildid cache functionality currently interferes with loading the correct symbols in environments which use separated debug images (Ubuntu, Fedora, ...)
This patch set modifies the image search behaviour to perform 2 passes: no image without a full symtab is considered for loading unless no image with a symtab is available at all. This avoids loading images from the buildid cache if a better image is available elsewhere. As a consequence, perf report and friends should now load the correct symbols on Ubuntu and friends, at the cost of an extra image loading pass.
This patch set does not change the way files get cached, but ought to be OK as the basis for further refactoring to replicate the same search behaviour in the caching code so all relevant images get cached.
The extra image loading pass should not need to occur on the common path once the caching behaviour modifications are done.
Dave Martin (3): perf tools: Factor out buildid reading and make it implicit in dso__load perf tools: remove extra build-id check factored into dso__load perf tools: improve debug image search when loading symbols
tools/perf/util/symbol.c | 206 +++++++++++++++++++++++++-------------------- 1 files changed, 114 insertions(+), 92 deletions(-)
On Mon, 2010-08-02 at 10:55 +0100, Dave Martin wrote:
Hi there,
I worked on a few patches to the help the userspace perf tools do better at loading debug symbols -- this is particularly of interest on Ubuntu and Linaro where separated debug symbols in .ddebs are typically not loaded even when present. Note that even with the patches, the contents of .ddebs are still not cached by perf along with the other profiled binaries, so you will still lose the symbols if you uninstall/upgrade your ddebs in the meantime. This should get fixed in the future.
Due to mailserver issues, Arnaldo kindly helped merge these patches for me. The patches are now merged to the linux-2.6-tip master branch.
You can find details of the commits here:
Thanks for passing this along! I am interested in figuring out whether we should backport the patches to Ubuntu Maverick.
I took a look at the first two patches that deal with the cwd issue. Although it fixes a real bug, I'm not sure it's worthwhile to pull it into the Ubuntu Maverick kernel because it's rather esoteric. For those unaware, the bug is triggered by starting a perf record while inside a directory along the path to the executable being recorded. For example:
/bin # perf record ls /bin # pert report
The report will work but it won't find the debug symbols in /usr/lib/debug/bin/ls. Moving to a different directory, like /tmp, while recording is a work around.
Since the bug is likely never to be seen except when running perf inside system directories (which doesn't seem like a good usage scenario), I'm inclined to recommend not backporting this for Ubuntu.
However, I still don't fully understand the issue surrounding the buildid cache bug fix. Can you provide more details on the issue this is fixing? It may be something we want to consider for Maverick, but I need more info to know what's going on.
Thanks,
-- Chase
Hi there,
On Mon, Aug 2, 2010 at 10:10 PM, Chase Douglas chase.douglas@canonical.com wrote:
On Mon, 2010-08-02 at 10:55 +0100, Dave Martin wrote:
[...]
Thanks for passing this along! I am interested in figuring out whether we should backport the patches to Ubuntu Maverick.
Certainly would be nice to have, if there are developers who'd like to use the tools. The perf tools look like a good asset to have, but unpatched, they are not very usable on Ubuntu yet (see below).
The patches only affect the userspace perf tools, so there's no risk of destabilising the kernel etc. by applying the patches.
I took a look at the first two patches that deal with the cwd issue. Although it fixes a real bug, I'm not sure it's worthwhile to pull it into the Ubuntu Maverick kernel because it's rather esoteric. For those
[...]
Since the bug is likely never to be seen except when running perf inside system directories (which doesn't seem like a good usage scenario), I'm inclined to recommend not backporting this for Ubuntu.
Indeed--- I hit this a lot when trying to debug the perf tools ;) but for most people it will be much more uncommon. When profiling an app from inside its own build tree, the debug will usually not have been separated out already so it should work fine.
So it seems reasonable to allow this change to propagate in its own time via mainline.
However, I still don't fully understand the issue surrounding the buildid cache bug fix. Can you provide more details on the issue this is fixing? It may be something we want to consider for Maverick, but I need more info to know what's going on.
perf cleverly caches binaries during perf record, to ensure that later analysis definitely uses the right symbols etc., even if binaries were rebuilt / upgraded / removed in the meantime.
Unfortunately, it doesn't cleverly cache separate debug images yet (i.e., ddebs contents). So when searching for symbols during perf report and friends, it hits in the cache and uses the stripped images which were observed by perf record. The result is that perf report will only see symbols from .dynsym even if the ddebs are available --- usually inadequate/misleading for profiling purposes.
The patch provides a workaround which looks in locations like /usr/lib/debug anyway, if no matching image is found in the cache with a non-empty .symtab. This should make the tools signifcantly more usable on Ubuntu. Without the patch, the workaround is to pass --no-buildid-cache to perf record, and to make sure there's no stale cache content when running the other tools (to keep deleting ~/.debug is the easiest way to ensure this -- perf buildid-cache is currently buggy)
Unfortunately, I hit another bug which causes code symbol locations to get skewed in non-PIE executable images --- see follow-ups (starting [PATCH 0/2] perf: symbol offset breakage with separated debug, to lkml and kernel-team@ubuntu). I believe there is no workaround for this other than applying the patch... or building everything as PIE ;P
Cheers ---Dave
-
Chase Douglas -
Dave Martin