For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
1) The source package's control file must contain a Vcs-Git entry
2) The Vcs-Git entry must reference a git tree which contains the _exact_ source code which appears in the source package.
* Such a tree must exist and must be publicly readable. It does not have to be on git.linaro.org (though this is the recommended place).
* Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
* Manufacturing the source package from the contents of multiple repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
* Referring to a tree which does not contain the whole contents of the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
3) Tagging of packaged kernels must be done in a standard way.
* I recommend <source package name>_<package version>, matching the Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
* Tree maintainers are of course free to add any other additional tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
* We _could_ standardise the following, but it is not essential:
* ubuntu/<release>: The tagged source for the _original_ kernel which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
4) No specific branch naming requirements exist.
* Release tags do not necessarily need to appear on any branch.
* We _could_ standardise the following, but it is not essential:
* ubuntu/latest - the tagged packaged source for the most recent kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
On Fri, Jan 13, 2012 at 6:32 AM, Dave Martin dave.martin@linaro.org wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
...
This seems reasonable to me. This month we are adding CI auto packaging/building for LT kernels so I will add the Vcs-Git setting to these CI scripts.
On Fri, Jan 13, 2012 at 3:52 PM, John Rigby john.rigby@linaro.org wrote:
On Fri, Jan 13, 2012 at 6:32 AM, Dave Martin dave.martin@linaro.org wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
...
This seems reasonable to me. This month we are adding CI auto packaging/building for LT kernels so I will add the Vcs-Git setting to these CI scripts.
I think at the moment, we're just pasting it from the vanilla kernel packaging, which is why we end up with a pointer to the wrong tree (but that's partly guesswork on my part)
Do you know if the TI packaged kernels already in a tree somewhere? I had someone trying to find that specifically.
Cheers ---Dave
The CI autopackaged TI 3.1 kernels all land here: https://github.com/jcrigby/packaged-linux-linaro-3.1-ci/commits/lt-omap/
All 3.1 LT and non LT kernel land there in different branches but each branch gets overwritten with each new merge so you need the tag to find a particular release.
On Fri, Jan 13, 2012 at 9:50 AM, Dave Martin dave.martin@linaro.org wrote:
On Fri, Jan 13, 2012 at 3:52 PM, John Rigby john.rigby@linaro.org wrote:
On Fri, Jan 13, 2012 at 6:32 AM, Dave Martin dave.martin@linaro.org wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
...
This seems reasonable to me. This month we are adding CI auto packaging/building for LT kernels so I will add the Vcs-Git setting to these CI scripts.
I think at the moment, we're just pasting it from the vanilla kernel packaging, which is why we end up with a pointer to the wrong tree (but that's partly guesswork on my part)
Do you know if the TI packaged kernels already in a tree somewhere? I had someone trying to find that specifically.
Cheers ---Dave
Linaro Android kernels are very easy to find
Browse to:
https://android-build.linaro.org/builds/~linaro-android/staging-origen/#buil...
Click on Downloads
You'll see:
http://snapshots.linaro.org/android/~linaro-android/staging-origen/142/sourc...
and
http://snapshots.linaro.org/android/~linaro-android/staging-origen/142/pinne...
The source-manifest contains:
<remote name="linaro-other" fetch="git://git.linaro.org/" review="review.android.git.linaro.org"/> <project path="kernel" name="landing-teams/working/samsung/kernel" remote="linaro-other" revision="android"/>
For the specific commit that got built in our CI loop use the pinned-manifest.xml:
<project name="landing-teams/working/samsung/kernel" path="kernel" remote="linaro-other" revision="41824b7e304cb4022dbcf893b7b0db272c2ab374"/>
Just
git clone git://git.linaro.org/landing-teams/working/samsung/kernel
Get the toolchain
wget http://android-build.linaro.org/builds/~linaro-android/toolchain-4.6-bzr/141...
tar -jxvf android-toolchain-eabi-4.6-*
Get the config
wget --no-check-certificate https://android-build.linaro.org/jenkins/job/linaro-android_staging-origen/1... or click on Raw
Get the build commands:
cat consoleText | grep ARCH=arm
Fix up the toolchain and bingo!
Amit is putting this all into a per build script, which you'll just download and run.
Enjoy!
On 13 January 2012 07:32, Dave Martin dave.martin@linaro.org wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
The source package's control file must contain a Vcs-Git entry
The Vcs-Git entry must reference a git tree which contains the
_exact_ source code which appears in the source package.
* Such a tree must exist and must be publicly readable. It does not have to be on git.linaro.org (though this is the recommended place).
* Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
* Manufacturing the source package from the contents of multiple repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
* Referring to a tree which does not contain the whole contents of the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
- Tagging of packaged kernels must be done in a standard way.
* I recommend <source package name>_<package version>, matching the Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
* Tree maintainers are of course free to add any other additional tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
* We _could_ standardise the following, but it is not essential:
* ubuntu/<release>: The tagged source for the _original_ kernel which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
- No specific branch naming requirements exist.
* Release tags do not necessarily need to appear on any branch.
* We _could_ standardise the following, but it is not essential:
* ubuntu/latest - the tagged packaged source for the most recent kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
I was specifically after solving the problem for the debianised kernels. Android is an analogous but completely separate problem -- if you already have a solution there then great.
https://android-build.linaro.org/builds/~linaro-android/staging-origen/#buil...
I assume people can tell from their deliverables which build to go to? Is the URL obvious / guessable?
Otherwise, it looks like all the needed info is there, though you're right that scripting could make it more convenient to access.
Cheers ---Dave
On 13 January 2012 11:36, Dave Martin dave.martin@linaro.org wrote:
I was specifically after solving the problem for the debianised kernels. Android is an analogous but completely separate problem -- if you already have a solution there then great.
https://android-build.linaro.org/builds/~linaro-android/staging-origen/#buil...
I assume people can tell from their deliverables which build to go to? Is the URL obvious / guessable?
Its not obvious, but we're going to change that. Right now people have to ask.
Otherwise, it looks like all the needed info is there, though you're right that scripting could make it more convenient to access.
Cheers ---Dave
On Fri, Jan 13, 2012 at 5:42 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 13 January 2012 11:36, Dave Martin dave.martin@linaro.org wrote:
I was specifically after solving the problem for the debianised kernels. Android is an analogous but completely separate problem -- if you already have a solution there then great.
https://android-build.linaro.org/builds/~linaro-android/staging-origen/#buil...
I assume people can tell from their deliverables which build to go to? Is the URL obvious / guessable?
Its not obvious, but we're going to change that. Right now people have to ask.
Cool; I guess once that's sorted it should work.
---Dave
Hi again,
After a report of yet another instance of un-findable source for a kernel released from a landing team, it would be good if we could move forward with this.
Does anyone have any significant disagreements with the proposal below?
If not, I can try to write up a formal specification somewhere.
Can we then plan to implement it?
If anyone has any preference for the common prefix for tag names, please speak up (otherwise we will proceed with the "ubuntu/" prefix).
Cheers ---Dave
On Fri, Jan 13, 2012 at 01:32:35PM +0000, Dave Martin wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
The source package's control file must contain a Vcs-Git entry
The Vcs-Git entry must reference a git tree which contains the
_exact_ source code which appears in the source package.
- Such a tree must exist and must be publicly readable. It does not
have to be on git.linaro.org (though this is the recommended place).
- Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not
acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
- Manufacturing the source package from the contents of multiple
repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
- Referring to a tree which does not contain the whole contents of
the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
Note: this means that the released binary packages must be reproducible from the tagged source using standard package build mechanisms, to the extent that the exact versions of build tools and other build-time dependencies used to build the originally released binaries are still avilable.
- Tagging of packaged kernels must be done in a standard way.
- I recommend <source package name>_<package version>, matching the
Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
- Tree maintainers are of course free to add any other additional
tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
We _could_ standardise the following, but it is not essential:
- ubuntu/<release>: The tagged source for the _original_ kernel
which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
- No specific branch naming requirements exist.
Release tags do not necessarily need to appear on any branch.
We _could_ standardise the following, but it is not essential:
- ubuntu/latest - the tagged packaged source for the most recent
kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
All released kernel source is available on git.linaro.org.
Specifically: Git: git://git.linaro.org/landing-teams/leb/<landing_team>/kernel.git HTML: http://git.linaro.org/git/landing-teams/leb/<landing_team>/kernel.git Gitweb: http://git.linaro.org/gitweb?p=landing-teams/leb/<landing_team>/kernel.git;a=summary
The Gitweb has a description of what you can expect from each tag.
Kind regards, Lee
On 24/01/12 10:18, Dave Martin wrote:
Hi again,
After a report of yet another instance of un-findable source for a kernel released from a landing team, it would be good if we could move forward with this.
Does anyone have any significant disagreements with the proposal below?
If not, I can try to write up a formal specification somewhere.
Can we then plan to implement it?
If anyone has any preference for the common prefix for tag names, please speak up (otherwise we will proceed with the "ubuntu/" prefix).
Cheers ---Dave
On Fri, Jan 13, 2012 at 01:32:35PM +0000, Dave Martin wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
The source package's control file must contain a Vcs-Git entry
The Vcs-Git entry must reference a git tree which contains the
_exact_ source code which appears in the source package.
- Such a tree must exist and must be publicly readable. It does not
have to be on git.linaro.org (though this is the recommended place).
- Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not
acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
- Manufacturing the source package from the contents of multiple
repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
- Referring to a tree which does not contain the whole contents of
the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
Note: this means that the released binary packages must be reproducible from the tagged source using standard package build mechanisms, to the extent that the exact versions of build tools and other build-time dependencies used to build the originally released binaries are still avilable.
- Tagging of packaged kernels must be done in a standard way.
- I recommend <source package name>_<package version>, matching the
Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
- Tree maintainers are of course free to add any other additional
tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
We _could_ standardise the following, but it is not essential:
- ubuntu/<release>: The tagged source for the _original_ kernel
which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
- No specific branch naming requirements exist.
Release tags do not necessarily need to appear on any branch.
We _could_ standardise the following, but it is not essential:
- ubuntu/latest - the tagged packaged source for the most recent
kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
On Tue, Jan 24, 2012 at 11:34:51AM +0000, Lee Jones wrote:
All released kernel source is available on git.linaro.org.
Specifically: Git: git://git.linaro.org/landing-teams/leb/<landing_team>/kernel.git HTML: http://git.linaro.org/git/landing-teams/leb/<landing_team>/kernel.git Gitweb: http://git.linaro.org/gitweb?p=landing-teams/leb/<landing_team>/kernel.git;a=summary
The Gitweb has a description of what you can expect from each tag.
Good, but I fear it's not enough.
The question was never whether the source exists somewhere in infospace, but whether a linaro consumer can find it and make use of it. Lets see how they get on in this typical scenario:
Suppose we have a board with linaro that we set up a while ago. We are experiencing some issues, and want to turn on some instrumentation in the kernel and/or add some patches of their own. This means we would like to reconfigure/change and rebuild the kernel that we have installed.
1) We know how to get source code on Ubuntu: run apt-get source
But, that specific package version may have disappeared from the server (Fixable by running apt-get upgrade, but then we get more problems ... see (4))
or
We get some source, but it's not for the expected kernel version (See http://ppa.launchpad.net/linaro-landing-team-samsung/linaro-samsung/ubuntu/p... It's actuall a 3.0.4 kernel. This may be a local mispackaging problem, but it's still wrong.)
or
We get some source, and it's the right version, but it's not much use to us because we might have some git branches we want to merge before building, so we really want to see the git repo, not a tarball.
2) apt-get source told us this package was in Git (via the Vcs-Git tag) at git://git.linaro.org/jcrigby/linux-linaro.git
But there is no such tree.
3) Feeling lucky, we fish about on git.linaro.org for alternatives
There's a bewildering variety of possible kernel trees for every platform:
e.g., for samsung:
http://git.linaro.org/gitweb?p=landing-teams/leb/samsung/kernel.git%3Ba=summ... http://git.linaro.org/gitweb?p=landing-teams/working/samsung/kernel.git%3Ba=... http://git.linaro.org/gitweb?p=people/angus/linux-samsung.git%3Ba=summary http://git.linaro.org/gitweb?p=people/tushar/linux-linaro-samsung.git%3Ba=su...
How do we know which one to look at?
I look at one with a tag matching my kernel version. I find none, except in generic trees that don't appear to be connected with my platform.
4) OK, we received a rumour about where the actual correct tree is, or just made a lucky guess let's try:
http://git.linaro.org/gitweb?p=landing-teams/leb/samsung/kernel.git%3Ba=summ...
The tree is nicely tagged with linaro releases.
But, we have no idea which release the installed kernel relates to, because we a) forgot, or b) ran apt-get upgrade to solve a disappeared- source problem in step (1), in which case the correspondence between installed packages and linaro releases has become indeterminate -- we are not linaro/ubuntu/debian experts, so we don't realise this is the case, but will just spend time searching for wrong things.
After somehow finding the kernel source for the release we installed and finding it doesn't match because we upgraded at some point...
...we only have the package name and version number to go on, because that's what debian/ubuntu packages have to identify them. Packages are not intrinsically mapped to a release; only the converse is true.
But, there is nothing in the git repository or package metadata to indicate how binary package versions map back to releases.
5) OK, we guessed or were told the correct release tag, and proceed to rebuild the binary packages.
The debian/ directory isn't in any of these repositories, so we can't build binary packages. Neither the build scripts or the kernel config are there.
To finally build valid binary packages, we need the appropriately- tagged debian*/ directories. And not just any ones, but those used for the release.
The chance of a linaro consumer making it from start to end of that process without falling over is minimal, and in my experience, it doesn't happen.
I think (hopefully) that my proposal robustly fixes this, but if it can be achieved more easily, I'm eager to know how.
Cheers ---Dave
On 24/01/12 10:18, Dave Martin wrote:
Hi again,
After a report of yet another instance of un-findable source for a kernel released from a landing team, it would be good if we could move forward with this.
Does anyone have any significant disagreements with the proposal below?
If not, I can try to write up a formal specification somewhere.
Can we then plan to implement it?
If anyone has any preference for the common prefix for tag names, please speak up (otherwise we will proceed with the "ubuntu/" prefix).
Cheers ---Dave
On Fri, Jan 13, 2012 at 01:32:35PM +0000, Dave Martin wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
The source package's control file must contain a Vcs-Git entry
The Vcs-Git entry must reference a git tree which contains the
_exact_ source code which appears in the source package.
- Such a tree must exist and must be publicly readable. It does not
have to be on git.linaro.org (though this is the recommended place).
- Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not
acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
- Manufacturing the source package from the contents of multiple
repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
- Referring to a tree which does not contain the whole contents of
the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
Note: this means that the released binary packages must be reproducible from the tagged source using standard package build mechanisms, to the extent that the exact versions of build tools and other build-time dependencies used to build the originally released binaries are still avilable.
- Tagging of packaged kernels must be done in a standard way.
- I recommend <source package name>_<package version>, matching the
Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
- Tree maintainers are of course free to add any other additional
tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
We _could_ standardise the following, but it is not essential:
- ubuntu/<release>: The tagged source for the _original_ kernel
which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
- No specific branch naming requirements exist.
Release tags do not necessarily need to appear on any branch.
We _could_ standardise the following, but it is not essential:
- ubuntu/latest - the tagged packaged source for the most recent
kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
-- Lee Jones Linaro ST-Ericsson Landing Team Lead M: +44 77 88 633 515 Linaro.org ??? Open source software for ARM SoCs Follow Linaro: Facebook | Twitter | Blog
On 24 January 2012 13:32, Dave Martin dave.martin@linaro.org wrote:
On Tue, Jan 24, 2012 at 11:34:51AM +0000, Lee Jones wrote:
All released kernel source is available on git.linaro.org.
Specifically: Git: git://git.linaro.org/landing-teams/leb/<landing_team>/kernel.git HTML: http://git.linaro.org/git/landing-teams/leb/
<landing_team>/kernel.git
<landing_team>/kernel.git;a=summary
The Gitweb has a description of what you can expect from each tag.
Good, but I fear it's not enough.
The question was never whether the source exists somewhere in infospace, but whether a linaro consumer can find it and make use of it. Lets see how they get on in this typical scenario:
Suppose we have a board with linaro that we set up a while ago. We are experiencing some issues, and want to turn on some instrumentation in the kernel and/or add some patches of their own. This means we would like to reconfigure/change and rebuild the kernel that we have installed.
- We know how to get source code on Ubuntu: run apt-get source
But, that specific package version may have disappeared from the server (Fixable by running apt-get upgrade, but then we get more problems ... see (4))
orWe get some source, but it's not for the expected kernel version (See http://ppa.launchpad.net/linaro-landing-team-samsung/linaro-samsung/ubuntu/p... It's actuall a 3.0.4 kernel. This may be a local mispackaging problem, but it's still wrong.)
orWe get some source, and it's the right version, but it's not much use to us because we might have some git branches we want to merge before building, so we really want to see the git repo, not a tarball.
- apt-get source told us this package was in Git (via the Vcs-Git tag)
at git://git.linaro.org/jcrigby/linux-linaro.git
But there is no such tree.
- Feeling lucky, we fish about on git.linaro.org for alternatives
There's a bewildering variety of possible kernel trees for every platform:
e.g., for samsung:
http://git.linaro.org/gitweb?p=landing-teams/leb/samsung/kernel.git%3Ba=summ...
http://git.linaro.org/gitweb?p=landing-teams/working/samsung/kernel.git%3Ba=... http://git.linaro.org/gitweb?p=people/angus/linux-samsung.git%3Ba=summary
http://git.linaro.org/gitweb?p=people/tushar/linux-linaro-samsung.git%3Ba=su...
How do we know which one to look at?
I look at one with a tag matching my kernel version. I find none, except in generic trees that don't appear to be connected with my platform.
- OK, we received a rumour about where the actual correct tree is,
or just made a lucky guess let's try:
http://git.linaro.org/gitweb?p=landing-teams/leb/samsung/kernel.git%3Ba=summ...
The tree is nicely tagged with linaro releases.
But, we have no idea which release the installed kernel relates to, because we a) forgot, or b) ran apt-get upgrade to solve a disappeared- source problem in step (1), in which case the correspondence between installed packages and linaro releases has become indeterminate -- we are not linaro/ubuntu/debian experts, so we don't realise this is the case, but will just spend time searching for wrong things.
After somehow finding the kernel source for the release we installed and finding it doesn't match because we upgraded at some point...
...we only have the package name and version number to go on, because that's what debian/ubuntu packages have to identify them. Packages are not intrinsically mapped to a release; only the converse is true.
But, there is nothing in the git repository or package metadata to indicate how binary package versions map back to releases.
- OK, we guessed or were told the correct release tag, and proceed to
rebuild the binary packages.
The debian/ directory isn't in any of these repositories, so we can't build binary packages. Neither the build scripts or the kernel config are there.
To finally build valid binary packages, we need the appropriately- tagged debian*/ directories. And not just any ones, but those used for the release.
The chance of a linaro consumer making it from start to end of that process without falling over is minimal, and in my experience, it doesn't happen.
I think (hopefully) that my proposal robustly fixes this, but if it can be achieved more easily, I'm eager to know how.
For Android we have:
https://android-build.linaro.org/builds/~linaro-android/panda-12.01-release/
we should have the same thing for Ubuntu:
ubuntu-build.linaro.org
with the similar information.
Cheers ---Dave
On 24/01/12 10:18, Dave Martin wrote:
Hi again,
After a report of yet another instance of un-findable source for a kernel released from a landing team, it would be good if we could move forward with this.
Does anyone have any significant disagreements with the proposal below?
If not, I can try to write up a formal specification somewhere.
Can we then plan to implement it?
If anyone has any preference for the common prefix for tag names, please speak up (otherwise we will proceed with the "ubuntu/" prefix).
Cheers ---Dave
On Fri, Jan 13, 2012 at 01:32:35PM +0000, Dave Martin wrote:
For everyone who packages kernel trees:
I've had some questions about getting the source for linaro kernel packaged, and it seems that this is still not straightforward:
Getting the debian package source (i.e., flat tarball) for a binary kernel is possible, but only if it is a non-superseded version.
Working out which source package you need is non-obvious (You have to check for installed kernel packages, and guess which source package you need, based on that. Non-linaro folks may not understand the difference between the various meta- and real kernel packages and may get pretty confused along the way.)
Finding the git tree and the relevant commit to reproduce the source (including that for superseded kernels) is hard or impossible. It requires guesswork and/or specific knowledge about the way the relevant team manages their trees. For some platforms, it looks like there may be no single tree containing the packaged kernel at all, in which case you would also need more guesswork and/or scripts or help from the relevant landing team in order actually to reproduce a release.
Am I correct in these conclusions?
If so, here's a proposal -- I welcome people's comments (and please do say if any of these problems are actually solved already!):
For every binary kernel package (.deb) publicly released by any linaro team, including those produced by the platform team and the landing teams:
The source package's control file must contain a Vcs-Git entry
The Vcs-Git entry must reference a git tree which contains the
_exact_ source code which appears in the source package.
- Such a tree must exist and must be publicly readable. It does not
have to be on git.linaro.org (though this is the recommended place).
- Referring to git://git.linaro.org/ubuntu/linux-linaro.git is not
acceptable, unless that repo is populated with the real source for that specific kernel as well as the packaging.
- Manufacturing the source package from the contents of multiple
repositories or branches at source package upload time is not acceptable, unless the result is also recorded as a tagged commit in the repository referenced by the Vcs-Git entry in the debian/control file contained in that same commit. The commit must have full history: importing tarballs directly into a repository for the purpose of release tagging is not acceptable.
- Referring to a tree which does not contain the whole contents of
the debian source package (for example, debian/ and other packaging files/dirs are missing) is not acceptable.
Note: this means that the released binary packages must be reproducible from the tagged source using standard package build mechanisms, to the extent that the exact versions of build tools and other build-time dependencies used to build the originally released binaries are still avilable.
- Tagging of packaged kernels must be done in a standard way.
- I recommend <source package name>_<package version>, matching the
Source field of debian/control and the version number of the most recent debian/changelog entry respectively (which must both be present in the repository as a consequence of (2)). If we want to avoid namespace pollution, we probably want to add a prefix such as debian/ or ubuntu/ to the tag name to indicate that the tag describes the source for a published .deb package. If so, we must standardise that prefix so that it is identical in all out trees.
- Tree maintainers are of course free to add any other additional
tags for their own use if they want to.
All teams already do release tagging of some sort, but the lack of consistency creates difficulties when anyone from outside the team tries to understand that team's trees.
We _could_ standardise the following, but it is not essential:
- ubuntu/<release>: The tagged source for the _original_ kernel
which was distributed in <release> (where <release> is a linaro monthly release such as 11.12 or 12.01)
- No specific branch naming requirements exist.
Release tags do not necessarily need to appear on any branch.
We _could_ standardise the following, but it is not essential:
- ubuntu/latest - the tagged packaged source for the most recent
kernel release made from this tree
(In the above, we could choose a diferent prefix instead of ubuntu/, but as described in (3) ,this should be chosen globally and _not_ on a per-tree basis).
Cheers ---Dave
-- Lee Jones Linaro ST-Ericsson Landing Team Lead M: +44 77 88 633 515 Linaro.org ??? Open source software for ARM SoCs Follow Linaro: Facebook | Twitter | Blog
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On Tue, Jan 24, 2012 at 9:40 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
[...]
For Android we have:
https://android-build.linaro.org/builds/~linaro-android/panda-12.01-release/
we should have the same thing for Ubuntu:
ubuntu-build.linaro.org
with the similar information.
I'm not sure about that: for Debian/Ubuntu there are established methods for getting source and provenance info. It's a solved problem, so we should just use the mature solution instead of insisting on inventing our own.
A key issue is that there is a fundamental difference between the way building and versioning works between the Debian and Android worlds.
In Android, if I understand correctly, the whole build is effectively done from a single tree, so you can meaningfully tag a whole release and bungle source for it without tagging individual components. Am I correct here?
In the Debian way of doing things, builds are incremental and continuous there is no single tree containing all the source for a release. Bootstrapping a whole release from pure source is a rare event, and involves a significant manual effort. Rather, a release is a particular set of versions of particular packages, not built as part of the release process, but instead the set of newest pre-built versions of the chosen packages at the time the release was defined. Also, once you have the platform running you can upgrade it piecemeal, package by package. So establishing metadata at the release level only is hard and makes little sense: the metadata must be tracked at the package level in any case.
All this means that the way we track a source project (such as the Linux kernel) which is common between both worlds must accommodate both worlds. If it fails to accommodate either, we will encounter trouble in one world or the other.
For the kernels, we do almost get things right for Ubuntu-land, but just not right _enough_ that finding the source works reliably in the same way as for every other package.
A UI is a good thing if it is built on firm foundations, but I fear that if we don't get the fundamentals correct, no amount of UI polishing is going to hide the instability that lurks beneath.
Cheers ---Dave
On 25 January 2012 12:18, Dave Martin dave.martin@linaro.org wrote:
On Tue, Jan 24, 2012 at 9:40 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
[...]
For Android we have:
https://android-build.linaro.org/builds/~linaro-android/panda-12.01-release/
we should have the same thing for Ubuntu:
ubuntu-build.linaro.org
with the similar information.
I'm not sure about that: for Debian/Ubuntu there are established methods for getting source and provenance info. It's a solved problem, so we should just use the mature solution instead of insisting on inventing our own.
A key issue is that there is a fundamental difference between the way building and versioning works between the Debian and Android worlds.
In Android, if I understand correctly, the whole build is effectively done from a single tree, so you can meaningfully tag a whole release and bungle source for it without tagging individual components. Am I correct here?
In the Debian way of doing things, builds are incremental and continuous there is no single tree containing all the source for a release. Bootstrapping a whole release from pure source is a rare event, and involves a significant manual effort. Rather, a release is a particular set of versions of particular packages, not built as part of the release process, but instead the set of newest pre-built versions of the chosen packages at the time the release was defined. Also, once you have the platform running you can upgrade it piecemeal, package by package. So establishing metadata at the release level only is hard and makes little sense: the metadata must be tracked at the package level in any case.
All this means that the way we track a source project (such as the Linux kernel) which is common between both worlds must accommodate both worlds. If it fails to accommodate either, we will encounter trouble in one world or the other.
For the kernels, we do almost get things right for Ubuntu-land, but just not right _enough_ that finding the source works reliably in the same way as for every other package.
A UI is a good thing if it is built on firm foundations, but I fear that if we don't get the fundamentals correct, no amount of UI polishing is going to hide the instability that lurks beneath.
That's all well and good, but the point is you need to answer the following question:
What kernel was used. Where can I get it. How can I rebuild it.
You may as well put that on a 'page' so that people who are not Debian people can easily find what they're looking for.
The point of the android-build pages is that it answers specific questions:
How do I use this? How do I rebuild this? Where does this come from? What works? Where can I get help?
Take a look at: https://android-build.linaro.org/builds/~linaro-android/landing-panda-12.01-...
Everything's in one place. Its not the way Android does this, but that doesn't matter, it giving our customers exactly what they want.
That's why an ubuntu-build.linaro.org is so important. Right now its hard to find Ubuntu stuff which is bad. As a Linaro user I should be able to find everything I need on one page without digging through out of date wikis or knowing someone.
Cheers ---Dave
On Wed, Jan 25, 2012 at 6:52 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 25 January 2012 12:18, Dave Martin dave.martin@linaro.org wrote:
On Tue, Jan 24, 2012 at 9:40 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
[...]
For Android we have:
https://android-build.linaro.org/builds/~linaro-android/panda-12.01-release/
we should have the same thing for Ubuntu:
ubuntu-build.linaro.org
with the similar information.
I'm not sure about that: for Debian/Ubuntu there are established methods for getting source and provenance info. It's a solved problem, so we should just use the mature solution instead of insisting on inventing our own.
A key issue is that there is a fundamental difference between the way building and versioning works between the Debian and Android worlds.
In Android, if I understand correctly, the whole build is effectively done from a single tree, so you can meaningfully tag a whole release and bungle source for it without tagging individual components. Am I correct here?
In the Debian way of doing things, builds are incremental and continuous there is no single tree containing all the source for a release. Bootstrapping a whole release from pure source is a rare event, and involves a significant manual effort. Rather, a release is a particular set of versions of particular packages, not built as part of the release process, but instead the set of newest pre-built versions of the chosen packages at the time the release was defined. Also, once you have the platform running you can upgrade it piecemeal, package by package. So establishing metadata at the release level only is hard and makes little sense: the metadata must be tracked at the package level in any case.
All this means that the way we track a source project (such as the Linux kernel) which is common between both worlds must accommodate both worlds. If it fails to accommodate either, we will encounter trouble in one world or the other.
For the kernels, we do almost get things right for Ubuntu-land, but just not right _enough_ that finding the source works reliably in the same way as for every other package.
A UI is a good thing if it is built on firm foundations, but I fear that if we don't get the fundamentals correct, no amount of UI polishing is going to hide the instability that lurks beneath.
That's all well and good, but the point is you need to answer the following question:
What kernel was used. Where can I get it. How can I rebuild it.
You may as well put that on a 'page' so that people who are not Debian people can easily find what they're looking for.
The point of the android-build pages is that it answers specific questions:
How do I use this? How do I rebuild this? Where does this come from? What works? Where can I get help?
Take a look at: https://android-build.linaro.org/builds/~linaro-android/landing-panda-12.01-...
Everything's in one place. Its not the way Android does this, but that doesn't matter, it giving our customers exactly what they want.
That's why an ubuntu-build.linaro.org is so important. Right now its hard to find Ubuntu stuff which is bad. As a Linaro user I should be able to find everything I need on one page without digging through out of date wikis or knowing someone.
I do agree -- it's just that when building such a thing for ubuntu, we need to build it on a firm foundation.
The foundation is called the apt archive, and it's already there. We don't quite use it right in some cases, though it sounds like jcrigby has already made good progress for the kernel CI builds. We also don't document it enough.
Once we can track and generate those reports automatically, then we can freely choose how to present them. There's no reason not to have a nice web page for it, so long as the underlying infrastructure is coherent.
Cheers ---Dave
On 01/26/2012 02:18 AM, Somebody in the thread at some point said:
I'm not sure about that: for Debian/Ubuntu there are established methods for getting source and provenance info. It's a solved problem, so we should just use the mature solution instead of insisting on inventing our own.
The established way of handling getting source in real distros is issue a kernel source package matching the binary. For a complete solution - including any sauce added at packaging time - I don't think there's an alternative to the packager providing source.
LTs are not naturally synchronized to what packagers are using and frankly at times when we're overloaded with keeping our head above water on tracking, or excitement with linux-linaro-tracking, dealing with these monthly releases is a headache. If packagers, who do have the exact source they are packaging, provide the sources (including sauce), there will be no possibility of loss of sync with what is provided no matter what the packaging side are doing.
For provenance, it has been a real problem even before monthly release effort, people have often said they have something going wrong on ".1234" ie, using packaged build revision ordinal, or some tree I never heard of. I am then reduced to asking "uh... what tree is that? It's based on our stuff?" and often they don't know either.
Locally I tag builds by passing in EXTRAVERSION to make, I tag it with "output directory name", which is "panda", then branch, eg, "3.3.0-panda_tracking-topic-audio-t31u+" which goes on to appear in boot log and become uname -r. In the past I added git HEAD hash too. However the space for that EXTRAVERSION info is limited in the kernel.
Maybe what we should do here is append to the compressed kernel at build-time some coordinates for getting the matching sources, eg
$ cat provenance # __WHERE_IS_THE_SOURCE__ git://git.linaro.org/landing-teams/working/ti/kernel.git tilt-3.2 8ae3d379998c7f48c73251d8f60b31b2b82a817d $ cat provenance >> uImage $ cat uImage | sed 1,'/# __WHERE_IS_THE_SOURCE/d' git://git.linaro.org/landing-teams/working/ti/kernel.git tilt-3.2 8ae3d379998c7f48c73251d8f60b31b2b82a817d
Existing boot flow doesn't care about appended data, it doesn't get loaded because it's past the U-Boot header length field. But it will then be easy to find out where to get matching sources even for non-booting kernel binary with no other information where it came from.
-Andy
-
Andy Green -
Dave Martin -
John Rigby -
Lee Jones -
Zach Pfeffer