Hello there,
This week I initiated a confused conversation during the techleads call about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
I seem to be hung up on having a way of saying "this hardware pack's kernel was built from this git tree with this config", so I wanted to explore the use cases a bit more:
- My #1 use case is, once I've installed a hardware pack, running into a problem and then being able to verify whether it contains a certain patch or was built with a certain config option. I'd like to know that because it's the first thing the KWG and LT people ask me when I go to report the bug.
- I think there's also a use case of wanting to know how to get that hwpack rebuilt with slightly updated kernel code or configs. If it's much easier to just document how to build and replace a kernel for a running image, that may cater for most of that use case, the exception being weird bugs caused by our build process and/or toolchain.
- I wonder if there's a use case for understanding and/or replacing the other contents of the hardware pack. I normally think of the hardware pack as "a built kernel and not much else" XXX.
Zach suggested SPDX (as in spdx.org) as a solution to this problem; I'm not sure I understand enough about it (Loïc's provided a sample file at http://spdx.org/wiki/sample-partial-spdx-file-geronimo) but here's my strawman proposal of what data we should give people quick access to:
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)
How does that sound? Too simplistic?
On 1 June 2011 11:41, Christian Robottom Reis kiko@linaro.org wrote:
I seem to be hung up on having a way of saying "this hardware pack's kernel was built from this git tree with this config", so I wanted to explore the use cases a bit more:
- My #1 use case is, once I've installed a hardware pack, running into a problem and then being able to verify whether it contains a certain patch or was built with a certain config option. I'd like to know that because it's the first thing the KWG and LT people ask me when I go to report the bug.
This problem could be easily solved with git::
All official Linaro builds are generated from a single git tree that has branches for different kernel versions that we build from being automatically updated during the build process. The git rev is embedded in the kernel package name (linaro-linux-<githash>-...) and also in the kernel uname so that it is immediately obvious what tree and set of patches it come from and the KWG or LT can go fix the issue in their private trees that then get pulled into the main git tree. We can also tag the git tree during a build and embed that into the kernel uname.
Zach suggested SPDX (as in spdx.org) as a solution to this problem; I'm not sure I understand enough about it (Loïc's provided a sample file at http://spdx.org/wiki/sample-partial-spdx-file-geronimo) but here's my strawman proposal of what data we should give people quick access to:
Quick look at it and to me it seems a bit heavy for just having a way to find out what's in the kernel. If we want to carry other information than that than it would be good to have all the meta-data in one place such as the SPDX file.
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?)
By patchset do you mean broken out patches as in a quilt stack or a changelog of the patches? If someone has the git url and git revision, they inherently know what patches are in the kernel.
- What kernel config was used to build it (A separate file in the hwpack directory?)
Do we have a repo where we store these? We probably want something similar to the kernel with an "official" repo used for build purposes that can be tagged at with the same tag as the kernel builds.
Thanks, ~Deepak
On Wed, Jun 01, 2011 at 12:51:12PM -0700, Deepak Saxena wrote:
All official Linaro builds are generated from a single git tree that has branches for different kernel versions that we build from being automatically updated during the build process. The git rev is embedded in the kernel package name (linaro-linux-<githash>-...) and also in the kernel uname so that it is immediately obvious what tree and set of patches it come from and the KWG or LT can go fix the issue in their private trees that then get pulled into the main git tree. We can also tag the git tree during a build and embed that into the kernel uname.
So your suggestion is that we have a tree to which we commit the result of the kernel source package build process? It's an interesting one; I'd like to know if this (or an approximation of it) is feasible.
And thinking about it, John should already have this tree somewhere, right?
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?)
By patchset do you mean broken out patches as in a quilt stack or a changelog of the patches? If someone has the git url and git revision, they inherently know what patches are in the kernel.
I was referring to the fact that for a source package (such as what we build the hwpacks out of) we have a base plus a set of patches which may not live in any tree. But answering the question above will probably answer this one too.
- What kernel config was used to build it (A separate file in the hwpack directory?)
Do we have a repo where we store these? We probably want something similar to the kernel with an "official" repo used for build purposes that can be tagged at with the same tag as the kernel builds.
Alas, we don't, and I think we definitely need one.
On 1 June 2011 12:58, Christian Robottom Reis kiko@linaro.org wrote:
On Wed, Jun 01, 2011 at 12:51:12PM -0700, Deepak Saxena wrote:
All official Linaro builds are generated from a single git tree that has branches for different kernel versions that we build from being automatically updated during the build process. The git rev is embedded in the kernel package name (linaro-linux-<githash>-...) and also in the kernel uname so that it is immediately obvious what tree and set of patches it come from and the KWG or LT can go fix the issue in their private trees that then get pulled into the main git tree. We can also tag the git tree during a build and embed that into the kernel uname.
So your suggestion is that we have a tree to which we commit the result of the kernel source package build process? It's an interesting one; I'd like to know if this (or an approximation of it) is feasible.
No, not a tree of packaged kernels. I meant a single git tree that consolidates all the kernel source trees that we possibly build from to provide a single location where developers can grab any kernel we've used for builds. This would also facilitate easy diffing between kernel versions.
And thinking about it, John should already have this tree somewhere, right?
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?)
By patchset do you mean broken out patches as in a quilt stack or a changelog of the patches? If someone has the git url and git revision, they inherently know what patches are in the kernel.
I was referring to the fact that for a source package (such as what we build the hwpacks out of) we have a base plus a set of patches which may not live in any tree. But answering the question above will probably answer this one too.
What's an example of such a set of patches?
~Deepak
On Wed, 1 Jun 2011 13:17:38 -0700, Deepak Saxena dsaxena@linaro.org wrote:
No, not a tree of packaged kernels. I meant a single git tree that consolidates all the kernel source trees that we possibly build from to provide a single location where developers can grab any kernel we've used for builds. This would also facilitate easy diffing between kernel versions.
I don't think that a centralised solution to this is worth pursuing.
Take the Landing Teams for instance, I'm sure they would appreciate some of these facilities for their private hwpacks, but we can't pull their trees in to a central place.
And thinking about it, John should already have this tree somewhere, right?
http://git.linaro.org/gitweb?p=people/jcrigby/ubuntu/linux-linaro-natty.git%...
I think that we can fairly cheaply put a file/version suffix/something in the kernel binary package that points to the tree + ref that it was built from.
We can then generate a report when building the hwpack that consolidates this information.
This is also useful in that it's not specific to the kernel, so if we put that file in other packages too they will be picked up (u-boot anyone?)
I was referring to the fact that for a source package (such as what we build the hwpacks out of) we have a base plus a set of patches which may not live in any tree. But answering the question above will probably answer this one too.
What's an example of such a set of patches?
I don't believe that this is the case for the kernel. Everything should be in John's tree referenced above, and you can use git to find out which patches are in any other tree that you care about (nico/linus etc.) if that information is needed for what you are doing.
I don't think we should be looking to attribute the provenance of every line of source that ends up in the hwpack in one report, we just need to shorten the chain to find the information that you care about.
One very cheap thing we could do is to produce a report when building the hwpack that tells you which archive each binary package that was used came from. You can sort of do this now (assuming there aren't clashing versions), but it's a pain. Once you know that you can find the source package. So this is a general solution, but we can do much better in specific cases.
Another cheap thing to do would be to dump the config from the kernel package in to the output dir, so you can see the config without having to download the hwpack or produce an image. This can be useful, much like the new .manifest that lists the packages included and their versions is useful if you want to know if the new hwpack build picked up the fix for some bug in the latest upload.
Thanks,
James
On 06/01/2011 10:21 PM, Somebody in the thread at some point said:
Another cheap thing to do would be to dump the config from the kernel package in to the output dir, so you can see the config without having to download the hwpack or produce an image. This can be useful, much like the new .manifest that lists the packages included and their versions is useful if you want to know if the new hwpack build picked up the fix for some bug in the latest upload.
At least for the config,
CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y
gets you a gzip of the config used in the kernel body readable by scripts/extract-ikconfig externally, and readable at runtime down /proc/config.gz. (static extract script is useful in the case the kernel doesn't boot or stay up long enough to query at runtime). That's more tightly bound to the actual kernel than doing it in packaging only.
My build scripts put the git branch in the uname -r, the make process already puts the git HEAD hash in there; packaging can do the same.
Nowhere says which repo was used at the moment. However a lot of bug reports just talk about packagenames like blah-1003 leaving the reader clueless about where to get the package let alone what it was built from.
If CONFIG_IKCONFIG_PROC was used everywhere, we / the packaging process could leverage that to add a dummy Kconfig like CONFIG_BUILD_SOURCE="git://..." and have it all available cleanly just from the uImage or at runtime.
-Andy
On Thu, 02 Jun 2011 08:48:31 +0100, Andy Green andy@warmcat.com wrote:
At least for the config,
CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y
gets you a gzip of the config used in the kernel body readable by scripts/extract-ikconfig externally, and readable at runtime down /proc/config.gz. (static extract script is useful in the case the kernel doesn't boot or stay up long enough to query at runtime). That's more tightly bound to the actual kernel than doing it in packaging only.
This is a useful thing to know about, thanks. Given that the packaging currently contains the full config in plaintext and I expect that jcrigby's skeleton packaging also does this, I don't think that this is a problem that needs solving right now.
If CONFIG_IKCONFIG_PROC was used everywhere, we / the packaging process could leverage that to add a dummy Kconfig like CONFIG_BUILD_SOURCE="git://..." and have it all available cleanly just from the uImage or at runtime.
This would be one trick that we could use with the plaintext config too.
My opinion is that we shouldn't make this kernel specific. We should put this info somewhere else (use Vcs-* perhaps?) so that we can provide this information for other packages too. The kernel isn't the entire universe after all.
Thanks,
James
On 6 June 2011 16:30, James Westby james.westby@canonical.com wrote:
We should put this info somewhere else (use Vcs-* perhaps?) so that we can provide this information for other packages too.
You can add additional user-defined fields to the control file if it's required or more convenient.
On Mon, 6 Jun 2011 16:45:35 +0000, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 June 2011 16:30, James Westby james.westby@canonical.com wrote:
We should put this info somewhere else (use Vcs-* perhaps?) so that we can provide this information for other packages too.
You can add additional user-defined fields to the control file if it's required or more convenient.
I'm well aware of that, thanks.
However, automatic insertion of generated new fields is not that easy, and can cause problems when interacting with e.g. version control.
In addition the information is used in a variety of places, and if the rules for how to specify where the information end up don't work for you use case you are forced to push the information in to places it doesn't really belong, at best causing bloat.
Thanks,
James
On Thu, Jun 02, 2011 at 08:48:31AM +0100, Andy Green wrote:
At least for the config,
CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y
Is this not used everywhere, and if not, why not?
On Wed, Jun 01, 2011 at 05:21:14PM -0400, James Westby wrote:
I don't think we should be looking to attribute the provenance of every line of source that ends up in the hwpack in one report, we just need to shorten the chain to find the information that you care about.
That is a very good point, and a better statement of my intent.
One very cheap thing we could do is to produce a report when building the hwpack that tells you which archive each binary package that was used came from. You can sort of do this now (assuming there aren't clashing versions), but it's a pain. Once you know that you can find the source package. So this is a general solution, but we can do much better in specific cases.
That's a good idea.
Another cheap thing to do would be to dump the config from the kernel package in to the output dir, so you can see the config without having to download the hwpack or produce an image. This can be useful, much like the new .manifest that lists the packages included and their versions is useful if you want to know if the new hwpack build picked up the fix for some bug in the latest upload.
That's also a good idea. +1 for JFDIing these parts of the solution.
On Tue, 7 Jun 2011 10:28:10 -0300, Christian Robottom Reis kiko@canonical.com wrote:
One very cheap thing we could do is to produce a report when building the hwpack that tells you which archive each binary package that was used came from. You can sort of do this now (assuming there aren't clashing versions), but it's a pain. Once you know that you can find the source package. So this is a general solution, but we can do much better in specific cases.
That's a good idea.
https://bugs.launchpad.net/linaro-image-tools/+bug/794303
Pretty easy I think.
Another cheap thing to do would be to dump the config from the kernel package in to the output dir, so you can see the config without having to download the hwpack or produce an image. This can be useful, much like the new .manifest that lists the packages included and their versions is useful if you want to know if the new hwpack build picked up the fix for some bug in the latest upload.
That's also a good idea. +1 for JFDIing these parts of the solution.
https://bugs.launchpad.net/linaro-image-tools/+bug/794304
a bit trickier, but as I say in the report some work that Mattias is doing may help here.
We'll try and pick these up for this month's release.
Thanks,
James
On Wed, 1 Jun 2011 15:41:05 -0300, Christian Robottom Reis kiko@linaro.org wrote:
This week I initiated a confused conversation during the techleadscall about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
Luckily I missed that call, I have no confusion and this sounds perfectly sensible to me :-)
- My #1 use case is, once I've installed a hardware pack, running into a problem and then being able to verify whether it contains a certain patch or was built with a certain config option. I'd like to know that because it's the first thing the KWG and LT people ask me when I go to report the bug.
The config is available in /boot isn't it? As for whether a patch is in, if we can't easily go from package version to git id and work from there, then we should definitely add that regardless of anything else, as being able to do that quickly is regularly useful.
- I think there's also a use case of wanting to know how to get that hwpack rebuilt with slightly updated kernel code or configs. If it's much easier to just document how to build and replace a kernel for a running image, that may cater for most of that use case, the exception being weird bugs caused by our build process and/or toolchain.
I've long wondered if a repack-hwpack command would be useful.
We're also just starting some work on adding a way to add in a custom kernel at linaro-media-create time to support validation work, and we think it will be useful elsewhere. We're not entirely sure that we can pull it off in a seamless manner yet, we'll be experimenting this week to be sure.
- I wonder if there's a use case for understanding and/or replacing the other contents of the hardware pack. I normally think of the hardware pack as "a built kernel and not much else" XXX.
This is becoming less true over time it seems, with accelerated components making there way in to the hwpacks (SGX for OMAP.)
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)How does that sound? Too simplistic?
This just seems to deal with the kernel?
My gut feeling is that all of this should be fairly straightforward, but the tough part will be co-ordinating across all of the different pieces to get it in one final report.
For instance, at hwpack creation time it's too late to know most of the things above, unless they are recorded in the kernel package. If they are then they can fairly easily be extracted and recorded.
Thanks,
James
On Wed, Jun 01, 2011 at 04:13:40PM -0400, James Westby wrote:
Luckily I missed that call, I have no confusion and this sounds perfectly sensible to me :-)
Though now I'm stuck with a confusing Subject line to compensate :-/
The config is available in /boot isn't it?
Good point.
As for whether a patch is in, if we can't easily go from package version to git id and work from there, then we should definitely add that regardless of anything else, as being able to do that quickly is regularly useful.
That's what I was thinking.
- I wonder if there's a use case for understanding and/or replacing the other contents of the hardware pack. I normally think of the hardware pack as "a built kernel and not much else" XXX.This is becoming less true over time it seems, with accelerated components making there way in to the hwpacks (SGX for OMAP.)
That's true, and I didn't finish my original sentence but I would have pointed out that more complete hardware packs would contain other vendor-supplied binaries. Having a version for them described would be great to have for debugging reasons, but I guess it's available in the packaging?
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)How does that sound? Too simplistic?
This just seems to deal with the kernel?
My main motivation is around the kernel (answering the question "what kernel is being used for this hwpack") though I guess it applies to anything we have source for. Apart from the kernel and bootloader, are we supplying anything in the hwpack we have source for?
On Tue, 7 Jun 2011 10:20:57 -0300, Christian Robottom Reis kiko@canonical.com wrote:
That's true, and I didn't finish my original sentence but I would have pointed out that more complete hardware packs would contain other vendor-supplied binaries. Having a version for them described would be great to have for debugging reasons, but I guess it's available in the packaging?
Yeah, they have a version number. Whether that's meaningful or not is up to the packager.
We now produce files like
http://snapshots.linaro.org/11.05-daily/linaro-hwpacks/omap3/20110607/0/imag...
to make it easy to see what's included without having to download and extract the hwpack.
My main motivation is around the kernel (answering the question "what kernel is being used for this hwpack") though I guess it applies to anything we have source for. Apart from the kernel and bootloader, are we supplying anything in the hwpack we have source for?
Taking a look at panda-x11-base it requests that it include:
linux-image-linaro-omap linux-headers-linaro-omap u-boot-linaro-omap4-panda x-loader-omap4-panda libegl1-sgx-omap4 libgles1-sgx-omap4 libgles2-sgx-omap4 libopenvg1-sgx-omap4
so just kernel and bootloader.
The actual hwpack contains a bit more (it contains everything that isn't already in the rootfs that is needed to satisfy the dependencies of the above too)
http://snapshots.linaro.org/11.05-daily/linaro-hwpacks/panda-x11-base/201106...
It contains more that we have source for, but given that they aren't the primary things that the hwpack is for I'm not sure that we need make it as easy to go to the source (so knowing the version and where the package was taken from to get you going is probably good enough for now.)
Therefore it looks like we mainly need to concern ourselves with the kernel and bootloader as you suggest.
Thanks,
James
On Wed, Jun 8, 2011 at 12:45 AM, James Westby james.westby@canonical.com wrote:
On Tue, 7 Jun 2011 10:20:57 -0300, Christian Robottom Reis kiko@canonical.com wrote:
That's true, and I didn't finish my original sentence but I would have pointed out that more complete hardware packs would contain other vendor-supplied binaries. Having a version for them described would be great to have for debugging reasons, but I guess it's available in the packaging?
Yeah, they have a version number. Whether that's meaningful or not is up to the packager.
We now produce files like
http://snapshots.linaro.org/11.05-daily/linaro-hwpacks/omap3/20110607/0/imag...
to make it easy to see what's included without having to download and extract the hwpack.
My main motivation is around the kernel (answering the question "what kernel is being used for this hwpack") though I guess it applies to anything we have source for. Apart from the kernel and bootloader, are we supplying anything in the hwpack we have source for?
Taking a look at panda-x11-base it requests that it include:
linux-image-linaro-omap linux-headers-linaro-omap u-boot-linaro-omap4-panda x-loader-omap4-panda libegl1-sgx-omap4 libgles1-sgx-omap4 libgles2-sgx-omap4 libopenvg1-sgx-omap4
so just kernel and bootloader.
+ graphics userspace and kernel driver ;)
The actual hwpack contains a bit more (it contains everything that isn't already in the rootfs that is needed to satisfy the dependencies of the above too)
just to mention: we probably want to regularly review hwpack contents and if we see that we pull in stuff that is not board specific it might indicate that we have to tweak our contents of rootfs (CCed Ricardo, so he can think about adding this to a regularly done review process of seeds and hwpack content or so).
http://snapshots.linaro.org/11.05-daily/linaro-hwpacks/panda-x11-base/201106...
It contains more that we have source for, but given that they aren't the primary things that the hwpack is for I'm not sure that we need make it as easy to go to the source (so knowing the version and where the package was taken from to get you going is probably good enough for now.)
Therefore it looks like we mainly need to concern ourselves with the kernel and bootloader as you suggest.
Yes, we can start with this for now. Later we can extend that based on how well this goes.
W dniu 01.06.2011 20:41, Christian Robottom Reis pisze:
Hello there,
This week I initiated a confused conversation during the techleadscall about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
I seem to be hung up on having a way of saying "this hardware pack's kernel was built from this git tree with this config", so I wanted to explore the use cases a bit more:
My use case is similar:
I'm an LAVA probe and I want to determine the very same information so that reporting framework can cross-reference if with the test results.
In LAVA we have introduced a concept of SoftwareSource - source information about binary packages on the system. Each source denotes a code branch (either bzr, git or svn) and revision (or tag). This allows you to pinpoint GCC revision, kernel tree and other important pieces of information. You can later use this to draw some interesting conclusions (especially for system-level performance, kernel quality and so on).
It would be nice to standardize this information and actually store it inside the hardware pack.
One of the things it does not capture currently is kernel configuration. Assuming you can cat /proc/config it would be easy to capture that as well but I would like to know what others think.
Zach suggested SPDX (as in spdx.org) as a solution to this problem; I'm not sure I understand enough about it (Loïc's provided a sample file at http://spdx.org/wiki/sample-partial-spdx-file-geronimo) but here's my strawman proposal of what data we should give people quick access to:
While using SPDX might be interesting I would encourage us to look back at LAVA so that either we also use SPDX or if it's not feasible we all use some other sensible tool.
In LAVA terms:
- What kernel tree it was built from (A URL to the git tree)
SoftwareSource.branch_vcs + SoftwareSource.branch_url
- What revision (A revision ID)
SoftwareSource.branch_revision
- What patches were applied on top of it (A URL to the patchset, maybe?)
TBD, it's hard to say what those patches were. I would actually like a different solution, have the tree with pre-applied patches (so that we can still pinpoint the tree from git repo) and have a informative description of patches that were applied for other reasons. This would allow us to compare results of tests with and without a specific kernel patch (easy and interesting) withuot resorting to git bisections.
- What kernel config was used to build it (A separate file in the hwpack directory?)
Attachment to test run.
Best regards Zygmunt Krynicki
On Wed, Jun 01, 2011 at 10:33:27PM +0200, Zygmunt Krynicki wrote:
One of the things it does not capture currently is kernel configuration. Assuming you can cat /proc/config it would be easy to capture that as well but I would like to know what others think.
James has rightly pointed out that this will be available under /boot, right?
- What patches were applied on top of it (A URL to the patchset, maybe?)TBD, it's hard to say what those patches were. I would actually like a different solution, have the tree with pre-applied patches (so that we can still pinpoint the tree from git repo) and have a informative description of patches that were applied for other reasons. This would allow us to compare results of tests with and without a specific kernel patch (easy and interesting) withuot resorting to git bisections.
Well, the reason I asked for this was because I had the probably erroneous conception that we would have patches applied to the kernel that would not be committed to git. I have since reflected that John's tree, for instance, already includes any the packaging-related and SAUCE patches, so it's not immediately necessary.
However, thinking a bit further, it would be nice to have a reference to both the final branch (which the package was generated from) and the original, sans-packaging branch which the packaging branch added to. However, does this separation exist for all hwpacks? Do Landing Teams maintain separate packaging branches, or do you, John?
On Tue, Jun 7, 2011 at 7:26 AM, Christian Robottom Reis kiko@linaro.org wrote:
On Wed, Jun 01, 2011 at 10:33:27PM +0200, Zygmunt Krynicki wrote:
One of the things it does not capture currently is kernel configuration. Assuming you can cat /proc/config it would be easy to capture that as well but I would like to know what others think.
James has rightly pointed out that this will be available under /boot, right?
- What patches were applied on top of it (A URL to the patchset, maybe?)
TBD, it's hard to say what those patches were. I would actually like a different solution, have the tree with pre-applied patches (so that we can still pinpoint the tree from git repo) and have a informative description of patches that were applied for other reasons. This would allow us to compare results of tests with and without a specific kernel patch (easy and interesting) withuot resorting to git bisections.
Well, the reason I asked for this was because I had the probably erroneous conception that we would have patches applied to the kernel that would not be committed to git. I have since reflected that John's tree, for instance, already includes any the packaging-related and SAUCE patches, so it's not immediately necessary.
However, thinking a bit further, it would be nice to have a reference to both the final branch (which the package was generated from) and the original, sans-packaging branch which the packaging branch added to. However, does this separation exist for all hwpacks? Do Landing Teams maintain separate packaging branches, or do you, John?
Landing Teams do their own packaging.
-- Christian Robottom Reis | [+55] 16 9112 6430 | http://launchpad.net/~kiko Linaro Engineering VP | [ +1] 612 216 4935 | http://async.com.br/~kiko
On Tue, 7 Jun 2011 10:26:20 -0300, Christian Robottom Reis kiko@linaro.org wrote:
However, thinking a bit further, it would be nice to have a reference to both the final branch (which the package was generated from) and the original, sans-packaging branch which the packaging branch added to. However, does this separation exist for all hwpacks? Do Landing Teams maintain separate packaging branches, or do you, John?
You could ask git for this based on the last merge point with nico's branch (or any other branch you cared about for that matter.)
Do we need to call this information out?
Thanks,
James
Zach suggested SPDX (as in spdx.org) as a solution to this problem; I'm not sure I understand enough about it (Loïc's provided a sample file at http://spdx.org/wiki/sample-partial-spdx-file-geronimo) but here's my strawman proposal of what data we should give people quick access to:
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)
I'll work with Kate to come up with an example. I think once we can play with something, spdx will look pretty good.
Speaking about Ubuntu packaged kernels only...
Everything that goes into the source package is in git. Every release has a signed tag. Ubuntu kernel package names have a strict naming convention that makes upgrading and abi checking work right, so from my perspective putting the tag name somewhere in the binary package is much more realistic than forcing a new naming convention on the package name. Kernel image packages have the config used to generate them in /boot we could easily add another file that contained the git tag. We could also put a git log from a known upstream Linus or stable release in there but that log could be long. It would have all the linaro patches plus all the Ubuntu patches.
john
I went through Kiko's request:
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)
and the spdx spec:
http://spdx.org/system/files/spdx-draft20110516_0.pdf
And wrote this:
PackageName: linux-linaro-omap 2.6.38-1002.3 #https://launchpad.net/ubuntu/+source/linux-linaro-omap PackageDownloadLocation: https://launchpad.net/ubuntu/+archive/primary/+files/linux-linaro-omap_2.6.3... PackageChecksum: SHA1: cf0b587742611328f095da4b329e9fc7 # from https://launchpad.net/ubuntu/+source/linux-linaro-omap/2.6.38-1002.3 SourceInfo: uses Linux v2.6.38.1 SourceInfo: uses linaro-linux-2.6.38-upstream-29Mar2011 SourceInfo: uses (fill in patch1) SourceInfo: uses (fill in patch2) SourceInfo: uses (fill in patch3) SourceInfo: uses <text> CONFIG_ etc... </text>
FileName: file1 FileName: file2 FileName: file3 FileChecksum: SHA1: calculated
SPDXVersion: SPDX-1.0 Creator: Person: Zach Pfeffer (zach.pfeffer@linaro.org) Created: 2011-06-02T13:18:00Z PackageLicenseDeclared: GPL-2.0 PackageVerificationCode: (fill in SHA1 of all souyrce files) LicenseConcluded: GPL-2.0 LicenseInfoFromFiles: GPL-2.0 PackageCopyrightText: <text> Copyright 2008-2010 John Smith </text>
I think this generally adheres to the spec (I put all the licensing info below the packaging into and also used # comment)
I've added Kate to the thread to tell me I'm wrong... :)
I can't really express how important it is for Linaro to get its legal house in order now. With binary blobs and various components coming together, we can get in front of this issue instead of "creating our own thing." Right now things are in Beta so if we have feed back we should give it.
-Zach
On 1 June 2011 13:41, Christian Robottom Reis kiko@linaro.org wrote:
Hello there,
This week I initiated a confused conversation during the techleads call about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
I seem to be hung up on having a way of saying "this hardware pack's kernel was built from this git tree with this config", so I wanted to explore the use cases a bit more:
- My #1 use case is, once I've installed a hardware pack, running into a problem and then being able to verify whether it contains a certain patch or was built with a certain config option. I'd like to know that because it's the first thing the KWG and LT people ask me when I go to report the bug.
- I think there's also a use case of wanting to know how to get that hwpack rebuilt with slightly updated kernel code or configs. If it's much easier to just document how to build and replace a kernel for a running image, that may cater for most of that use case, the exception being weird bugs caused by our build process and/or toolchain.
- I wonder if there's a use case for understanding and/or replacing the other contents of the hardware pack. I normally think of the hardware pack as "a built kernel and not much else" XXX.
Zach suggested SPDX (as in spdx.org) as a solution to this problem; I'm not sure I understand enough about it (Loïc's provided a sample file at http://spdx.org/wiki/sample-partial-spdx-file-geronimo) but here's my strawman proposal of what data we should give people quick access to:
- What kernel tree it was built from (A URL to the git tree) - What revision (A revision ID) - What patches were applied on top of it (A URL to the patchset, maybe?) - What kernel config was used to build it (A separate file in the hwpack directory?)
How does that sound? Too simplistic?
Christian Robottom Reis | [+55] 16 9112 6430 | http://launchpad.net/~kiko Linaro Engineering VP | [ +1] 612 216 4935 | http://async.com.br/~kiko
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Hi,
Apologies for asking you directly what could probably be looked up, but the spec isn't very easy to digest.
On Thu, 2 Jun 2011 16:59:46 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
PackageName: linux-linaro-omap 2.6.38-1002.3 #https://launchpad.net/ubuntu/+source/linux-linaro-omap PackageDownloadLocation: https://launchpad.net/ubuntu/+archive/primary/+files/linux-linaro-omap_2.6.3...
This isn't the full source that was built. The source package has three parts. Can we link to three things here? If we can only link to one it should probably be the .dsc which is the description for the whole thing.
SourceInfo: uses Linux v2.6.38.1 SourceInfo: uses linaro-linux-2.6.38-upstream-29Mar2011 SourceInfo: uses (fill in patch1) SourceInfo: uses (fill in patch2) SourceInfo: uses (fill in patch3)
What's the constraints on what we put here? What's the use for it?
What's listed here seems fairly tricky to produce automatically.
FileName: file1 FileName: file2 FileName: file3 FileChecksum: SHA1: calculated
This is all the files in the source?
Creator: Person: Zach Pfeffer (zach.pfeffer@linaro.org)
What option do we have here? Given this is going to be produced automatically I'm not sure we should blame you for all of the mistakes ;-)
PackageLicenseDeclared: GPL-2.0
Is this is single choice field? Does it cover source or binary?
PackageVerificationCode: (fill in SHA1 of all souyrce files)
SHA1 of all source calculated how?
LicenseConcluded: GPL-2.0 LicenseInfoFromFiles: GPL-2.0
What do these mean?
Thanks,
James
On 6 June 2011 11:45, James Westby james.westby@canonical.com wrote:
Hi,
Apologies for asking you directly what could probably be looked up, but the spec isn't very easy to digest.
On Thu, 2 Jun 2011 16:59:46 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
PackageName: linux-linaro-omap 2.6.38-1002.3 #https://launchpad.net/ubuntu/+source/linux-linaro-omap PackageDownloadLocation: https://launchpad.net/ubuntu/+archive/primary/+files/linux-linaro-omap_2.6.3...
This isn't the full source that was built. The source package has three parts. Can we link to three things here? If we can only link to one it should probably be the .dsc which is the description for the whole thing.
For reference:
http://www.spdx.org/system/files/spdx-draft20110516_0.pdf
Right now the spec has this as:
4.3.3Cardinality: Mandatory, one.
Kate would have to comment if this could change to:
4.3.3Cardinality: Mandatory, one or more.
SourceInfo: uses Linux v2.6.38.1 SourceInfo: uses linaro-linux-2.6.38-upstream-29Mar2011 SourceInfo: uses (fill in patch1) SourceInfo: uses (fill in patch2) SourceInfo: uses (fill in patch3)
What's the constraints on what we put here? What's the use for it?
The spec says:
4.6Source Information 4.6.1Purpose: This is a free form text field that contains additional comments about the origin of the package. For instance, this field might include comments indicating whether the package been pulled from a source code management system or has been repackaged. 4.6.2Intent: Here, by providing a freeform field, reviewers can provide any additional information to describe any anomalies, or discoveries, in the determination of the origin of the package. 4.6.3Cardinality: Optional, one 4.6.4Data Format: single line of free form text 4.6.5Tag: SourceInfo Example: SourceInfo: uses glibc-2_11-branch from git://sourceware.org/git/glibc.git.
What's listed here seems fairly tricky to produce automatically.
What part do you think would be tricky?
FileName: file1 FileName: file2 FileName: file3 FileChecksum: SHA1: calculated
This is all the files in the source?
Yeah.
Creator: Person: Zach Pfeffer (zach.pfeffer@linaro.org)
What option do we have here? Given this is going to be produced automatically I'm not sure we should blame you for all of the mistakes
Ha! This is just the Creator of the SPDX file. It will probably become the PoC. Kate is there a specific field for, ask this person questions about the package? Perhaps we need
SpdxCreator: Person: Zach Pfeffer (zach.pfeffer@linaro.org) PackageCreator: Person: Not Zach Pfeffer :)
;-)
PackageLicenseDeclared: GPL-2.0
Is this is single choice field? Does it cover source or binary?
You link all the licenses together with ANDs and ORs. Looks like it covers both, Kate?
PackageVerificationCode: (fill in SHA1 of all souyrce files)
SHA1 of all source calculated how?
4.5.4Algorithm: verificationcode = 0 filelist = “” for all files in package { if file is an “excludes” file, skip it /* exclude SPDX analysis file itself */ appended filelist with “SHA1(file) || string(file)” } sort filelist in ascending order by SHA1 value verificationcode = SHA1(filelist)
LicenseConcluded: GPL-2.0
From the spec:
The licensing that the preparer of this SPDX document has concluded, based on the evidence, actual applies to the package.
I think this is where the lawyer would say, this is the license.
LicenseInfoFromFiles: GPL-2.0
This is a field that has all the license found in the package.
What do these mean?
Thanks,
James
On Mon, 6 Jun 2011 14:37:18 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
The spec says:
4.6Source Information 4.6.1Purpose: This is a free form text field that contains additional comments about the origin of the package. For instance, this field might include comments indicating whether the package been pulled from a source code management system or has been repackaged. 4.6.2Intent: Here, by providing a freeform field, reviewers can provide any additional information to describe any anomalies, or discoveries, in the determination of the origin of the package. 4.6.3Cardinality: Optional, one 4.6.4Data Format: single line of free form text 4.6.5Tag: SourceInfo Example: SourceInfo: uses glibc-2_11-branch from git://sourceware.org/git/glibc.git.
So it looks like we'd have to define our own microformat here (though it's going to be consumed by humans at least to start with, so consistency doesn't really matter at this stage)
What's listed here seems fairly tricky to produce automatically.
What part do you think would be tricky?
It depends when we are generating this file, but the format of what you specify seems a little clever for humans.
If the content is freeform then we can obviously choose something that is easy to generate.
FileName: file1 FileName: file2 FileName: file3 FileChecksum: SHA1: calculated
This is all the files in the source?
Yeah.
I guess the cost of that in a kernel build is pretty small.
You only list one FileChecksum here. Can that line follow every FileName line?
LicenseConcluded: GPL-2.0
From the spec:
The licensing that the preparer of this SPDX document has concluded, based on the evidence, actual applies to the package.
I think this is where the lawyer would say, this is the license.
Yeah. Again my question is source or binary?
I presume this can be an AND/OR list again?
LicenseInfoFromFiles: GPL-2.0
This is a field that has all the license found in the package.
Just a dumping ground of every license found?
My overally impression is that this is rather a large additional overhead to just be able to say
the kernel was built from 0A2E345 of git://git.linaro.org/jcrigby/linux-linaro-natty.git
which is the main thrust of kiko's request as I understand it.
Debian packages already contain licensing info (they also have a proposed standard to make that info machine readable.) Is it worth Linaro's time to try and move everything to one of these new formats at this stage?
Thanks,
James
On Mon, 2011-06-06 at 16:37 -0400, James Westby wrote:
On Mon, 6 Jun 2011 14:37:18 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
The spec says:
4.6Source Information 4.6.1Purpose: This is a free form text field that contains additional comments about the origin of the package. For instance, this field might include comments indicating whether the package been pulled from a source code management system or has been repackaged. 4.6.2Intent: Here, by providing a freeform field, reviewers can provide any additional information to describe any anomalies, or discoveries, in the determination of the origin of the package. 4.6.3Cardinality: Optional, one 4.6.4Data Format: single line of free form text 4.6.5Tag: SourceInfo Example: SourceInfo: uses glibc-2_11-branch from git://sourceware.org/git/glibc.git.
So it looks like we'd have to define our own microformat here (though it's going to be consumed by humans at least to start with, so consistency doesn't really matter at this stage)
If there are some pretty common trends, we can look at adding fields to support. Its in beta right now so this sort of feedback can be gathered after all ;)
What's listed here seems fairly tricky to produce automatically.
What part do you think would be tricky?
It depends when we are generating this file, but the format of what you specify seems a little clever for humans.
If the content is freeform then we can obviously choose something that is easy to generate.
FileName: file1 FileName: file2 FileName: file3 FileChecksum: SHA1: calculated
This is all the files in the source?
Yeah.
I guess the cost of that in a kernel build is pretty small.
You only list one FileChecksum here. Can that line follow every FileName line?
After every FileName: there should be a FileChecksum: field.
For each file listed in the package, the fields that are mandatory and should show up are: - FileName: - FileChecksum: - LicenseConcluded: - LicensesInfoInFile: - CopyrightText:
The rest are optional, and can be included or not.
LicenseConcluded: GPL-2.0
From the spec:
The licensing that the preparer of this SPDX document has concluded, based on the evidence, actual applies to the package.
I think this is where the lawyer would say, this is the license.
Yeah. Again my question is source or binary?
This is captured as an optional field (FileType:) at the file level.
I presume this can be an AND/OR list again?
LicenseInfoFromFiles: GPL-2.0
This is a field that has all the license found in the package.
Just a dumping ground of every license found?
If a package has multiple file, its a way of having a summary of all the licenses encountered in those files. Ideally the LicenseConcluded: at the package level, will match the LicenseInfoFromFiles:. By making this visible, its hoped over time that the package owners will look into clearing up the ambiguities.
My overally impression is that this is rather a large additional overhead to just be able to say
the kernel was built from 0A2E345 of git://git.linaro.org/jcrigby/linux-linaro-natty.git
which is the main thrust of kiko's request as I understand it.
If you just want to say the build origins, then yes its overkill for that. If you want to be able to pass on a package that has a set of patches off a known public kernel (with its own SPDX file ;) ) then it permits the licensing and copyrights to be clearly articulated.
Debian packages already contain licensing info (they also have a proposed standard to make that info machine readable.) Is it worth Linaro's time to try and move everything to one of these new formats at this stage?
Debian package standard has no way to track the accuracy of the file level information, so it could change underneath without there being a way of detecting it - which causes problems. If you look at: http://dep.debian.net/deps/dep5/ (machine readable info), you'll see this aspect missing (the wildcards preclude it). The DEP5 proposal was analyzed extensively last year but commercial companies didn't feel they could rely on the accuracy information provided through it, but there is a lot of common ground on the fields provided, file paragraphs, separate licensing section, etc. We've had a team of volunteer corporate lawyers (Motorola, HP, etc. ) working on the SPDX for the last year and reviewing and modifying the proposed fields extensively.
Kate
Hi Kate,
Thanks for the information. I have some responses inline. To be clear I'm veering in to commentary on the format itself for a lot of this, rather than specific statements on whether I think Linaro should use it or not.
On Mon, 06 Jun 2011 18:05:54 -0500, Kate Stewart kate.stewart@canonical.com wrote:
If there are some pretty common trends, we can look at adding fields to support. Its in beta right now so this sort of feedback can be gathered after all ;)
That's good to know.
After every FileName: there should be a FileChecksum: field.
For each file listed in the package, the fields that are mandatory and should show up are:
- FileName:
- FileChecksum:
- LicenseConcluded:
- LicensesInfoInFile:
- CopyrightText:
The rest are optional, and can be included or not.
You say "for each file listed in the package," is it mandatory to list all files? It seems to me that it would be much better to allow for gradual implementation of what will be a large job for some projects, so I hope it is optional.
I guess it's intended for the format to always be output from some tool rather than hand-edited, unless it's possible to put glob patterns or similar in the FileName field?
LicenseConcluded: GPL-2.0
From the spec:
The licensing that the preparer of this SPDX document has concluded, based on the evidence, actual applies to the package.
I think this is where the lawyer would say, this is the license.
Yeah. Again my question is source or binary?
This is captured as an optional field (FileType:) at the file level.
Well, this applies to the whole package by my understanding, so the particular file types may not matter.
Take for instance a package which is dual licensed GPL-2 and something else at the source level. If you choose to build this package and link it against GPL-2 only code then some people's interpretation of the GPL would say that the binary is GPL-2 only, despite the source being dual licensed.
Perhaps that's a minority opinion, but I expect there are other cases where the source and binary licenses may not match, so I am wondering if the expectation that this defines the source or binary concluded license.
Perhaps what you are saying is that there will be different SPDX files for the source and binary, and the difference would be captured there?
My overally impression is that this is rather a large additional overhead to just be able to say
the kernel was built from 0A2E345 of git://git.linaro.org/jcrigby/linux-linaro-natty.git
which is the main thrust of kiko's request as I understand it.
If you just want to say the build origins, then yes its overkill for that. If you want to be able to pass on a package that has a set of patches off a known public kernel (with its own SPDX file ;) ) then it permits the licensing and copyrights to be clearly articulated.
Yeah. Is the kernel going to be released with an SPDX file that we can base our work on?
Debian package standard has no way to track the accuracy of the file level information, so it could change underneath without there being a way of detecting it - which causes problems. If you look at: http://dep.debian.net/deps/dep5/ (machine readable info), you'll see this aspect missing (the wildcards preclude it). The DEP5 proposal was analyzed extensively last year but commercial companies didn't feel they could rely on the accuracy information provided through it, but there is a lot of common ground on the fields provided, file paragraphs, separate licensing section, etc. We've had a team of volunteer corporate lawyers (Motorola, HP, etc. ) working on the SPDX for the last year and reviewing and modifying the proposed fields extensively.
Sure, I don't doubt that a lot of effort has been put in to SPDX, and I don't want to get in to a debate about whether it is a good idea to have wildcards or not. My point was more that I don't see it being worth Linaro engineering time to push for adoption of either format inside Debian packages at this time.
Thanks,
James
On Mon, 2011-06-06 at 20:19 -0400, James Westby wrote:
On Mon, 06 Jun 2011 18:05:54 -0500, Kate Stewart kate.stewart@canonical.com wrote:
After every FileName: there should be a FileChecksum: field.
For each file listed in the package, the fields that are mandatory and should show up are:
- FileName:
- FileChecksum:
- LicenseConcluded:
- LicensesInfoInFile:
- CopyrightText:
The rest are optional, and can be included or not.
You say "for each file listed in the package," is it mandatory to list all files? It seems to me that it would be much better to allow for gradual implementation of what will be a large job for some projects, so I hope it is optional.
Afraid not. Each file gets in a "package/tarball/what ever you choose to group together", gets its own file name listed. Package could be interpreted as tarball equally well as a .deb.
So, yeah, its a large job for some projects. Problem is that right now alot of folks are doing the work in multiple formats already, as a necessary condition of license compliance for shipping code. So, some of the original files may not be produced by the project itself, but rather by some 3rd party, that then can be 'reviewed' and signed off by others.
Yes, tools will help. I'd like to get some open source tools available to do the generation, and am working with Ninka and FOSSology on some prototypes. (help is always welcome ;) ). There are some commercial tool providers who have built up plans to produce the RDF varient of files already with their tools - BlackDuck, Open Logic, Protecode are all active participants. Tag value and the RDF have been designed to be interchangeable for this reason.
I guess it's intended for the format to always be output from some tool rather than hand-edited, unless it's possible to put glob patterns or similar in the FileName field?
For small projects or even single binary files that get packaged, it should be possible to hand generate. For larger packages with multiple source files, yeah, tools are going to be the key here. See comments above. ;)
LicenseConcluded: GPL-2.0
From the spec:
The licensing that the preparer of this SPDX document has concluded, based on the evidence, actual applies to the package.
I think this is where the lawyer would say, this is the license.
Yeah. Again my question is source or binary?
This is captured as an optional field (FileType:) at the file level.
Well, this applies to the whole package by my understanding, so the particular file types may not matter.
Actually if you're shipping a binary file as its own package, like the linux kernel, then its just a package with one file, there's just one file, and the FileType: applies. If you're shipping a source tar ball with 30K+ files, in it, then the Type doesn't make much sense at the Package level, since there could be multiple types involved. There are firmware binaries in the linux kernel sources, for instance. ;)
Take for instance a package which is dual licensed GPL-2 and something else at the source level. If you choose to build this package and link it against GPL-2 only code then some people's interpretation of the GPL would say that the binary is GPL-2 only, despite the source being dual licensed.
This is the sort of argument that lead us down the notion of ConcludedLicense vs. LicenseInfoInFile (at the file and package levels).
Perhaps that's a minority opinion, but I expect there are other cases where the source and binary licenses may not match, so I am wondering if the expectation that this defines the source or binary concluded license.
Yup, its not just source and binary though that cause conflicts. BTW.
Perhaps what you are saying is that there will be different SPDX files for the source and binary, and the difference would be captured there?
We're still figuring out all the use cases, but I would expect the source to be packaged separately from the binary (thinking about how the archive work, this seems to be the case in practice), and there would be different SPDX files as a result.
My overally impression is that this is rather a large additional overhead to just be able to say
the kernel was built from 0A2E345 of git://git.linaro.org/jcrigby/linux-linaro-natty.git
which is the main thrust of kiko's request as I understand it.
If you just want to say the build origins, then yes its overkill for that. If you want to be able to pass on a package that has a set of patches off a known public kernel (with its own SPDX file ;) ) then it permits the licensing and copyrights to be clearly articulated.
Yeah. Is the kernel going to be released with an SPDX file that we can base our work on?
That's been the request from others as well. Am working with Ninka/FOSSology to see if we can make it happen in an Open Source way, and then it will be a packaging step. Linux is fairly clean license wise compared to most large packages. If you're curious: http://repo.fossology.org/?mod=nomoslicense&show=detail&upload=149&a... (this takes about 7 minutes for FOSSology to generate, for instance), so it feasible to add in SPDX file generation to it - since the hard part is recognizing those licenses. The socialization of Linux generating this as a deliverable with the releases is being pursued.
Proprietary tool vendors will be providing RDF varients for specific instances of linux from time to time as well.
My point was more that I don't see it being worth Linaro engineering time to push for adoption of either format inside Debian packages at this time.
This is a format that can go outside the packages (as well as in), so can be used without marshalling the entire Debian community to adopt it. So, am not advocating it be pushed by Linaro engineering for adoption inside Debian packages. Just be considered for use when describing the licensing and copyrights of what you're providing, rather than come up with yet another version... ;)
The problem SPDX is trying to solve is the software bill of materials one, so that the supply chain can know what licenses they have to comply with from a licencing and cooyright perspective, and each sender/recipient doesn't have to scan through the same files over and over again, as they build on the work of others.
Thanks, Kate
On Mon, 06 Jun 2011 20:29:38 -0500, Kate Stewart kate.stewart@canonical.com wrote:
This is a format that can go outside the packages (as well as in), so can be used without marshalling the entire Debian community to adopt it. So, am not advocating it be pushed by Linaro engineering for adoption inside Debian packages. Just be considered for use when describing the licensing and copyrights of what you're providing, rather than come up with yet another version... ;)
Right. The trigger of this discussion wasn't about describing the licensing and copyrights of what we are providing.
Given that I don't think it's going to work for our use case, given that we have to specify the license details of every file we are shipping (files here will be .deb packages), just in order so say
this is based on 03E520A of git://git.linaro.org/jcrigby/linux-linaro-natty.git
I certainly think that it's a good idea to consider it if we do wish to describe the licenses of the files we are providing in a machine-readable manner.
Thanks,
James
On 7 June 2011 17:54, James Westby james.westby@linaro.org wrote:
On Mon, 06 Jun 2011 20:29:38 -0500, Kate Stewart kate.stewart@canonical.com wrote:
This is a format that can go outside the packages (as well as in), so can be used without marshalling the entire Debian community to adopt it. So, am not advocating it be pushed by Linaro engineering for adoption inside Debian packages. Just be considered for use when describing the licensing and copyrights of what you're providing, rather than come up with yet another version... ;)
Right. The trigger of this discussion wasn't about describing the licensing and copyrights of what we are providing.
Given that I don't think it's going to work for our use case, given that we have to specify the license details of every file we are shipping (files here will be .deb packages), just in order so say
Perhaps Kate can chime in. I think you just have to specify the license as a whole.
this is based on 03E520A of git://git.linaro.org/jcrigby/linux-linaro-natty.git
I certainly think that it's a good idea to consider it if we do wish to describe the licenses of the files we are providing in a machine-readable manner.
Thanks,
James
Hi Zach,
In addition I realised that some of the information requested isn't explicit in what you propose.
On Thu, 2 Jun 2011 16:59:46 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
I went through Kiko's request:
- What kernel tree it was built from (A URL to the git tree)
- What revision (A revision ID)
These two aren't really present.
Is there some way to include these as well?
Thanks,
James
On 6 June 2011 13:40, James Westby james.westby@canonical.com wrote:
Hi Zach,
In addition I realised that some of the information requested isn't explicit in what you propose.
On Thu, 2 Jun 2011 16:59:46 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
I went through Kiko's request:
- What kernel tree it was built from (A URL to the git tree)
For this I listed:
SourceInfo: uses Linux v2.6.38.1
since I figured people would just go to:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git%3Ba=summary
(which has a bad name now) 2.6 is soooo 10 years ago.
- What revision (A revision ID)
This is baked in to the previous value, but we could list it as well with something like
SourceInfo: uses git://.... 83729abc4...
These two aren't really present.
Is there some way to include these as well?
Thanks,
James
On Tue, Jun 07, 2011 at 09:59:32AM -0500, Zach Pfeffer wrote:
On 6 June 2011 13:40, James Westby james.westby@canonical.com wrote:
Hi Zach,
In addition I realised that some of the information requested isn't explicit in what you propose.
On Thu, 2 Jun 2011 16:59:46 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
I went through Kiko's request:
- What kernel tree it was built from (A URL to the git tree)
For this I listed:
SourceInfo: uses Linux v2.6.38.1
We really want a URL here, not a text rendition.
On 01/06/11 19:41, Christian Robottom Reis wrote:
Hello there,
This week I initiated a confused conversation during the techleadscall about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
Sorry I'm getting into this conversation a bit late but is there also a need to figure out what toolchain was used to build this hwpack and the way in which the toolchain was configured (v7-a, neon, vfp , vfpv3-d16) and what version of the toolchain was used. If this information is already captured or there is an easy way of getting back to this. This is if you think there is a use-case of regenerating the hwpack to investigate any issues that might come up
cheers Ramana
On Thu, 2 Jun 2011 23:17:02 +0100, Ramana Radhakrishnan ramana.radhakrishnan@linaro.org wrote:
Sorry I'm getting into this conversation a bit late but is there also a need to figure out what toolchain was used to build this hwpack and the way in which the toolchain was configured (v7-a, neon, vfp , vfpv3-d16) and what version of the toolchain was used. If this information is already captured or there is an easy way of getting back to this. This is if you think there is a use-case of regenerating the hwpack to investigate any issues that might come up
While this would be a useful thing to record it's not that easy given that the toolchain used is a couple of steps removed from the hwpack. It currently works like
Source package -> Binary package -> Hardware pack
with the toolchain being used in the first translation. We would need to record the toolchain used in the binary packages to provide this information. There's currently no mechanism to do this generally, but again if we are modifying the packaging of each piece to do this they could run something to discover the toolchain information and store this in a similar way to git-describe output used to identify the ref that was built.
Thanks,
James
On Fri, Jun 3, 2011 at 12:17 AM, Ramana Radhakrishnan ramana.radhakrishnan@linaro.org wrote:
On 01/06/11 19:41, Christian Robottom Reis wrote:
Hello there,
This week I initiated a confused conversation during the techleads call about having a way to describe what the hardware pack was built from. We had a couple of false starts but I think we agreeing that there needs to be some form of manifest that describes what the hardware pack contains.
Sorry I'm getting into this conversation a bit late but is there also a need to figure out what toolchain was used to build this hwpack and the way in which the toolchain was configured (v7-a, neon, vfp , vfpv3-d16) and what version of the toolchain was used. If this information is already captured or there is an easy way of getting back to this. This is if you think there is a use-case of regenerating the hwpack to investigate any issues that might come up
Is there a gcc fingerprints in binaries that we could use to extract the toolchain info?
Otherwise, we probably would have to fiddle this out of the build logs and so on and it feels like quite a challenge to get this right. e.g. some packages work around toolchain bugs by compiling objects, so's and binaries with non-default compiler flags etc. so even if we try to track this manually, everything short of having that info directly in the binaries would yield to inaccurate info is my guess.
-
Alexander Sack -
Andy Green -
Christian Robottom Reis -
Christian Robottom Reis -
Deepak Saxena -
Fathi Boudra -
James Westby -
James Westby -
John Rigby -
Kate Stewart -
Ramana Radhakrishnan -
Zach Pfeffer -
Zygmunt Krynicki