Now that we have an Android build for every board and Gerrit and LAVA I'd like to unify how we handle kernels for each so that we can work more efficiently, start to unify the kernels and provide a means for external Android users to start to improve these trees.
First, since we control the kernels that go into our Android builds I'd like to follow AOSP convention and have:
kernel/common.git (john's tree) kernel/imx53.git kernel/omap.git kernel/origen.git kernel/snowball.git
(right now we have:
panda, beagle <remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/jstultz/android" revision="linaro-android-3.0" remote="linaro-other"/>
panda-leb <remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/andygreen/kernel-tilt" revision="tilt-jstultz-linaro-android-3.0" remote="linaro-other"/>
leb-snowball <remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="bsp/st-ericsson/linux-3.0-android-ux500" revision="master" remote="linaro-other"/>
leb-origen <remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/angus/linux-linaro-3.0" remote="linaro-other" revision="android"/>
leb-imx53 <remote fetch="git://git.linaro.org/" name="linaro-other"/> <project name="people/bernhardrosenkranzer/android-kernel-iMX53" path="kernel" remote="linaro-other" revision="3d981d9418c53e3e1a079582088121c641588791"/> )
I'd like these to be at git://android.git.linaro.org/. I'd like to not mirror.
Each tree would accept patches via Gerrit and be maintainable through standard git by the tree maintainers. This should allow upgrades to each, without needing to push all the upgrade patches through Gerrit.
Next, we'd like to point to a tip branch for each of these trees. This branch would be were development would be happening. Tracking tip is fundamental to continuous integration, if its broken then the primary job of everyone should be getting it going again. I'd like to suggest the branches be named the same and follow John's branch naming convention:
linaro-android-3.0...etc
Lastly, I'd like to request (and we may be able to automate this once all the kernels are co-located) that once a pinned-manifest.xml comes out, referencing a specific sha1, I'd like to lay a tag on that sha so it sticks around. Its better to tag after the fact because it saves having to do another build/test/qa cycle and will ensure that our pinned-manifest.xml continue to work.
I suspect a few growing pains, but I think this is going to be great. Once the kernels are co-located we should be able to look at Gerrit extensions that allow easier upstream patch submission and tracking. We should also be able to more easily refactor things. Using the Gerrit flow and extending it to support Android upstreaming is exactly the thing Linaro was built for. Since Gerrit is such an integral part of Android development, extensions to allow patch propagation would allow a lot of the good work to flow back into the mainline kernel.
-Zach
On Mon, 29 Aug 2011 08:08:17 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
Now that we have an Android build for every board and Gerrit and LAVA I'd like to unify how we handle kernels for each so that we can work more efficiently, start to unify the kernels and provide a means for external Android users to start to improve these trees.
First, since we control the kernels that go into our Android builds I'd like to follow AOSP convention and have:
kernel/common.git (john's tree)
Correction: kernel/common is AOSP upstream, John's tree is kernel/linux-android .
kernel/imx53.git kernel/omap.git kernel/origen.git kernel/snowball.git
These looks ok and in line with existing AOSP's naming conventions.
(right now we have:
panda, beagle
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/jstultz/android" revision="linaro-android-3.0" remote="linaro-other"/>
panda-leb
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/andygreen/kernel-tilt" revision="tilt-jstultz-linaro-android-3.0" remote="linaro-other"/>
leb-snowball
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="bsp/st-ericsson/linux-3.0-android-ux500" revision="master" remote="linaro-other"/>
leb-origen
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/angus/linux-linaro-3.0" remote="linaro-other" revision="android"/>
leb-imx53
<remote fetch="git://git.linaro.org/" name="linaro-other"/> <project name="people/bernhardrosenkranzer/android-kernel-iMX53" path="kernel" remote="linaro-other" revision="3d981d9418c53e3e1a079582088121c641588791"/> )
I'd like these to be at git://android.git.linaro.org/. I'd like to not mirror.
Each tree would accept patches via Gerrit and be maintainable through standard git by the tree maintainers. This should allow upgrades to each, without needing to push all the upgrade patches through Gerrit.
Next, we'd like to point to a tip branch for each of these trees. This branch would be were development would be happening. Tracking tip is fundamental to continuous integration, if its broken then the primary job of everyone should be getting it going again. I'd like to suggest the branches be named the same and follow John's branch naming convention:
linaro-android-3.0...etc
Lastly, I'd like to request (and we may be able to automate this once all the kernels are co-located) that once a pinned-manifest.xml comes out, referencing a specific sha1, I'd like to lay a tag on that sha so it sticks around.
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Its better to tag after the fact because it saves having to do another build/test/qa cycle and will ensure that our pinned-manifest.xml continue to work.
I suspect a few growing pains, but I think this is going to be great. Once the kernels are co-located we should be able to look at Gerrit extensions that allow easier upstream patch submission and tracking. We should also be able to more easily refactor things. Using the Gerrit flow and extending it to support Android upstreaming is exactly the thing Linaro was built for. Since Gerrit is such an integral part of Android development, extensions to allow patch propagation would allow a lot of the good work to flow back into the mainline kernel.
-Zach
On 29 August 2011 08:22, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 29 Aug 2011 08:08:17 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
Now that we have an Android build for every board and Gerrit and LAVA I'd like to unify how we handle kernels for each so that we can work more efficiently, start to unify the kernels and provide a means for external Android users to start to improve these trees.
First, since we control the kernels that go into our Android builds I'd like to follow AOSP convention and have:
kernel/common.git (john's tree)
Correction: kernel/common is AOSP upstream, John's tree is kernel/linux-android .
kernel/imx53.git kernel/omap.git kernel/origen.git kernel/snowball.git
These looks ok and in line with existing AOSP's naming conventions.
(right now we have:
panda, beagle
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/jstultz/android" revision="linaro-android-3.0" remote="linaro-other"/>
panda-leb
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/andygreen/kernel-tilt" revision="tilt-jstultz-linaro-android-3.0" remote="linaro-other"/>
leb-snowball
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="bsp/st-ericsson/linux-3.0-android-ux500" revision="master" remote="linaro-other"/>
leb-origen
<remote name="linaro-other" fetch="git://git.linaro.org/"/> <project path="kernel" name="people/angus/linux-linaro-3.0" remote="linaro-other" revision="android"/>
leb-imx53
<remote fetch="git://git.linaro.org/" name="linaro-other"/> <project name="people/bernhardrosenkranzer/android-kernel-iMX53" path="kernel" remote="linaro-other" revision="3d981d9418c53e3e1a079582088121c641588791"/> )
I'd like these to be at git://android.git.linaro.org/. I'd like to not mirror.
Each tree would accept patches via Gerrit and be maintainable through standard git by the tree maintainers. This should allow upgrades to each, without needing to push all the upgrade patches through Gerrit.
Next, we'd like to point to a tip branch for each of these trees. This branch would be were development would be happening. Tracking tip is fundamental to continuous integration, if its broken then the primary job of everyone should be getting it going again. I'd like to suggest the branches be named the same and follow John's branch naming convention:
linaro-android-3.0...etc
Lastly, I'd like to request (and we may be able to automate this once all the kernels are co-located) that once a pinned-manifest.xml comes out, referencing a specific sha1, I'd like to lay a tag on that sha so it sticks around.
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
That sounds good. Perhaps we should lay down the branch across every git on a successful build. It could be like a cursor that would track the latest. We'd still do the pinned-manifest.xml but users would be able to
Its better to tag after the fact because it saves having to do another build/test/qa cycle and will ensure that our pinned-manifest.xml continue to work.
I suspect a few growing pains, but I think this is going to be great. Once the kernels are co-located we should be able to look at Gerrit extensions that allow easier upstream patch submission and tracking. We should also be able to more easily refactor things. Using the Gerrit flow and extending it to support Android upstreaming is exactly the thing Linaro was built for. Since Gerrit is such an integral part of Android development, extensions to allow patch propagation would allow a lot of the good work to flow back into the mainline kernel.
-Zach
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
Is it perhaps possible to improve "repo" instead?
-Andy
On 29 August 2011 10:13, Andy Green andy.green@linaro.org wrote:
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
I think this can just happen as part of the build cycle. If we track the tree directly the build system can lay down a branch automatically after release.
We could probably just update the branch after the automerge step. For those who are just getting their feet wet with CI the flow is:
Sync build Apply patch Build Regress build on regress success, merge patch Save build, gen pinned-manifest
Anyway, this isn't an issue with repo, its a sha1 reachability issue. repo 's just a foreach git tool.
Is it perhaps possible to improve "repo" instead?
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
On 08/29/2011 11:41 PM, Somebody in the thread at some point said:
On 29 August 2011 10:13, Andy Greenandy.green@linaro.org wrote:
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
I think this can just happen as part of the build cycle. If we track the tree directly the build system can lay down a branch automatically after release.
Sorry where's this branch going, some internal shadowing repo is it?
We could probably just update the branch after the automerge step. For those who are just getting their feet wet with CI the flow is:
Fine but what's so special about a branch in this "CI flow"? It should be just as happy with git reset --hard abcdef which is on no branch so long as abcdef wasn't garbage collected away.
Anyway, this isn't an issue with repo, its a sha1 reachability issue. repo 's just a foreach git tool.
What do you mean "SHA1 reachability"? I can "reach" arbitrary HEADs using a hash even if they're not tagged so long as I didn't garbage collect. If I tagged them they're guaranteed to not be garbage collected. I can always "reach" them for checkout. So what is this "reachability" issue?
The way Paul described it, it sounds like a limitation with this repo script that it depends on specifically a branch has been checked out.
Is it perhaps possible to improve "repo" instead?
-Andy
On 29 August 2011 10:58, Andy Green andy.green@linaro.org wrote:
On 08/29/2011 11:41 PM, Somebody in the thread at some point said:
On 29 August 2011 10:13, Andy Greenandy.green@linaro.org wrote:
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
I think this can just happen as part of the build cycle. If we track the tree directly the build system can lay down a branch automatically after release.
Sorry where's this branch going, some internal shadowing repo is it?
We could probably just update the branch after the automerge step. For those who are just getting their feet wet with CI the flow is:
Fine but what's so special about a branch in this "CI flow"? It should be just as happy with git reset --hard abcdef which is on no branch so long as abcdef wasn't garbage collected away.
Anyway, this isn't an issue with repo, its a sha1 reachability issue. repo 's just a foreach git tool.
What do you mean "SHA1 reachability"? I can "reach" arbitrary HEADs using a hash even if they're not tagged so long as I didn't garbage collect. If I tagged them they're guaranteed to not be garbage collected. I can always "reach" them for checkout. So what is this "reachability" issue?
The way Paul described it, it sounds like a limitation with this repo script that it depends on specifically a branch has been checked out.
All repo is doing with a pinned-manifest.xml is:
foreach git in this manifest checkout sha1
so as long as you can checkout the sha1 everything is cool. The reason we're using this is because we can automatically create a manifest with the sha1's of every sub git head that we built. This allows someone to reproduce the build exactly and allows us to ship what we test. If you can apply the same tag across all gits, which we can do now, and that'll keep the sha's around than that's what we should do.
Is it perhaps possible to improve "repo" instead?
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
On 08/30/2011 01:15 AM, Somebody in the thread at some point said:
Anyway, this isn't an issue with repo, its a sha1 reachability issue. repo 's just a foreach git tool.
What do you mean "SHA1 reachability"? I can "reach" arbitrary HEADs using a hash even if they're not tagged so long as I didn't garbage collect. If I tagged them they're guaranteed to not be garbage collected. I can always "reach" them for checkout. So what is this "reachability" issue?
The way Paul described it, it sounds like a limitation with this repo script that it depends on specifically a branch has been checked out.
All repo is doing with a pinned-manifest.xml is:
foreach git in this manifest checkout sha1
so as long as you can checkout the sha1 everything is cool. The reason
I think you missed my point, if I tagged it, you CAN check it out. So "reachability" is not the issue.
When you check out a hash that is not a branch head, git acts slightly differently on some commands because the HEAD hash does not match any branch. I guess this repo script is not proof against those differences, which is easy to happen if you only tested it on branches, and probably not that difficult to solve either.
Is it perhaps possible to improve "repo" instead?
-Andy
On Tue, 30 Aug 2011 10:19:35 +0800, Andy Green andy.green@linaro.org wrote:
I think you missed my point, if I tagged it, you CAN check it out. So "reachability" is not the issue.
I think there are two things here:
The first is that repo currently only fetches branches, and not tags. This means that simply tagging isn't enough to ensure that you have the object in your repo.
Paul has a patch for this and sent it upstream, but there's no response yet. If we use that change then tagging is sufficient.
When you check out a hash that is not a branch head, git acts slightly differently on some commands because the HEAD hash does not match any branch. I guess this repo script is not proof against those differences, which is easy to happen if you only tested it on branches, and probably not that difficult to solve either.
I'm not sure that's the issue with unreferenced objects.
If you have a revision that isn't reachable from any head (tag or branch), and not referenced by the reflog or anything, does "git clone" transfer that revision, or does it only fetch reachable objects?
If it doesn't fetch them, then it's pretty hard for us to support them, given the number of times repos are transferred in just building Android on our build service (let alone the fact that it would be pretty difficult for someone to grab the tree and look at the revision with just git.)
Thanks,
James
On 08/30/2011 10:56 AM, Somebody in the thread at some point said:
On Tue, 30 Aug 2011 10:19:35 +0800, Andy Greenandy.green@linaro.org wrote:
I think you missed my point, if I tagged it, you CAN check it out. So "reachability" is not the issue.
I think there are two things here:
The first is that repo currently only fetches branches, and not tags. This means that simply tagging isn't enough to ensure that you have the object in your repo.
Not sure what "your repo" means but I get the message.
Paul has a patch for this and sent it upstream, but there's no response yet. If we use that change then tagging is sufficient.
Sounds good. Presumably you can deploy this patched repo version anyway and I can continue to tag pushes and be done.
When you check out a hash that is not a branch head, git acts slightly differently on some commands because the HEAD hash does not match any branch. I guess this repo script is not proof against those differences, which is easy to happen if you only tested it on branches, and probably not that difficult to solve either.
I'm not sure that's the issue with unreferenced objects.
If you have a revision that isn't reachable from any head (tag or branch), and not referenced by the reflog or anything, does "git clone" transfer that revision, or does it only fetch reachable objects?
If it doesn't fetch them, then it's pretty hard for us to support them, given the number of times repos are transferred in just building Android on our build service (let alone the fact that it would be pretty difficult for someone to grab the tree and look at the revision with just git.)
You're quite right but we're talking at cross purposes. I tag my pushes currently so they are always reachable in my repo, will turn up in clones, etc.
The suggestion was that there was nothing to improve with repo due to the problem being somehow with "unreachable objects": I'm glad to hear that it could be patched to use tags and has been.
-Andy
On Tue, 30 Aug 2011 11:44:35 +0800, Andy Green andy.green@linaro.org wrote:
The first is that repo currently only fetches branches, and not tags. This means that simply tagging isn't enough to ensure that you have the object in your repo.
Not sure what "your repo" means but I get the message.
Yeah, sorry, I meant "the resulting repository."
Paul has a patch for this and sent it upstream, but there's no response yet. If we use that change then tagging is sufficient.
Sounds good. Presumably you can deploy this patched repo version anyway and I can continue to tag pushes and be done.
Unfortunately I'm not sure it's as straightwforward as this, given that we want developers to also be able to do builds locally.
We can deploy to the couple of places in the build service that would need the update, but that will then be supporting something that will break for developers until they update.
Maybe we want the transition plan to be "update your install of repo when you find that it is broken," but I don't think that's very satisfying. It will mainly affect the Android team though, so I'll let them chime in.
Thanks,
James
On Mon, 29 Aug 2011 12:15:15 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
We could probably just update the branch after the automerge step.
What do you mean "SHA1 reachability"? I can "reach" arbitrary HEADs using a hash even if they're not tagged so long as I didn't garbage collect. If I tagged them they're guaranteed to not be garbage collected. I can always "reach" them for checkout. So what is this "reachability" issue?
The way Paul described it, it sounds like a limitation with this repo script that it depends on specifically a branch has been checked out.
All repo is doing with a pinned-manifest.xml is:
foreach git in this manifest checkout sha1
so as long as you can checkout the sha1 everything is cool. The reason we're using this is because we can automatically create a manifest with the sha1's of every sub git head that we built. This allows someone to reproduce the build exactly and allows us to ship what we test. If you can apply the same tag across all gits, which we can do now, and that'll keep the sha's around than that's what we should do.
Yes, and we also should use *that tag* in the manifest!
Is it perhaps possible to improve "repo" instead?
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
On Mon, 29 Aug 2011 10:41:59 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 29 August 2011 10:13, Andy Green andy.green@linaro.org wrote:
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
I think this can just happen as part of the build cycle. If we track the tree directly the build system can lay down a branch automatically after release.
We could probably just update the branch after the automerge step. For those who are just getting their feet wet with CI the flow is:
Sync build Apply patch Build Regress build on regress success, merge patch Save build, gen pinned-manifest
Anyway, this isn't an issue with repo, its a sha1 reachability issue. repo 's just a foreach git tool.
Let's just look at it from different side - repo is not designed to work with manifests containing raw SHA revs, google doesn't bless that ;-)
Is it perhaps possible to improve "repo" instead?
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
Hello Andy,
On Mon, 29 Aug 2011 23:13:07 +0800 Andy Green andy.green@linaro.org wrote:
On 08/29/2011 09:22 PM, Somebody in the thread at some point said:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Sorry... this means for a rebase tree, we have to spawn a new branch per push to public git? We already spawn a tag per push to keep a finger on the tree so it won't be garbage collected: no great problem, but a branch per push?
That's why I was talking about LT and Android team doing it in manual sync (before release) for now. But yes, such transient branches (instead of tags) are needed (repo fetched all branches, but only tags directly accessible from those branches).
Is it perhaps possible to improve "repo" instead?
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
-Andy
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue?
Because we push everything upstream.
On 30 August 2011 10:43, Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue? -- Christian Robottom Reis, Engineering VP Brazil (GMT-3) | [+55] 16 9112 6430 | [+1] 612 216 4935 Linaro.org: Open Source Software for ARM SoCs
On Tue, Aug 30, 2011 at 10:46:55AM -0500, Zach Pfeffer wrote:
Because we push everything upstream.
While I agree with that blanket statement, there's no reason we wouldn't provide a [potentially temporary] version of repo that included the changes we're pushing upstream.
We do this for every one of the components we ship, so I see no philosophical reason why we wouldn't do so for repo.
Is there a technical reason?
On Tue, 30 Aug 2011 12:50:51 -0300, Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 10:46:55AM -0500, Zach Pfeffer wrote:
Because we push everything upstream.
While I agree with that blanket statement, there's no reason we wouldn't provide a [potentially temporary] version of repo that included the changes we're pushing upstream.
We do this for every one of the components we ship, so I see no philosophical reason why we wouldn't do so for repo.
Is there a technical reason?
Nope, there are two things as I see it:
1. knowing the the patch will go upstream eventually, or being willing to maintain a fork forever, or do the rework necessary to fix it. There are also related social issues, such as the possibility of being seen as a hostile fork shipping bad code or something.
2. Getting people to use our repo version with our trees. This is both a time problem (that doesn't really go away with upstreaming,) and a problem of inconveniencing people who are already working with repo elsewhere.
We make these decisions all the time in Linaro, and we should evaluate these things here.
Any other issues that I've missed? Where should we come down in this case?
Thanks,
James
On 30 August 2011 10:58, James Westby james.westby@canonical.com wrote:
On Tue, 30 Aug 2011 12:50:51 -0300, Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 10:46:55AM -0500, Zach Pfeffer wrote:
Because we push everything upstream.
While I agree with that blanket statement, there's no reason we wouldn't provide a [potentially temporary] version of repo that included the changes we're pushing upstream.
We do this for every one of the components we ship, so I see no philosophical reason why we wouldn't do so for repo.
Is there a technical reason?
Nope, there are two things as I see it:
1. knowing the the patch will go upstream eventually, or being willing to maintain a fork forever, or do the rework necessary to fix it. There are also related social issues, such as the possibility of being seen as a hostile fork shipping bad code or something.
2. Getting people to use our repo version with our trees. This is both a time problem (that doesn't really go away with upstreaming,) and a problem of inconveniencing people who are already working with repo elsewhere.
We make these decisions all the time in Linaro, and we should evaluate these things here.
Any other issues that I've missed? Where should we come down in this case?
I say we stay the course and fix the issue of sha's disappearing. I feel this way because its:
1. Technically feasible in the short term. 2. Saves the substantial burden of redirecting to the user. 3. Supports the only way that you can guarantee 100% build fidelity.
That said I think we should upstream to repo to help out the general community and based on the results of that see where we go.
Thanks,
James
On Tue, Aug 30, 2011 at 11:13:08AM -0500, Zach Pfeffer wrote:
Any other issues that I've missed? Where should we come down in this case?
I say we stay the course and fix the issue of sha's disappearing. I feel this way because its:
- Technically feasible in the short term.
- Saves the substantial burden of redirecting to the user.
- Supports the only way that you can guarantee 100% build fidelity.
Can you restate "fix the issue of sha's [sic] disappearing" in a way which I can understand? As far as I know, the actual problem is that repo isn't fetching all tags.
Or are you describing the symptom as seen from the build system? Because my suggestion (ship a forked repo temporarily) would address that.
On 1 September 2011 05:59, Christian Robottom Reis kiko@canonical.com wrote:
On Tue, Aug 30, 2011 at 11:13:08AM -0500, Zach Pfeffer wrote:
Any other issues that I've missed? Where should we come down in this case?
I say we stay the course and fix the issue of sha's disappearing. I feel this way because its:
- Technically feasible in the short term.
- Saves the substantial burden of redirecting to the user.
- Supports the only way that you can guarantee 100% build fidelity.
Can you restate "fix the issue of sha's [sic] disappearing" in a way which I can understand? As far as I know, the actual problem is that repo isn't fetching all tags.
Or are you describing the symptom as seen from the build system? Because my suggestion (ship a forked repo temporarily) would address that.
Here's the symptom. Android uses a tool called repo to fetch all the source code in a project. repo reads a manifest file that lists each git it needs to clone. The syntax used to specify the git includes a feature that allows you to fetch a branch, tag or specific revision. An example of an entry of this manifest file that fetches a particular branch is:
<remote name="linaro-android" fetch="git://android.git.linaro.org/" review="review.android.git.linaro.org"/> <project path="device/linaro/beagleboard" name="device/linaro/beagleboard" revision="linaro_android_2.3.5" remote="linaro-android"/>
We use this for our tip builds.
An example that fetches a particular revision is:
<remote fetch="git://android.git.linaro.org/" name="linaro-android" review="review.android.git.linaro.org"/> <project name="platform/build" path="build" remote="linaro-android" revision="45f7969d21930a1ffa8776cd643e1637a88ce632">
We use entries like this for our release builds.
We do this because the sha1 allows for the exact state of the git to be checked out. Its better than a tag or a branch because both are moveable and the tags tend to pileup. Its also very easy to create a "pinned-manifest" that just lists each sha; you can do it in automation, and you don't have to write the tags back to the subgits which you may not control.
The pinned-manifest.xml allows a user to create a build that what we built exactly; this is useful if someone is trying to bisect a bug on the exact version of a build or they want to perform an experiment on the exact version.
Developers who just want to work on tip, which is most developers, can just pull the manifest that tracks a particular branch. Developers are encouraged to just use tip since the pinned-manifests may not reflect the current build state.
When I sync a build using the pinned-manifest.xml then I essentially do a for each git in manifest x operation. If the sha doesn't exist the repo sync fails and the user has to start over from scratch.
Why would a sha disappear?
git-gc handles clean up of unreachable objects. git-gc (http://www.kernel.org/pub/software/scm/git/docs/v1.7.6.1/git-gc.html)
Commits that are reachable from (taken from man):
Objects referenced by your current set of branches and tags Objects referenced by the index Remote-tracking branches Refs saved by git filter-branch in refs/original/ Reflogs (which may reference commits in branches that were later amended or rewound).
So then what happens in a typical repo init, repo sync
People who want to use a pinned-manifest.xml will run something like:
repo init -u git://git.linaro.org/android/platform/manifests.git -b linaro-android-11.08-release -m LEB-panda.xml repo sync
Prefix both with REPO_TRACE=1 to see exactly what git commands get executed.
We create release builds using the pinned-manifest.xml since we want to reproduce the builds exactly. Again we use the sha's because we may not be able to lay a tag on a git we don't own.
under the covers repo sync is doing a few things (see REPO_TRACE=1 for the whole list), but it basically comes down to:
For each git: git init git fetch git pack-refs git update-ref (sha should be valid, points HEAD to branch head or sha)
For each git git gc, to save space
For each git git read-tree ..., to bring the tree up-to-date
So repo is really doing a good job. Its saving space by getting rid of things, limiting disk-space usage, etc...and is really just using some low level git commands to accomplish its job.
Anyhow, we have a few ways to solve this issue. I would say that as long as we reference trees whose shas are reachable from the branches in the remote git, then we should be fine. From a manifest perspective, we like to sync a tip branch, once the build is completed the sha will be the tip of that tip branch, as long as that sha sticks around in another branch in the repo everything should work.
-- Christian Robottom Reis | [+55 16] 3376 0125 | http://launchpad.net/~kiko Canonical Ltd. | [+55 16] 9112 6430 | http://async.com.br/~kiko
On Thu, 1 Sep 2011 16:51:25 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Anyhow, we have a few ways to solve this issue. I would say that as long as we reference trees whose shas are reachable from the branches in the remote git, then we should be fine. From a manifest perspective, we like to sync a tip branch, once the build is completed the sha will be the tip of that tip branch, as long as that sha sticks around in another branch in the repo everything should work.
Andy's question earlier in this threads was whether we had to use branches here. It seems you are saying that we are going to use branches?
Thanks,
James
On 1 September 2011 17:04, James Westby james.westby@canonical.com wrote:
On Thu, 1 Sep 2011 16:51:25 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Anyhow, we have a few ways to solve this issue. I would say that as long as we reference trees whose shas are reachable from the branches in the remote git, then we should be fine. From a manifest perspective, we like to sync a tip branch, once the build is completed the sha will be the tip of that tip branch, as long as that sha sticks around in another branch in the repo everything should work.
Andy's question earlier in this threads was whether we had to use branches here. It seems you are saying that we are going to use branches?
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Thanks,
James
On Thu, 1 Sep 2011 17:10:03 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Paul is working on a change to make tags usable as well. It seems to me that we have a couple of options
1) Use branches long term. Paul can continue to push the patch for correctness sake, but we don't plan to use it.
2) Use branches for now. Paul fixes repo to make tags usable, at which point we allow people (e.g. Andy) to use those too.
Which would you advocate?
Thanks,
James
On 1 September 2011 17:21, James Westby james.westby@canonical.com wrote:
On Thu, 1 Sep 2011 17:10:03 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Paul is working on a change to make tags usable as well. It seems to me that we have a couple of options
1) Use branches long term. Paul can continue to push the patch for correctness sake, but we don't plan to use it.
#1 because from what I can see, that is more inline with the original repo vision, if Paul's change gets in and makes people rethink things than I think that'll be great!
2) Use branches for now. Paul fixes repo to make tags usable, at which point we allow people (e.g. Andy) to use those too.
Which would you advocate?
Thanks,
James
On Thu, 1 Sep 2011 17:54:55 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 1 September 2011 17:21, James Westby james.westby@canonical.com wrote:
On Thu, 1 Sep 2011 17:10:03 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Paul is working on a change to make tags usable as well. It seems to me that we have a couple of options
1) Use branches long term. Paul can continue to push the patch for correctness sake, but we don't plan to use it.
#1 because from what I can see, that is more inline with the original repo vision, if Paul's change gets in and makes people rethink things than I think that'll be great!
Ok, then I guess that leaves Andy unhappy, but I think everyone has a clearer picture of the current situation and the plans now.
Thanks,
James
On Thu, 01 Sep 2011 18:21:30 -0400 James Westby james.westby@canonical.com wrote:
On Thu, 1 Sep 2011 17:10:03 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Paul is working on a change to make tags usable as well. It seems to me that we have a couple of options
- Use branches long term. Paul can continue to push the patch for
correctness sake, but we don't plan to use it.
- Use branches for now. Paul fixes repo to make tags usable, at
which point we allow people (e.g. Andy) to use those too.
Which would you advocate?
But we already seem to have decided to use patched repo, no??
Thanks,
James
On 2 September 2011 12:39, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Thu, 01 Sep 2011 18:21:30 -0400 James Westby james.westby@canonical.com wrote:
On Thu, 1 Sep 2011 17:10:03 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
We've had two instances of this problem for all the builds we've ever done. One instance was trying to sync u-boot and the other was trying to sync the TI LT kernel. John Rigby suggested the branch approach and it seems to have worked, so I'd like to go with that.
Paul is working on a change to make tags usable as well. It seems to me that we have a couple of options
1) Use branches long term. Paul can continue to push the patch for correctness sake, but we don't plan to use it.
2) Use branches for now. Paul fixes repo to make tags usable, at which point we allow people (e.g. Andy) to use those too.
Which would you advocate?
But we already seem to have decided to use patched repo, no??
Seems to be the consensus.
Would you write up where to pull Linaro's repo from and why its different? Make sure the release engineers are in the loop. Adding fabo.
Thanks,
James
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hello Fathi,
On Fri, 2 Sep 2011 13:19:11 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
But we already seem to have decided to use patched repo, no??
Seems to be the consensus.
Would you write up where to pull Linaro's repo from and why its different? Make sure the release engineers are in the loop. Adding fabo.
The patches are available for review at:
http://review.android.git.linaro.org/155 http://review.android.git.linaro.org/156
As soon as they approved, I'll be bale to provide downlike link to make a release. Fathi, would it make sense for you to register in Gerrit, to be able to be added as reviewer for patches which may be of interest to you, to stay in loop of their lifecycle? If so, please follow https://wiki.linaro.org/Platform/Android/Gerrit#Creating_Gerrit_Account
Also note that teh issue affects 11.08 release, so please be ready to update release notes, errata for it.
In the meantime, I drafted a description page, feel free to update it as needed: https://wiki.linaro.org/Platform/Android/PatchedRepo
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as: http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Hello Fathi,
On Fri, 2 Sep 2011 13:19:11 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
But we already seem to have decided to use patched repo, no??
Seems to be the consensus.
Would you write up where to pull Linaro's repo from and why its different? Make sure the release engineers are in the loop. Adding fabo.
The patches are available for review at:
http://review.android.git.linaro.org/155 http://review.android.git.linaro.org/156
As soon as they approved, I'll be bale to provide downlike link to make a release. Fathi, would it make sense for you to register in Gerrit, to be able to be added as reviewer for patches which may be of interest to you, to stay in loop of their lifecycle? If so, please follow https://wiki.linaro.org/Platform/Android/Gerrit#Creating_Gerrit_Account
Also note that teh issue affects 11.08 release, so please be ready to update release notes, errata for it.
In the meantime, I drafted a description page, feel free to update it as needed: https://wiki.linaro.org/Platform/Android/PatchedRepo
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
Cheers,
Fathi
On 6 September 2011 08:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
the patched version is available on: http://releases.linaro.org/platform/linaro-n/android/11.08/repo/repo-linaro-... http://launchpad.net/linaro-android/11.11/11.08/+download/repo-linaro-1.7.5-...
and linked from: http://www.linaro.org/downloads/ http://wiki.linaro.org/Cycles/1108/Release#Android_Components
Cheers,
Fathi
On Tue, Sep 6, 2011 at 4:25 PM, Fathi Boudra fathi.boudra@linaro.orgwrote:
On 6 September 2011 08:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep...
that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
the patched version is available on:
http://releases.linaro.org/platform/linaro-n/android/11.08/repo/repo-linaro-...
http://launchpad.net/linaro-android/11.11/11.08/+download/repo-linaro-1.7.5-...
and linked from: http://www.linaro.org/downloads/ http://wiki.linaro.org/Cycles/1108/Release#Android_Components
hmm. the AOSP repo can be wgetted without unpacking ...you basically just download the lightweight wrapper script instead of a tarball. IMO we should offer the same on top of the whole tarball as a component.
Also this is not a 1108 component. it's 11.09 if anything.
On 6 September 2011 17:28, Alexander Sack asac@linaro.org wrote:
On Tue, Sep 6, 2011 at 4:25 PM, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 08:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
the patched version is available on:
http://releases.linaro.org/platform/linaro-n/android/11.08/repo/repo-linaro-...
http://launchpad.net/linaro-android/11.11/11.08/+download/repo-linaro-1.7.5-...
and linked from: http://www.linaro.org/downloads/ http://wiki.linaro.org/Cycles/1108/Release#Android_Components
hmm. the AOSP repo can be wgetted without unpacking ...you basically just download the lightweight wrapper script instead of a tarball. IMO we should offer the same on top of the whole tarball as a component.
I don't see any difference between wget && chmod +x vs wget && bunzip2 I don't have a strong opinion on how AOSP does vs how Linaro components does.
Also this is not a 1108 component. it's 11.09 if anything.
Paul mentioned an 11.08 errata.
On Tue, 6 Sep 2011 17:34:11 +0300 Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 17:28, Alexander Sack asac@linaro.org wrote:
On Tue, Sep 6, 2011 at 4:25 PM, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 08:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
the patched version is available on:
http://releases.linaro.org/platform/linaro-n/android/11.08/repo/repo-linaro-...
http://launchpad.net/linaro-android/11.11/11.08/+download/repo-linaro-1.7.5-...
and linked from: http://www.linaro.org/downloads/ http://wiki.linaro.org/Cycles/1108/Release#Android_Components
hmm. the AOSP repo can be wgetted without unpacking ...you basically just download the lightweight wrapper script instead of a tarball. IMO we should offer the same on top of the whole tarball as a component.
I don't see any difference between wget && chmod +x vs wget && bunzip2 I don't have a strong opinion on how AOSP does vs how Linaro components does.
As pointed out by folks, it's rather expectable to fetch bootstrap script and be running it directly. As to where to place it, we don't have direct access to android.git.linaro.org, so either that needs to go thru IS (James, could you handle this?), or I'd suggest putting it at somewhere like
http://releases.linaro.org/platform/android/repo
(yep, non-milestoned meta-util).
Also this is not a 1108 component. it's 11.09 if anything.
Paul mentioned an 11.08 errata.
Yes, I meant that patched repo we release now is needed to properly fetch previous release 11.08 (and likely, older too). So, for 11.08 release notes, errata should be issued. On asac's request, I'm doing complete matrix testing of 11.08 fetchability, and will provide detailed results a bit later.
On 6 September 2011 19:03, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Tue, 6 Sep 2011 17:34:11 +0300 Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 17:28, Alexander Sack asac@linaro.org wrote:
On Tue, Sep 6, 2011 at 4:25 PM, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 08:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote: > > On Mon, 5 Sep 2011 14:36:48 +0300 > Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote: > > Ok, patched repo is available as: > > > http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... > that file should be downloaded and named as "repo". > Apparently, it should be mirrored at download servers. >
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
the patched version is available on:
http://releases.linaro.org/platform/linaro-n/android/11.08/repo/repo-linaro-...
http://launchpad.net/linaro-android/11.11/11.08/+download/repo-linaro-1.7.5-...
and linked from: http://www.linaro.org/downloads/ http://wiki.linaro.org/Cycles/1108/Release#Android_Components
hmm. the AOSP repo can be wgetted without unpacking ...you basically just download the lightweight wrapper script instead of a tarball. IMO we should offer the same on top of the whole tarball as a component.
I don't see any difference between wget && chmod +x vs wget && bunzip2 I don't have a strong opinion on how AOSP does vs how Linaro components does.
As pointed out by folks, it's rather expectable to fetch bootstrap script and be running it directly.
I'll put the plain script as you request but it doesn't change the fact that you need to write 2 commands in any case.
As to where to place it, we don't have direct access to android.git.linaro.org, so either that needs to go thru IS (James, could you handle this?), or I'd suggest putting it at somewhere like
http://releases.linaro.org/platform/android/repo
(yep, non-milestoned meta-util).
I'll prefer to have it milestoned like any Linaro released components: http://releases.linaro.org/platform/linaro-n/android/11.08/repo/
If for unknown reason, you need to release another version, latest released version is available with: http://releases.linaro.org/platform/linaro-n/android/latest/repo/
Also this is not a 1108 component. it's 11.09 if anything.
Paul mentioned an 11.08 errata.
Yes, I meant that patched repo we release now is needed to properly fetch previous release 11.08 (and likely, older too). So, for 11.08 release notes, errata should be issued. On asac's request, I'm doing complete matrix testing of 11.08 fetchability, and will provide detailed results a bit later.
On 6 September 2011 00:28, Fathi Boudra fathi.boudra@linaro.org wrote:
On 6 September 2011 00:15, Alexander Sack asac@linaro.org wrote:
On Mon, Sep 5, 2011 at 10:37 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 5 Sep 2011 14:36:48 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Ok, patched repo is available as:
http://android.git.linaro.org/gitweb?p=tools/repo.git%3Ba=blob_plain%3Bf=rep... that file should be downloaded and named as "repo". Apparently, it should be mirrored at download servers.
Right. where shall we put it? i think for AOSP you get it from an android.git.k.o URL. do we want to do something equivalent (e.g. android.git.l.o) or just put it on releases.linaro.org?
It should be treated as a released components: releases.l.o, update release wiki/website download links. If it could be provided as upstream does (on android.git.l.o), +1.
Even though we're cutting our own repo as an interm solution, we should maintain the same repo usage where repo downloads the latest version of itself (from us via git) which means we should put it somewhere where the path won't change.
Cheers,
Fathi
On 30 August 2011 10:50, Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 10:46:55AM -0500, Zach Pfeffer wrote:
Because we push everything upstream.
While I agree with that blanket statement, there's no reason we wouldn't provide a [potentially temporary] version of repo that included the changes we're pushing upstream.
We do this for every one of the components we ship, so I see no philosophical reason why we wouldn't do so for repo.
Is there a technical reason?
There's no reason to do it now, because its solving the wrong problem. The problem is sha's disappear. We use sha's because the provide immovable references to the state of a set of git trees so that people can reproduce builds exactly. We don't need the change to tag a build after its been deemed correct, we just need to implement a function to tag across the gits so that all the shas continue to exist.
In addition, telling people to use Linaro'r repo over Android's is an additional burden to the user and a barrier to the use of our builds. If the changes value is greater than this burden than it would be good to direct people to use our repo, if we can take a different approach that saves us this redirection, while that change is upstreamed we should do do that thing since redirecting people to use our repo is a significant burden to the user.
-- Christian Robottom Reis, Engineering VP Brazil (GMT-3) | [+55] 16 9112 6430 | [+1] 612 216 4935 Linaro.org: Open Source Software for ARM SoCs
On Tue, 30 Aug 2011 11:10:10 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
There's no reason to do it now, because its solving the wrong problem. The problem is sha's disappear. We use sha's because the provide immovable references to the state of a set of git trees so that people can reproduce builds exactly. We don't need the change to tag a build after its been deemed correct, we just need to implement a function to tag across the gits so that all the shas continue to exist.
I'm sorry, but I'm not sure what course of action you are advocating here?
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
Thanks,
James
On Wed, Aug 31, 2011 at 2:09 AM, James Westby james.westby@canonical.comwrote:
On Tue, 30 Aug 2011 11:10:10 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
There's no reason to do it now, because its solving the wrong problem. The problem is sha's disappear. We use sha's because the provide immovable references to the state of a set of git trees so that people can reproduce builds exactly. We don't need the change to tag a build after its been deemed correct, we just need to implement a function to tag across the gits so that all the shas continue to exist.
I'm sorry, but I'm not sure what course of action you are advocating here?
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
I think what he is saying is that everytime we produce a pinned-manifest.xml we would at best be able to prevent those revisions from getting garbage collected.
I think thats a reasonable vision in general. My main concern to start with is about tag inflation. Is there a thing like a 'hidden' tag in git that would allow that folks looking at a tree to still spot the 'real' tags and only see all the pinned/manifest tags upon request?
On 30 August 2011 19:48, Alexander Sack asac@linaro.org wrote:
On Wed, Aug 31, 2011 at 2:09 AM, James Westby james.westby@canonical.com wrote:
On Tue, 30 Aug 2011 11:10:10 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
There's no reason to do it now, because its solving the wrong problem. The problem is sha's disappear. We use sha's because the provide immovable references to the state of a set of git trees so that people can reproduce builds exactly. We don't need the change to tag a build after its been deemed correct, we just need to implement a function to tag across the gits so that all the shas continue to exist.
I'm sorry, but I'm not sure what course of action you are advocating here?
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
I think what he is saying is that everytime we produce a pinned-manifest.xml we would at best be able to prevent those revisions from getting garbage collected.
I think thats a reasonable vision in general. My main concern to start with is about tag inflation. Is there a thing like a 'hidden' tag in git that would allow that folks looking at a tree to still spot the 'real' tags and only see all the pinned/manifest tags upon request?
The tag thing is a means to an end, to keep the sha's around. If we just tracked history trees we'd be fine, but all the rebasing causes refs to become unreachable. All trees in Android should be history trees so we shouldn't have a problem, but not everyone see's things that way so we have the current disappearing sha problem.
--
- Alexander
On Tue, 30 Aug 2011 22:40:57 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
The tag thing is a means to an end, to keep the sha's around. If we just tracked history trees we'd be fine, but all the rebasing causes refs to become unreachable. All trees in Android should be history trees so we shouldn't have a problem, but not everyone see's things that way so we have the current disappearing sha problem.
Well, asking integration (aka Landing) teams to stop using integration trees (those which are rebased up and down), a legitimate and best practice for their kind of usage, is for sure too high a price for wanting to use SHAs in manifests.
On 30 August 2011 19:09, James Westby james.westby@canonical.com wrote:
On Tue, 30 Aug 2011 11:10:10 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
There's no reason to do it now, because its solving the wrong problem. The problem is sha's disappear. We use sha's because the provide immovable references to the state of a set of git trees so that people can reproduce builds exactly. We don't need the change to tag a build after its been deemed correct, we just need to implement a function to tag across the gits so that all the shas continue to exist.
I'm sorry, but I'm not sure what course of action you are advocating here?
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
Yup.
Thanks,
James
On Tue, 30 Aug 2011 22:38:22 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
Yup.
We can't currently do this though.
Paul has a patch intended to allow us to do this, but it's not been accepted upstream.
So in the short term we either have to have some other approach, or we have to use a customized version of repo.
If you don't want to use a patched repo, then what's the alternative approach that we will use until a fix has been accepted upstream?
Thanks,
James
On 31 August 2011 08:06, James Westby james.westby@canonical.com wrote:
On Tue, 30 Aug 2011 22:38:22 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
It sounds like you are advocating using a model where we use tags to ensure that the referenced revisions are reachable from a head, and then refer to the sha ids of the revisions in the manifest. Is that correct?
Yup.
We can't currently do this though.
Paul has a patch intended to allow us to do this, but it's not been accepted upstream.
So in the short term we either have to have some other approach, or we have to use a customized version of repo.
If you don't want to use a patched repo, then what's the alternative approach that we will use until a fix has been accepted upstream?
Can we tag across all the gits in the manifest?
Thanks,
James
On Wed, 31 Aug 2011 13:12:30 -0500, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Can we tag across all the gits in the manifest?
I think you can do anything you want with
repo forall git tag foo
However, we've already established that tags aren't currently sufficient to keep all these refs accessible, so this seems like it's still a solution depending on changing repo (which they have so far rejected.)
Thanks,
James
On 30 August 2011 10:50, Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 10:46:55AM -0500, Zach Pfeffer wrote:
Because we push everything upstream.
While I agree with that blanket statement, there's no reason we wouldn't provide a [potentially temporary] version of repo that included the changes we're pushing upstream.
We do this for every one of the components we ship, so I see no philosophical reason why we wouldn't do so for repo.
Is there a technical reason?
repo bootstraps itself by syncing the repo guts from google, google signs the repo guts so that people know that they're using a trusted source. As our repo in not a trusted source, people may be reluctant to use it and will instead use Google's version which won't work 100% of the time for the reasons already discussed.
-- Christian Robottom Reis, Engineering VP Brazil (GMT-3) | [+55] 16 9112 6430 | [+1] 612 216 4935 Linaro.org: Open Source Software for ARM SoCs
On Fri, Sep 02, 2011, Zach Pfeffer wrote:
repo bootstraps itself by syncing the repo guts from google, google signs the repo guts so that people know that they're using a trusted source. As our repo in not a trusted source, people may be reluctant to use it and will instead use Google's version which won't work 100% of the time for the reasons already discussed.
I think there's an env var to disable the auto-update of repo; we could set it in our instructions or in build wrappers, and/or we could patch repo to update itself from linaro.org instead.
Hello Loïc,
On Fri, 2 Sep 2011 23:18:33 +0200 Loïc Minier loic.minier@linaro.org wrote:
On Fri, Sep 02, 2011, Zach Pfeffer wrote:
repo bootstraps itself by syncing the repo guts from google, google signs the repo guts so that people know that they're using a trusted source. As our repo in not a trusted source, people may be reluctant to use it and will instead use Google's version which won't work 100% of the time for the reasons already discussed.
I think there's an env var to disable the auto-update of repo; we could set it in our instructions or in build wrappers, and/or we could patch repo to update itself from linaro.org instead.
There's switch telling where to update from, it is set to the right value on Android Build. There's actually even switch which disables auto-update (or one which you'd think does that), but it doesn't work, surprise (typical surprise with all those repo/gerrit tools though).
Hello Christian,
On Tue, 30 Aug 2011 12:43:40 -0300 Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue?
By the same reason we don't fork entire Android itself and fixing it to work "right"? ;-) "repo" is kind of foundation of Android development, axiom or something. It's like saying that someone can't do kernel development because git is broken, so one must stop, fix git in incompatible way and then distribute one's tree requiring that git version. Hardly would work. Productive way is: 1) adjust to existing set of axioms; 2) slowly push for them to be changed if adjusting caused continued everyday pain.
On Wed, Aug 31, 2011 at 02:19:08PM +0300, Paul Sokolovsky wrote:
Hello Christian,
On Tue, 30 Aug 2011 12:43:40 -0300 Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue?
By the same reason we don't fork entire Android itself and fixing it to work "right"? ;-)
Either I'm missing the point or you are. We "fork" Android and everything else all the time; we then proceed to send the patches upstream and help them through the process, and so for me the "fork" is more like a cooperative branch.
Look, this is a weird conversation and I want to get out of it as soon as I can. But you guys are overblowing the issue -- I'm just suggesting that if the patch will take a while to go upstream, you can ship a separate repo tool. We do this all the time in Linaro (just look at www.linaro.org/downloads).
It doesn't mean the patch won't go upstream eventually, nor that you won't find another solution -- it just means this thread can end and we can go on working without getting other teams all excited about having to change their source control working models.
On Thu, Sep 1, 2011 at 1:05 PM, Christian Robottom Reis kiko@linaro.orgwrote:
On Wed, Aug 31, 2011 at 02:19:08PM +0300, Paul Sokolovsky wrote:
Hello Christian,
On Tue, 30 Aug 2011 12:43:40 -0300 Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue?
By the same reason we don't fork entire Android itself and fixing it to work "right"? ;-)
Either I'm missing the point or you are. We "fork" Android and everything else all the time; we then proceed to send the patches upstream and help them through the process, and so for me the "fork" is more like a cooperative branch.
Look, this is a weird conversation and I want to get out of it as soon as I can. But you guys are overblowing the issue -- I'm just suggesting that if the patch will take a while to go upstream, you can ship a separate repo tool. We do this all the time in Linaro (just look at www.linaro.org/downloads).
I agree with kiko here. If the repo patch would have a clearer indication in gerrit by now that this is not acceptable upstream I would have agreed to think about a solution on the git branch policy side. But if the lack of --tag hurts us now badly we should work around somehow by patching repo etc. until that patch discussion gives us more assurance that adopting everyone's workflow is not just for the sake of a temporary bandaid.
On Thu, 1 Sep 2011 13:23:37 +0200 Alexander Sack asac@linaro.org wrote:
On Thu, Sep 1, 2011 at 1:05 PM, Christian Robottom Reis kiko@linaro.orgwrote:
On Wed, Aug 31, 2011 at 02:19:08PM +0300, Paul Sokolovsky wrote:
Hello Christian,
On Tue, 30 Aug 2011 12:43:40 -0300 Christian Robottom Reis kiko@linaro.org wrote:
On Tue, Aug 30, 2011 at 01:25:15PM +0300, Paul Sokolovsky wrote:
Yeah, I have patch for that. But we cannot use before upstream accepts it (because people will get errors checking out tree) and I don't hold my breath for that at all.
Why wouldn't we provide our own version of repo that worked around this issue?
By the same reason we don't fork entire Android itself and fixing it to work "right"? ;-)
Either I'm missing the point or you are. We "fork" Android and everything else all the time; we then proceed to send the patches upstream and help them through the process, and so for me the "fork" is more like a cooperative branch.
Look, this is a weird conversation and I want to get out of it as soon as I can. But you guys are overblowing the issue -- I'm just suggesting that if the patch will take a while to go upstream, you can ship a separate repo tool. We do this all the time in Linaro (just look at www.linaro.org/downloads).
I agree with kiko here. If the repo patch would have a clearer indication in gerrit by now that this is not acceptable upstream I
It seems that they're really not against it. My concern was that tag fetching was omitted deliberately to save on traffic and they would be reluctant to add it. But it turned out that's actually not that easy to do that on git's side using raw git fetch, in particular, "git fetch --tags", like I patched it first, indeed fetches *only* tags (and git docs are confusing in this respect). But they even suggested how to do it right, which I just have tested to work for our case:
https://pastebin.linaro.org/220/ https://pastebin.linaro.org/221/ https://pastebin.linaro.org/222/
review.source.android.com is down now (hmm, is it hosted on the same host as android.git.kernel.org ?), but I'll prepare new version of the patch when it's up.
would have agreed to think about a solution on the git branch policy side. But if the lack of --tag hurts us now badly we should work around somehow by patching repo etc. until that patch discussion gives us more assurance that adopting everyone's workflow is not just for the sake of a temporary bandaid.
Well, it's very easy for me to prepare a new repo branch and hand out a download link to release managers if that's what we want. But as it's just milestone start, let's hope it might land in upstream indeed.
On Mon, Aug 29, 2011 at 3:22 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Paul, what happened to the repo patch we had that would also fetch tags? was that ever submit this to gerrit? on the mailing list we didn't receive any resistance to fix this. If its an easy landing upstream I would prefer that over adding "create branch for everything" policy.
On Mon, 29 Aug 2011 18:42:29 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Aug 29, 2011 at 3:22 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Paul, what happened to the repo patch we had that would also fetch tags? was that ever submit this to gerrit? on the mailing list we didn't receive any resistance to fix this. If its an easy landing upstream I would prefer that over adding "create branch for everything" policy.
It waited its turn in queue. Well, now here it is: https://review.source.android.com/25843
On 30 August 2011 05:46, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Mon, 29 Aug 2011 18:42:29 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Aug 29, 2011 at 3:22 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Paul, what happened to the repo patch we had that would also fetch tags? was that ever submit this to gerrit? on the mailing list we didn't receive any resistance to fix this. If its an easy landing upstream I would prefer that over adding "create branch for everything" policy.
It waited its turn in queue. Well, now here it is: https://review.source.android.com/25843
Paul, you may want to proactively find someone to review that. If you don't it'll just sit here.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Tue, Aug 30, 2011 at 12:46 PM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
On Mon, 29 Aug 2011 18:42:29 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Aug 29, 2011 at 3:22 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
It's not enough if you still want to refer to it via SHA, due to repo peculiarities. It should be also reachable from one of the live branches (so, instead of a tag, a branch can be created right away).
Paul, what happened to the repo patch we had that would also fetch tags? was that ever submit this to gerrit? on the mailing list we didn't receive any resistance to fix this. If its an easy landing upstream I would prefer that over adding "create branch for everything" policy.
It waited its turn in queue. Well, now here it is: https://review.source.android.com/25843
fantastic! Seems it's going well so far.
-
Alexander Sack -
Andy Green -
Christian Robottom Reis -
Christian Robottom Reis -
Fathi Boudra -
James Westby -
James Westby -
Loïc Minier -
Paul Sokolovsky -
Zach Pfeffer