In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
-Zach
On Mon, 2011-07-11 at 09:11 -0500, Zach Pfeffer wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
Yea. I'm totally fine with adding tags, and we're working on the kernel team to get a consistent tagging method created.
That said, wouldn't using the specific sha1s in the manifest be a lower overhead solution?
Otherwise you run into oddball issues where a "-final" tag is created, and a build is made, and a bug is found, and fix committed, and that fix is tested and then we say ok "-final-forrealthistime", etc.
Otherwise, we need to do a loop where the current tree is built, tested, and "acked" in some way. Then we make the tag and build the "official" release.
So if build reproduction is the goal, creating the manifests using the current sha1 hashes allows for that in a very simple way with less overhead or complications.
That said, having human usable names is definitely useful, and the plan is for there to be consistent tagging between trees so we can easily tell what release is based on what point in a tree (so we know which linaro tree the linaro+android tree contains). However, these tags will most likely be date based (something like: linaro-3.0-11.07.12-android-15), and thus somewhat too coarse if there are numerous last minute builds (in which case sha1s are better because they are much more exact).
thanks -john
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
As Alexander says, we need to be able to reference the sha since our "pinned-manifest" list all the subgit syncables by sha.
Like this:
https://android-build.linaro.org/jenkins/job/linaro-android_beagle-11.07-rel...
-Zach
On 11 July 2011 16:18, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
--
- Alexander
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
Cheers,
On Tue, Jul 12, 2011 at 1:07 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
If I read the prior emails correctly then the commit needs to be reachable by a head but not necessarily "the" head as in master so I believe that I can still rebase master (or whatever) as long as released commits are still accessible from some other head. So if each time a release happens a branch is created as well as a tag then there is no problem other than the noise of having these extra branches that only exist for keep tags or commits accessible. Please someone correct me if I am wrong.
--john
On Tue, Jul 12, 2011 at 5:13 AM, John Rigby john.rigby@linaro.org wrote:
On Tue, Jul 12, 2011 at 1:07 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
If I read the prior emails correctly then the commit needs to be reachable by a head but not necessarily "the" head as in master so I believe that I can still rebase master (or whatever) as long as released commits are still accessible from some other head. So if each time a release happens a branch is created as well as a tag then there is no problem other than the noise of having these extra branches that only exist for keep tags or commits accessible. Please someone correct me if I am wrong.
If using only released code is enough, then I believe this is a solution.
Problem is when Android starts to do continuous integration by building and using the git tree, but then don't know if there's a solution, unless we work to fix the tools.
Cheers,
Hello John,
On Tue, 12 Jul 2011 02:13:40 -0600 John Rigby john.rigby@linaro.org wrote:
[]
If I read the prior emails correctly then the commit needs to be reachable by a head but not necessarily "the" head as in master so I believe that I can still rebase master (or whatever) as long as released commits are still accessible from some other head. So if each time a release happens a branch is created as well as a tag then there is no problem other than the noise of having these extra branches that only exist for keep tags or commits accessible. Please someone correct me if I am wrong.
Yes, this should be the case, i.e. besides creating a tag, also to create a branch from the same underlying revision. Yes, that will lead to proliferation of branches (just as proliferation of tags is often the problem for may projects), but let's treat that as temporary solution while we settle this issue. We might end up with something else, or make policy to clean up old branches/tags after good timeout.
In the meantime, I posted about the problem to repo mailing list (cc: linaro-dev), initial response was positive, so maybe it would be fixed on repo side.
--john
On Tue, Jul 12, 2011 at 10:13 AM, John Rigby john.rigby@linaro.org wrote:
On Tue, Jul 12, 2011 at 1:07 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
If I read the prior emails correctly then the commit needs to be reachable by a head but not necessarily "the" head as in master so I believe that I can still rebase master (or whatever) as long as released commits are still accessible from some other head. So if each time a release happens a branch is created as well as a tag then there is no problem other than the noise of having these extra branches that only exist for keep tags or commits accessible. Please someone correct me if I am wrong.
you are on the spot; we are trying to fix repo upstream so you don't need a head (just a tag); until then keeping commits reachable through a head would be great!
On Tue, 2011-07-12 at 04:07 -0300, Ricardo Salveti wrote:
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Er. Forgive me if you were talking about JohnR, but I don't rebase my tree. I do create new branches for each kernel version, but the old branches remain.
thanks -john
It is fragile, but I think its the only way to do it and still give people the ability to create arbitrary builds easily, with 100% fidelity using a method that's easy to automate and that generally works. Since we need to use repo -r, we don't want to have people using side gits if we can help it, and we want to follow Google's processes as much as we can.
-Zach
On 12 July 2011 02:07, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Mon, Jul 11, 2011 at 6:18 PM, Alexander Sack asac@linaro.org wrote:
On Mon, Jul 11, 2011 at 4:11 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
In-order to make reproducible builds we create pinned manifests with each commit explicitly listed. We also use this method to create a release. We depend on these pinned commits - if they don't exist the "released" builds can no longer be reproduced.
One amend: the commits need to exist AND need to be reachable through a head.
In other words: due to how the repo tool work, tagging and then rebasing will not be good enough.
For me this seems to be quite fragile, as you're expecting the upstream tree for a component to not rebase the tree.
At least when looking at what happened with u-boot-linaro, where a rebase is expected by the way John is maintaining his tree, this method will fail unless you're building against a tag (as I believe git will respect the tag even if the tree was rebased in some way).
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
Cheers,
Ricardo Salveti de Araujo
On Tue, Jul 12, 2011 at 9:07 AM, Ricardo Salveti <ricardo.salveti@linaro.org
wrote:
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
if the problem is that you rely on external maintainers to 1) create a tag or a ref that reaches the commits you depend on for your release, or 2) do not rebase so that you don't loose your commit. it's likely that the only option is to mirror all trees you need so that you can tag them and have no external dependency. that's one of the reason why all projects are cloned on git.omapzoom.org, because we needed to get reproducible trees since OMAP Android releases are also based on 'static' manifests
Hello Nicolas,
On Wed, 13 Jul 2011 08:22:08 +0200 "Dechesne, Nicolas" n-dechesne@ti.com wrote:
On Tue, Jul 12, 2011 at 9:07 AM, Ricardo Salveti <ricardo.salveti@linaro.org
wrote:
Is there other way to fix this at the tool instead of forcing the component tree owner to not rebase the tree?
if the problem is that you rely on external maintainers to 1) create a tag or a ref that reaches the commits you depend on for your release, or 2) do not rebase so that you don't loose your commit. it's likely that the only option is to mirror all trees you need so that you can tag them and have no external dependency. that's one of the reason why all projects are cloned on git.omapzoom.org, because we needed to get reproducible trees since OMAP Android releases are also based on 'static' manifests
Well, yes, mirroring everything of course will be the most general solution, but would add burden on our side and complicate getting quick answer to the most basic question "what was changed in Linaro tree comparing to the upstream". So, while we're busy with setting up the rest of Android-related infrastructure, maybe it's worth trying to optimize existing set up even if it's heuristical and depends on some external conditions, maybe it will be quite workable in practice after all.
-
Alexander Sack -
Dechesne, Nicolas -
John Rigby -
John Stultz -
Paul Sokolovsky -
Ricardo Salveti -
Zach Pfeffer