Hello,
There was weird problem with android git mirror again (https://android-build.linaro.org/jenkins/job/patrik-ryd_lt-panda/4/consoleFu...):
error: revision tilt-linaro-android.38 in git.linaro.org/people/andygreen/kernel-tilt not found Archiving artifacts Finished: FAILURE
This was 100% reproducible, but that revision exists in the git repository on git.linaro.org, and actually it's being mirrored, as
git clone git://us-east-1.ec2-git-mirror.linaro.org/git.linaro.org/people/andygreen/kernel-tilt
Worked pretty well. But "repo sync" gave the error above, which well, pointed at some repo peculiarity.
There were similar issues already, with newly added git repos which lacked master branch. Adding master branch seemed to solve the issue. But kernel-tilt had master branch.
So, I for some time suspected that repo --mirror we use for mirroring doesn't make faithful enough mirror of git forest, and this time decided to dig out what is known about it.
First of all, while there's no --verbose or --debug switch for the repo, it's possible to make it dump git commands it executes by setting "REPO_TRACE=1" in the environment. I verified that very same git command, when run directly against git.linaro.org, works well, but leads to repo error quoted above if run against mirror.
I then googled for known issues with repo --mirror, couldn't find much, but there're few reports of obscure issues:
http://groups.google.com/group/repo-discuss/browse_thread/thread/b9752498836...
This is even more interesting, with Google people confirming that repo --mirror works within some bounds and limitations:
http://groups.google.com/group/repo-discuss/browse_thread/thread/401656c3ad0...
Finally, here's typical usage scenario for repo --mirror (which is pretty underdocumented):
http://www.excentral.org/archives/2011/02/24/android-repo-mirroring
So, like "repo help init" says, mirror created with repo --mirror should later be used with repo --reference, with both mirror and checkout using it to be located on local filesystem. I couldn't find referenced that repo --mirror is intended for use like we do - to serve it using git-daemon.
My next step was trying to do real git mirror (git clone --mirror), and already did it and prepared to move it into intended location, when I did ls -l in the mirror directory:
drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-02 00:11 kernel-tilt.git drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-01 12:30 kernel-tilt.git.git
Once I moved that ".git.git" repo away, original issue with repo sync resolved. I have no idea that double-.git repo was created. Based on the timestamp, it was created first, and later barely updated, with mirror and just git clone using the proper repo, while repo sync on the build slave managing to pick up wrong one. I wonder, if that (creation of 2 repos) could be a result of some race condition.
And I wish this issue could be called resolved, but as quoted above, there're (more) issues are known with git --mirror, so I decided to put up this mail with the links, for the next time such issues may come.
On Thu, 2 Jun 2011 10:24:12 +0300, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
There was weird problem with android git mirror again (https://android-build.linaro.org/jenkins/job/patrik-ryd_lt-panda/4/consoleFu...):
error: revision tilt-linaro-android.38 in git.linaro.org/people/andygreen/kernel-tilt not found Archiving artifacts Finished: FAILURE
This was 100% reproducible, but that revision exists in the git repository on git.linaro.org, and actually it's being mirrored, as
git clone git://us-east-1.ec2-git-mirror.linaro.org/git.linaro.org/people/andygreen/kernel-tilt
Worked pretty well. But "repo sync" gave the error above, which well, pointed at some repo peculiarity.
There were similar issues already, with newly added git repos which lacked master branch. Adding master branch seemed to solve the issue. But kernel-tilt had master branch.
So, I for some time suspected that repo --mirror we use for mirroring doesn't make faithful enough mirror of git forest, and this time decided to dig out what is known about it.
First of all, while there's no --verbose or --debug switch for the repo, it's possible to make it dump git commands it executes by setting "REPO_TRACE=1" in the environment. I verified that very same git command, when run directly against git.linaro.org, works well, but leads to repo error quoted above if run against mirror.
I then googled for known issues with repo --mirror, couldn't find much, but there're few reports of obscure issues:
http://groups.google.com/group/repo-discuss/browse_thread/thread/b9752498836...
This is even more interesting, with Google people confirming that repo --mirror works within some bounds and limitations:
http://groups.google.com/group/repo-discuss/browse_thread/thread/401656c3ad0...
Finally, here's typical usage scenario for repo --mirror (which is pretty underdocumented):
http://www.excentral.org/archives/2011/02/24/android-repo-mirroring
So, like "repo help init" says, mirror created with repo --mirror should later be used with repo --reference, with both mirror and checkout using it to be located on local filesystem. I couldn't find referenced that repo --mirror is intended for use like we do - to serve it using git-daemon.
My next step was trying to do real git mirror (git clone --mirror), and already did it and prepared to move it into intended location, when I did ls -l in the mirror directory:
drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-02 00:11 kernel-tilt.git drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-01 12:30 kernel-tilt.git.git
!
Once I moved that ".git.git" repo away, original issue with repo sync resolved. I have no idea that double-.git repo was created. Based on the timestamp, it was created first, and later barely updated, with mirror and just git clone using the proper repo, while repo sync on the build slave managing to pick up wrong one. I wonder, if that (creation of 2 repos) could be a result of some race condition.
I bet it is.
And I wish this issue could be called resolved, but as quoted above, there're (more) issues are known with git --mirror, so I decided to put up this mail with the links, for the next time such issues may come.
As you pointed out to me fairly soon after you started, we don't actually need to run "repo sync" on the server at all -- it's not hard to figure out the "git clone" commands we need to run, and so maybe we (well, you :-p) should just do that.
Cheers, mwh
Hello Michael,
On Thu, 02 Jun 2011 21:18:44 +1200 Michael Hudson-Doyle michael.hudson@linaro.org wrote:
[]
Once I moved that ".git.git" repo away, original issue with repo sync resolved. I have no idea that double-.git repo was created. Based on the timestamp, it was created first, and later barely updated, with mirror and just git clone using the proper repo, while repo sync on the build slave managing to pick up wrong one. I wonder, if that (creation of 2 repos) could be a result of some race condition.
I bet it is.
And it's not the only case, but at least not regular:
ubuntu@android-build:/mnt/mirror$ find -type d -name '*.git.git' ./git.omapzoom.org/kernel/omap.git.git
What's interesting is that this is again a kernel repository. I'm not sure what to do about it now, considering just to set up a cronjob which will use that find command (need to figure out how to send mail from ec2).
And I wish this issue could be called resolved, but as quoted above, there're (more) issues are known with git --mirror, so I decided to put up this mail with the links, for the next time such issues may come.
As you pointed out to me fairly soon after you started, we don't actually need to run "repo sync" on the server at all -- it's not hard to figure out the "git clone" commands we need to run, and so maybe we (well, you :-p) should just do that.
Yes, I still keep that in mind, wanted to try it for this case, when it suddenly resolved. All in all, current mirror system seems to work (even though sometimes workarounds are required), while some stuff still missing, so I guess this will need to wait a bit...
Cheers, mwh
Paul
On Thu, Jun 2, 2011 at 9:24 AM, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
This is even more interesting, with Google people confirming that repo --mirror works within some bounds and limitations:
http://groups.google.com/group/repo-discuss/browse_thread/thread/401656c3ad0...
Finally, here's typical usage scenario for repo --mirror (which is pretty underdocumented):
http://www.excentral.org/archives/2011/02/24/android-repo-mirroring
So, like "repo help init" says, mirror created with repo --mirror should later be used with repo --reference, with both mirror and checkout using it to be located on local filesystem. I couldn't find referenced that repo --mirror is intended for use like we do - to serve it using git-daemon.
My next step was trying to do real git mirror (git clone --mirror), and already did it and prepared to move it into intended location, when I did ls -l in the mirror directory:
drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-02 00:11 kernel-tilt.git drwxr-xr-x 7 git-mirror nogroup 4096 2011-06-01 12:30 kernel-tilt.git.git
Once I moved that ".git.git" repo away, original issue with repo sync resolved. I have no idea that double-.git repo was created. Based on the timestamp, it was created first, and later barely updated, with mirror and just git clone using the proper repo, while repo sync on the build slave managing to pick up wrong one. I wonder, if that (creation of 2 repos) could be a result of some race condition.
And I wish this issue could be called resolved, but as quoted above, there're (more) issues are known with git --mirror, so I decided to put up this mail with the links, for the next time such issues may come.
I have spent some time in the past with repo --mirror (we have internal mirrors within TI). After several attempts my conclusion was that it was not the best approach. the main problem is that repo --mirror expects that you give a manifest tree. If you have several branches in your manifest tree (like it's the case for the omapmanifest on git.omapzoom.org), repo sync will only sync the branch you've used initially. So if you have new or different trees on other branches they aren't sync'd. that pretty much means that you need 1 mirror for each manifest branch. (in fact I am the one that asked the repo --mirror question you've found!)
so i ended it doing a git mirror, and I wrote this simple helper script https://github.com/ndechesne/git-mirror which can be used to mirror a complete 'git server' using gitweb. for example, you just need to add this in a cron:
git-mirror -o /data/git/repositories/ -s http://git.omapzoom.org -g git:// git.omapzoom.org
and the script will mirror the server locally. everytime it runs: - it creates new trees that have appeared on the remote - updates all existing trees - removed trees that have disappeared from the remote.
i have been using this script on several mirror (1 at every site) quite reliably for ~2 years. prior to that I was using repo sync and frequently had failures...
hope this helps.
Hello Nicolas,
On Thu, 2 Jun 2011 23:04:42 +0200 "Dechesne, Nicolas" n-dechesne@ti.com wrote:
Paul
On Thu, Jun 2, 2011 at 9:24 AM, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
This is even more interesting, with Google people confirming that repo --mirror works within some bounds and limitations:
http://groups.google.com/group/repo-discuss/browse_thread/thread/401656c3ad0...
[]
I have spent some time in the past with repo --mirror (we have internal mirrors within TI). After several attempts my conclusion was that it was not the best approach. the main problem is that repo --mirror expects that you give a manifest tree. If you have several branches in your manifest tree (like it's the case for the omapmanifest on git.omapzoom.org), repo sync will only sync the branch you've used initially. So if you have new or different trees on other branches they aren't sync'd. that pretty much means that you need 1 mirror for each manifest branch. (in fact I am the one that asked the repo --mirror question you've found!)
But, as far as I understood from the mailing list thread above, it's possible for one mirror to contain different manifest branches, one just need to sync them explicitly (unlike one would think mirror would do). (That's what we do now, I just want to be sure we won't any additional surprises soon.)
so i ended it doing a git mirror, and I wrote this simple helper script https://github.com/ndechesne/git-mirror which can be used to mirror a complete 'git server' using gitweb. for example, you just need to add this in a cron:
git-mirror -o /data/git/repositories/ -s http://git.omapzoom.org -g git:// git.omapzoom.org
Thanks, this looks interesting.
and the script will mirror the server locally. everytime it runs:
- it creates new trees that have appeared on the remote
- updates all existing trees
- removed trees that have disappeared from the remote.
i have been using this script on several mirror (1 at every site) quite reliably for ~2 years. prior to that I was using repo sync and frequently had failures...
Nice, thanks much for sharing the experience! We'd probably still want to use manifest as the source for what to mirror (that's the only thing that our mirror service takes as input actually), but it's great to know that using git clone --mirror works well with repo and actually improves reliability.
hope this helps.
-- Best regards, Paul
-
Dechesne, Nicolas -
Michael Hudson-Doyle -
Paul Sokolovsky