On Tue, Feb 11, 2020 at 2:10 PM Nicolas Dechesne
<nicolas.dechesne@linaro.org> wrote:
>
> On Tue, Feb 11, 2020 at 1:29 PM Ryan Harkin <ryan.harkin@linaro.org> wrote:
> >
> >
> >
> > On Tue, 11 Feb 2020, 12:19 Nicolas Dechesne, <nicolas.dechesne@linaro.org> wrote:
> >>
> >> hey,
> >>
> >>
> >> On Tue, Feb 11, 2020 at 12:59 PM Ryan Harkin <ryan.harkin@linaro.org> wrote:
> >> >
> >> > Hi Nico,
> >> >
> >> > On Tue, 11 Feb 2020 at 08:34, Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote:
> >> >>
> >> >> hi,
> >> >>
> >> >> we are hitting some errors with repo init since last night.
> >> >
> >> >
> >> > Yes, I was investigating this failure too. Thanks very much for working it out.
> >> >
> >> >
> >> >>
> >> >> After
> >> >> looking into it , I came up with the following explanation and fix..
> >> >> If I am correct, it will impact all our OE RPB build setup..
> >> >
> >> >
> >> > It definitely impacts on my WaRP7 builds, which are failing since last night too.
> >> >
> >> >>
> >> >>
> >> >> Can you please review that carefully?
> >> >>
> >> >> The PR is on github:
> >> >> https://github.com/96boards/oe-rpb-manifest/pull/121
> >> >>
> >> >
> >> > I added myself as a reviewer and approved it.
> >>
> >> thanks. Looks like Fathi merged it.
> >
> >
> > Good. That's a quick turnaround.
> >
> >> I will backport to other branches
> >> and push the same fix. You are probably using your own branch anyways,
>
> argh.. now it gets funny.. our builds have restarted, but still
> fail... the reason is because of this line:
>
> https://git.linaro.org/ci/job/configs.git/tree/rpb-openembedded/builders.sh#n77
>
> so basically we fetch the 'previous' manifest, and try to deduce the
> changes since the last build. However the old manifest contains the
> symlink with relative path, so the new repo command fails
>
> 00:05:27.092 + repo diffmanifests /srv/oe/pinned-manifest-latest.xml
> /srv/oe/pinned-manifest.xml
> 00:05:27.233 Traceback (most recent call last):
> 00:05:27.233 File "/srv/oe/.repo/repo/main.py", line 555, in <module>
> 00:05:27.233 _Main(sys.argv[1:])
> 00:05:27.233 File "/srv/oe/.repo/repo/main.py", line 530, in _Main
> 00:05:27.233 result = run()
> 00:05:27.233 File "/srv/oe/.repo/repo/main.py", line 523, in <lambda>
> 00:05:27.233 run = lambda: repo._Run(name, gopts, argv) or 0
> 00:05:27.233 File "/srv/oe/.repo/repo/main.py", line 212, in _Run
> 00:05:27.233 result = cmd.Execute(copts, cargs)
> 00:05:27.233 File "/srv/oe/.repo/repo/subcmds/diffmanifests.py",
> line 195, in Execute
> 00:05:27.233 manifest1.Override(args[0], load_local_manifests=False)
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 174, in Override
> 00:05:27.233 self._Load()
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 480, in _Load
> 00:05:27.233 self._ParseManifest(nodes)
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 588, in
> _ParseManifest
> 00:05:27.233 project = self._ParseProject(node)
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 910, in
> _ParseProject
> 00:05:27.233 self._ParseLinkFile(project, n)
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 1045,
> in _ParseLinkFile
> 00:05:27.233 self._ValidateFilePaths('linkfile', src, dest)
> 00:05:27.233 File "/srv/oe/.repo/repo/manifest_xml.py", line 1026,
> in _ValidateFilePaths
> 00:05:27.233 '<%s> invalid "src": %s: %s' % (element, src, msg))
> 00:05:27.233 error.ManifestInvalidPathError: <linkfile> invalid "src":
> ../../.repo/manifests/setup-environment: bad component: ..
>
> not sure how to best fix that... maybe we should check the old
> manifest and try to detect the symlink with relative path and skip
> repo diffmanifest.. it really gets ugly...
I am not proud ;-)
:-)
but I think this should fix the problem for all our
builds? Can you please check:
https://review.linaro.org/c/ci/job/configs/+/34254
Well, that'll do it!
I can't see a neater way around it. Disabling the diffmanifests call until we've all got successful builds is an option. But it's just a different way of hacking it.
>
> >
> >
> > Yes, I have a clone with some small patches on top.
> >
> >> so you should cherry-pick for your project(s) !
> >
> >
> > Same difference, but I rebase my branch when anything new is committed.
> >
> >>
> >> >
> >> > You've highlighted problems below, but in the end, your fix is better than no fix. And anything else we do could be on top of this fix.
> >> >
> >> >>
> >> >> I put a copy of the commit message below to bring it to your attention!
> >> >>
> >> >> cheers
> >> >> nico
> >> >>
> >> >> ==
> >> >> All along we have been using a relative symlink for setup-environment
> >> >> which points outside of the project path. While we've enjoyed this
> >> >> hack for several years, a recent change was in repo, and repo init now
> >> >> fails with:
> >> >>
> >> >> error.ManifestInvalidPathError: <linkfile> invalid "src":
> >> >> ../../.repo/manifests/setup-environment: bad component: .repo
> >> >>
> >> >> The change in repo is this one:
> >> >> https://gerrit.googlesource.com/git-repo/+/07392ed32662006c029299bc06617310cfe17957
> >> >>
> >> > Yes, I also worked out this was the culprit.
> >> >
> >> >>
> >> >> The repo documentation states: The symlink is created at “dest”
> >> >> (relative to the top of the tree) and points to the path specified by
> >> >> “src” which is a path in the project.
> >> >>
> >> >> As such, repo clearly expects that 'src' is in the project path, which
> >> >> is not the cause when using ../../.repo/ path.
> >> >>
> >> >> This patch changes our folder structure in such a way that:
> >> >> * the manifest git is checked out as ./conf/ folder in the top level
> >> >> directory
> >> >> * setup-environment link is created by repo, as a link to
> >> >> conf/setup-environment-internal
> >> >>
> >> >> The main drawback is that it feels non standard to duplicate the whole
> >> >> manifest project again (since it's already checked out in .repo)
> >> >> however using .repo content also meant we were making assumptions on
> >> >>
> >> >> implementation details of repo itself.
> >> >
> >> >
> >> > I don't see an obvious way around this. We are using the repo contents, but we're not allowed to use it directly.
> >> >
> >> > So unless we re-work how we do setup-environment, I don't see a way to avoid this. The only nagging thought I have is that setup-environment and default.xml are two distinct things, so we could split it into two repos. But as the repo is so small, I don't see the added value of doing this. It seems to be more overhead. That said, when I clone the manifest repo for my own projects, all I'm doing is editing default.xml, and nothing else. So it might make sense anyway, so I can get changes to things like setup-env without have to rebase my manifest repo. But again, it's such a small repo, it's not really a problem to have it like this. And probably makes life easier in the long term.
> >> >
> >> > Cheers,
> >> > Ryan.
> >> >
> >> >>
> >> >>
> >> >> Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
> >> >> _______________________________________________
> >> >> OpenEmbedded mailing list
> >> >> OpenEmbedded@lists.linaro.org
> >> >> https://lists.linaro.org/mailman/listinfo/openembedded