Paul Sokolovsky

May 20 09:35 (ADT)

Fruther tries: I commented "RewriteMap prg:" completely, just in case if it applies to each request by error. Didn't help.

I then tried to just wget -r http://android.git.linaro.org/git-http/platform/manifest.git to see how long that will take, and what I found is that there's http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/pack-a51ef40ec4ecf6ecb1e8b8001e5fd6edb301ff81.pack which is 660MB. Of course, if that pack is download with normal git clone, it got to be slow. And if it isn't, then why such a big pack lies in repo dir at all?

All in all, I didn't have much progress investigating this, so I revert apache config to what it was, suggest Android people to switch to git:// for now, and this bug is stays in queue, waiting for peorper investigation and solution.

Paul Sokolovsky

May 20 09:06 (ADT)

Ok, I did to things:

1) temporarily disabled RewriteLock to see if it will help performance - it did not, took 6 min before it failed.

2) Urls with "/git-http/" are not subject to rewriting, and it still both slow and gives errors:

$ time git clone http://android.git.linaro.org/git-http/platform/manifest.git
Cloning into 'manifest'...
error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/pack-9b13e86787a1d775996ed1c3f27b17efe00e13d1.idx

error: Unable to get pack index http://android.git.linaro.org/git-http/platform/manifest.git/objects/pack/pack-e7620cb111dfa2530cd07083ae6982ed23acc64d.idx

^C

real 7m18.513s

Paul Sokolovsky

May 20 08:37 (ADT)

Ben:

1) What could be explanation of need for lock - that script doesn't do any context-sensitive processing.
2) We should try to get rid of it at all

Paul Sokolovsky

May 20 08:30 (ADT)

Vishal: this ticket is for issue reported by Jon, and other http access hiccups as confirmed by me.

"branch 'stable' has not been signed" is separate and tracked at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1181907

Ben Copeland (IT Services)

May 20 08:22 (ADT)

Hi Paul,

I'm not too sure what I can do to speed up the http process. The bottleneck is down to the python script, We have had to put add "RewriteLock" to Apache otherwise users are getting these clone errors. The downside of this is it means it is preventing parallel requests.

When I start to clone http://android.git.linaro.org/git-ro/platform/manifest.git, watching apache log there are many 404 links "GET /git-ro/platform/manifest.git/objects/75/0313d9ddb35959103d75ff8c450f4d7b6596d9 HTTP/1.1" 404 506 "-" "git/1.8.1.2" at the start. These do stop, but its going process each http request per file, its going to be a lengthy wait along with this process lock.

Ben

Vishal Bhoj

May 20 07:59 (ADT)

I still face the same issue which happens on android-build with this command:
repo init -u http://android.git.linaro.org/git-ro/platform/manifest.git -b linaro_android_4.2.2 -m default.xml --repo-url=http://android.git.linaro.org/git-ro/tools/repo

Get http://android.git.linaro.org/git-ro/tools/repo/clone.bundle

fatal: branch 'stable' has not been signed

Paul Sokolovsky

May 20 07:54 (ADT)

Reprod info for the issue I see:

$ time git clone git://android.git.linaro.org/platform/manifest.git
[]
real 0m7.706s

$ time git clone http://android.git.linaro.org/git-ro/platform/manifest.git
[]
real 14m17.214s

Jon Medhurst (Tixy)

May 20 07:53 (ADT)

I'm getting no problems now, thanks.

I had already synced up my local system using the git protocol, so I
restored my Android directory from my Friday backup and did a repo sync.
Dozens of git directories were updated by http:// URLs without any
issues.

Ben Copeland (IT Services)

May 20 07:34 (ADT)

Hello Jon,

We had some issues with the git server regarding the git-ro directory. I have made a similar change to the apache config which hopefully is linked to your problem.

Can you try again and let us know if you are still getting problems?

Thanks

Ben

Paul Sokolovsky

May 20 07:34 (ADT)

All in all, I can confirm that there're issues with http:// access on android.git.linaro.org after migration - plain repo init takes very long, and I see errors like:

error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pack-55bb9d9172b2bb8fce748cc3507b6bf3e47f07fd.idx

error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pack-a6d8325efd61cdb3ca2f4e576b373851397b7b40.idx

error: Unable to get pack index http://android.git.linaro.org/git-ro/platform/manifest.git/objects/pack/pack-a356c43f34e65287e1cd5cff210b66182a43519a.idx

(Those are non-deterministic, re-trying them manually works.)

Jon Medhurst (Tixy)

May 20 07:10 (ADT)

On Mon, 2013-05-20 at 12:43 +0300, Paul Sokolovsky wrote:
> Hello Jon,
>
> On Mon, 20 May 2013 09:35:00 +0100
> "Jon Medhurst (Tixy)" <tixy@linaro.org> wrote:
>
> > On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote:
> > > Android builds are failing in repo init phase. I have opened a bug
> > > for the same:
> > > https://bugs.launchpad.net/bugs/1181907
> > >
> > > repo init and sync worked for me locally. Can someone please have
> > > a look.
> >
> > I get several errors like this, with various repos...
> >
> > error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7
> > under http://android.git.linaro.org/git-ro/platform/ndk
> > Cannot obtain needed blob
> > 51bd0a1525c7a722ad64a8954d90c6df0b8546a7
> > while processing commit
> > 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e.
> > error: Fetch failed.
>
> This is different issue from one report by Vishal.

Oops, yes. I didn't read the bug, sorry. I jumped to conclusions after
seeing Vishal and Bero's emails appeared at about the same time.

>
> >
> > As I said in reply to bero's mail, I always seem to get these since
> > the move to http URLs
>
> Move to HTTP happened last couple of days during LCA13 in Hong Kong, do
> you want to say that you experienced the error as quoted above since
> then? Was it reported to some bug/ticket? (Do you have a reference?)

I didn't raise a bug as I had a workaround and I assumed if it was a
widespread problem then it would get sorted out eventually.

> The situation known to me is:
>
> 1. After flipping the switch in HK, git:// became unstable (because it
> was throttled to stop wasting resources as it always did). It took some
> time to migrate everyone to http://.

If my memory serves me right, it wasn't long after this I noticed my
problems, which is why it occurred to me to change the URLs in the
manifests back to the old git:// (They'll get restored to http when the
manifests are changed, which is why I notice it as a continuing
problem.)

I perhaps only do a repo sync a couple of times a month as I don't
really work much on Android, which is perhaps why a) I see this problem?
b) I've not been motivated to get to the bottom of it.

--
Tixy

Paul Sokolovsky

May 20 06:44 (ADT)

Hello Jon,

On Mon, 20 May 2013 09:35:00 +0100
"Jon Medhurst (Tixy)" <tixy@linaro.org> wrote:

> On Mon, 2013-05-20 at 09:15 +0530, Vishal Bhoj wrote:
> > Android builds are failing in repo init phase. I have opened a bug
> > for the same:
> > https://bugs.launchpad.net/bugs/1181907
> >
> > repo init and sync worked for me locally. Can someone please have
> > a look.
>
> I get several errors like this, with various repos...
>
> error: Unable to find 51bd0a1525c7a722ad64a8954d90c6df0b8546a7
> under http://android.git.linaro.org/git-ro/platform/ndk
> Cannot obtain needed blob
> 51bd0a1525c7a722ad64a8954d90c6df0b8546a7
> while processing commit
> 7b3bda5d569a69c4f9b87225c5d21da4a41bf12e.
> error: Fetch failed.

This is different issue from one report by Vishal.

>
> As I said in reply to bero's mail, I always seem to get these since
> the move to http URLs

Move to HTTP happened last couple of days during LCA13 in Hong Kong, do
you want to say that you experienced the error as quoted above since
then? Was it reported to some bug/ticket? (Do you have a reference?)

The situation known to me is:

1. After flipping the switch in HK, git:// became unstable (because it
was throttled to stop wasting resources as it always did). It took some
time to migrate everyone to http://.
2. http:// was stable since then, until migration of git.linaro.org
happened weekened May 11.
3. Immediately after migration, there were issues with
http://git.linaro.org which manifested pretty much like the error
above. It was resolved Mon/Tue last week.
4. Friday GMT evening, android.git.linaro.org was migrated to new
setup. Initial builds seem to have gone ok. Detailed status is not
known (your telling that error above was received after May 17,
17:00GMT would help).
5. Additionally, today issue reported by Vishal popped up.

All in all, git issues now should be cc: to its@linaro.org (added), ITS
guys are now the owners of new infrastructure, and can resolve issues
pretty well (p.3 was resolved by them).

> and I work around it by changing the manifest
> to have the old git URLs.

This may work as a personal workaround, but won't scale to official
manifests - we need to get http:// working right.

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog