Hello John,
On Thu, 28 Aug 2014 10:28:06 -0700 John Stultz john.stultz@linaro.org wrote:
On Thu, Aug 28, 2014 at 10:05 AM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Recently, we had DoS-like episodes on the main Linaro git server, http://git.linaro.org , which affected number of Linaro users, including users of Gerrit system, http://review.linaro.org .
These episodes were related to unfriendly usage of native protocol, git:// (service port 9418). The implementation of this protocol is known to be resource-hungry and not scale to many connections and users. The issue itself is not new, it is something which affected us in waves over last 3 years, and a resolution for which was established a year ago, providing 2 HTTP-based protocols (so called "dump" and "smart" protocols) as more scalable replacement.
So, this is a gentle reminder that use of git:// protocol by is discouraged for Linaro engineers, and completely unsupported(*1) for third parties. Based on the analysis and outcome of the current DoS-like activity, we may need to make git:// access more limited and strict. So, please kindly:
So why does this affect us but not kernel.org?
Regarding kernel.org, Milo Casagrande did some correspondence with them and can share specific details. IIRC, they use some custom infrastructure.
- Check URLs you use for cloning and updating your local trees. If
you use "ssh://" or "http(s)://" protocols, you're ok. If you use git://, please switch to using http-based protocol instead. In most cases, this requires just replacing "git://" schema with "http://". If in doubt, please visit gitweb page for your repositories, which lists all supported URLS to clone a repository, e.g.: https://git.linaro.org/arm/arm-trusted-firmware.git
- If you set up of oversee CI or automated build jobs, please
audit and apply similar changes to them.
So this is problematic, because there are folks out there in the community who already use the git:// urls for fetching work from the Linaro repos. (The 0day build/test bot, for instance..).
So, it would be nice if they updated to use http://. We actually can be proactive and contact them regarding this change (we could use your help with that, or just with compiling list of parties who should be contacted).
While the git:// urls are now off the gitweb (which is good for future users), this wasn't the case previously.
We already went through one painful transition where our URLs got scrambled, and I've had a few situations where folks have just recently realized that we still had trees, but the URLs were just different. So its quite frustrating to have to go through that again.
I'm not sure which transition you mean, but the matter of deprecating git:// and switching to http:// indeed comes not the first time. And previous time there were conservative (or skeptic) responses too, which made transition be far complete, and now we need to approach the same matter again, instead of having done it once and for all.
But otherwise, the world is dynamic place and changes all the time. For example, in the summer 2011 aforementioned kernel.org was down for unbelievable 1.5 months, and what, people coped. But we're not going to do it like kernel.org and go down, but instead going to start as seamless as possible transition (I hope you didn't get any other idea from this mail). Just for that, we need some help of the users, first of all, internal users, as about their access and its stability we care the most.
What would be required to just make the git:// urls work properly?
There can be different technical and organizational answers to this question, but the most productive I can give is: Systems and ITS will be working towards that; in the meantime, all parties which would like a sustainable service are encouraged to upgrade to http:// protocol.
Is this mainly an issue with the Android repos?
It used to be. It was a big problem in that summer 2011, when with kernel.org down, AOSP tree went down too and after couple of weeks people rushed to fetch from us. But this time, it affected git.linaro.org straight.
If we reduce the git:// url load on the wort users, would that improve things enough? Do you have stats on which trees are hardest hit?
The case we have with git:// is that small number of users can hog almost all resources of a server. This can happen at release time and block work of Linaro engineers, something like that happened this time. So, we're working on technical means to avoid hogging all resources, but we also would like to be sure that we won't affect internal users, and the most productive way for that is them to use a scalable protocol.
(*1) Unsupported in the current context means that "git://" URLs are not published in up-to-date information, and there's no warranty that any 3rd party will be able to complete a clone successfully using this protocol.
So as someone who has sent git pull requests in the past with the git urls, this is terrifying (and makes me hesitant to further use the linaro infrastructure). Do you have a pointer to why the git urls aren't coherent?
Oh, I'm sorry for leaving ambiguity for such interpretation. I just meant that we are going to serve git:// to 3rd-party users on "best effort" basis, and apply measures to give priority to internal users. For example, if there's important build doing fetch, and an external user makes 5 (or maybe just 2) git:// connections, they may get connection reset. Note that this does not change status quo - for example, 2 days ago, *any* user who tried to connect was getting connection refused.
So, kernel.org being down for 1.5 months is terrifying. Myself trying to build OABI toolchain and seeing all support being removed from everywhere, and finding aligned historical releases of toolchain parts almost impossible (while OABI hardware is still in use!) - that's terrifying. But I don't see anything terrifying with being frank about our git access policies and giving users a choice - either get reliable service with http:// or "best effort" with git://.
thanks -john