I do not see why you guys should limit git usage. Do you guys have ssh keys in place? also if git uses UDP then that can be another source of said activity. I have seen any UDP ports which I have open on my network usually get flooded by packets inducing a denial of service like situation.
On Fri, Aug 29, 2014 at 6:30 AM, John Stultz john.stultz@linaro.org wrote:
On Thu, Aug 28, 2014 at 2:51 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Thu, 28 Aug 2014 10:28:06 -0700 John Stultz john.stultz@linaro.org wrote:
On Thu, Aug 28, 2014 at 10:05 AM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote: So this is problematic, because there are folks out there in the community who already use the git:// urls for fetching work from the Linaro repos. (The 0day build/test bot, for instance..).
So, it would be nice if they updated to use http://. We actually can be proactive and contact them regarding this change (we could use your help with that, or just with compiling list of parties who should be contacted).
Well, I can try to ping the users I know of, but the problematic spot is users I am not aware of.
We already went through one painful transition where our URLs got scrambled, and I've had a few situations where folks have just recently realized that we still had trees, but the URLs were just different. So its quite frustrating to have to go through that again.
I'm not sure which transition you mean, but the matter of deprecating
So moving from what we had before to gitolite broke the git URLs that existed previously. I know folks tried to add compatibility urls, but those URLs were slightly different then what we had previously. So folks who were pulling regularly from our tree just stopped being able to connect and get any updates.
git:// and switching to http:// indeed comes not the first time. And previous time there were conservative (or skeptic) responses too, which made transition be far complete, and now we need to approach the same matter again, instead of having done it once and for all.
But otherwise, the world is dynamic place and changes all the time. For example, in the summer 2011 aforementioned kernel.org was down for unbelievable 1.5 months, and what, people coped. But we're not going to do it like kernel.org and go down, but instead going to start as seamless as possible transition (I hope you didn't get any other idea from this mail). Just for that, we need some help of the users, first of all, internal users, as about their access and its stability we care the most.
Ok. I appreciate the transition. I was worried git:// url access was about to be turned off.
If we reduce the git:// url load on the wort users, would that improve things enough? Do you have stats on which trees are hardest hit?
The case we have with git:// is that small number of users can hog almost all resources of a server. This can happen at release time and block work of Linaro engineers, something like that happened this time.
Do we have a sense of who those users (IPs? which tree they are pulling?) are?
(*1) Unsupported in the current context means that "git://" URLs are not published in up-to-date information, and there's no warranty that any 3rd party will be able to complete a clone successfully using this protocol.
So as someone who has sent git pull requests in the past with the git urls, this is terrifying (and makes me hesitant to further use the linaro infrastructure). Do you have a pointer to why the git urls aren't coherent?
Oh, I'm sorry for leaving ambiguity for such interpretation. I just meant that we are going to serve git:// to 3rd-party users on "best effort" basis, and apply measures to give priority to internal users. For example, if there's important build doing fetch, and an external user makes 5 (or maybe just 2) git:// connections, they may get connection reset. Note that this does not change status quo - for example, 2 days ago, *any* user who tried to connect was getting connection refused.
Ok. I was worried you were claiming that the git:// url might serve different (possibly stale) data then the https:// urls. If I was making a pull request and someone only got half of the commits, that would be a major infrastructure trust issue.
If its just the connections would be slow or refused, that's enough to bother folks but not something that would make folks think or worry our infrastructure was compromised.
So, kernel.org being down for 1.5 months is terrifying. Myself trying to build OABI toolchain and seeing all support being removed from everywhere, and finding aligned historical releases of toolchain parts almost impossible (while OABI hardware is still in use!) - that's terrifying. But I don't see anything terrifying with being frank about our git access policies and giving users a choice - either get reliable service with http:// or "best effort" with git://.
So yea, kernel.org being down (and some git URLs changing) was a big deal, but it was also due to a major compromise of the system, which required a from-scratch rebuild. Hopefully things aren't (ever, we can dream) so bad for us.
Ok. I'll try to make the change with my own uses, and I really think removing the git:// url on the gitweb was the right call for the best first step here (ie: stop any new users of the git urls).
My main concern was that we might break our current users just because we had under-provisioned infrastructure. So as long as the git urls continue to work for legacy users, I'm happy, and I can work on changing my own uses, and trying to pester folks I know to change their git remote urls.
Though I think having some tracking done via the connection logs and notifying repo owners if their repo is a problem would be good in helping get the worst offenders fixed up.
Also I think continuing discussion w/ the kernel.org folks to understand their infrastructure would be good. They really started taking things seriously after their compromise, and it would be good for us to learn from their experience and take things similarly seriously before any such problems arise for us.
Thanks again for the explanations here! -john
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev