Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
If you really need to send it to linaro-dev, would you mind at least giving more information and writing the email properly to avoid just using the subject?
Thanks.
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
Is Twitter really that much better to inform us it's broken? I would have missed the news entirely, for one random data point.
On 10 May 2012 15:20, Christian Robottom Reis kiko@linaro.org wrote:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
Thanks Kiko.
Yes, we sorted the issue out and we're back up. The builds are currently in flight.
Is Twitter really that much better to inform us it's broken? I would have missed the news entirely, for one random data point. -- Christian Robottom Reis, Engineering VP Brazil (GMT-3) | [+55] 16 9112 6430 | [+1] 612 216 4935 Linaro.org: Open Source Software for ARM SoCs
+++ Christian Robottom Reis [2012-05-10 17:20 -0300]:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
I don't like subject-only content-free mails either, but this one wasn't entirely useless so I guess it's fair enough.
But as Ricardo's started an email-moan thread I'll just get something off my chest that's been bugging me for a while.
Everytime we get a new person arriving a load of people send pointless mails to us _all_ saying 'hi there'. Those mails are great to send to the actual newbie, but not to everyone, _please_.
Call me a miserable bastard, but really, we all get more than enough mail (especially from bloody launchpad :-), please try not to send this sort of spam to all - just people who will finds it useful.
Ah, that's better...
Wookey (who has lost his sig)
On Thu, May 10, 2012 at 1:37 PM, Wookey wookey@wookware.org wrote:
+++ Christian Robottom Reis [2012-05-10 17:20 -0300]:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
I don't like subject-only content-free mails either, but this one wasn't entirely useless so I guess it's fair enough.
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.
But as Ricardo's started an email-moan thread I'll just get something off my chest that's been bugging me for a while.
Everytime we get a new person arriving a load of people send pointless mails to us _all_ saying 'hi there'. Those mails are great to send to the actual newbie, but not to everyone, _please_.
Call me a miserable bastard, but really, we all get more than enough mail (especially from bloody launchpad :-), please try not to send this sort of spam to all - just people who will finds it useful.
Ah, that's better...
++1 :-)
Cheers,
On Fri, May 11, 2012 at 12:24 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Thu, May 10, 2012 at 1:37 PM, Wookey wookey@wookware.org wrote:
+++ Christian Robottom Reis [2012-05-10 17:20 -0300]:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
I don't like subject-only content-free mails either, but this one wasn't entirely useless so I guess it's fair enough.
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.]
Actually, I think LAVA outage was announced. I poked for getting more status updates, so more mails would have been great.
Same goes for ci.linaro.org ... if our CI service used for everything but android is not available, I want to get a mail that this is the case.
On Thu, May 10, 2012 at 3:30 PM, Alexander Sack asac@linaro.org wrote:
On Fri, May 11, 2012 at 12:24 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Thu, May 10, 2012 at 1:37 PM, Wookey wookey@wookware.org wrote:
+++ Christian Robottom Reis [2012-05-10 17:20 -0300]:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
I don't like subject-only content-free mails either, but this one wasn't entirely useless so I guess it's fair enough.
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.]
Actually, I think LAVA outage was announced. I poked for getting more status updates, so more mails would have been great.
Same goes for ci.linaro.org ... if our CI service used for everything but android is not available, I want to get a mail that this is the case.
Sure, but then I believe it'd be better to have another mailing list for such purposes.
Linaro-dev is used for development related discussion, and it's the first most folks first subscribe to. I don't have the current number of folks participating at this list, but I'd prefer to still keep it dev-focused, and avoid off-topic discussions to try to keep it sane.
Cheers,
On 10 May 2012 23:34, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Thu, May 10, 2012 at 3:30 PM, Alexander Sack asac@linaro.org wrote: Linaro-dev is used for development related discussion, and it's the first most folks first subscribe to. I don't have the current number of folks participating at this list, but I'd prefer to still keep it dev-focused, and avoid off-topic discussions to try to keep it sane.
Personally I think the most useful thing we could do with linaro-dev is get the kernel folks to take their patch emails to another list. None of the other teams spam linaro-dev with patchmail AFAICT, and I think it would be nicer for that traffic to go to somewhere that's only subscribed to by the people who actually care about it...
-- PMM
On Fri, 2012-05-11 at 08:15 +0100, Peter Maydell wrote:
On 10 May 2012 23:34, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Thu, May 10, 2012 at 3:30 PM, Alexander Sack asac@linaro.org wrote: Linaro-dev is used for development related discussion, and it's the first most folks first subscribe to. I don't have the current number of folks participating at this list, but I'd prefer to still keep it dev-focused, and avoid off-topic discussions to try to keep it sane.
Personally I think the most useful thing we could do with linaro-dev is get the kernel folks to take their patch emails to another list.
I agree, though I solved this problem personally by filtering mails from the linaro-dev list to drop any that are To: or Cc: linux-arm-kernel@lists.infradead.org
(I'm subscribed to that anyway, but want to see mails in there relevant context.)
On Fri, 11 May 2012 00:30:26 +0200, Alexander Sack asac@linaro.org wrote:
On Fri, May 11, 2012 at 12:24 AM, Ricardo Salveti
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.]
Actually, I think LAVA outage was announced. I poked for getting more status updates, so more mails would have been great.
Same goes for ci.linaro.org ... if our CI service used for everything but android is not available, I want to get a mail that this is the case.
So, what this discussion points to is: we need a process for handling disruptions to the services we provide. When the **** hits the fan, the last think you want people to be doing is _thinking_, or at least, thinking about things that could have been thought through ahead of time and are not totally specific to the incident at hand.
Just recently within the LAVA team, we've started following such a process:
https://wiki.linaro.org/Internal/LAVA/Incidents
(apologies to the non-Linaro insiders for the internal link). The process will look very familiar to anyone who works at Canonical...
Creating a wiki page for each incident can feel a bit heavyweight, but having some kind of defined place for recording details has two massive values:
1. It means there's a canonical place to go for information while the incident is still in progress.[1]
2. It means that at the end of the month or quarter or whatever you can look back and have _actual data_ for how often various issues come up, rather than relying on vague feelings like "it seems we run out of disk space a lot".
I created a Google spreadsheet & form for adding details to it in an attempt to reduce the overhead of recording an incident, but after exactly two incidents, we already have an incident that was recorded in a wiki page but not the spreadsheet, so maybe that was premature optimization on my part.
It's only early days but I already feel happier for having this process in place. I'm happy to donate this policy to the wider set of services Linaro runs if there is consensus it would be useful :-)
There is already a page on a related topic:
https://wiki.linaro.org/Internal/Process/DealingWithCrisis
but that seems to me to be aimed at bigger issues than android-build or LAVA being unreachable for an hour.
One thing this thread points out to me though is that our policy does not really cover communication, either within the team or with our users. I'll work on a proposal for that today.
Cheers, mwh
[1] In particular, if an incident goes on for long enough to require hand overs between people working on in, then a wiki page like this is downright essential.
On Fri, 11 May 2012 12:11:36 +1200, Michael Hudson-Doyle michael.hudson@canonical.com wrote:
On Fri, 11 May 2012 00:30:26 +0200, Alexander Sack asac@linaro.org wrote:
On Fri, May 11, 2012 at 12:24 AM, Ricardo Salveti
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.]
Actually, I think LAVA outage was announced. I poked for getting more status updates, so more mails would have been great.
Same goes for ci.linaro.org ... if our CI service used for everything but android is not available, I want to get a mail that this is the case.
So, what this discussion points to is: we need a process for handling disruptions to the services we provide. When the **** hits the fan, the last think you want people to be doing is _thinking_, or at least, thinking about things that could have been thought through ahead of time and are not totally specific to the incident at hand.
Just recently within the LAVA team, we've started following such a process:
https://wiki.linaro.org/Internal/LAVA/Incidents(apologies to the non-Linaro insiders for the internal link). The process will look very familiar to anyone who works at Canonical...
Creating a wiki page for each incident can feel a bit heavyweight,
It turns out that moin has a funky NewPage macro (https://wiki.linaro.org/HelpOnMacros#Others) that one can use to make this really easy. So we've scrapped the Google document.
Cheers, mwh
I was hoping we could have one system for this instead of two. Are you sure that the existing IR process can't be tweaked and used for this purpose?
On Thu, May 10, 2012 at 8:59 PM, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
On Fri, 11 May 2012 12:11:36 +1200, Michael Hudson-Doyle michael.hudson@canonical.com wrote:
On Fri, 11 May 2012 00:30:26 +0200, Alexander Sack asac@linaro.org wrote:
On Fri, May 11, 2012 at 12:24 AM, Ricardo Salveti
Sure, I just think there are better places for it :-) Based on issues we had with LAVA and Jenkins at the previous cycle, if I had one email for every issue, I'd send at least 20 of them, which is useful but that still doesn't make me send them to the list.]
Actually, I think LAVA outage was announced. I poked for getting more status updates, so more mails would have been great.
Same goes for ci.linaro.org ... if our CI service used for everything but android is not available, I want to get a mail that this is the case.
So, what this discussion points to is: we need a process for handling disruptions to the services we provide. When the **** hits the fan, the last think you want people to be doing is _thinking_, or at least, thinking about things that could have been thought through ahead of time and are not totally specific to the incident at hand.
Just recently within the LAVA team, we've started following such a process:
https://wiki.linaro.org/Internal/LAVA/Incidents
(apologies to the non-Linaro insiders for the internal link). The process will look very familiar to anyone who works at Canonical...
Creating a wiki page for each incident can feel a bit heavyweight,
It turns out that moin has a funky NewPage macro (https://wiki.linaro.org/HelpOnMacros#Others) that one can use to make this really easy. So we've scrapped the Google document.
Cheers, mwh
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 05/11/2012 08:50 AM, Joey STANFORD wrote:
I was hoping we could have one system for this instead of two. Are you sure that the existing IR process can't be tweaked and used for this purpose?
My fear is that we'd wind up spamming your list of incidents with stuff like "panda14's SD card died". ie - alot of our incidents probably won't be high profile. But maybe that doesn't matter?
Ok, let's try your way and see how it works. Can you please just link from the IR to your lava bit so there's a two-way link? Thanks...
On Fri, May 11, 2012 at 9:22 AM, Andy Doan andy.doan@linaro.org wrote:
On 05/11/2012 08:50 AM, Joey STANFORD wrote:
I was hoping we could have one system for this instead of two. Are you sure that the existing IR process can't be tweaked and used for this purpose?
My fear is that we'd wind up spamming your list of incidents with stuff like "panda14's SD card died". ie - alot of our incidents probably won't be high profile. But maybe that doesn't matter?
On Fri, 11 May 2012 07:50:48 -0600, Joey STANFORD joey@linaro.org wrote:
I was hoping we could have one system for this instead of two. Are you sure that the existing IR process can't be tweaked and used for this purpose?
Well, the main reason we now have two processes is that I wasn't aware of / had forgotten about the wider process when I wrote up the one for LAVA!
When I did read the DealingWithCrisis page, I felt that the incident it had in mind was something more like a leak of embargoed information than a service disruption. Not all LAVA disruptions will fall into the "wake people up" category and I definitely don't want people to avoid creating an incident report because the process implies that all incidents are a big deal that require disturbing people's sleep.
If we were to have a general incident management policy, I think we would need to come up with some guidelines for gauging the severity of an incident.
Whether we do that or not, it would be easy to put the LAVA incident reports under Internal/Process/DealingWithCrisis/IncidentReports rather than Internal/LAVA/Incidents/Reports/ but use some naming convention so that we could still have a LAVA specific list of incidents and process[0]. Let me know if you'd like to do this.
Cheers, mwh
[0] Something that I wouldn't want to lose -- even if the LAVA specific process pointed to some general page for general things, I still want a no-thinking process for the LAVA team to follow when a LAVA disruption happens.
On Thu, May 10, 2012 at 1:20 PM, Christian Robottom Reis kiko@linaro.org wrote:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
Well, just saying it's broken doesn't help much either. If you really want a better and more up-to-date place to show this info even IRC would be better.
Is Twitter really that much better to inform us it's broken? I would have missed the news entirely, for one random data point.
Better than email at least :-) While it it's useful, did it actually help you in any front?
Twitter is the general place used by most of projects for status update, that's why I thought that it'd be the best way to go.
Cheers,
On Fri, May 11, 2012 at 12:21 AM, Ricardo Salveti ricardo.salveti@linaro.org wrote:
On Thu, May 10, 2012 at 1:20 PM, Christian Robottom Reis kiko@linaro.org wrote:
On Thu, May 10, 2012 at 12:55:26PM -0700, Ricardo Salveti wrote:
Sorry to say that, but I hate these kind of emails at Linaro Dev, as I believe we have other places (and better ones) to report such issues. Twitter would probably be the way to go.
Well, to be honest I don't really see the problem with letting the list know that there are issues, and I interpret Zach's brevity as "we're totally focused on getting the problem solved". I expect he's going to come back and explain what's broken when the issue is resolved.
Well, just saying it's broken doesn't help much either. If you really want a better and more up-to-date place to show this info even IRC would be better.
Is Twitter really that much better to inform us it's broken? I would have missed the news entirely, for one random data point.
Better than email at least :-) While it it's useful, did it actually help you in any front?
Twitter is the general place used by most of projects for status update, that's why I thought that it'd be the best way to go.
I would have preferred to get a bit more context as well, but as kiko said, getting an notification is definitely better than nothing.
Especially if you are in the mids of a firedrill you might not have the time to explain the context, nor might you understand whats going on at all etc.
* IRC is nice for this kind of announcements; you can even put it in the #linaro topic
* perhaps we can just add a banner to android-build.linaro.org?
* perhaps a Linaro services dashboard is what we miss? e.g. "Android build system: working" (green), or "broken (see LP #1234)"; Joey is running a basic HTTP status monitor as part of Linaro IS
* notifications about build system going down could go to the same places where build failures / successful builds are reported; perhaps we need a dedicated "build-notifications" list or something like that
Hello,
On Fri, 11 May 2012 11:09:59 +0200 Loïc Minier loic.minier@linaro.org wrote:
IRC is nice for this kind of announcements; you can even put it in the #linaro topic
perhaps we can just add a banner to android-build.linaro.org?
perhaps a Linaro services dashboard is what we miss? e.g. "Android build system: working" (green), or "broken (see LP #1234)"; Joey is running a basic HTTP status monitor as part of Linaro IS
Well, lack of context is indeed very bad thing. There was no known outages to Android Build as a service. So, my understanding is that Zack meant that for that particular day, there were lot of build failures, which is true, I looked at them too, identified something we can improve on Infra side (https://bugs.launchpad.net/linaro-android-infrastructure/+bug/997551), the rest where genuine compile errors. For those errors, https://android-build.linaro.org/ is *the dashboard*.
Ah, as it's confession time, some time ago that page was prefixed with content, not really related to the Build System function, which exactly complicates access to build dashboard (requires lot of vertical scrolling). IMHO, that text is misplaced there - it belongs to releases.linaro.org/snapshot.linaro.org, which are gateways for community access, and android-build.linaro.org is first of all internal build system. Having extra stuff there complicates access and maintenance.
- notifications about build system going down could go to the same places where build failures / successful builds are reported;
perhaps we need a dedicated "build-notifications" list or something like that
-
Alexander Sack -
Andy Doan -
Christian Robottom Reis -
Joey STANFORD -
Jon Medhurst (Tixy) -
Loïc Minier -
Michael Hudson-Doyle -
Michael Hudson-Doyle -
Paul Sokolovsky -
Peter Maydell -
Ricardo Salveti -
Wookey -
Zach Pfeffer