Hi All,
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
To address this I'd like to suggest the following.
The production of the LEB and related builds should continue to be on a monthly release cycle.
However outside of that process, for the WGs and such, we should go to a process based on continuous integration.
As work groups, landing teams and so on, we commit to the LEB process that we have some set of blueprints IN QUEUE, meaning they are being worked on right now. When they go complete if they are LEB bound, they are delivered to the staging overlay. Items in the staging overlay are candidates for promotion to the overlay. When an item in the staging overlay is judged as being ready (bugs are at least triaged and deemed to not be blocking) it is promoted into the overlay from which daily builds are made and the monthly release is made.
Packages in the staging overlay are the subject of functional test.IE Does it work? Packages in the overlay are the subject of integration test. IE Does it break the system?
Work Groups, Landing Teams will also commit to having a priority list of blueprints and bugs that are being addressed or will be addressed. This keeps the release process and management team aware of what is coming and when. The # of completed work items per week (or other counts) can also help the management team measure forward momentum and progress.
In some respects it's subtle process tweek. But I feel shifting part of linaro to a continuous integration / ship when ready model makes sense. In the work groups we are more connected upstream as we stay tune with upstream release boundaries but still want to showcase results in the monthly LEBs. Likewise focus on the LEB continues with higher quality software. It does shift the goal of creating great software by date X, to creating great software and ship when it's ready.
Perhaps this would be a good discussion for 1Q12LC.
Hi Tom,
On 10 January 2012 19:14, Tom Gall tom.gall@linaro.org wrote:
Hi All,
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
It seems there's some confusion here. The WGs/LTs deliveries aren't tight to LEB monthly releases, you have your own goals. It's fine to skip a monthly release (like many have done) or provide a snapshot of your current work (like LTs have done).
monthly releases != monthly planning.
It doesn't prevent that PMs/TLs should know what will make the targeted delivery, independently of the release. I'd like to believe that the hack-a-thons/rush is done to meet your own planned date and not the LEB release date.
To address this I'd like to suggest the following.
The production of the LEB and related builds should continue to be on a monthly release cycle.
However outside of that process, for the WGs and such, we should go to a process based on continuous integration.
All the chain should be based on continuous integration, including our builds. That's the goal we're aiming and working on.
As work groups, landing teams and so on, we commit to the LEB process that we have some set of blueprints IN QUEUE, meaning they are being worked on right now. When they go complete if they are LEB bound, they are delivered to the staging overlay. Items in the staging overlay are candidates for promotion to the overlay. When an item in the staging overlay is judged as being ready (bugs are at least triaged and deemed to not be blocking) it is promoted into the overlay from which daily builds are made and the monthly release is made.
Packages in the staging overlay are the subject of functional test.IE Does it work? Packages in the overlay are the subject of integration test. IE Does it break the system?
The picture is pretty clear and has been discussed at last Connect: https://blueprints.launchpad.net/linaro-project-management/+spec/linaro-gene...
See "The road to Continuous Integration"
Work Groups, Landing Teams will also commit to having a priority list of blueprints and bugs that are being addressed or will be addressed. This keeps the release process and management team aware of what is coming and when. The # of completed work items per week (or other counts) can also help the management team measure forward momentum and progress.
In which way is it different from current workflow? AFAICS the only difference is LTs, as things stand today they don't use blueprint but rally for their project management.
In some respects it's subtle process tweek. But I feel shifting part of linaro to a continuous integration / ship when ready model makes sense. In the work groups we are more connected upstream as we stay tune with upstream release boundaries but still want to showcase results in the monthly LEBs. Likewise focus on the LEB continues with higher quality software. It does shift the goal of creating great software by date X, to creating great software and ship when it's ready.
The goal is to create great software and ship it upstream, when it's ready. Though we do project management, planning, estimate deliveries, and it isn't tight to LEB monthly releases. The monthly releases are provided to showcase our results, and is also a playground for our improvements. If you don't have anything to show at the end of a cycle, or aren't interested to test your improvements in a real world environment, just skip it. Re-stating, my initial comment: it seems there's some confusion.
Perhaps this would be a good discussion for 1Q12LC.
I think we want to talk about the monthly planning.
-- Regards, Tom
"Where's the kaboom!? There was supposed to be an earth-shattering kaboom!" Marvin Martian Multimedia Tech Lead | Linaro.org │ Open source software for ARM SoCs w) tom.gall att linaro.org w) tom_gall att vnet.ibm.com h) tom_gall att mac.com
Cheers,
On 01/11/2012 03:34 AM, Somebody in the thread at some point said:
Hi -
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
It seems there's some confusion here. The WGs/LTs deliveries aren't tight to LEB monthly releases, you have your own goals. It's fine to skip a monthly release (like many have done) or provide a snapshot of your current work (like LTs have done).
I think there is confusion inside Linaro about what these releases are. What you're saying is correct, but it doesn't always reflect what's happening on the ground.
For LT the 'beat' we work to is kernel cycle and as you say monthly release is something downstream of what we are anyway doing for us, it should just be a staging post for what popped out of our sausage machine recently.
However people responsible for making the releases feel under pressure to maximize what's in there for a deadline that's not inherently meaningful, when at times when we may be dedicated to having to do something else do to lack of sync with our underlying 'beat'. At its worst, the drama of having a release leads to a false sense of urgency and bad decisions that are not in Linaro's overall interest.
However outside of that process, for the WGs and such, we should go to a process based on continuous integration.
All the chain should be based on continuous integration, including our builds. That's the goal we're aiming and working on.
I completely agree about CI approach will lead to best results in medium and long term and we should be all about that, not wasting time polishing the coprolite of old releases - especially someone else's old releases.
-Andy
On Wed, Jan 11, 2012 at 1:09 AM, Andy Green andy.green@linaro.org wrote:
On 01/11/2012 03:34 AM, Somebody in the thread at some point said:
Hi -
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
It seems there's some confusion here. The WGs/LTs deliveries aren't tight to LEB monthly releases, you have your own goals. It's fine to skip a monthly release (like many have done) or provide a snapshot of your current work (like LTs have done).
I think there is confusion inside Linaro about what these releases are. What you're saying is correct, but it doesn't always reflect what's happening on the ground.
For LT the 'beat' we work to is kernel cycle and as you say monthly release is something downstream of what we are anyway doing for us, it should just be a staging post for what popped out of our sausage machine recently.
However people responsible for making the releases feel under pressure to maximize what's in there for a deadline that's not inherently meaningful, when at times when we may be dedicated to having to do something else do to lack of sync with our underlying 'beat'. At its worst, the drama of having a release leads to a false sense of urgency and bad decisions that are not in Linaro's overall interest.
Once we agree that with CI we'll always try to avoid having regressions (bugs on new features and development is expected), I don't see why the LEBs would not just follow the LT trees directly. But, unfortunately, when the features are closely related with binary blobs from the vendor, that is not always the truth, and things can be quite broken over the time.
Guess we should probably thing better on how we're integrating the content produced by the LTs and the LEBs. While I agree that we want to the release snapshot to be something stable and working without critical regressions, that's not the case atm, and fighting against this situation will for sure just add a lot of pressure at folks all around.
For other projects than Kernel, I don't see our cycle as an issue, as once something is released (or in a shape we can merge at our LEBs), we can then plan the integration work and work on getting them available at our images (by staging PPA and such, as described by Tom).
Cheers,
On 01/12/2012 09:15 AM, Somebody in the thread at some point said:
On Wed, Jan 11, 2012 at 1:09 AM, Andy Greenandy.green@linaro.org wrote:
On 01/11/2012 03:34 AM, Somebody in the thread at some point said:
Hi -
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
It seems there's some confusion here. The WGs/LTs deliveries aren't tight to LEB monthly releases, you have your own goals. It's fine to skip a monthly release (like many have done) or provide a snapshot of your current work (like LTs have done).
I think there is confusion inside Linaro about what these releases are. What you're saying is correct, but it doesn't always reflect what's happening on the ground.
For LT the 'beat' we work to is kernel cycle and as you say monthly release is something downstream of what we are anyway doing for us, it should just be a staging post for what popped out of our sausage machine recently.
However people responsible for making the releases feel under pressure to maximize what's in there for a deadline that's not inherently meaningful, when at times when we may be dedicated to having to do something else do to lack of sync with our underlying 'beat'. At its worst, the drama of having a release leads to a false sense of urgency and bad decisions that are not in Linaro's overall interest.
Once we agree that with CI we'll always try to avoid having regressions (bugs on new features and development is expected), I
I think maybe there's some gap about what CI is meaning for people. In kernel, there are frequent releases, the key part is follow HEAD closely. For others I think CI is thinking about a single history tree, with "knowledgeless" adding, testing and reverting patches that seemed to make the tests fail. For kernel we get big advantages from following a rebase model that's not really compatible with that.
don't see why the LEBs would not just follow the LT trees directly. But, unfortunately, when the features are closely related with binary blobs from the vendor, that is not always the truth, and things can be quite broken over the time.
Right this second breakage of syslink on 3.2 tilt tree is related to changes about generic iommu in mainline kernel, unfortunately we can't blame the binary blobs ^^ As you know "syslink 3" / rpmsg stuff that will deprecate current syslink 2 code isn't quite here for gstreamer case either. However we're still expecting to be able to fix 3.1 syslink uplevel code in 3.2 and tracking shortly.
Guess we should probably thing better on how we're integrating the content produced by the LTs and the LEBs. While I agree that we want to the release snapshot to be something stable and working without critical regressions, that's not the case atm, and fighting against this situation will for sure just add a lot of pressure at folks all around.
There's a relationship between introduction of regression and keeping on latest basis underlying this, as we can see above.
In fact while we never want regression, and normally go to some lengths to avoid it, if the feature really is broken for longer than a short while on HEAD that does reflect the reality that we lost control of it in the present.
If we try to mask that by sticking at last known good release, we introduce other problems like WG content not appearing in it since that's all in later kernels suppressed by the pinning. Then literally regressive logic begins to appear like demand to backport other stuff to the last-known-good one while the originally lost feature remains broken and out of control at HEAD, and we start to go to Hell because we're working on a rotting past where we did have things under control instead of working towards a future where we still have things under control.
Other times we got around it by providing both last release and tracking kernel packages, so people can get completeness and latest stuff while we almost solely work on latest stuff, I think that'll square the circle enough here too if we can't get it solved quickly.
For other projects than Kernel, I don't see our cycle as an issue, as once something is released (or in a shape we can merge at our LEBs), we can then plan the integration work and work on getting them available at our images (by staging PPA and such, as described by Tom).
Yeah if the other projects don't have such a strong beat as the kernel then they won't experience widespread breakage after each upstream release which is common for us. So depending on when the monthly release samples tracking compared to kernel release cycle, it can be anywhere inbetween ahead of last release in features and quality, or unbootable junk. CI in the 'knowledgeless loop' form can't help with that since its reaction would be revert everything since last release all the time ^^ when the correct answer is more work.
-Andy
-
Andy Green -
Fathi Boudra -
Ricardo Salveti -
Tom Gall