On 01/12/2012 09:15 AM, Somebody in the thread at some point said:
On Wed, Jan 11, 2012 at 1:09 AM, Andy Greenandy.green@linaro.org wrote:
On 01/11/2012 03:34 AM, Somebody in the thread at some point said:
Hi -
I had some thoughts and a suggestion this morning about the monthly release cycle that I'd like to share. Perhaps I'll get booed off the stage, perhaps not. I'm sure it can be improved on.
There is a great deal of stress around what blueprints fit into what monthly release boundary. For deliveries like the LEB that combine the fruits of our Linaro labors this makes great sense.
Outside of the process for LEB creation and release, I'd like to suggest it's less than efficient and creates a bit of stress as we have the monthly rush to make the release dates. PMs and TLs continually have to be watching for what will and what won't be making dates that fit with the LEB. This passes on the stress to the engineering team, causes some late night hack-a-thons which in turn I believe caused us to rush software what was less than ready to be included in a LEB. I'm sure we all have examples we could point to.
It seems there's some confusion here. The WGs/LTs deliveries aren't tight to LEB monthly releases, you have your own goals. It's fine to skip a monthly release (like many have done) or provide a snapshot of your current work (like LTs have done).
I think there is confusion inside Linaro about what these releases are. What you're saying is correct, but it doesn't always reflect what's happening on the ground.
For LT the 'beat' we work to is kernel cycle and as you say monthly release is something downstream of what we are anyway doing for us, it should just be a staging post for what popped out of our sausage machine recently.
However people responsible for making the releases feel under pressure to maximize what's in there for a deadline that's not inherently meaningful, when at times when we may be dedicated to having to do something else do to lack of sync with our underlying 'beat'. At its worst, the drama of having a release leads to a false sense of urgency and bad decisions that are not in Linaro's overall interest.
Once we agree that with CI we'll always try to avoid having regressions (bugs on new features and development is expected), I
I think maybe there's some gap about what CI is meaning for people. In kernel, there are frequent releases, the key part is follow HEAD closely. For others I think CI is thinking about a single history tree, with "knowledgeless" adding, testing and reverting patches that seemed to make the tests fail. For kernel we get big advantages from following a rebase model that's not really compatible with that.
don't see why the LEBs would not just follow the LT trees directly. But, unfortunately, when the features are closely related with binary blobs from the vendor, that is not always the truth, and things can be quite broken over the time.
Right this second breakage of syslink on 3.2 tilt tree is related to changes about generic iommu in mainline kernel, unfortunately we can't blame the binary blobs ^^ As you know "syslink 3" / rpmsg stuff that will deprecate current syslink 2 code isn't quite here for gstreamer case either. However we're still expecting to be able to fix 3.1 syslink uplevel code in 3.2 and tracking shortly.
Guess we should probably thing better on how we're integrating the content produced by the LTs and the LEBs. While I agree that we want to the release snapshot to be something stable and working without critical regressions, that's not the case atm, and fighting against this situation will for sure just add a lot of pressure at folks all around.
There's a relationship between introduction of regression and keeping on latest basis underlying this, as we can see above.
In fact while we never want regression, and normally go to some lengths to avoid it, if the feature really is broken for longer than a short while on HEAD that does reflect the reality that we lost control of it in the present.
If we try to mask that by sticking at last known good release, we introduce other problems like WG content not appearing in it since that's all in later kernels suppressed by the pinning. Then literally regressive logic begins to appear like demand to backport other stuff to the last-known-good one while the originally lost feature remains broken and out of control at HEAD, and we start to go to Hell because we're working on a rotting past where we did have things under control instead of working towards a future where we still have things under control.
Other times we got around it by providing both last release and tracking kernel packages, so people can get completeness and latest stuff while we almost solely work on latest stuff, I think that'll square the circle enough here too if we can't get it solved quickly.
For other projects than Kernel, I don't see our cycle as an issue, as once something is released (or in a shape we can merge at our LEBs), we can then plan the integration work and work on getting them available at our images (by staging PPA and such, as described by Tom).
Yeah if the other projects don't have such a strong beat as the kernel then they won't experience widespread breakage after each upstream release which is common for us. So depending on when the monthly release samples tracking compared to kernel release cycle, it can be anywhere inbetween ahead of last release in features and quality, or unbootable junk. CI in the 'knowledgeless loop' form can't help with that since its reaction would be revert everything since last release all the time ^^ when the correct answer is more work.
-Andy