As you might know already, Linaro is moving to a monthly release of technology preview for most of the components being worked on. This obviously includes the kernel, so some coordination amongst all people involved is required.
It was decided that those releases would happen on the last Thursday of each month, so for the 11.06 release this is June 30th.
To give a chance for the kernel to be stable by that time, I'll effectively freeze it one week earlier i.e. on June 23rd for the 11.06 release. The last week before the actual release should be dedicated to testing and bug fixing only.
So... what this means is that you need to send me any patches and/or pull requests for the Linaro kernel at least _8_ DAYS BEFORE THE RELEASE. Obviously, you can send me patches way before that deadline as well, which would be highly appreciated. Today is June 14th, meaning that there are less than 9 days left (less so if you don't include the weekend).
We also hope that everyone will try to perform a bit more testing than usual during the last week, and also get involved in bug fixing activities until the release.
If you have some concerns about this please let me know so we can find ways to improve things. Hopefully we'll get up to speed with this pace and there won't be any major problems.
Nicolas
Nico, Is there any dependency on the Toolchain WG? Does the latest kernel have to be built using the latest Toolchain release? Does the kernel have to boot under the latest QEMU release? Thanks, Mounir
On Tue, Jun 14, 2011 at 5:03 PM, Nicolas Pitre nicolas.pitre@linaro.orgwrote:
As you might know already, Linaro is moving to a monthly release of technology preview for most of the components being worked on. This obviously includes the kernel, so some coordination amongst all people involved is required.
It was decided that those releases would happen on the last Thursday of each month, so for the 11.06 release this is June 30th.
To give a chance for the kernel to be stable by that time, I'll effectively freeze it one week earlier i.e. on June 23rd for the 11.06 release. The last week before the actual release should be dedicated to testing and bug fixing only.
So... what this means is that you need to send me any patches and/or pull requests for the Linaro kernel at least _8_ DAYS BEFORE THE RELEASE. Obviously, you can send me patches way before that deadline as well, which would be highly appreciated. Today is June 14th, meaning that there are less than 9 days left (less so if you don't include the weekend).
We also hope that everyone will try to perform a bit more testing than usual during the last week, and also get involved in bug fixing activities until the release.
If you have some concerns about this please let me know so we can find ways to improve things. Hopefully we'll get up to speed with this pace and there won't be any major problems.
Nicolas
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 14 June 2011 15:16, Mounir Bsaibes mounir.bsaibes@linaro.org wrote:
Nico, Is there any dependency on the Toolchain WG? Does the latest kernel have to be built using the latest Toolchain release?
Good question. As we get the CI loop going, I imagine that we'll want the kernel building with multiple tool chains that the WG releases but for now I think the right answer is probably for Nicolas and others to use the latest Linaro toolchain for their manual testing.
Does the kernel have to boot under the latest QEMU release?
I would say ideally yes if we have a proper defconfig for the platform under emulation. Again, I think this is something that is more important as we start automating testing.
~Deepak
On Tue, 14 Jun 2011, Mounir Bsaibes wrote:
Nico, Is there any dependency on the Toolchain WG? Does the latest kernel have to be built using the latest Toolchain release?
No, it doesn't have to. That would be a good thing to do nevertheless.
Does the kernel have to boot under the latest QEMU release?
For device tree support I would think so. An older Linaro release for U-Boot should be fine too. Maybe John Crigby could provide more details here.
Nicolas
On 15 June 2011 00:47, Nicolas Pitre nicolas.pitre@linaro.org wrote:
On Tue, 14 Jun 2011, Mounir Bsaibes wrote:
Does the kernel have to boot under the latest QEMU release?
For device tree support I would think so. An older Linaro release for U-Boot should be fine too. Maybe John Crigby could provide more details here.
Mounir, are you asking: (a) "Does the kernel require the latest QEMU release to boot, i.e. we don't support running newer kernels on older QEMUs" or (b) "Do we require the kernel to boot on the latest QEMU release, i.e. if it doesn't we hold release until we fix one or the other" ?
The first is true in practice anyway. The second is a nice ideal but could be tricky to achieve in practice...
-- PMM
On Wed, Jun 15, 2011 at 12:00 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 June 2011 00:47, Nicolas Pitre nicolas.pitre@linaro.org wrote:
On Tue, 14 Jun 2011, Mounir Bsaibes wrote:
Does the kernel have to boot under the latest QEMU release?
For device tree support I would think so. An older Linaro release for U-Boot should be fine too. Maybe John Crigby could provide more details here.
Mounir, are you asking: (a) "Does the kernel require the latest QEMU release to boot, i.e. we don't support running newer kernels on older QEMUs" or (b) "Do we require the kernel to boot on the latest QEMU release, i.e. if it doesn't we hold release until we fix one or the other" ?
The first is true in practice anyway. The second is a nice ideal but could be tricky to achieve in practice...
Obviously, we should should ensure latest kernel boots on qemu and qemu can run latest kernel version :)
CI will ensure that we get notified of problems on this sooner; for now we will probably notice during integration - latest. Such a bug would certainly be release critical.
Like with all release critical bugs the decision what to do if this doesn't get fixed will be made by the release team.
Since we do time based releases my guess is that we probably won't delay release. This leaves us with two options:
+ hold back release of new kernel/qemu and re-release the previous one instead of the most recent one + release note this problem and send out the bits anyway
Anyway, I really hope this is unlikely to happen for now and as I mentioned above we are working on improving our CI story which should probably make this even less likely to happen.
On 15 June 2011 11:14, Alexander Sack asac@linaro.org wrote:
Anyway, I really hope this is unlikely to happen for now
[I should start this email with a disclaimer: this isn't intended to be finger pointing, just an explanation of why we should expect and plan for QEMU breakage rather than hoping it is unlikely.]
History tells us that QEMU often breaks when the kernel gains functionality to use new bits of the hardware. The most recent example is that a change which went into the 1105 final release omap3 kernel at some point between the 20110520 snapshot and the 20110526 final release did break QEMU (graphics output stopped working). [Technical detail: the kernel now probes for the presence of a monitor and QEMU wasn't emulating the hardware on the I2C bus which returns EDID info so the kernel thought there was no monitor present and didn't turn on the display. This missing feature in QEMU will be fixed in the 2011.06 release.]
Other examples we've seen: * kernel access to non-existent device registers triggering so many warnings from QEMU as to render it unusable [also present in 1105; worked around in qemu 2011.06] * the move to the omap-specific serial driver required model changes * probing for cp15 perf registers hit qemu bugs where they weren't implemented * bugs in the MMC controller model tickled by a u-boot MMC driver rewrite * newer kernels use the ARM1136r1 TLS registers but QEMU's 1136 model doesn't implement them
The underlying cause here is that QEMU's models are not tested in any formal way against a specification or against a test suite used for validating the hardware. The main test is "does it boot Linux?". So it's inevitable that new kernel features will be exercising essentially untested QEMU code, and breakage is quite likely.
CI will help to flag this kind of problem up sooner (and I have a blueprint for this cycle to work with the validation folks to expand the range of QEMU automated testing and benchmarking), but if we want to guarantee that QEMU and the kernel work together I think we really need to pretty much freeze the kernel two weeks before QEMU's release date, in order to have a fighting chance at catching and fixing problems. Alternatively the kernel team could refuse to merge qemu-breaking changes, but that seems to me like putting the cart before the horse.
(Rolling back to previous qemu release is generally not a possible fix because typically the bug has been in qemu all along and is not a regression.)
-- PMM
On Wed, 15 Jun 2011, Peter Maydell wrote:
The underlying cause here is that QEMU's models are not tested in any formal way against a specification or against a test suite used for validating the hardware. The main test is "does it boot Linux?". So it's inevitable that new kernel features will be exercising essentially untested QEMU code, and breakage is quite likely.
CI will help to flag this kind of problem up sooner (and I have a blueprint for this cycle to work with the validation folks to expand the range of QEMU automated testing and benchmarking), but if we want to guarantee that QEMU and the kernel work together I think we really need to pretty much freeze the kernel two weeks before QEMU's release date, in order to have a fighting chance at catching and fixing problems.
I don't think it is reasonable to freeze the kernel for two weeks in a ~4 week cycle. Therefore we might have to consider simply not having the same release dates for all components. We could pipeline the dependencies instead of being all synchronous.
Nicolas
On 15 June 2011 15:40, Nicolas Pitre nicolas.pitre@linaro.org wrote:
On Wed, 15 Jun 2011, Peter Maydell wrote:
if we want to guarantee that QEMU and the kernel work together I think we really need to pretty much freeze the kernel two weeks before QEMU's release date, in order to have a fighting chance at catching and fixing problems.
I don't think it is reasonable to freeze the kernel for two weeks in a ~4 week cycle. Therefore we might have to consider simply not having the same release dates for all components. We could pipeline the dependencies instead of being all synchronous.
This would mostly work, although you'd end up with the slightly odd effect that qemu-linaro released at the same time as the rest of the toolchain components but nominally as part of the previous month's release...
-- PMM
Or we could just use continuous integration where there are no freezes, just per change regression tests.
-Zach
On 15 June 2011 11:17, Peter Maydell peter.maydell@linaro.org wrote:
On 15 June 2011 15:40, Nicolas Pitre nicolas.pitre@linaro.org wrote:
On Wed, 15 Jun 2011, Peter Maydell wrote:
if we want to guarantee that QEMU and the kernel work together I think we really need to pretty much freeze the kernel two weeks before QEMU's release date, in order to have a fighting chance at catching and fixing problems.
I don't think it is reasonable to freeze the kernel for two weeks in a ~4 week cycle. Therefore we might have to consider simply not having the same release dates for all components. We could pipeline the dependencies instead of being all synchronous.
This would mostly work, although you'd end up with the slightly odd effect that qemu-linaro released at the same time as the rest of the toolchain components but nominally as part of the previous month's release...
-- PMM
linaro-kernel mailing list linaro-kernel@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-kernel
On Thu, Jun 23, 2011 at 3:20 AM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Or we could just use continuous integration where there are no freezes, just per change regression tests.
thats the idealistic goal yes, but getting even close there takes a bit and I am not sure if full continuous integration in the sense that you can land even massive patchsets the day before release just relying on your automated tests is something that can ever happen. e.g. you will always end up with some explicit release process with some kind of restrictions on what type of change is acceptable right before the release etc.
Basically as long as you ever discover any issue through manual testing that you didn't catch through automation before means you are still not ready to do full continuous integration.
On 23 June 2011 03:45, Alexander Sack asac@linaro.org wrote:
On Thu, Jun 23, 2011 at 3:20 AM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Or we could just use continuous integration where there are no freezes, just per change regression tests.
thats the idealistic goal yes, but getting even close there takes a bit and I am not sure if full continuous integration in the sense that you can land even massive patchsets the day before release just relying on your automated tests is something that can ever happen. e.g. you will always end up with some explicit release process with some kind of restrictions on what type of change is acceptable right before the release etc.
With CI loops you tend to not do massive patches, which helps things out. Your point is taken though. CI solves the "end of the month" problem because all the work is already done, you just scrape the results. Since you're always integrating then there's non of this back and forth with tarballs and patches, its just totally streamlined. It dramatically increases the quality and quantity of software that you can produce.
Basically as long as you ever discover any issue through manual testing that you didn't catch through automation before means you are still not ready to do full continuous integration.
Its a spectrum. In reality you still need to bake things. But the CI loop solves the software delivery into a common image beautifully.
linaro-kernel@lists.linaro.org
-
Alexander Sack -
Deepak Saxena -
Mounir Bsaibes -
Nicolas Pitre -
Peter Maydell -
Zach Pfeffer