Greetings,
Here is the post-mortem and lessons learned review for Linaro release 11.10. Thanks to the teams who have contributed to this.
==================== Release Review 11.10 ====================
Android ======= Highlights
* Gerrit CI was well tested when deployed. * Making test results visible created quality awareness across the organization.
Issues
* Infrastructure capacity (git server) was inadequate and affected developer performance. * The one-month cycle leaves no room for events like Connect and ELC-E. * Heavy reliance on landing teams as most issues are theirs. * Bugs fixed in Developer are marked fixed even if not working on Android. * Landing teams seem to get too little support from member company developers. * The manual test frenzy at release time should be reduced or better shared. * Engineers that also have interrupt type duties struggle with Milestone delivery accuracy. * Daily test errors did not get immediate attention.
Developer Platform ================== Highlights
* Successful release on Oneiric ahead of schedule * CI on packaged kernels working * Communications need to be examined * John released U-Boot in time! That's great and keep up :) * Release process went fairly well
Issues
* Blueprints were not updated very well. * Launchpad issues blocked many blueprints * builders in LP were blocking some blueprints * Offspring and LP related issues take time to resolve * Lauchpad: to many bugs on derived distro support * testing was not done well on the LP side * Issues with the WGs: some did not release on time * communications were not smooth or even non-existent * transparency of plans is the problem; platform teams need to be aligned about plans * Unity tree not available on time, released only on Friday * Once tested, it did not work on Panda - it seems that Unity is not tested on Panda * The working group should continuously push to public * lack of communication and support for hardware problems * LAVA interface very slow - needs rework * requires better error reporting of the LAVA web service (error on job submission with wrong hardware pack) * Need better navigation of job from build to result in the dashboard UI * Linux-linaro quality isn't really good (no strict tracking of features etc.) * device tree is missing regularly. * Test cases should be provided well in advance of testing * Test cases in spreadsheet are vague, some not valid to test
Lessons Learned
* set up a point of contact with working groups for any issues during the release. * establish a best-practice process for the graphics group to continuously push to public * tests could have been better defined; next step is developing it.
Kernel ====== Highlights
* Pinctrl core and pinmux are now in linux-next. currently, answering late review comments and merging smaller patches. * Continued to implement Device Tree support for Linaro member platforms, focusing on changes to drivers and subsystems inlcuding IRQ controllers, GPIO, serial devices, MMC devices, and regulators across various SOCs. All i.MX basic drivers are ready, audio and usb are still big missing pieces. * omap-hsmmc dt conversion completed. Prepared single device tree enabled board file for smdkv310 and Origen boards and tested device tree support for the following modules: UART, SDHCI, Keypad, GPIO keys, DMA, RTC, I2C, WDT, GPIO, IRQ. * Continued with SoC Tree maintenance, code review, merging and verifying patches. * Continued work on fixes and changes to the config fragment merge_config.sh script, resubmitted merge_config.sh script to lkml. * Continued the cleanup and consolidation of various kernel headers that will allow for building of a single kernel across multiple SOCs the work included: patches to rework the low level UART debugging code on OMAP1, OMAP2 and Davinci, patches for mach/memory.h removal, patches for appending of a device tree binary to the kernel zImage and patches for removing all instances of mach/vmalloc.h.
Multimedia ========== Highlights
* There was a speedy reaction to get headlines andacceptance defined faster. Teams had those in place (90% of the BPs) by the deadline. * Managed to move ahead with some of the issues we have been facing eg Unity upstreaming in GFX.
Issues
* Quite a few blueprints did not complete. This can be attributed to a few possible reasons: * work was larger than initially thought * assignee was distracted to other activities * trees needed to be more carefully constructed at times to avoid mistakes from being based on old versions of upstream * assignee not "gardening" blueprints diligently * Sometimes the work is done but the blueprint doesn't reflect the reality
General lessons learned
* Maintain a tree always available. If tree is broken we should have a good idea when it will be available again. * Always work on the tip and fix things as they appear - that is easier said than done for some projects. So we check the cases one by one on the projects involved * Raise the risk flag early: kudos to those of you who did so this month. Perhaps an improvement next would be to try and anticipate the risk with each work item rather than on the blueprint level. * Even if you think that the release is not worth the effort, for example an optimization in performance did not show up, it is always good to make the release if you have patches which do not break the component. You will get more eyeballs looking at the components this way maybe that will help unearth other issues or suggest new ideas.
Power Management ================ Highlights
* Thermal management is working on Samsumg platform. However, the code is not pushed to mainline yet. * OMAP Thermal management integrated and delivered to TI. But, still under testing and not completely working. * PM QA testsuite now includes tests for hotplug integrated with LAVA, studying LAVA test runs for accuracy.
Issues
Toolchain ========= Highlights
* Released Linaro GDB 7.3 11.10 * Released Linaro GCC 4.6 2011.10 and Linaro GCC 4.5 2011.10 * Released Linaro QEMU 2011.10
Issues
Validation ========== Highlights
* Initial release of bundle format documentation * Ability to display reports on the front page.
Issues
* Oneiric complicated packaging, weakened testing and development focus.
Infrastructure ============== Highlights
* Released linaro-image-tools 2011.10 * Didn't have to keep up with board changes due to hwpacks v2, already a success for that project.
Issues
* Had to change to cope with a breaking change in the format of rootfs created by live-build. We should add metadata to the rootfs to avoid this problem in future. Session registered at Connect to discuss it. https://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-p-imag...
Lessons Learned
Headlines * Headlines assumptions did not come true * To be ahead of the curb, in the PMWG, we created the monthly headline one week ahead of time * Not all headlines completed * wait until after the release to do the headline
Quality * Quality status not known at all times * Important to ensure quality awareness across the organization * Less quality focus * Document what we test and then keep testing it
Blueprints * Blueprints do not reflect reality * Invalid functionality * Work in small chunks on the days up to the release. Plan to release early and often as work items are done. Do not wait to have some mad dash in the hours before the release
-- David Zinman Linaro Release Manager | Project Manager Linaro.org | Open source software for ARM SoCs