Greetings,
Here is the post-mortem and lessons learned review for Linaro release 11.09. Thanks to the teams who have contributed to this.
Release Review: 2011.09
Android
Highlights
Earlier Toolchain release date was very helpful. Tracking toolchain tip was helpful. LTs were more responsive to bugs.
Issues
BPs can sometimes be unclear. What to do and and why. Suggestions: Make clear where the BP comes from - traceability to requirement. Be prudent with Headline and Done criteria. Make sure BPs are gardened regulaly and WIs are updated. LTs and WGs are not agreeing with all our plans that involve them. Suggestions: PM to add affected groups to each BP and Bug and talk one-on-one with them and their PM at the start of the cycle (maybe co-team kickoffs)?
Some tasks could not be completed in the cycle. Suggestions: If something doesn't work after a week of effort, lets regroup on it. Its hard to see which groups need to work on a BP. Suggestions: Assignee must assert that co-workers are subscribed and properly bugged for progress. Use the prescribed syntax in WI for describing who does what. The PM should also be bugging the other teams PM. iMX53 was difficult. Suggestions: We need to gear up with JTAG debuggers. We need to get the iMX53 LT more engaged. Sometimes we can't have our way (probably we'd have had less trouble if we had started with the LT kernel than with trying to build a LT/jstultz hybrid). Get SoC documentation.
Ask LT members for PoCs. Work the community sites.
Read commit logs for PoCs. Post on android-building and android-platform.
Blueprints that have unfinished items should not be marked as 'implemented' until a review is done. This was the case with https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-snowbal....
Work Items
https://blueprints.launchpad.net/linaro-android/+spec/enable-faketsd-on-leb-... done - But after release, noted in blueprint This was a verification issue that fell through the cracks
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-integra... postponed items broken out into a new blueprint - not a bug since the items need discussion before the work will be done
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-snowbal... Many items here have been postponed, yet the BP is marked implemented.
https://blueprints.launchpad.net/linaro-android/+spec/update-gerrit-and-lava... postponed items - New blueprint created for 11.10: https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-enhance...
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-fix-pow... Bugs to be logged against
Developer Platform
Highlights
Production development cycle since there was more time for development. CI for daily builds was started. Evaluation of derived distro worked well. Debug symbols for kernel delivered: makes tool building easier.
Issues
Work items should be more descriptive in order to be more understandable. Some components from the working groups were not delivered on Thursday Communicating respin: getting better, but a defined process is needed. Separate upstreaming activities from monthly cycle.
Work Items
Cross Buildable Nano PPA, Natty based did not make the release. Rationale: technical development issues Suggestions: Better estimation and sizing effort
OMAP 4 SPL USB support at U-Boot did not make the release. Rationale: Partial Delivery? Suggestions:
Star Rating process definition and documentation did not make the release. Rationale: This blueprint has been stagnating Partial Delivery? Suggestions:
CI with the Linux Linaro packages was split. Unfinished work items have been re-targeted to CI with the Linux Linaro packages - Jenkins and LAVA Integration for release 11.10. Issues: poorly planned and estimated, was put off track by resource dependency.
Bug #865547 has been logged against Cross Buildable Nano PPA, Natty based This blueprint was planned poorly and badly estimated.
OMAP 4 SPL USB support at U-Boot has been moved to the backlog since it has not been started and has been stagnating for a while.
The issue of enabling boot loader testing should be brought to TSC concerning blueprint CI with the U-Boot-Linaro packages Requires improved LAVA support A possible solution could be SPL USB booting support rather than JTAG
Added Bug #865573 to track postponed item from blueprint Change hwpacks to use hwpacks v2
Generic Toolchain Improvements for 11.09 has been moved to the backlog in order to re-plan
Bug 867825 has been logged against postponed item in Resyncing U-Boot-Linaro with upstream U-Boot (11.09)
Bug 867938 has been logged against blueprint Native Gcc-Linaro packages
Bug 867935 has been logged against blueprint Native Gcc-Linaro packages and Cross Native Gcc-Linaro packages
Power Management
Highlights
Team is well set with the process, no issue was reported.
Lessons Learned: each month PM and TL set and assign the blueprints and work items in launchpad then tell the team about them. In another word buffer the engineers from Launchpad modification.
Issues
Constantly in the planning mode, as soon as a month is over, the next starts. Maintaining the plan is currently very expensive.
Toolchain
Highlights
More than one person can push release candidates builds, now.One can push the build and another can analyze the results. Making use of the time zone difference, this method works great, one can push the builds go home, others can analyze the build when they login
Issues
Too many bugs before the release, need to address bugs before they accumulate
Validation
Highlights
Issues
lava-dashboard and lava-server were released later than expected. lava-scheduler-tool and linaro-python-dashboard-bundle weren't planned to be released. Blueprint changes:
https://blueprints.launchpad.net/lava-qatracker/+spec/linaro-platforms-o-lav... (milestone target: 2011.09->2011.10, priority: high->medium)
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-di...: high->low)
Infrastructure
Highlights
Linaro CI build service is now available for wider beta testing for Linaro engineering teams.
Issues
https://blueprints.launchpad.net/linaro-ci/+spec/all-ci-jobs-to-bzr Several blocked items - created bug 869419 to cover it Several postponed items - covered in bug 859473
https://blueprints.launchpad.net/linaro-ci/+spec/jenkins-maintenance-upgrade... started, retarget to 11.10 CI kernel fails to reboot successfully: Bug #860556, re-targeted to 11.10
Release
Highlights
Freescale and ST-Ericsson LT have notified their release, respinned their hwpacks and asked for approval. Overall the release is running smoothly. Each monthly release is going better than the previous one.
Issues
Android: Linaro Android RC images announced later than expected. The URLs to the images should be communicated as we need them for the call for testing. Monday 16:00 UTC should be targeted.
Graphics WG: glproxy wasn't released as expected.
LT: Kernel has been released without notification. Tarball created/released while a tag wasn't available. I'm not even sure the source code tarball is the one used in the hwpack.
Developer Platform: TI LT kernel packaged from git tag. respinned lt-panda hwpack without approval. u-boot-linaro was released later than expected. last minute integration issues with u-boot-linaro (regression spotted with RC).
Validation: lava-dashboard and lava-server were released later than expected. lava-scheduler-tool and linaro-python-dashboard-bundle weren't planned to be released.
Toolchain WG: cortex-strings release delayed. It required to poke TL/PM to get a status. General: Still a lot of work to do on QA and release testing. We need lava-qatracker and mandatory testing. Headlines came late in the cycle. They should be defined during the planning phase, right after the release. Ubuntu-build jobs should be scheduled at 16:00 UTC instead of 00:00 UTC. Image respin should be communicated (including rationale, bugs, etc...). It's like domino: if components aren't released in time, Platform have less time to resolve issues spotted on integration. Release Team should be in the loop for each topic affecting the release: delay, issues, respin, etc...
Lessons Learned
Monthly Release Pm & TL do most of the Launchpad work (leg work) better team acceptance of the process continue doing until team is ready to handle themselves
roles Make sure every role has a backup, has 2 people, that can do it better coverage, when the main responsible is on leave
bugs Too many bugs to handle before the release should address bugs continuously and let them accumulate
Monthly Release Post-mortem came late There was not enough time to deliver the post-mortem quickly after the release Other issues delayed Start the post-mortem process earlier within the release cycle