Hey everyone.
I'd like to update you on where we are with BUILD-INFO support.
Over the past few days I've been testing how our existing, mock/test BUILD-INFO.txt are handled by the current build and delivery system. The outcome of this tests show that it has a number of shortcomings that make it impractical to really deploy the new binary overlays there. Builds protected by OpenID cannot be integrated at all and all other builds would need to use the word 'blob' in the Jenkins configuration name for anything else to work.
To address this, the current PHP-based codebase was rewritten as a Django application. We are currently waiting for two RT tickets to allow that application to land on a new staging server. As this is still 'far away' according to IS we need to come up with a new plan that would allow us to make progress and untangle us from a slow and frustrating RT-bound development cycle.
I've discussed this with Asac just now and we came up with the following plan:
1) I will locally deploy the new Django codebase and feed it with some of the data from the snapshot server. This will allow me to verify how the codebase reacts to various BUILD-INFO.txt without really building anything or waiting for RT to land a file in certain spot. This will allow us to ensure that all the BUILD-INFO.txt overlays I've created work as intended.
2) I will manually patch and request uploads of three vendor tarballs: one for origen and two for snowball. We currently have two snowball overlays that have been patched but due to bugs detected later in testing they need to be updated again.
3) We will wait for the RT tickets 504 and 518 to land. This will allow us to have the new Django codebase on a staging server. Then, in coordination with Linaro Infrastructure and Canonical IS we'll do the switch. This part is the most risky step in my opinion. The switch involves two objects: our server software and tip configurations of all existing builds.
The risky step is to prevent any old builds from bein downloaded without relying on the old PHP rules to serve EULAs. I will be working with gesha to ensure we have planned this step well.
To allow us to track 1-3 we need to make some changes to our existing blueprints. Currently we have three blueprints that cover BUILD-INFO work:
* https://blueprints.launchpad.net/linaro-android/+spec/include-build-info-txt
This blueprint is already marked as implemented although Asac has raised concerns as to this being inappropriate as it clearly contradicts the acceptance criteria. Still I would like to keep it closed and just focus on the remaining two:
* https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt
This blueprint currently captures testing in existing environment, generating real overlay tarballs and updating the configuration for tip builds. I'd like to extend it to cover testing in the new django codebase and anything that comes out of that.
* https://blueprints.launchpad.net/linaro-license-protection/+spec/django-rewr...
This blueprint covers the new django rewrite and the deployment. It also has some aspects of BUILD-INFO such as job migration. Since there is overlap here I'd like to remove it from one of the blueprints so that there is exactly one person that is responsible for doing it and there is no confusion.
One of the thing that is still problematic is the unfinished aspect of the rewritten code. According to this blueprint the final specification of BUILD-INFO is not yet finished. I'd like to avoid the situation where code bugs or bugs coupled with specification changes render existing vendor tarballs broken.
I think we need to have a very clear idea on how to implement this to avoid any transition risks and know how to track progress. I'd like to propose the following actions to happen:
[pfefferz or asac]
* ack a change to include-build-info-txt Headline and Acceptance fields so that it can stay implemented
* ack the production transition plan created by gesha
* provide parts of snapshots.linaro.org files to zyga to recreate some of the builds for testing
[zyga]
* deploy the django rewrite locally, syncing with gesha if needed
* test (on django codebase), fix if needed and upload snowball and origen tarballs to snapshots.linaro.org
[gesha]
* describe how to transition from old PHP to Django server and how the new staging instance is used in that transition. The critical part here is to understand what happens to files published without BUILD-INFO or writ damaged/wrong BUILD-INFO.
* get IS to assist with the transition, ensure that we can revert to the old system if something goes wrong
Once we all agree on the following plan I'd like to document it in our blueprints.
Thanks. ZK
On high level I see the following steps for completing roll out of BUILD-INFO for snapshots:
1. validate that the BUILD-INFO.txt files we have for open, eula and openid work properly. For that install django locally so you can test without having all the legacy code around.
2. Update vendor tarballs with known-to-be-working BUILD-INFO.txt
3. Land proper BUILD-INFO.txt vendor tarballs on _all_ android builds.
4. Get staging server up with a mirrored filesystem. Deploy django app there and drop all legacy/php code. Validate that everything still works as expected and that all builds have proper access control/restrictions.
5. switch DNS of staging over to be new production host.
then: tackle releases.linaro.org
On Thu, Jul 26, 2012 at 5:13 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'd like to update you on where we are with BUILD-INFO support.
Over the past few days I've been testing how our existing, mock/test BUILD-INFO.txt are handled by the current build and delivery system. The outcome of this tests show that it has a number of shortcomings that make it impractical to really deploy the new binary overlays there. Builds protected by OpenID cannot be integrated at all and all other builds would need to use the word 'blob' in the Jenkins configuration name for anything else to work.
To address this, the current PHP-based codebase was rewritten as a Django application. We are currently waiting for two RT tickets to allow that application to land on a new staging server. As this is still 'far away' according to IS we need to come up with a new plan that would allow us to make progress and untangle us from a slow and frustrating RT-bound development cycle.
I've discussed this with Asac just now and we came up with the following plan:
- I will locally deploy the new Django codebase and feed it with some of
the data from the snapshot server. This will allow me to verify how the codebase reacts to various BUILD-INFO.txt without really building anything or waiting for RT to land a file in certain spot. This will allow us to ensure that all the BUILD-INFO.txt overlays I've created work as intended.
- I will manually patch and request uploads of three vendor tarballs: one
for origen and two for snowball. We currently have two snowball overlays that have been patched but due to bugs detected later in testing they need to be updated again.
- We will wait for the RT tickets 504 and 518 to land. This will allow us
to have the new Django codebase on a staging server. Then, in coordination with Linaro Infrastructure and Canonical IS we'll do the switch. This part is the most risky step in my opinion. The switch involves two objects: our server software and tip configurations of all existing builds.
The risky step is to prevent any old builds from bein downloaded without relying on the old PHP rules to serve EULAs. I will be working with gesha to ensure we have planned this step well.
To allow us to track 1-3 we need to make some changes to our existing blueprints. Currently we have three blueprints that cover BUILD-INFO work:
https://blueprints.launchpad.net/linaro-android/+spec/include-build-info-txt
This blueprint is already marked as implemented although Asac has raised concerns as to this being inappropriate as it clearly contradicts the acceptance criteria. Still I would like to keep it closed and just focus on the remaining two:
https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt
This blueprint currently captures testing in existing environment, generating real overlay tarballs and updating the configuration for tip builds. I'd like to extend it to cover testing in the new django codebase and anything that comes out of that.
https://blueprints.launchpad.net/linaro-license-protection/+spec/django-rewr...
This blueprint covers the new django rewrite and the deployment. It also has some aspects of BUILD-INFO such as job migration. Since there is overlap here I'd like to remove it from one of the blueprints so that there is exactly one person that is responsible for doing it and there is no confusion.
One of the thing that is still problematic is the unfinished aspect of the rewritten code. According to this blueprint the final specification of BUILD-INFO is not yet finished. I'd like to avoid the situation where code bugs or bugs coupled with specification changes render existing vendor tarballs broken.
I think we need to have a very clear idea on how to implement this to avoid any transition risks and know how to track progress. I'd like to propose the following actions to happen:
[pfefferz or asac]
- ack a change to include-build-info-txt Headline and Acceptance fields so
that it can stay implemented
ack the production transition plan created by gesha
provide parts of snapshots.linaro.org files to zyga to recreate some of
the builds for testing
[zyga]
deploy the django rewrite locally, syncing with gesha if needed
test (on django codebase), fix if needed and upload snowball and origen
tarballs to snapshots.linaro.org
[gesha]
- describe how to transition from old PHP to Django server and how the new
staging instance is used in that transition. The critical part here is to understand what happens to files published without BUILD-INFO or writ damaged/wrong BUILD-INFO.
- get IS to assist with the transition, ensure that we can revert to the old
system if something goes wrong
Once we all agree on the following plan I'd like to document it in our blueprints.
Thanks. ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
Django implementation has backwards compatibility with EULA/OPEN-EULA protection, it means that old builds won't be impacted by build-info stuff, the behavior will be the same as previously.
On 07/26/2012 07:56 PM, Alexander Sack wrote:
On high level I see the following steps for completing roll out of BUILD-INFO for snapshots:
- validate that the BUILD-INFO.txt files we have for open, eula and
openid work properly. For that install django locally so you can test without having all the legacy code around.
That was done during development, but testing with real data is good point
Update vendor tarballs with known-to-be-working BUILD-INFO.txt
Land proper BUILD-INFO.txt vendor tarballs on _all_ android builds.
Get staging server up with a mirrored filesystem. Deploy django app
there and drop all legacy/php code. Validate that everything still works as expected and that all builds have proper access control/restrictions.
There won't be mirrored filesystem, only directories structure and configurations.
- switch DNS of staging over to be new production host.
Staging servers are here for pre-release testing of all code that infra team produces, so they won't be used as new production servers. They are functional copy of production servers so we can do testing and preparing the deployment steps without depending and interacting with IS and then deploy with guarantee on production servers.
then: tackle releases.linaro.org
On Thu, Jul 26, 2012 at 5:13 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'd like to update you on where we are with BUILD-INFO support.
Over the past few days I've been testing how our existing, mock/test BUILD-INFO.txt are handled by the current build and delivery system. The outcome of this tests show that it has a number of shortcomings that make it impractical to really deploy the new binary overlays there. Builds protected by OpenID cannot be integrated at all and all other builds would need to use the word 'blob' in the Jenkins configuration name for anything else to work.
To address this, the current PHP-based codebase was rewritten as a Django application. We are currently waiting for two RT tickets to allow that application to land on a new staging server. As this is still 'far away' according to IS we need to come up with a new plan that would allow us to make progress and untangle us from a slow and frustrating RT-bound development cycle.
I've discussed this with Asac just now and we came up with the following plan:
- I will locally deploy the new Django codebase and feed it with some of
the data from the snapshot server. This will allow me to verify how the codebase reacts to various BUILD-INFO.txt without really building anything or waiting for RT to land a file in certain spot. This will allow us to ensure that all the BUILD-INFO.txt overlays I've created work as intended.
- I will manually patch and request uploads of three vendor tarballs: one
for origen and two for snowball. We currently have two snowball overlays that have been patched but due to bugs detected later in testing they need to be updated again.
- We will wait for the RT tickets 504 and 518 to land. This will allow us
to have the new Django codebase on a staging server. Then, in coordination with Linaro Infrastructure and Canonical IS we'll do the switch. This part is the most risky step in my opinion. The switch involves two objects: our server software and tip configurations of all existing builds.
The risky step is to prevent any old builds from bein downloaded without relying on the old PHP rules to serve EULAs. I will be working with gesha to ensure we have planned this step well.
To allow us to track 1-3 we need to make some changes to our existing blueprints. Currently we have three blueprints that cover BUILD-INFO work:
https://blueprints.launchpad.net/linaro-android/+spec/include-build-info-txt
This blueprint is already marked as implemented although Asac has raised concerns as to this being inappropriate as it clearly contradicts the acceptance criteria. Still I would like to keep it closed and just focus on the remaining two:
https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt
This blueprint currently captures testing in existing environment, generating real overlay tarballs and updating the configuration for tip builds. I'd like to extend it to cover testing in the new django codebase and anything that comes out of that.
https://blueprints.launchpad.net/linaro-license-protection/+spec/django-rewr...
This blueprint covers the new django rewrite and the deployment. It also has some aspects of BUILD-INFO such as job migration. Since there is overlap here I'd like to remove it from one of the blueprints so that there is exactly one person that is responsible for doing it and there is no confusion.
One of the thing that is still problematic is the unfinished aspect of the rewritten code. According to this blueprint the final specification of BUILD-INFO is not yet finished. I'd like to avoid the situation where code bugs or bugs coupled with specification changes render existing vendor tarballs broken.
I think we need to have a very clear idea on how to implement this to avoid any transition risks and know how to track progress. I'd like to propose the following actions to happen:
[pfefferz or asac]
- ack a change to include-build-info-txt Headline and Acceptance fields so
that it can stay implemented
ack the production transition plan created by gesha
provide parts of snapshots.linaro.org files to zyga to recreate some of
the builds for testing
[zyga]
deploy the django rewrite locally, syncing with gesha if needed
test (on django codebase), fix if needed and upload snowball and origen
tarballs to snapshots.linaro.org
[gesha]
- describe how to transition from old PHP to Django server and how the new
staging instance is used in that transition. The critical part here is to understand what happens to files published without BUILD-INFO or writ damaged/wrong BUILD-INFO.
- get IS to assist with the transition, ensure that we can revert to the old
system if something goes wrong
Once we all agree on the following plan I'd like to document it in our blueprints.
Thanks. ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
On 26 July 2012 10:13, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'd like to update you on where we are with BUILD-INFO support.
Over the past few days I've been testing how our existing, mock/test BUILD-INFO.txt are handled by the current build and delivery system. The outcome of this tests show that it has a number of shortcomings that make it impractical to really deploy the new binary overlays there. Builds protected by OpenID cannot be integrated at all and all other builds would need to use the word 'blob' in the Jenkins configuration name for anything else to work.
To address this, the current PHP-based codebase was rewritten as a Django application. We are currently waiting for two RT tickets to allow that application to land on a new staging server. As this is still 'far away' according to IS we need to come up with a new plan that would allow us to make progress and untangle us from a slow and frustrating RT-bound development cycle.
I've discussed this with Asac just now and we came up with the following plan:
- I will locally deploy the new Django codebase and feed it with some of
the data from the snapshot server. This will allow me to verify how the codebase reacts to various BUILD-INFO.txt without really building anything or waiting for RT to land a file in certain spot. This will allow us to ensure that all the BUILD-INFO.txt overlays I've created work as intended.
- I will manually patch and request uploads of three vendor tarballs: one
for origen and two for snowball. We currently have two snowball overlays that have been patched but due to bugs detected later in testing they need to be updated again.
- We will wait for the RT tickets 504 and 518 to land. This will allow us
to have the new Django codebase on a staging server. Then, in coordination with Linaro Infrastructure and Canonical IS we'll do the switch. This part is the most risky step in my opinion. The switch involves two objects: our server software and tip configurations of all existing builds.
The risky step is to prevent any old builds from bein downloaded without relying on the old PHP rules to serve EULAs. I will be working with gesha to ensure we have planned this step well.
To allow us to track 1-3 we need to make some changes to our existing blueprints. Currently we have three blueprints that cover BUILD-INFO work:
https://blueprints.launchpad.net/linaro-android/+spec/include-build-info-txt
This blueprint is already marked as implemented although Asac has raised concerns as to this being inappropriate as it clearly contradicts the acceptance criteria. Still I would like to keep it closed and just focus on the remaining two:
https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt
This blueprint currently captures testing in existing environment, generating real overlay tarballs and updating the configuration for tip builds. I'd like to extend it to cover testing in the new django codebase and anything that comes out of that.
https://blueprints.launchpad.net/linaro-license-protection/+spec/django-rewr...
This blueprint covers the new django rewrite and the deployment. It also has some aspects of BUILD-INFO such as job migration. Since there is overlap here I'd like to remove it from one of the blueprints so that there is exactly one person that is responsible for doing it and there is no confusion.
One of the thing that is still problematic is the unfinished aspect of the rewritten code. According to this blueprint the final specification of BUILD-INFO is not yet finished. I'd like to avoid the situation where code bugs or bugs coupled with specification changes render existing vendor tarballs broken.
I think we need to have a very clear idea on how to implement this to avoid any transition risks and know how to track progress. I'd like to propose the following actions to happen:
[pfefferz or asac]
- ack a change to include-build-info-txt Headline and Acceptance fields so
that it can stay implemented
ack the production transition plan created by gesha
provide parts of snapshots.linaro.org files to zyga to recreate some of
the builds for testing
[zyga]
deploy the django rewrite locally, syncing with gesha if needed
test (on django codebase), fix if needed and upload snowball and origen
tarballs to snapshots.linaro.org
[gesha]
- describe how to transition from old PHP to Django server and how the new
staging instance is used in that transition. The critical part here is to understand what happens to files published without BUILD-INFO or writ damaged/wrong BUILD-INFO.
- get IS to assist with the transition, ensure that we can revert to the old
system if something goes wrong
Once we all agree on the following plan I'd like to document it in our blueprints.
Lets get a meeting together. It sounds like we're trying to replace the entire infrastructure at the same time that we're trying to change the implementation. I think we just need to stage these things a little bit.
1. Get the new infrastructure set up 2. Demo the BUILD-INFO.txt 3. Switch to it
I'm not sure why Zyga needs to replicate the entire Django installation, seems like a task for the infrastructure team that undertook this massive effort.
I think we need to step back and think about a few things. Will schedule a call for tomorrow.
Thanks. ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
linaro-android@lists.linaro.org
-
Alexander Sack -
Georgy Redkozubov -
Zach Pfeffer -
Zygmunt Krynicki