I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
Thanks, Paul Larson
On 24 April 2012 08:40, Paul Larson paul.larson@linaro.org wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
Yeah, we should because we should make sure these scripts work. Even your own experience is good.
With a good network connection on my W520 I've seen a build get done in an hour.
Thanks, Paul Larson
On Mon, Apr 23, 2012 at 11:55 PM, Zach Pfeffer zach.pfeffer@linaro.orgwrote:
On 24 April 2012 08:40, Paul Larson paul.larson@linaro.org wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time
was
due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much
more
free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2
didn't
exist (it did... just not in the path after it did a cd - submitted a
small
bug for this which should be a very easy fix). So I'm trying again, but
my
question is, should we really be including this test for every build,
given
that it takes so long to run? How long does this normally take to run?
Yeah, we should because we should make sure these scripts work. Even your own experience is good.
With a good network connection on my W520 I've seen a build get done in an hour.
Yeah, I don't think it's the build time that's killing me. Not everyone has a 50Mb/s connection :)
On 24 April 2012 10:36, Paul Larson paul.larson@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:55 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 08:40, Paul Larson paul.larson@linaro.org wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
Yeah, we should because we should make sure these scripts work. Even your own experience is good.
With a good network connection on my W520 I've seen a build get done in an hour.
Yeah, I don't think it's the build time that's killing me. Not everyone has a 50Mb/s connection :)
Hehe...
Andy,
This may be a good thing to put a note in the script about:
1. It may take a long time for things to download 2. If the script fails in the download - print something out about starting over. You may want to leave the contents that were already downloaded intact, unless people pass a special flag in).
On Tue, Apr 24, 2012 at 12:08 AM, Zach Pfeffer zach.pfeffer@linaro.orgwrote:
On 24 April 2012 10:36, Paul Larson paul.larson@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:55 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 08:40, Paul Larson paul.larson@linaro.org wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First
time
was due to a lack of space. The drive I tried to run it on had 6G left,
but
apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again,
but
my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to
run?
Yeah, we should because we should make sure these scripts work. Even your own experience is good.
With a good network connection on my W520 I've seen a build get done in an hour.
Yeah, I don't think it's the build time that's killing me. Not everyone
has
a 50Mb/s connection :)
Hehe...
Andy,
This may be a good thing to put a note in the script about:
- It may take a long time for things to download
- If the script fails in the download - print something out about
starting over. You may want to leave the contents that were already downloaded intact, unless people pass a special flag in).
It already seems to ask/confirm that you want to reuse the existing android directory, which was *really* nice, since it let me continue. So about 12 hours for the download + 2 hours to build and I was finally able to successfully build it using the script.
Andy,
This may be a good thing to put a note in the script about:
- It may take a long time for things to download
- If the script fails in the download - print something out about
starting over. You may want to leave the contents that were already downloaded intact, unless people pass a special flag in).
Please open bug and assign to me. I'll try and fix it possibly next week.
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
On Tue, Apr 24, 2012 at 1:30 AM, Jon Medhurst (Tixy) tixy@linaro.orgwrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12
hours, I
finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd -
submitted
a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
Thanks Tixy, that echoes *exactly* what I was trying to get at. These tests are unreasonably long, and unless they can be automated I think they are of limited value. The tests themselves are really not testing the images, but rather the scripts that are intended to make it easier for people to do these tasks.
Thanks, Paul Larson
On Tue, 2012-04-24 at 01:39 -0500, Paul Larson wrote:
On Tue, Apr 24, 2012 at 1:30 AM, Jon Medhurst (Tixy) tixy@linaro.orgwrote:
[...]
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
Thanks Tixy, that echoes *exactly* what I was trying to get at. These tests are unreasonably long, and unless they can be automated I think they are of limited value. The tests themselves are really not testing the images, but rather the scripts that are intended to make it easier for people to do these tasks.
Well, the scripts are something we're distributing so they should have some testing.
Assuming that the scripts for different boards are essentially the same but with different parameters couldn't we just test one board a month?
Alternatively, as they are meant to build the entire Android system, why don't we produce all the release images using these scripts? That way the final binaries we test and distribute should be pretty much identical to what a user would produce using the same scripts.
On 24 April 2012 12:51, Jon Medhurst (Tixy) tixy@linaro.org wrote:
On Tue, 2012-04-24 at 01:39 -0500, Paul Larson wrote:
On Tue, Apr 24, 2012 at 1:30 AM, Jon Medhurst (Tixy) tixy@linaro.orgwrote:
[...]
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
Thanks Tixy, that echoes *exactly* what I was trying to get at. These tests are unreasonably long, and unless they can be automated I think they are of limited value. The tests themselves are really not testing the images, but rather the scripts that are intended to make it easier for people to do these tasks.
Well, the scripts are something we're distributing so they should have some testing.
Assuming that the scripts for different boards are essentially the same but with different parameters couldn't we just test one board a month?
I think once the builds are all the same except the kernel that'll work. We're not there yet, but perhaps in the next few months we will be.
Alternatively, as they are meant to build the entire Android system, why don't we produce all the release images using these scripts? That way the final binaries we test and distribute should be pretty much identical to what a user would produce using the same scripts.
I actually think that's a good idea since it would "close the loop." Right now the script that builds things on Jenkins is close to this script, but this script is intended to be a human readable version tuned directly to a particular build.
-- Tixy
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now,
and
I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with
much
more free space. This is running on a core i7, 8GB ram. After 12
hours, I
finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd -
submitted
a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
That's what I'm hoping - see my 2nd note in this thread where I suggested that android build use this. Then it would be doing this step for us effectively. Thanks, Paul Larson
On 24 April 2012 13:28, Paul Larson paul.larson@linaro.org wrote:
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much more free space. This is running on a core i7, 8GB ram. After 12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
That's what I'm hoping - see my 2nd note in this thread where I suggested that android build use this. Then it would be doing this step for us effectively.
Aye, and using linaro-android-media-create to program the images. Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time. Adding Paul S for visibility.
Thanks, Paul Larson
On 04/24/2012 03:53 AM, Zach Pfeffer wrote:
That's what I'm hoping - see my 2nd note in this thread where I suggested
that android build use this. Then it would be doing this step for us effectively.
Aye, and using linaro-android-media-create to program the images. Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time. Adding Paul S for visibility.
Its not quite that simple because Paul's build scripts create these build scripts. Also, the current build scripts require manual steps for snowball/origen so that they can get the vendor.tar.gz specified.
Its all just code and could be worked around. Paul's opinion will be the important one.
Hello,
[Alexander, Joey, at the end are some thought on how we got to known EC2 usage, and how we can "optimize" it.]
Sorry, I'm a bit out if context, so adding my comments wherever I see something to comment ;-).
On Tue, 24 Apr 2012 14:23:09 +0530 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 13:28, Paul Larson paul.larson@linaro.org wrote:
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much
Google says 30Gb free space is needed for their builds, we need more for our builds. linaro-android-build-cmds.sh really should check for 50Gb and refuse to run otherwise.
more free space. This is running on a core i7, 8GB ram. After
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
It's just the same as Google's own check for 64-bitness in their makefiles. Can the build process work on 32bits? Of course it can, it's just not supported. So it all happens strictly in the following order 1) users get visible indication that what they do is unsupported; 2) users go to patch that out because they obviously know better; 3) users continue, in full awareness that they (and only they) put themselves in a position for problems.
12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path
That's why I was great proponent of adding these build scripts - to be able to add loads of validation *before* starting the build, to avoid cases like spending hours (days) to get only errors in result. To build is very easy, setting up proper environment turns out to be very hard, based on the feedback we saw. IMHO, build scripts should be 5-10Kb, and 90% of that should be environment validation for all possible and impossible environment problems which may happen on users' machines (essentially, whenever we hear a problem report, we add a validation for that). The alternative is a user frustration, which we already saw quite a lot on IRC.
after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
That's what I'm hoping - see my 2nd note in this thread where I suggested that android build use this. Then it would be doing this step for us effectively.
Aye, and using linaro-android-media-create to program the images. Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time. Adding Paul S for visibility.
Creating images with linaro-android-media-create on android-build was already in talks, following concerns were raised:
1. The way it was suggested is to add linaro-android-media-create processing at the end of build. Android builds are already very complicated and monolithic - it's for example very hard to test any changes to build process already.
2. That produces even more artifacts (arguable, with low "hit" ratio), with us going out of space all the time already.
3. It requires running root. We don't have root for build scripts on android-build. And with restricted builds launch, I guess it really should stay that way.
More generally, recently I wondered that we turn our Jenkins instances from build systems into generic script runners. I didn't share that speculation previously, because I thought "indeed, why not". But now we know what's the problem with that - it costs too much (re: this month's hot discussion of us having gone over EC2 budget, I don't know if that's widely known).
We use cloud for Android builds because android *build* (i.e. compiling gigabytes of C++ files, produced by humans, and thus expected to contain errors) requires enormous resources and time. It's plain expensive to use that for file archive munging. So, what about old good cronjobs? (Alternatively, cheap instances could be used for that kind of work, but someone first need to implement that, and that's still inefficient cost-wise and arguably more complicated than setting a cronjob).
Another way is to rethink what and how we do. For example:
Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time.
At first, this seems like a good idea, but thinking more, the machine code which is executed by CPU is the same in Android standard tarballs and in images produced by l-i-t. So, regarding testing of android *code* the latter is as helpful as wrapping a tarball in other bzip2 round (but it takes extra resources to do that).
I don't say that l-i-t images shouldn't be tested - apparently, they should be, but that's *different* kind of testing, with different priority, schedule, etc.
Thanks, Paul Larson
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On 25 April 2012 13:59, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
[Alexander, Joey, at the end are some thought on how we got to known EC2 usage, and how we can "optimize" it.]
Sorry, I'm a bit out if context, so adding my comments wherever I see something to comment ;-).
On Tue, 24 Apr 2012 14:23:09 +0530 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 13:28, Paul Larson paul.larson@linaro.org wrote:
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much
Google says 30Gb free space is needed for their builds, we need more for our builds. linaro-android-build-cmds.sh really should check for 50Gb and refuse to run otherwise.
more free space. This is running on a core i7, 8GB ram. After
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
A lot of developers have 8GB of memory and we build Android ICS on EC2 large instance with 7.5 GB memory afaik.
It's just the same as Google's own check for 64-bitness in their makefiles. Can the build process work on 32bits? Of course it can, it's just not supported. So it all happens strictly in the following order
- users get visible indication that what they do is unsupported; 2)
users go to patch that out because they obviously know better; 3) users continue, in full awareness that they (and only they) put themselves in a position for problems.
12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path
That's why I was great proponent of adding these build scripts - to be able to add loads of validation *before* starting the build, to avoid cases like spending hours (days) to get only errors in result. To build is very easy, setting up proper environment turns out to be very hard, based on the feedback we saw. IMHO, build scripts should be 5-10Kb, and 90% of that should be environment validation for all possible and impossible environment problems which may happen on users' machines (essentially, whenever we hear a problem report, we add a validation for that). The alternative is a user frustration, which we already saw quite a lot on IRC.
after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
That's what I'm hoping - see my 2nd note in this thread where I suggested that android build use this. Then it would be doing this step for us effectively.
Aye, and using linaro-android-media-create to program the images. Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time. Adding Paul S for visibility.
Creating images with linaro-android-media-create on android-build was already in talks, following concerns were raised:
- The way it was suggested is to add linaro-android-media-create
processing at the end of build. Android builds are already very complicated and monolithic - it's for example very hard to test any changes to build process already.
- That produces even more artifacts (arguable, with low "hit" ratio),
with us going out of space all the time already.
- It requires running root. We don't have root for build scripts on
android-build. And with restricted builds launch, I guess it really should stay that way.
More generally, recently I wondered that we turn our Jenkins instances from build systems into generic script runners. I didn't share that speculation previously, because I thought "indeed, why not". But now we know what's the problem with that - it costs too much (re: this month's hot discussion of us having gone over EC2 budget, I don't know if that's widely known).
We use cloud for Android builds because android *build* (i.e. compiling gigabytes of C++ files, produced by humans, and thus expected to contain errors) requires enormous resources and time. It's plain expensive to use that for file archive munging. So, what about old good cronjobs? (Alternatively, cheap instances could be used for that kind of work, but someone first need to implement that, and that's still inefficient cost-wise and arguably more complicated than setting a cronjob).
Another way is to rethink what and how we do. For example:
Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time.
At first, this seems like a good idea, but thinking more, the machine code which is executed by CPU is the same in Android standard tarballs and in images produced by l-i-t. So, regarding testing of android *code* the latter is as helpful as wrapping a tarball in other bzip2 round (but it takes extra resources to do that).
I don't say that l-i-t images shouldn't be tested - apparently, they should be, but that's *different* kind of testing, with different priority, schedule, etc.
Thanks, Paul Larson
Thanks, Paul
On 25 April 2012 18:25, Fathi Boudra fathi.boudra@linaro.org wrote:
On 25 April 2012 13:59, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
[Alexander, Joey, at the end are some thought on how we got to known EC2 usage, and how we can "optimize" it.]
Sorry, I'm a bit out if context, so adding my comments wherever I see something to comment ;-).
On Tue, 24 Apr 2012 14:23:09 +0530 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 13:28, Paul Larson paul.larson@linaro.org wrote:
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote:
I was trying to run the linaro-android-build-cmds-sh a few times now, and I've never actually been able to get all the way through it. First time was due to a lack of space. The drive I tried to run it on had 6G left, but apparently that's not enough. So I ran it again on a drive with much
Google says 30Gb free space is needed for their builds, we need more for our builds. linaro-android-build-cmds.sh really should check for 50Gb and refuse to run otherwise.
more free space. This is running on a core i7, 8GB ram. After
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
I build android on my 4gb ram laptop in 2 hrs. the requirement mentioned
from google is not mandatory but preferable.
A lot of developers have 8GB of memory and we build Android ICS on EC2
large instance with 7.5 GB memory afaik.
It's just the same as Google's own check for 64-bitness in their makefiles. Can the build process work on 32bits? Of course it can, it's just not supported. So it all happens strictly in the following order
- users get visible indication that what they do is unsupported; 2)
users go to patch that out because they obviously know better; 3) users continue, in full awareness that they (and only they) put themselves in a position for problems.
12 hours, I finally got far enough to get an error telling me that vendor.tar.bz2 didn't exist (it did... just not in the path
That's why I was great proponent of adding these build scripts - to be able to add loads of validation *before* starting the build, to avoid cases like spending hours (days) to get only errors in result. To build is very easy, setting up proper environment turns out to be very hard, based on the feedback we saw. IMHO, build scripts should be 5-10Kb, and 90% of that should be environment validation for all possible and impossible environment problems which may happen on users' machines (essentially, whenever we hear a problem report, we add a validation for that). The alternative is a user frustration, which we already saw quite a lot on IRC.
after it did a cd - submitted a small bug for this which should be a very easy fix). So I'm trying again, but my question is, should we really be including this test for every build, given that it takes so long to run? How long does this normally take to run?
I didn't run these tests when I did the vexpress testing because I knew the large amount of data which would need to be downloaded and disk space required.
The other linaro-android-build-env build tests also would require me to setup two new Ubuntu chroots and do that enormous download an build twice again.
I justify to myself that I didn't need to run them by using the fact that the tests weren't marked with and 'm' to say they should be run monthly. But to be honest, after 5 hour testing the release I wanted to be doing other things, not having my work laptop out of action for what would probably be the best part of another day.
If these tests are meant to run monthly, can't we automate this in 'the cloud' somewhere?
That's what I'm hoping - see my 2nd note in this thread where I suggested that android build use this. Then it would be doing this step for us effectively.
Aye, and using linaro-android-media-create to program the images. Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time. Adding Paul S for visibility.
Creating images with linaro-android-media-create on android-build was already in talks, following concerns were raised:
- The way it was suggested is to add linaro-android-media-create
processing at the end of build. Android builds are already very complicated and monolithic - it's for example very hard to test any changes to build process already.
- That produces even more artifacts (arguable, with low "hit" ratio),
with us going out of space all the time already.
- It requires running root. We don't have root for build scripts on
android-build. And with restricted builds launch, I guess it really should stay that way.
More generally, recently I wondered that we turn our Jenkins instances from build systems into generic script runners. I didn't share that speculation previously, because I thought "indeed, why not". But now we know what's the problem with that - it costs too much (re: this month's hot discussion of us having gone over EC2 budget, I don't know if that's widely known).
We use cloud for Android builds because android *build* (i.e. compiling gigabytes of C++ files, produced by humans, and thus expected to contain errors) requires enormous resources and time. It's plain expensive to use that for file archive munging. So, what about old good cronjobs? (Alternatively, cheap instances could be used for that kind of work, but someone first need to implement that, and that's still inefficient cost-wise and arguably more complicated than setting a cronjob).
Another way is to rethink what and how we do. For example:
Would mean we'd be automatically testing exactly the way users would use the builds which would save a bunch of time.
At first, this seems like a good idea, but thinking more, the machine code which is executed by CPU is the same in Android standard tarballs and in images produced by l-i-t. So, regarding testing of android *code* the latter is as helpful as wrapping a tarball in other bzip2 round (but it takes extra resources to do that).
I don't say that l-i-t images shouldn't be tested - apparently, they should be, but that's *different* kind of testing, with different priority, schedule, etc.
Thanks, Paul Larson
Thanks, Paul
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
Hello,
On Thu, 26 Apr 2012 00:32:14 +0530 Vishal Bhoj vishal.bhoj@linaro.org wrote:
On 25 April 2012 18:25, Fathi Boudra fathi.boudra@linaro.org wrote:
On 25 April 2012 13:59, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
[Alexander, Joey, at the end are some thought on how we got to known EC2 usage, and how we can "optimize" it.]
Sorry, I'm a bit out if context, so adding my comments wherever I see something to comment ;-).
On Tue, 24 Apr 2012 14:23:09 +0530 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 24 April 2012 13:28, Paul Larson paul.larson@linaro.org wrote:
On Apr 24, 2012 1:30 AM, "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Mon, 2012-04-23 at 22:10 -0500, Paul Larson wrote: > I was trying to run the linaro-android-build-cmds-sh a few > times now, and > I've never actually been able to get all the way through it. > First time was due to a lack of space. The drive I tried > to run it on had 6G left, but apparently that's not > enough. So I ran it again on a drive with much
Google says 30Gb free space is needed for their builds, we need more for our builds. linaro-android-build-cmds.sh really should check for 50Gb and refuse to run otherwise.
> more free space. This is running on a core i7, 8GB ram. > After
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
I build android on my 4gb ram laptop in 2 hrs.
Build, like your working directory, or rebuild from scratch (a new checkout)? Can't believe that, the checkout alone may take more ;-). But if you know some trick, let us know - maybe we can go from very expensive XLARGE instances to LARGE.
the requirement mentioned from google is not mandatory but preferable.
Yes, I elaborated my IMHO on what all those requirements mean - something for user to override. But again IMHO, it's still good idea to use them as the baseline, to set users' expectations right.
A lot of developers have 8GB of memory and we build Android ICS on EC2 large instance with 7.5 GB memory afaik.
No we're compliant with upstream requirements - how could we do it otherwise? We use XLARGE instances with 15Gb, and maybe the lacking gigabyte is the cause why some of instances hang after a build (just kidding ;-) ).
It's just the same as Google's own check for 64-bitness in their makefiles. Can the build process work on 32bits? Of course it can, it's just not supported. So it all happens strictly in the following order 1) users get visible indication that what they do is unsupported; 2) users go to patch that out because they obviously know better; 3) users continue, in full awareness that they (and only they) put themselves in a position for problems.
> 12 hours, I > finally got far enough to get an error telling me that > vendor.tar.bz2 didn't exist (it did... just not in the path
That's why I was great proponent of adding these build scripts - to be able to add loads of validation *before* starting the build, to avoid cases like spending hours (days) to get only errors in result. To build is very easy, setting up proper environment turns out to be very hard, based on the feedback we saw. IMHO, build scripts should be 5-10Kb, and 90% of that should be environment validation for all possible and impossible environment problems which may happen on users' machines (essentially, whenever we hear a problem report, we add a validation for that). The alternative is a user frustration, which we already saw quite a lot on IRC.
[]
On Thu, 2012-04-26 at 12:25 +0300, Paul Sokolovsky wrote:
Hello,
On Thu, 26 Apr 2012 00:32:14 +0530 Vishal Bhoj vishal.bhoj@linaro.org wrote:
On 25 April 2012 13:59, Paul Sokolovsky paul.sokolovsky@linaro.org
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
I build android on my 4gb ram laptop in 2 hrs.
Build, like your working directory, or rebuild from scratch (a new checkout)? Can't believe that, the checkout alone may take more ;-). But if you know some trick, let us know - maybe we can go from very expensive XLARGE instances to LARGE.
As an experiment I booted my 8GB laptop with mem=4096M and built Android from clean in 55 minutes. That was with make -j4 and memory usage came perilously close to the 3.5GB that the 'free' command said was the total system RAM. (Note, I don't have swap enabled so all this worked in real memory.)
With make -j8 the build failed, though I was also using Mumble at the time as well.
Hello Jon,
On Thu, 26 Apr 2012 15:13:34 +0100 "Jon Medhurst (Tixy)" tixy@linaro.org wrote:
On Thu, 2012-04-26 at 12:25 +0300, Paul Sokolovsky wrote:
Hello,
On Thu, 26 Apr 2012 00:32:14 +0530 Vishal Bhoj vishal.bhoj@linaro.org wrote:
On 25 April 2012 13:59, Paul Sokolovsky paul.sokolovsky@linaro.org
Google says 16Gb is the requirement, linaro-android-build-cmds.sh really should check and refuse to run otherwise.
I build android on my 4gb ram laptop in 2 hrs.
Build, like your working directory, or rebuild from scratch (a new checkout)? Can't believe that, the checkout alone may take more ;-). But if you know some trick, let us know - maybe we can go from very expensive XLARGE instances to LARGE.
As an experiment I booted my 8GB laptop with mem=4096M and built Android from clean in 55 minutes. That was with make -j4 and memory usage came perilously close to the 3.5GB that the 'free' command said was the total system RAM. (Note, I don't have swap enabled so all this worked in real memory.)
With make -j8 the build failed, though I was also using Mumble at the time as well.
Ok, I also set up a build on LARGE (instead of our usual XLARGE) instance: https://android-build.linaro.org/jenkins/job/pfalcon_vexpress-rtsm-ics-gcc46...
So, it works, but takes twice as much time (4:15h). EC2-wise, that will be rounded to 5hrs, so no economical effect. So, doesn't make any sense to switch to LARGE build slaves.
But coming back to the point of generating dosk images in the cloud, we could set up in the similar manner SMALL instance, allow sudo on it, and generate images there with separate job (which would probably generate images for all recent builds in a batch).
linaro-android@lists.linaro.org
-
Andy Doan -
Fathi Boudra -
Jon Medhurst (Tixy) -
Paul Larson -
Paul Sokolovsky -
Vishal Bhoj -
Zach Pfeffer