On Wed, 29 May 2013 17:11:11 +0100 James Tunnicliffe james.tunnicliffe@linaro.org wrote:
Issues raised in http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/tr...
Getting a token: I see this as the service that starts the job has some secret allowing it to request the token, which is then passed onto the job.
Yes, just like that: we have fixed (and thus trusted) job scheduler, which can reliably authenticate itself to a token server, get a token and pass that token to a much less trusted builder.
This may require a Jenkins plugin.
Yes, that's one possibility, as line 94 of the doc you quote says. Another possibility is to have extra frontend on top of Jenkins, which would request a token and inject it into Jenkins job (as a paremeter during start). That's possible right away with android-build and would be possible with other Jenkins with original ci-frontend plan.
Open Embedded Builds/rsync: This is not required. OE has a list of package sources and versions associated with each build so we should be able to write a simple "upload packages if that version doesn't exist" script. This probably requires us to host files in openembedded/sources/<package name>/<version>/<package file>, but that is a simple enough change.
James, for me, what you said translates into something "let's dig into adhoc peculiarities of one usecase, because it seems that it's easy to handle that, as if we don't know that it the end it still won't be that easy, but the result will not be reusable". Nope, let's shoot for general solution of problem of publishing arbitrary file sets. That's also "easy". The hardest part of that is to accept that we'll duplicate best-practice solution (rsync) and by duplicating it, we'll lose extra features like intra-file optimizations (because *that* we for sure don't want to duplicate).
That's why I'd like who anyone has strong reasons for going with rsync is come up with them before we decide on architecture (keeping in mind that rsync comes with friends like ssh, LDAP, PAM, Kerberos, etc.)
publish --token=<token> --type=<build_type> --strip=<strip> <build_id> <glob_pattern>...
This seems like a reasonable starting point. Lets make sure that it uses a configuration file to specify what to do with those build types etc. Preferably one that it can update from a public location so we don't have to re-spin the tool to add a new build type (though I guess we normally check it out of VCS as we go, so that works too).
Well, on client side, it's ideally just a single file which just handle obvious filtering options (like <glob_pattern> or --strip=) locally and passes the rest to API/service. Server-side can handle the options in any way it wants, note that options above don't require much "configuration", for example --type= just maps to top-level download dir.
Except for, well, we already have adhoc behavioral idiosyncrasies, like Android builds flattening happening on server. You hardly can "configure" that (though lambdas in YAML sounds cool (for debugging :-D)). Better approach would be to move that stuff to client side and have simple well-defined publishing semantics.
James
On 29 May 2013 16:57, James Tunnicliffe james.tunnicliffe@linaro.org wrote:
Hi Paul,
Thanks for this. I need a mechanism for publishing from CI runtime jobs so this is important to me. I did look into using SSH/SFTP and it is simple to do in a reasonably insecure way, it would be much better to have an HTTP[S] based solution that uses temporary authentication tokens.
I was looking at this today because I woke up early worrying about it. Clearly I need more interesting stuff to think about! (and now, more sleep).
Anyway, it should be possible to do in Django:
https://docs.djangoproject.com/en/dev/topics/http/file-uploads/ https://pypi.python.org/pypi/django-transfer/0.2-2 http://wiki.nginx.org/HttpUploadModule
My own notes are more focused on an internal server that would upload files to releases/snapshots but could retain files until disk space was needed to act as a cache. I was going to look at extending linaro-licenese-protection for this so there was no way to use the cache to avoid licenses. I was also going to have completely private files that you could only access if you had the authentication token that a job was given.
https://docs.google.com/a/linaro.org/document/d/1_ewb-xFDJc8Adk7AijV95XthGMv...
Feel free to add comments or insert more information and thoughts.
Note that for high performance uploads we probably want to hand off the upload to a web server. That django-transfer module doesn't support any Apache related upload stuff, which may mean that it doesn't exist. Moving to an nginx based solution would be easy enough if we needed to (we could replace the mod-xsendfile with the equivalent nginx call).
I think a prototype branch is ~1 day of engineering effort (no nginx, token based upload call in place, probably some kind of token request call, probably limited security, no proxy-publishing). Adding the rest of the features, testing etc probably takes it to more like 1 week.
James
On 29 May 2013 16:26, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Begin forwarded message:
Date: Wed, 29 May 2013 17:19:31 +0300 From: Paul Sokolovsky Paul.Sokolovsky@linaro.org To: Tyler Baker tyler.baker@linaro.org, Alan Bennett alan.bennett@linaro.org Cc: Senthil Kumaran senthil.kumaran@linaro.org, Fathi Boudra fathi.boudra@linaro.org Subject: Re: New publishing infra prototype report
Hello Tyler,
As brought up today on IRC, it's a month since the below report and proposal for further steps, and I don't remember any reply. This whole publishing thing is peculiar, as it sits still when its not needed, but it's actually in ambush there to cause havoc at any time.
For example, today Senthil came up with the question on how to publish to snapshots. Fortunately, it turned out that it was request for publishing a single file manually. But I know the guys are working on Fedora builds, and that definitely will need automated publishing (unless initial requirements as provided by Fathi changed). And it's definitely needed for CBuild migration, which I assume will be worked on next month.
Btw, I discovered that a BP for that was submitted yet by Danilo: https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/file-publi...
Thanks, Paul
On Mon, 29 Apr 2013 18:58:39 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
Last month I worked on a blueprint https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/prototy... to prototype an implementation of publishing framework which wouldn't depend on particular Jenkins features (and misfeatures) and could be reused for other services across Linaro CI infrastructure. Among these other projects are:
- OpenEmbedded builds - efficient ("fresh only") publishing of
source tarballs and cache files. 2. CBuild - publishing of toolchain build artifacts and logs. 3. Fedora/Lava - publishing of build artifacts and logs.
So, the good news is that was possible to implement a publishing system whose interface is a single script which hides all the publishing complexity underneath. Implementation was a cumbersome, because existing publishing backend was reused, but it already opens possibility for better logging, debugging, profiling, etc.
With proof-of-concept client-side available, the main complexity still lies in server-side backend. It's clear that current "SFTP
- SSH trigger script" approach doesn't scale well in terms of
ease of setup and security. I added my considerations to address that topic in " Conclusions and Future Work" section of http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/tr...
So, action items I suggest based on this report:
- Tyler to consult with Fathi (Fedora), Marcin (OE) and me
(CBuild) and prepare architecture/spec for the general publishing system. It would be nice to BP this task to start in 13.05. 2. Depending on the time required to prepare spec, implementation can be scheduled right away, or postponed until LCE13, so we had another chance to discuss it face to face (as adhoc meeting, or as a session, if it's really worth it).
Thanks, Paul
Linaro.org | Open source software for ARM SoCe Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- James Tunnicliffe