Hello James,

On Wed, 29 May 2013 16:57:50 +0100 James Tunnicliffe james.tunnicliffe@linaro.org wrote:

Hi Paul,

Thanks for this. I need a mechanism for publishing from CI runtime jobs so this is important to me. I did look into using SSH/SFTP and it is simple to do in a reasonably insecure way, it would be much better to have an HTTP[S] based solution that uses temporary authentication tokens.

I was looking at this today because I woke up early worrying about it. Clearly I need more interesting stuff to think about! (and now, more sleep).

Well, I guess each of us (individual engineers) was hit by publishing issues already, so that's clearly a thing to worry about and I share your concerns. So, thanks for joining the discussion (and let's nag Tyler for spec together ;-) ).

Anyway, it should be possible to do in Django:

https://docs.djangoproject.com/en/dev/topics/http/file-uploads/ https://pypi.python.org/pypi/django-transfer/0.2-2 http://wiki.nginx.org/HttpUploadModule

My own notes are more focused on an internal server that would upload files to releases/snapshots but could retain files until disk space was needed to act as a cache. I was going to look at extending linaro-licenese-protection for this so there was no way to use the cache to avoid licenses. I was also going to have completely private files that you could only access if you had the authentication token that a job was given.

https://docs.google.com/a/linaro.org/document/d/1_ewb-xFDJc8Adk7AijV95XthGMv...

Feel free to add comments or insert more information and thoughts.

I'll look in more detail later. Just having a small window before restarting work on Crowd API auth ;-).

Note that for high performance uploads we probably want to hand off the upload to a web server. That django-transfer module doesn't support any Apache related upload stuff, which may mean that it doesn't exist. Moving to an nginx based solution would be easy enough if we needed to (we could replace the mod-xsendfile with the equivalent nginx call).

Yeah, I had experience with all that stuff at SourceForge.net, where high availability and extra features (like progress indicator on web page) were a must. I'd think that we can start with simple Django-level PUT handler, but yes, we should be prepared that it'll have performance issues and for need to upgrade architecture.

I think a prototype branch is ~1 day of engineering effort (no nginx, token based upload call in place, probably some kind of token request call, probably limited security, no proxy-publishing). Adding the rest of the features, testing etc probably takes it to more like 1 week.

Well, I'd really like us (all) to first consider architecture well enough. I also lean towards having that as a web service, but I'd like someone who has other opinion to have their say too.

James

On 29 May 2013 16:26, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:

...

Begin forwarded message:

Date: Wed, 29 May 2013 17:19:31 +0300 From: Paul Sokolovsky Paul.Sokolovsky@linaro.org To: Tyler Baker tyler.baker@linaro.org, Alan Bennett alan.bennett@linaro.org Cc: Senthil Kumaran senthil.kumaran@linaro.org, Fathi Boudra fathi.boudra@linaro.org Subject: Re: New publishing infra prototype report

Hello Tyler,

As brought up today on IRC, it's a month since the below report and proposal for further steps, and I don't remember any reply. This whole publishing thing is peculiar, as it sits still when its not needed, but it's actually in ambush there to cause havoc at any time.

For example, today Senthil came up with the question on how to publish to snapshots. Fortunately, it turned out that it was request for publishing a single file manually. But I know the guys are working on Fedora builds, and that definitely will need automated publishing (unless initial requirements as provided by Fathi changed). And it's definitely needed for CBuild migration, which I assume will be worked on next month.

Btw, I discovered that a BP for that was submitted yet by Danilo: https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/file-publi...

Thanks, Paul

On Mon, 29 Apr 2013 18:58:39 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:

...

Hello,

Last month I worked on a blueprint https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/prototy... to prototype an implementation of publishing framework which wouldn't depend on particular Jenkins features (and misfeatures) and could be reused for other services across Linaro CI infrastructure. Among these other projects are:

1. OpenEmbedded builds - efficient ("fresh only") publishing of

source tarballs and cache files. 2. CBuild - publishing of toolchain build artifacts and logs. 3. Fedora/Lava - publishing of build artifacts and logs.

So, the good news is that was possible to implement a publishing system whose interface is a single script which hides all the publishing complexity underneath. Implementation was a cumbersome, because existing publishing backend was reused, but it already opens possibility for better logging, debugging, profiling, etc.

With proof-of-concept client-side available, the main complexity still lies in server-side backend. It's clear that current "SFTP + SSH trigger script" approach doesn't scale well in terms of ease of setup and security. I added my considerations to address that topic in " Conclusions and Future Work" section of http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/tr...

So, action items I suggest based on this report:

1. Tyler to consult with Fathi (Fedora), Marcin (OE) and me

(CBuild) and prepare architecture/spec for the general publishing system. It would be nice to BP this task to start in 13.05. 2. Depending on the time required to prepare spec, implementation can be scheduled right away, or postponed until LCE13, so we had another chance to discuss it face to face (as adhoc meeting, or as a session, if it's really worth it).

Thanks, Paul

Linaro.org | Open source software for ARM SoCe Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog

-- Best Regards, Paul

Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog

-- Best Regards, Paul

Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog