Hello,
Short background:
1. We need to access license-protected files as hosted on snapshots.linaro.org in automated (or mediated) way as part if our CI workflow. 2. We used to have multiple ways to handle that, mostly adhoc, and most of which have bitrotted by now. 3. There were also concerns of specific means they're implemented (supposedly as not providing due diligence of license acceptance handling comparing to default procedure of browser download).
So, after confirming that we still have more than one client usage for this, I proceeded to implement what is intended to be single tool to handle this matter. The requirements are at: https://bugs.launchpad.net/linaro-license-protection/+bug/1130810 . The tool can be got as:
bzr branch lp:linaro-license-protection/download
or by following "download file" link at http://bazaar.launchpad.net/~linaro-automation/linaro-license-protection/dow...
(unfortunately, Launchpad Bzr browser doesn't seem to have means of getting download link to HEAD revision of a file, which is misfeature).
Please review both requirements collected and implementation, any suggestion for improvements are welcome.
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool. We should put the complexity in the server code and make clients trivial. Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from the web interface with different templates) to implement something like:
releases.linaro.org/api/path/to/directory = get listing releases.linaro.org/api/path/to/directory/file/license = get license text
If we want some clients to have unrestricted access without going through the license download/accept cycle we could have an API key based authentication mechanism. Keys could be managed in the Django admin interface and only given out to run our own services, such as regression tools. releases.linaro.org/api/path/to/directory/file?api_key=<api key>
If we want anyone who accepts a license to be able to download through the API we could use the same license accept mechanism as the web interface - provide a license hash along with the request to prove you have downloaded the license and indicate that you accept it: releases.linaro.org/api/path/to/directory/file?accept_license=license_md5
I would personally avoid the last option because it would be very tempting to post links that use this API to give people direct access to files. We could easily work around this by salting the license file by appending "Downloaded on <time stamp>" to each download. If the time stamp is accurate to the minute we can just allow hashes for now and last minute.
Once this is in place clients are trivial and sharing links doesn't work for very long.
James
On 22 February 2013 15:56, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
Short background:
- We need to access license-protected files as hosted on
snapshots.linaro.org in automated (or mediated) way as part if our CI workflow. 2. We used to have multiple ways to handle that, mostly adhoc, and most of which have bitrotted by now. 3. There were also concerns of specific means they're implemented (supposedly as not providing due diligence of license acceptance handling comparing to default procedure of browser download).
So, after confirming that we still have more than one client usage for this, I proceeded to implement what is intended to be single tool to handle this matter. The requirements are at: https://bugs.launchpad.net/linaro-license-protection/+bug/1130810 . The tool can be got as:
bzr branch lp:linaro-license-protection/download
or by following "download file" link at http://bazaar.launchpad.net/~linaro-automation/linaro-license-protection/dow...
(unfortunately, Launchpad Bzr browser doesn't seem to have means of getting download link to HEAD revision of a file, which is misfeature).
Please review both requirements collected and implementation, any suggestion for improvements are welcome.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Fri, Feb 22, 2013 at 05:08:43PM +0000, James Tunnicliffe wrote:
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool. We should put the complexity in the server code and make clients trivial. Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from the web interface with different templates) to implement something like:
releases.linaro.org/api/path/to/directory = get listing releases.linaro.org/api/path/to/directory/file/license = get license text
If we want some clients to have unrestricted access without going through the license download/accept cycle we could have an API key based authentication mechanism. Keys could be managed in the Django admin interface and only given out to run our own services, such as regression tools. releases.linaro.org/api/path/to/directory/file?api_key=<api key>
If we want anyone who accepts a license to be able to download through the API we could use the same license accept mechanism as the web interface - provide a license hash along with the request to prove you have downloaded the license and indicate that you accept it: releases.linaro.org/api/path/to/directory/file?accept_license=license_md5
I would personally avoid the last option because it would be very tempting to post links that use this API to give people direct access to files. We could easily work around this by salting the license file by appending "Downloaded on <time stamp>" to each download. If the time stamp is accurate to the minute we can just allow hashes for now and last minute.
Once this is in place clients are trivial and sharing links doesn't work for very long.
You could use HTTP headers instead of GET parameters for transmitting API keys and license acceptance data. Passing custom headers is trivial in headless clients, and impossible to represent in URL's so you don't have this problem with links.
On 25 February 2013 13:35, Antonio Terceiro antonio.terceiro@linaro.org wrote:
On Fri, Feb 22, 2013 at 05:08:43PM +0000, James Tunnicliffe wrote:
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool. We should put the complexity in the server code and make clients trivial. Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from the web interface with different templates) to implement something like:
releases.linaro.org/api/path/to/directory = get listing releases.linaro.org/api/path/to/directory/file/license = get license text
If we want some clients to have unrestricted access without going through the license download/accept cycle we could have an API key based authentication mechanism. Keys could be managed in the Django admin interface and only given out to run our own services, such as regression tools. releases.linaro.org/api/path/to/directory/file?api_key=<api key>
If we want anyone who accepts a license to be able to download through the API we could use the same license accept mechanism as the web interface - provide a license hash along with the request to prove you have downloaded the license and indicate that you accept it: releases.linaro.org/api/path/to/directory/file?accept_license=license_md5
I would personally avoid the last option because it would be very tempting to post links that use this API to give people direct access to files. We could easily work around this by salting the license file by appending "Downloaded on <time stamp>" to each download. If the time stamp is accurate to the minute we can just allow hashes for now and last minute.
Once this is in place clients are trivial and sharing links doesn't work for very long.
You could use HTTP headers instead of GET parameters for transmitting API keys and license acceptance data. Passing custom headers is trivial in headless clients, and impossible to represent in URL's so you don't have this problem with links.
Excellent point. That would be a far less complicated workaround.
Hello,
On Fri, 22 Feb 2013 17:08:43 +0000 James Tunnicliffe james.tunnicliffe@linaro.org wrote:
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool.
Yes, but *one* tool. Previously, we had (well, still have) bunch of tool breaking in random placed and times. So, having single tool to fix is already big improvement over what we had and exactly the requirement put into implementing that.
We should put the complexity in the server code and make clients trivial.
Well, most of our clients for that are shell-based, so we need shell-based "API layer" anyway.
Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from
Famous last words ;-). We definitely need general publishing/access API for our file storage infrastructure which would cover all usecases we've been hitting for a long time. Designing, passing thru review, implementing such API is not a 1 day task though...
So, the ideas are good and I appreciate them (having similar feeling for the need of it), but let's first get everyone on the same line regarding using one supported tool, and perfectalize its implementation later.
In that regard, I'd really like to get feedback on licensing handling (UI interaction wise) - unless you're lawyer, you can never be sure it's done right, and I'd hate something obvious to be missed and pop up later.
[]
On 26 February 2013 11:52, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
On Fri, 22 Feb 2013 17:08:43 +0000 James Tunnicliffe james.tunnicliffe@linaro.org wrote:
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool.
Yes, but *one* tool. Previously, we had (well, still have) bunch of tool breaking in random placed and times. So, having single tool to fix is already big improvement over what we had and exactly the requirement put into implementing that.
The reason why we have so many tools is because we didn't officially support one. I seem to recall that this was because we didn't have a clear answer about how such a tool should work in terms of presenting licenses and caching acceptances. I did write a tool/library that was pretty close to allowing licenses to be displayed and permanently accepted, but there wasn't any enthusiasm for getting those extra features finished so it remained yet another download bypassing the license protection tool. I don't think writing yet another tool is better than fixing an existing one and supporting it.
We should put the complexity in the server code and make clients trivial.
Well, most of our clients for that are shell-based, so we need shell-based "API layer" anyway.
That is why I suggested an API that could easily be interacted with using pretty much anything.
Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from
Famous last words ;-). We definitely need general publishing/access API for our file storage infrastructure which would cover all usecases we've been hitting for a long time. Designing, passing thru review, implementing such API is not a 1 day task though...
I agree that review is always an unpredictable delay, but I really don't think writing it to what Antonio and I have discussed would take longer than a day since it is just adding a couple of URLs to the existing web interface. We are talking about adding a couple of JSON listings and a new GET method that looks for the license hash in a custom header instead of reading it from a cookie. You could probably modify the existing code to make it look for the license hash in the header first and if that failed look for a cookie in a few lines. It is such a small change that adding it as an experimental interface is the easiest way to see if it works for everyone!
So, the ideas are good and I appreciate them (having similar feeling for the need of it), but let's first get everyone on the same line regarding using one supported tool, and perfectalize its implementation later.
In that regard, I'd really like to get feedback on licensing handling (UI interaction wise) - unless you're lawyer, you can never be sure it's done right, and I'd hate something obvious to be missed and pop up later.
Which is why I stuck to an API that required the client to interact with the license in the same way that the web interface works (which uses a hash of a license stored in a cookie to accept the license). We already can't enforce that the user sees the license and the scripts that avoid it completely aren't helping.
On Tue, 26 Feb 2013 12:23:10 +0000 James Tunnicliffe james.tunnicliffe@linaro.org wrote:
On 26 February 2013 11:52, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
On Fri, 22 Feb 2013 17:08:43 +0000 James Tunnicliffe james.tunnicliffe@linaro.org wrote:
Hi Paul,
Thanks for looking at this. The problem with this approach is changes to the web interface will break the tool.
Yes, but *one* tool. Previously, we had (well, still have) bunch of tool breaking in random placed and times. So, having single tool to fix is already big improvement over what we had and exactly the requirement put into implementing that.
The reason why we have so many tools is because we didn't officially support one. I seem to recall that this was because we didn't have a clear answer about how such a tool should work in terms of presenting licenses and caching acceptances.
Exactly, that's the problem we have - organizational one, not technical. So, I finally decided to get out of the cave and do something about it ;-).
I did write a tool/library that was pretty close to allowing licenses to be displayed and permanently accepted, but there wasn't any enthusiasm for getting those extra features finished so it remained yet another download bypassing the license protection tool. I don't think writing yet another tool is better than fixing an existing one and supporting it.
Everyone did write such tool/library. Most (all?) of them in piles now. I saw few, and decided that it's easier to write one from scratch to adhere to well defined requirements as specified in https://bugs.launchpad.net/linaro-license-protection/+bug/1130810 (collected over long time), critical of which being applying due diligence to license handling (not just being close to it) and being easy to use (which includes not being dug deep inside some repo where nobody can find it and not requiring any dependencies which can be done away with).
We should put the complexity in the server code and make clients trivial.
Well, most of our clients for that are shell-based, so we need shell-based "API layer" anyway.
That is why I suggested an API that could easily be interacted with using pretty much anything.
Adding an API to linaro-license-protection that is independent of page rendering wouldn't be difficult (1 day of work - it is mostly copy/paste from
Famous last words ;-). We definitely need general publishing/access API for our file storage infrastructure which would cover all usecases we've been hitting for a long time. Designing, passing thru review, implementing such API is not a 1 day task though...
I agree that review is always an unpredictable delay, but I really don't think writing it to what Antonio and I have discussed would take longer than a day since it is just adding a couple of URLs to the existing web interface. We are talking about adding a couple of JSON listings and a new GET method that looks for the license hash in a custom header instead of reading it from a cookie. You could probably modify the existing code to make it look for the license hash in the header first and if that failed look for a cookie in a few lines. It is such a small change that adding it as an experimental interface is the easiest way to see if it works for everyone!
I'd be glad to modify existing code, but unfortunately, I'm assigned to work on other code, and it's at risk. I took this because I'm on the maintenance and have to react to incoming requests. So, I had to resolve issue Dave was having problem with that in LAVA, and I just made attempt to resolve it right, because nobody else did it for months. That's where it stands - I actually wasn't assigned to it, and it all is just (well-thought) RFC. If you're interested to follow it up, you're welcome, if your schedule allows that.
So, the ideas are good and I appreciate them (having similar feeling for the need of it), but let's first get everyone on the same line regarding using one supported tool, and perfectalize its implementation later.
In that regard, I'd really like to get feedback on licensing handling (UI interaction wise) - unless you're lawyer, you can never be sure it's done right, and I'd hate something obvious to be missed and pop up later.
Which is why I stuck to an API that required the client to interact with the license in the same way that the web interface works (which uses a hash of a license stored in a cookie to accept the license). We already can't enforce that the user sees the license and the scripts that avoid it completely aren't helping.
The script I wrote works as much as possible close to how the web interface works. It shows license *unconditionally* on downloading a license protected file. Yes, it outputs it to stdout, and user can
/dev/null that, but then in the web interface user can not read license
and click button right away. So, we cannot enforce user to see license (that probably would be violation of their human rights ;-) ), but we can enforce that our tools show it, that's what web interface does, that's what the script I wrote does.
Not only that, my script's primary way to pre-accept license is to specify its id hash (default mode is still interactive acceptance). Yeah, there's a switch to preaccept any license, because, heck, that's what we need ourselves most of the time. I thought about adding big fat warning when such switch is used, but you know, it's like adding a sign "this backdoor is always open".
linaro-validation@lists.linaro.org
-
Antonio Terceiro -
James Tunnicliffe -
Paul Sokolovsky