Hi, all,
There's still a WI in squid proxy blueprint. That is to clean up the old cache. I jumped into linaro cache directory on v.l.o, and found it cost about 100GB.
control:/linaro/images/cache$ du * -sh 328M ci.linaro.org 36M people.linaro.org 3.4G releases-linaro-org 10G releases.linaro.org 21G snapshots-linaro-org 63G snapshots.linaro.org
control:/linaro/images/cache$ ll total 32 drwxr-xr-x 8 root root 4096 2012-05-11 23:26 ./ drwxr-xr-x 7 root root 4096 2012-01-25 01:16 ../ drwxr-xr-x 3 root root 4096 2012-03-30 12:07 ci.linaro.org/ drwxr-xr-x 3 root root 4096 2012-04-03 05:34 people.linaro.org/ drwxr-xr-x 4 root root 4096 2012-05-24 16:27 releases-linaro-org/ drwxr-xr-x 10 root root 4096 2012-05-09 14:31 releases.linaro.org/ drwxr-xr-x 3 root root 4096 2012-05-11 23:26 snapshots-linaro-org/ drwxr-xr-x 6 root root 4096 2012-05-02 18:55 snapshots.linaro.org/
For some health check jobs, they need some cache. I think it's releases-linaro-org/ and snapshots-linaro-org/.
So can we clean up others?
On 06/07/2012 02:43 AM, Spring Zhang wrote:
Hi, all,
There's still a WI in squid proxy blueprint. That is to clean up the old cache. I jumped into linaro cache directory on v.l.o, and found it cost about 100GB.
control:/linaro/images/cache$ du * -sh 328M ci.linaro.org http://ci.linaro.org 36M people.linaro.org http://people.linaro.org 3.4G releases-linaro-org 10G releases.linaro.org http://releases.linaro.org 21G snapshots-linaro-org 63G snapshots.linaro.org http://snapshots.linaro.org
control:/linaro/images/cache$ ll total 32 drwxr-xr-x 8 root root 4096 2012-05-11 23:26 ./ drwxr-xr-x 7 root root 4096 2012-01-25 01:16 ../ drwxr-xr-x 3 root root 4096 2012-03-30 12:07 ci.linaro.org/ http://ci.linaro.org/ drwxr-xr-x 3 root root 4096 2012-04-03 05:34 people.linaro.org/ http://people.linaro.org/ drwxr-xr-x 4 root root 4096 2012-05-24 16:27 releases-linaro-org/ drwxr-xr-x 10 root root 4096 2012-05-09 14:31 releases.linaro.org/ http://releases.linaro.org/ drwxr-xr-x 3 root root 4096 2012-05-11 23:26 snapshots-linaro-org/ drwxr-xr-x 6 root root 4096 2012-05-02 18:55 snapshots.linaro.org/ http://snapshots.linaro.org/
For some health check jobs, they need some cache. I think it's releases-linaro-org/ and snapshots-linaro-org/.
So can we clean up others?
Clean up the others for now, but I actually just noticed a problem in our current code. The only thing that we should now be managing cache for are the root.tgz and boot.tgz files. However, we aren't caching pre-built images properly. In master.py we are still calling download_with_cache for pre-built images rather than downloading via the proxy.
I have a small local branch with a fix for this. I forgot about it, so I'm going to test on our staging instance to make sure it hasn't suffered from bit rot. Once that's fixed, I think we can deploy the fix, and remove all non boot/root tarballs from the cache.
Does this make sense?
-andy
Andy Doan andy.doan@linaro.org writes:
On 06/07/2012 02:43 AM, Spring Zhang wrote:
Hi, all,
There's still a WI in squid proxy blueprint. That is to clean up the old cache. I jumped into linaro cache directory on v.l.o, and found it cost about 100GB.
control:/linaro/images/cache$ du * -sh 328M ci.linaro.org http://ci.linaro.org 36M people.linaro.org http://people.linaro.org 3.4G releases-linaro-org 10G releases.linaro.org http://releases.linaro.org 21G snapshots-linaro-org 63G snapshots.linaro.org http://snapshots.linaro.org
control:/linaro/images/cache$ ll total 32 drwxr-xr-x 8 root root 4096 2012-05-11 23:26 ./ drwxr-xr-x 7 root root 4096 2012-01-25 01:16 ../ drwxr-xr-x 3 root root 4096 2012-03-30 12:07 ci.linaro.org/ http://ci.linaro.org/ drwxr-xr-x 3 root root 4096 2012-04-03 05:34 people.linaro.org/ http://people.linaro.org/ drwxr-xr-x 4 root root 4096 2012-05-24 16:27 releases-linaro-org/ drwxr-xr-x 10 root root 4096 2012-05-09 14:31 releases.linaro.org/ http://releases.linaro.org/ drwxr-xr-x 3 root root 4096 2012-05-11 23:26 snapshots-linaro-org/ drwxr-xr-x 6 root root 4096 2012-05-02 18:55 snapshots.linaro.org/ http://snapshots.linaro.org/
For some health check jobs, they need some cache. I think it's releases-linaro-org/ and snapshots-linaro-org/.
So can we clean up others?
Clean up the others for now, but I actually just noticed a problem in our current code. The only thing that we should now be managing cache for are the root.tgz and boot.tgz files. However, we aren't caching pre-built images properly. In master.py we are still calling download_with_cache for pre-built images rather than downloading via the proxy.
Whoops.
I have a small local branch with a fix for this. I forgot about it, so I'm going to test on our staging instance to make sure it hasn't suffered from bit rot. Once that's fixed, I think we can deploy the fix, and remove all non boot/root tarballs from the cache.
Sure. I don't think we should worry /too/ much about deleting things from the cache, after all its just a cache and if we have to regenerate the tarballs for the health jobs, that's not the worst thing in the world.
Cheers, mwh
linaro-validation@lists.linaro.org
-
Andy Doan -
Michael Hudson-Doyle -
Spring Zhang