BOSH Stemcells are very slow to download about bosh-linux-stemcell-builder HOT 10 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/166018258

The labels on this github issue will be updated when the story is started.

from bosh-linux-stemcell-builder.

I've got some actual numbers from GCP. I uploaded a stemcell to GCS and then measured downloading it from there and then from the original endpoint.

GCP

$ time curl https://storage.googleapis.com/xoebus-test/bosh-stemcell-315.34-warden-boshlite-ubuntu-xenial-go_agent.tgz -o gcp.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  492M  100  492M    0     0  56.8M      0  0:00:08  0:00:08 --:--:-- 68.9M

real	0m8.681s
user	0m0.966s
sys	0m1.397s

AWS

$ time curl https://s3.amazonaws.com/bosh-core-stemcells/315.34/bosh-stemcell-315.34-warden-boshlite-ubuntu-xenial-go_agent.tgz -o aws.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  492M  100  492M    0     0  2820k      0  0:02:58  0:02:58 --:--:-- 2992k

real	2m58.867s
user	0m2.028s
sys	0m4.893s

8 seconds vs. nearly 180 seconds makes it seem like there's something fishy with the bucket setup.

from bosh-linux-stemcell-builder.

It just took me nearly 2 hours to download a BOSH-lite stemcell. 😭

Can I help you with any work to improve this? I expect we're wasting a tonne of time across the org. waiting for stemcell downloads (on workstations and in pipelines).

from bosh-linux-stemcell-builder.

At one point some downloads were being fronted by CloudFront CDN; not sure if/when/why that may have changed. I think there was some code to recognize and convert supported buckets to the CDN. Let me know if I can help.

from bosh-linux-stemcell-builder.

We enabled transfer acceleration from S3 a while ago, but it seems that you have to do some more leg work to actually use it. We are looking into that today.

Update:

It looks like we will have to change all the URLs to the accelerated endpoint.

from bosh-linux-stemcell-builder.

I also did a little experiment to mirror all of the stemcells into a GCS bucket using their storage transfer service. It was able to mirror everything (500GB) in around two minutes. This makes it seem like it might be an issue with our office (the peering between AWS and our ISP?) rather than the bucket itself?

from bosh-linux-stemcell-builder.

Interesting. That might be another issue.

We were able to assert that turning on the acceleration brings the downloads closer to GCP levels. Here are the links:

Fast

time curl https://bosh-core-stemcells.s3-accelerate.amazonaws.com/315.34/bosh-stemcell-315.34-warden-boshlite-ubuntu-xenial-go_agent.tgz -o aws.tar.gz

Slow

time curl https://s3.amazonaws.com/bosh-core-stemcells/315.34/bosh-stemcell-315.34-warden-boshlite-ubuntu-xenial-go_agent.tgz -o aws.tar.gz

So for now, you can use the "fast" link as a template for future downloads as we figure out how to change all the URLs.

from bosh-linux-stemcell-builder.

Just tried it out and it works great - thank you!

Want me to leave this open until the https://bosh.io change is made or should I close it?

from bosh-linux-stemcell-builder.

We can leave it open.

from bosh-linux-stemcell-builder.

Closing - stemcells were moved to accelerated buckets in https://www.pivotaltracker.com/story/show/168994545

from bosh-linux-stemcell-builder.