leroy-merlin-br / action-s3-cache Goto Github PK

https://github.com/peak/s5cmd

For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively.

can we use this to make the s3 powered cache faster?

https://github.com/actions/cache allows giving it multiple restore-keys and it'll try to prefix-match them in order against existing cache entries. This allows useful setups like having a cache per repository branch, with a fallback to get the cache of the default branch when a branch is new (i.e. a PR just got opened). It would be nice if actions-s3-cache supported this too.

I'd like to set up my S3 cache so that CI runs on pull requests can read from the cache, but for security reasons they shouldn't be able to write to it. I tried this by

• making AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY GitHub Secrets. GitHub Actions make sure that they're only available when built in a branch of the repository, not on PRs.
• making the S3 bucket publicly readable

Now, unfortunately, action-s3-cache doesn't seem to be able to read public S3 buckets. If I omit the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID variables when calling it, it fails with an error

2021/04/25 23:45:09 NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

What's the best way to make this work?

Hi, thanks for this great project.

It would be cool if it was possible to change the S3 storage class. For caches, OneZoneIA is enough and probably cheaper than Standard.

Hi!

Opening this ticket to keep track of this potential feature : it would be great to be able to use Minio or Scaleway S3 providers with this lib.

I'm not a Golang developer, but I may give it a try in the upcoming weeks. It seems feasible.

Thanks for this action 🙏 , you may want to chime in actions/cache#354 and let them know!

Hi,

Sometimes I get the following error when getting a cache from S3:

2023-04-19T11:32:04.0904284Z 2023/04/19 11:32:04 Cache downloaded successfully, containing 8870952960 bytes
2023-04-19T11:32:04.0904886Z 2023/04/19 11:32:04 zip: not a valid zip file

It seems that the problem is not with the zip file that's stored in a bucket cause if I rerun the failed GitHub Actions job everything works fine.

What are the possible causes of such error?

Let me know if you require any additional information.