heroku / heroku-buildpack-awscli Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
On Heroku-20 the build output showed the errors seen in #12 however the compile of this buildpack did not exit non-zero, so the build was marked as a success.
I believe this would be fixed by using the bash pipefail
option, which would stop the |& indent
here (and elsewhere) from swallowing the failure:
heroku-buildpack-awscli/bin/compile
Line 38 in 784a787
See also:
http://redsymbol.net/articles/unofficial-bash-strict-mode/
Hi there!
I tried using the buildpack in the Heroku CI and it failed installing with the following messages:
-----> Fetching heroku-community/awscli buildpack...
buildpack downloaded
-----> AWS CLI app detected
-----> Installing AWS CLI
/tmp/buildpacks/737ef5e961c7064fa7fd8cc69b95612443dcbec3b4f9832c1b4849dd69f3156dc19a15b3b06af89a6aff0ccee7d9df74d241d01980c5ca6d440c8dbec62b779b/bin/compile: line 34: /app/.awscli/awscli-bundle/install: No such file or directory
It appears to me that the build, app and tmp directories have a slightly different setup in the Heroku CI than in the slug itself which leads to the deletion of the freshly unzipped folder.
Where is the aws binary installed to? Having trouble getting the command to work.
I have the buildpack in index=1, with ruby and node as well. Have tried aws
and /app/.awscli/bin/aws
Building an app using this buildpack on Heroku-20 resulted in this output (the build itself didn't fail; for which I'll file a separate issue):
-----> Installing AWS CLI
Running cmd: /usr/bin/python -m venv /app/.awscli
Traceback (most recent call last):
File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 223, in <module>
main()
File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 199, in main
create_install_structure(working_dir, opts.install_dir)
File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 77, in create_install_structure
create_virtualenv(location=install_dir, working_dir=working_dir)
File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 84, in _create_virtualenv_internal
run('%s -m venv %s' % (sys.executable, location))
File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 56, in run
raise BadRCError("Bad rc (%s) for cmd '%s': %s" % (
__main__.BadRCError: Bad rc (1) for cmd '/usr/bin/python -m venv /app/.awscli': The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
Failing command: ['/app/.awscli/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']
This is because on Heroku-20, python
is now Python 3, not Python 2, and because under Python 3.7+ AWS CLI v1 tries to use venv
rather than the virtualenv
package:
# From the install script in https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
if GTE_PY37:
create_virtualenv = _create_virtualenv_internal
else:
create_virtualenv = _create_virtualenv_external
Looking at the latest AWS CLI docs, they now tell users to install AWS CLI v2 instead:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
Although that version only supports Python 3.7+, so would only work on Heroku-18+, plus has other breaking changes:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
It seems like our options are to:
python3-venv
package (which isn't a package we've previously made available)@dzuelke @dmathieu Thoughts? (Context: This blocks W-8287887).
This build pack is no longer compiling on Heroku stack-22.
Started happening within the past 1-2 hours.
Howdy!
For my application, I need to download an s3 bucket to a local folder whenever the dyno restarts (which resets my ephemeral file system). This currently takes ~ 4 min.
I'd like to set some configuration parameters to try to speed this up. How should I go about configuring this?
aws configure set default.s3.max_concurrent_requests 1000
aws configure set default.s3.max_queue_size 100000
Should I run these during the Heroku build phase? Will they be added to the slug? Or should I set them every time before I run aws s3 sync
?
Running commands on a one-off dyno with this buildback works nicely:
~ $ aws --version
aws-cli/1.16.208 Python/2.7.15rc1 Linux/4.4.0-1048-aws botocore/1.12.198
But if I try to run aws help
I get the following error:
~ $ aws help
Could not find executable named "groff"
Should groff/less be added as a dependency for this buildback or should this be fixed by aws?
Related aws-cli issue: aws/aws-cli#1957
First: Thanks for providing this buildpack; it's very useful!
We were successfully using this via heroku-community/awscli
(which resolves to here), but the recent changes on master
caused the following error:
/bin/sh: 1: aws: not found
The only other buildpacks we use are heroku/nodejs
and heroku/ruby
.
It looks like the AWS CLI binary is now placed in a different location by this buildpack, and that our fairly simple setup chokes on that.
Should we be doing something differently?
We have this as a workaround for now:
heroku-community/awscli
buildpack.https://github.com/heroku/heroku-buildpack-awscli.git#01f21a8ac4a3964b0a0475444e0716ec37109256
as the buildpack name.This seems to successfully overload the "You can optionally specify a tag or a branch ..." functionality described here, but is obviously not a long-term fix.
-----> AWS CLI app detected
-----> Installing AWS CLI
Running cmd: /app/.heroku/python/bin/python virtualenv.py --no-download --python /app/.heroku/python/bin/python /app/.awscli
Running cmd: /app/.awscli/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz
Traceback (most recent call last):
File "/tmp/build_f27eb55f2fee6100ee0ef83c1f744b20/.awscli/awscli-bundle/install", line 223, in <module>
main()
File "/tmp/build_f27eb55f2fee6100ee0ef83c1f744b20/.awscli/awscli-bundle/install", line 205, in main
pip_install_packages(opts.install_dir)
File "/tmp/build_f27eb55f2fee6100ee0ef83c1f744b20/.awscli/awscli-bundle/install", line 151, in pip_install_packages
_install_setup_deps(pip_script, '.')
File "/tmp/build_f27eb55f2fee6100ee0ef83c1f744b20/.awscli/awscli-bundle/install", line 168, in _install_setup_deps
pip_script, setup_package_dir, setuptools_scm_tarball))
File "/tmp/build_f27eb55f2fee6100ee0ef83c1f744b20/.awscli/awscli-bundle/install", line 66, in run
p.returncode, cmd, output))
__main__.BadRCError: Bad rc (1) for cmd '/app/.awscli/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz': Traceback (most recent call last):
File "/app/.awscli/bin/pip", line 5, in <module>
from pip._internal.main import main
ModuleNotFoundError: No module named 'pip._internal'
I'm using this build pack in the hopes that it will allow the package.json "preinstall" hook to fire our aws codeartifact login ..
command. This works in other CI environments using secrets/variables.
The build pack does successfully get installed, and it appears as though the login works: "Login expires in 12 hours at .." — however, when the install goes to fetch the package it fails with "401 Unauthorized".
Question 1 —> Can this build pack enable the functionality I am looking for? (e.g. run the aws command within a "preinstall" hook)
Question 2 —> If yes to the above, what am I missing? I've got the config vars set as documented in the readme.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.