aws / aws-lambda-builders Goto Github PK

Description:

sam build failed on Oracle java 12 environment (on mac).

Steps to reproduce the issue:

1. Setup an Oracle java 12 development environment on mac. (maven and sam-cli were installed by brew)
2. Execute sam build command in project directory.

Observed result:

$ sam build
2019-03-20 20:08:43 Building resource 'HelloWorldFunction'
Traceback (most recent call last):
File "/usr/local/bin/sam", line 11, in
load_entry_point('aws-sam-cli==0.13.0', 'console_scripts', 'sam')()
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 95, in cli
skip_pull_image, parameter_overrides) # pragma: no cover
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 133, in do_cli
artifacts = builder.build()
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 99, in build
lambda_function.runtime)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 190, in _build_function
runtime)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 210, in _build_function_in_process
executable_search_paths=config.executable_search_paths)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/builder.py", line 110, in build
return workflow.run()
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 46, in wrapper
valid_path = validator.validate(executable_path)
File "/usr/local/Cellar/aws-sam-cli/0.13.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/java_maven/maven_validator.py", line 31, in validate
if int(jvm_mv) > 8:
ValueError: invalid literal for int() with base 10: '12, vendor: Oracle Corporation'

Expected result:

$ sam build
2019-03-21 09:03:18 Building resource 'HelloWorldFunction'
2019-03-21 09:03:19 /usr/local/bin/mvn is using a JVM with major version 12 which is newer than 8 that is supported by AWS Lambda. The compiled function code may not run in AWS Lambda unless the project has been configured to be compatible with Java 8 using 'maven.compiler.target' in Maven.
2019-03-21 09:03:19 Running JavaMavenWorkflow:CopySource
2019-03-21 09:03:19 Running JavaMavenWorkflow:MavenBuild
2019-03-21 09:07:01 Running JavaMavenWorkflow:MavenCopyDependency
2019-03-21 09:08:15 Running JavaMavenWorkflow:MavenCopyArtifacts

Build Succeeded

Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml

Additional environment details

1. Mac 10.14.3 (18D109)
2. Oracle java version "12" 2019-03-19
   Java(TM) SE Runtime Environment (build 12+33)
   Java HotSpot(TM) 64-Bit Server VM (build 12+33, mixed mode, sharing)
3. Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
   Maven home: /usr/local/Cellar/maven/3.6.0/libexec
   Java version: 12, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-12.jdk/Contents/Home
   Default locale: en_US, platform encoding: UTF-8
   OS name: "mac os x", version: "10.14.3", arch: "x86_64", family: "mac"

Reason

According to the code logic at [0], a regular expression is used to match the version string, it try to get out the java version number from the string after the prefix Java version:\s+, the expression assumes the version number follows the prefix and fill the rest of the string line, so in the code a * is used in the group (\d.*) which is a greedy match operator. However in my case, from my output of the mvn -version command executed by the logic at [1], you can see beside version number part vendor part is listed in the same line as well, and finally the int() call failed and ValueError exception raised up.

Solution proposal

My idea is simple, use non-greedy match in the regular expression, to replace char * by ? at [0], it works for me, the build succeeded after the hotfix in my local.

I'd like to commit the fix by a tiny PR if the solution works for you.

thanks,
zhiyan

Code link
[0] https://github.com/awslabs/aws-lambda-builders/blob/v0.2.1/aws_lambda_builders/workflows/java_maven/maven_validator.py#L46
[1] https://github.com/awslabs/aws-lambda-builders/blob/v0.2.1/aws_lambda_builders/workflows/java_maven/maven_validator.py#L55

Description:
This method does not have any unit or functional tests. Functional tests are better for this because the real devil is in the details.

Description:
Every workflow has to make sure it has a compatible version of Python/Node/NPM/PIP etc in the shell before processing. This is necessary to make sure customers are building in a version of programming language that is compatible with AWS Lambda runtime (ex: py3.6 vs py2.7 are incompatible).

This is grunt work that could be abstracted into the BaseWorkflow similar to BaseWorkflow.SUPPORTED_MANIFESTS. This will also help us generate a standard error message on failure to satisfy prerequisites across all workflows.

Description:

I'm working on a Python project that indirectly uses a native library through ctypes, so I need to supply the library, libspatialindex.so, in the Lambda function's package. However, when I run sam build, the file does not get copied into the output. It looks like this is due to the EXCLUDED_FILES list for PythonPipWorkflow.

Steps to reproduce the issue:

1. Place a file ending in .so in the directory pointed to by CodeUri
2. Run sam build

Observed result:
The file should be copied into .aws-sam/build/FunctionName/

Expected result:
The file is missing

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
I am using SAM CLI version 0.16.0 on Debian Stretch

Description:
We can support several optimizations to improve the build process. This is a feature request to support optimization flags that can be passed to each build action. If an action cannot support a particular optimization, it should error out:

Optimization 1: Use prebuilt binaries

Benefit: Makes build faster

When building dependencies with native modules, use pre-built binaries that are available locally or download from internet. This saves time in recompiling the same binaries over and over again. It can also make the build more stable because sometimes compilation fails or times out.

Requested by build actions:

1. node-npm
2. python-pip

Optimization 2: Remove unnecessary dependencies

Benefit: Produces slim zips

Remove unnecessary dependency libraries, or files or folders from the output artifact. For example, remove tests, test data files, or dependencies that are already available in transitive closure.

Requested by build actions:

1. node-npm
2. python-pip

Description:

• [Works] Windows/Linux Py2.7, Py3.6 using the python_pip packager
• [Works] pip install numpy==1.15.4 manually on Windows Py3.7
• [Fails] Linux Py3.7 using python_pip packager
• [Fails] Windows Py3.7 using the python_pip packager

See full log here
Windows - https://ci.appveyor.com/project/sanathkr/aws-lambda-builders-3bxwl/build/job/qqwsv5iq9iwow801
Linux - https://travis-ci.org/awslabs/aws-lambda-builders/jobs/455144509

The failure happens in the following line. The packager is trying to download without a wheel and fails is my understanding. But there is a wheel already - https://pypi.org/project/numpy/#files

           with self._osutils.tempdir() as tempdir:
                wheels, packages_without_wheels = self._download_dependencies(
                    tempdir, requirements_filepath)
                self._install_wheels(tempdir, target_directory, wheels)
            if packages_without_wheels:
>               raise MissingDependencyError(packages_without_wheels)

Description:
Several workflows python_pip, node_npm share the same osutils class. Fix this by moving common functionality into a library that can be shared across workflows.

Steps to reproduce the issue:
1.
2.
3.

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:

Runtime Validation for python works by invoking the python executable in the path to check if the local python runtime version matches the required runtime.

This relies on how the path is setup to find the appropriate version of python.

ISSUE

• When trying to build python2.7 lambda applications from within a sam installation housed in a python3.6 environment, one sees the following error message.

(venv) srirammv@localrainbow:~/sam-app$sam build
2018-11-26 13:40:16 Building resource 'HelloWorldFunction'
Build Failed
Error: python executable found in your path does not match runtime.
 Expected version: python2.7, Found version: python3.6.
 Possibly related: https://github.com/awslabs/aws-lambda-builders/issues/30

WORKAROUND:

Fixing this involves following steps:

• Moving up python executable of version python2.7 on the path

export PATH=/usr/bin:$PATH
python --version
2.7.10

(venv) srirammv@localrainbow::~/sam-app$sam build
2018-11-26 13:58:56 Building resource 'HelloWorldFunction'
2018-11-26 13:58:56 Running PythonPipBuilder:ResolveDependencies
2018-11-26 13:58:57 Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

PERMANENT FIX

A more permanent fix is in the works: #35

line in question: https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/validate.py#L17

RELATED GITHUB ISSUE

• aws/aws-sam-cli#797

After upgrading to Pop OS(Ubuntu) 20.04, Python was upgraded to 3.8. Running sam build produces the following error:

Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pip/_internal/main.py", line 14, in main
    from pip._internal.utils.entrypoints import _wrapper
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/entrypoints.py", line 3, in <module>
    from pip._internal.cli.main import main
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 19, in <module>
    from distutils.util import strtobool
ModuleNotFoundError: No module named 'distutils.util'

I do have distutils installed under /usr/lib/python3.8. I noticed that the traceback above was produced by pip from python3/dist-packages, however I do have a later version of pip installed under my .local/lib/python3.8/site-packages. The two symptoms together make me think that when the builder runs pip as a subprocess, it's not picking up any 3.8 packages.

I have tried the sample project in a clean install 20.04 environment, and SAM builds fine there. So this is not necessarily a SAM issue, and must be something broken during the upgrade. I'm just hoping you can point me to how python/pip is executed by the builder.

Description:
We want the workflows to print logs that have a standardized format. It also allows us to dial the log-level up and down per workflow. This task is to provide a logger instance to each workflow instance that they can use.

It would be great to directly specify the path of the executable required for executing the workflow, instead of relying on looking up the the executable in precedence order in $PATH

Description:
Parent tracking issue for the python-pip build

@stealthycoin please edit the issue and add tasks breakdown, if you'd like.

Task Breakdown:

• Task 1
• Task 2

Description:
There is a TODO in the code that talks about trying to discover a virtualenv where the packager can run. Is this a real requirement? Can we just live with a Python and PIP executable available in the shell?

If this is a real requirement, what are some strategies to discover the virtualenv and fail if we can't find one?

Description:

If the import pip fails during python -c "import pip; print(pip.__version__)", we see an error SAM CLI: Error: PythonPipBuilder:ResolveDependencies - invalid literal for int() with base 10: '' in SAM CLI. This is caused by trying to cast the output ofprint(pip.version)` to an int. The works in most cases, but have seen it fail in python2.7 when the pip module is not found. Instead of giving this weird message back to customers, we should fail the workflow with a better message to customers on what is wrong (no pip found) and how to resolve (install pip into that python version).

Description:
Missing -mod=${VENDOR_PATH} support for go build. This prevent the use of external private repositories eventually pushed inside the repository

Steps to reproduce the issue:

1. Declare under go.mod a private repository
2. sam build

Observed result:

Build Failed
Error: GoModulesBuilder:Build - Builder Failed: go: ... unknown revision ...

Expected result:
Build without error like env GOOS=linux ARCH=amd64 go build -mod=${VENDOR_PATH} -o ${OUTPUT_PATH} ${TARGET}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
No additional environment details

Description:
Customers using Postgress with Lamnbda in Python run into the problem where this library does not load in AWS Lambda. This is a build time problem and not a runtime one. AWS Lambda does not include necessary libpq.so file required by Postgress in the runtime. So people have to compile & statically link this library.

See: https://github.com/jkehler/awslambda-psycopg2
More: https://stackoverflow.com/questions/44855531/no-module-named-psycopg2-psycopg-modulenotfounderror-in-aws-lambda

Steps to reproduce the issue:
1.
2.
3.

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:

When the artifacts_dir is enclosed within the source_dir, CopySourceAction will infinitely recurse and try to copy the artifacts_dir within itself.

To fix this problem, detect if the artifacts_dir is within the source_dir. If so, pass an excludes list that skips artifacts_dir when copying source.

Steps to reproduce the issue:
1.
2.
3.

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:

Cannot install dependency (pyodbc)

Steps to reproduce the issue:

1. sam build --template template.yaml --manifest requirements.txt --use-container --debug

Observed result:

Final missing wheels: {pyodbc==4.0.30(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    requirements_path=self.manifest_path,
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 201, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pyodbc==4.0.30(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pyodbc==4.0.30(sdist)}
Builder workflow failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    requirements_path=self.manifest_path,
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 201, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pyodbc==4.0.30(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pyodbc==4.0.30(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/__main__.py", line 126, in main
    mode=params.get("mode", None),
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/builder.py", line 125, in build
    return workflow.run()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 76, in wrapper
    func(self, *args, **kwargs)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 276, in run
    raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {pyodbc==4.0.30(sdist)}
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {pyodbc==4.0.30(sdist)}"}}


Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 23588, 'exi
tReason': 'BuildError', 'exitCode': 1, 'requestId': '99fac21e-d468-4997-a9a5-703291aeb9b3', 'installationId': '072d408a-950f-4a21-bb6b-cea85e65da65', 'sessionId': '1b16b
ffd-c24d-4843-88e9-09dc0912dff0', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.43.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - {pyodbc==4.0.30(sdist)}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
OS: Windows 10 1909
Docker: 19.03.05
Runtime: python3.7
sam: 0.43.0

Description:

CAVEAT: Today is the first time I've looked into how aws-sam-cli works so if I'm missing something glaringly obvious, I apologize. I'm also fairly new to python, so if there's an easy fix just with a requirements.txt flag or something, I'm all ears. However, I've tried a few different things to force it to not use the incompatible wheel and nothing has worked so far.

I'm trying to build a function on the Python 3.6 runtime that uses BeatifulSoup 4.7.1. The function fails because the version of beautifulsoup that gets installed is only compatible with Python2. There are separate versions on pypi for py2 and py3. It's possible that part of this issue may be poor naming on the part of the package maintainer, but all the information needed is included, but possibly not in the way the builder is looking for it.

On pypi.org, there are two wheel files for recent versions:

beautifulsoup4-4.7.1-py2-none-any.whl
beautifulsoup4-4.7.1-py3-none-any.whl

The issue is that on this line: https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/packager.py#L375

Adding some additional debugging to the builder code reveals that it's downloading the py2 version first and then deciding that it's good enough. The code just exits out because it sees the abi of none without checking the implementation as well. Again, may be this is an incorrectly named package, but I don't know enough about the Python package naming system to know for sure. However, it seems like there could be an additional check before returning to validate that the implementation version matches.

Steps to reproduce the issue:

1. Create a Python3.6 lambda function and add beautifulsoup4 to the requirements.txt. Import it into your function.
2. Build your function with sam build
3. Run the function and see the syntax error - The error message is really bad, but if you trace it down you find that it comes from a line in the beautifulsoup4 __init__.py where there is a string in there designed to fail on python3 telling you that you have a version for python2 and need to rebuild it for python3.

Observed result:
Syntax error in module 'app': invalid syntax (__init__.py, line 52)

Expected result:
Function should actually run

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws-sam-cli 0.10.0

Description:

In certain Gradle projects, running sam build may include files in the bundle that aren't needed.
This occurs because the logic for grabbing the project's jar is actually pulling the contents of all defined archive artifacts, some of which may be for other purposes entirely. In most cases, this doesn't "break" anything, but just increases the size of the deployment bundle silently.

One example of a situation that would cause this is a Java project with the built-in "application" plugin applied, which then produces additional zip and tar distribution artifacts.

https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle#L62

One potential fix would be to change https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle#L27 from def artifactJars = t.project.configurations.archives.artifacts.files.files to def artifactJars = configurations.archives.artifacts.findAll { it.extension == 'jar' }*.file.

Steps to reproduce the issue:

1. Run sam init --runtime java8 --dependency-manager gradle --name gradlebuildtest
2. Run cd gradlebuildtest/
3. Run sam build && find .aws-sam/build/HelloWorldFunction -type f (Expected result)
4. Remove the previous result: rm -rf .aws-sam/
5. Edit HelloWorldFunction/build.gradle as follows
   • Add id 'application' to the plugins block
   • Add mainClassName = 'helloworld.App' (it's not actually executable yet, but that doesn't matter for demonstrating this behavior)
6. Run sam build && find .aws-sam/build/HelloWorldFunction -type f (Observed result)
7. Note that there is now an unnecessary HelloWorldFunction directory included.

Observed result:

mac-dcarr:gradlebuildtest dcarr$ sam build && find .aws-sam/build/HelloWorldFunction -type f
Building resource 'HelloWorldFunction'
Running JavaGradleWorkflow:GradleBuild
Running JavaGradleWorkflow:CopyArtifacts

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

.aws-sam/build/HelloWorldFunction/helloworld/GatewayResponse.class
.aws-sam/build/HelloWorldFunction/helloworld/App.class
.aws-sam/build/HelloWorldFunction/HelloWorldFunction/bin/HelloWorldFunction
.aws-sam/build/HelloWorldFunction/HelloWorldFunction/bin/HelloWorldFunction.bat
.aws-sam/build/HelloWorldFunction/HelloWorldFunction/lib/aws-lambda-java-core-1.2.0.jar
.aws-sam/build/HelloWorldFunction/HelloWorldFunction/lib/HelloWorldFunction.jar
.aws-sam/build/HelloWorldFunction/META-INF/MANIFEST.MF
.aws-sam/build/HelloWorldFunction/lib/aws-lambda-java-core-1.2.0.jar
mac-dcarr:gradlebuildtest dcarr$

Expected result:

mac-dcarr:gradlebuildtest dcarr$ sam build && find .aws-sam/build/HelloWorldFunction -type f
Building resource 'HelloWorldFunction'
Running JavaGradleWorkflow:GradleBuild
Running JavaGradleWorkflow:CopyArtifacts

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

.aws-sam/build/HelloWorldFunction/helloworld/GatewayResponse.class
.aws-sam/build/HelloWorldFunction/helloworld/App.class
.aws-sam/build/HelloWorldFunction/META-INF/MANIFEST.MF
.aws-sam/build/HelloWorldFunction/lib/aws-lambda-java-core-1.2.0.jar
mac-dcarr:gradlebuildtest dcarr$

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Scripts were run on Mac OS X 10.14.6; SAM CLI, version 0.23.0.

Unable to switch from AWS Layer to specifying locally built editable pip3 packages when executing function:

I'm both learning python and attempting to build multiple lambdas that will both store and process Spotify listening history. I originally wrote one lambda with all dependencies packaged in the same directory and imported using relative paths.

Once creating similar functions which use the same dependencies, I've split the dependencies from the function and built them using pip3 and setup.py. When running pip3 list I can see the packages as expected. To execute the function locally using sam build && sam local invoke SpotifyPlayHistoryListener --env-vars env.json --event events/event.json and verify success, I'm specifying a layer I've built - this works!

To avoid having to rebuild the layer and modifying my function, I'd like to be able to revert to using a requirements.txt file and removing the specified layer.

Steps to reproduce the issue:

1. Build local editable package - pip3 install -e .
2. Specify in requirements.txt
3. sam build && sam local invoke SpotifyPlayHistoryListener --env-vars env.json --event events/event.json

Observed result:

Building resource 'SpotifyPlayHistoryListener'
Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: spotify-client

Expected result:

Build succeeds and function executes successfully.

Additional environment details:

I'm able to execute the function locally using sam build && sam local invoke SpotifyPlayHistoryListener --env-vars env.json --event events/event.json when specfying dependencies using a Layer

I'm open to being told I'm completely off the mark here. Though I feel like I'm not a million miles away.

Description:

Many common names for python virtual environments (such as venv and ENV) are excluded, but env is not in the list. Because of that sam build fails when an environment called env is present in the directory.

File location: https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/workflow.py

Description:

When sam build fails on a ruby project, the error output from bundler just gets lost. Even with --debug, it doesn't print out.

In the same vein (debugging ruby build issues), when --debug is passed to sam, this should IMO automatically pass the --verbose flag to bundler.

Steps to reproduce the issue:

1. Create a SAM project with the ruby runtime
2. Add a bogus gem to Gemfile
3. Run sam build

Observed result:

2019-02-05 18:03:16 'ruby' runtime has not been validated!
2019-02-05 18:03:16 Running RubyBundlerBuilder:CopySource
2019-02-05 18:03:16 Running RubyBundlerBuilder:RubyBundle
Build Failed
Error: RubyBundlerBuilder:RubyBundle - Bundler Failed:

Expected result:

Bundler error about unknown gem should be displayed.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

aws-lambda-builders 0.0.5
aws-sam-cli 0.10.0
aws-sam-translator 1.9.0

Getting an error when running sam build in an existing project of mine,
where the sam template has a CodeUri that points to a JAR file in a reactor build submodule.

Steps to reproduce the issue:

1. existing java maven project using reactor builds with
   1.1 a shaded jar generated in a maven submodule's target directory
   (<project-top-level>/lambdas/mainlambda/target/lambda.jar in my case)
   1.2 a template.yaml in the <project-top-level> directory using
   CodeUri: lambdas/mainlambda/target/lambda.jar
2. cd <project-top-level>
3. sam build --debug

Observed result:

2019-04-28 19:31:55 Collected default values for parameters: {}
2019-04-28 19:31:55 12 resources found in the template
2019-04-28 19:31:55 Found Serverless function with name='LatestFunction' and CodeUri='lambdas/mainlambda/target/lambda.jar'
2019-04-28 19:31:55 Found Serverless function with name='LiveFunction' and CodeUri='lambdas/mainlambda/target/lambda.jar'
2019-04-28 19:31:55 Found Serverless function with name='TestRunnerFunction' and CodeUri='lambdas/testlambda/target/lambda.jar'
2019-04-28 19:31:55 Building resource 'LatestFunction'
2019-04-28 19:31:55 Looking for a supported build workflow in following directories: ['<scrubbed>/GitRepos/someproject/lambdas/mainlambda/target/lambda.jar', '<scrubbed>/GitRepos/someproject']
2019-04-28 19:31:55 Loading workflow module 'aws_lambda_builders.workflows'
2019-04-28 19:31:55 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-04-28 19:31:55 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-04-28 19:31:55 Found workflow 'JavaMavenWorkflow' to support capabilities 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-04-28 19:31:55 Running workflow 'JavaMavenWorkflow'
2019-04-28 19:31:55 Running JavaMavenWorkflow:CopySource
2019-04-28 19:31:55 JavaMavenWorkflow:CopySource raised unhandled exception
Traceback (most recent call last):
  File "<scrubbed>/.virtualenvs/aws/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 248, in run
    action.execute()
  File "<scrubbed>/.virtualenvs/aws/lib/python3.6/site-packages/aws_lambda_builders/actions.py", line 101, in execute
    copytree(self.source_dir, self.dest_dir, ignore=shutil.ignore_patterns(*self.excludes))
  File "<scrubbed>/.virtualenvs/aws/lib/python3.6/site-packages/aws_lambda_builders/utils.py", line 43, in copytree
    names = os.listdir(source)
NotADirectoryError: [Errno 20] Not a directory: '<scrubbed>/GitRepos/someproject/lambdas/mainlambda/target/lambda.jar'

Expected result:

I would expect it to run the equivalent of

(in <project-top-level> folder)
mvn package -Dmaven.test.skip=true -pl lambdas/mainlambda -am

Which is what i run when I want to just build lambda.jar

• -pl lambdas/mainlambda because that is the module folder of the target folder where lambda.jar is located.
• -am because otherwise reactor builds will fail
• -Dmaven.test.skip=true because when run from the aws-toolkit, you don't want to rerun the whole test suite (?)

This sam build failure is preventing me from creating launch configurations using aws-toolkit for Intellij.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

I don't believe my environment makes any difference in this issue, but I'll answer any request for more information if that is deemed necessary

Description:

Local execution manually copies dependencies and classes. This works for most of the cases - when we test just the code itself. But when we try to optimise dependencies and tune maven-shade-plugin
configuration, this local execution becomes useless. We cannot test it because SAM ignores the final artifact.

To address this issue I implemented my own Makefile:

.PHONY: build

name = $(shell mvn help:evaluate -Dexpression=project.build.finalName -q -DforceStdout)

build:
	mvn ${settings} clean install
	unzip -q target/${name}.jar -d target/sam-local

run-local:
	sam local start-lambda -v target/sam-local/com -t template.yaml

So next actions are performed:

1. Identify finalName of jar file (just to support possible customisations).
2. Rebuild the jar file and extract to some folder.
3. Start lambda from that folder.

Maybe it's worth using the same approach in SAM itself?

Pros:

• always test what's inside the fat jar, what will be deployed in AWS
• no additional dependencies (I saw that you already use zipfile, at least in tests)

Cons:

• minor delay to execute maven (Java users are patient anyway)
• modification of stable code, potential stability impact (just a risk)

Happy to contribute.

P.S.: Note the trailing com in the value of -v: that was the only way to make it work. Looks funny, but did not find a better way.

Description:
It is helpful for applications to sometimes structure themselves as multiple, separate packages. sam build currently can't require those local packages because the pip subprocess is run outside of the requirements.txt's directory.

Steps to reproduce the issue:

1. Init a sam project. Create a sub-package mypackage with its own setup.py.
2. Add ./mypackages to your requriements.txt
3. Run sam build

Observed result:
Sam cannot find the local package ./mypackage

Note: this does work currently when sam build --use-container

Expected result:
Sam would find, and install the local package ./mypackage, regardless of building in a container or not.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:
CopySourceAction has an option to pass a list of files to ignore when copying source. It should come built-in with a safe list of commonly ignored files per runtime.

Python:

ignore_files = ('*.pyc', '.git', '__pycache__', "*.egg-info", "*.egg", "pip-log.txt")

Description:

sam build does not work with pip 19.3.

Building resource 'NotifyLowCreditsFunction'
Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'module' object is not callable

Steps to reproduce the issue:

1. activate virtual environment with python 3.7
2. make sure pip is the latest version pip install -U pip
3. run sam build

Observed result:

Building resource 'MyLambdaFunction'
Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'module' object is not callable

Expected result:

Building resource 'MyLambdaFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

First noticed the issue during CI/CD builds.
These build run in a docker container using the python:3.7 image.
Confirmed that the same error exists when using the lambci/lambda:build-python3.7 docker image.
Confirmed that the same error exists with python 3.7.4 on macos.

Downgrading pip with pip install 'pip==19.2.3' makes the build work again.

Description:

Cross posting issue here for awareness that something might be wrong with 0.9.0 and aws lambda us-west-2.

lambci/docker-lambda#272 has all the details but I will capture some of them here as well.

Steps to reproduce the issue:

I will work on creating a minimal working example and update this ticket in a few hours.

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:

When I run sam build on my Java project with Gradle 3.4.1, it always fails.
This appears to be due to https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle#L18 setting the project buildDir property to a Path, when it isn't prepared for it. Adding .toFile() to this line appears to fix the problem.

It appears that Gradle 3.5+ support resolving Path values.

Steps to reproduce the issue:

1. sam init --runtime java8 --dependency-manager gradle --name gradlebuildtest
2. cd gradlebuildtest/
3. (cd HelloWorldFunction && sdk use gradle 3.4.1 && gradle wrapper) && sam build

Observed result:

Using gradle version 3.4.1 in this shell.
:wrapper

BUILD SUCCESSFUL

Total time: 1.31 secs
Building resource 'HelloWorldFunction'
Running JavaGradleWorkflow:GradleBuild

Build Failed
Error: JavaGradleWorkflow:GradleBuild - Gradle Failed: FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'HelloWorldFunction'.
> Cannot convert the provided notation to a File or URI: /var/folders/p1/w3tg9xz54lj_4xgwj60tffb126v92f/T/tmptseldu67/4abbb2503507efca0fbeaf9d14459fc8cdd6af90/build.
  The following types/formats are supported:
    - A String or CharSequence path, for example 'src/main/java' or '/usr/include'.
    - A String or CharSequence URI, for example 'file:/usr/include'.
    - A File instance.
    - A URI or URL instance.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
mac-dcarr:gradlebuildtest dcarr$

Expected result:

mac-dcarr:gradlebuildtest dcarr$ (cd HelloWorldFunction && sdk use gradle 5.6.3 && gradle wrapper) && sam build

Using gradle version 5.6.3 in this shell.

BUILD SUCCESSFUL in 727ms
1 actionable task: 1 executed
Building resource 'HelloWorldFunction'
Running JavaGradleWorkflow:GradleBuild
Running JavaGradleWorkflow:CopyArtifacts

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

mac-dcarr:gradlebuildtest dcarr$

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
Scripts were run on Mac OS X 10.14.6, and use https://sdkman.io/ for Gradle version management.

Description:
Based on experience building claudia pack, the team learnt that people kept wanting to add certain tweaks (eg skip the .npmrc file, skip security audits, don’t create package.lock etc). NPM keeps adding these steps into their builds. So they introduced the ability to pass arbitrary flags to NPM.

Pros: Easy to support these one off cases
Cons: This breaks one of the tenets - "make this builder super easy to reason about". Arbitrary flags make the builder hard to reason about because the output one person gets on his machine will be different from output another person get, because of the flags they have configured.

Ideally, we should implement 80% of these requested features in the build action. But I am opening this feature request to collect use cases for passing arbitrary options to build actions.

Request by build actions:

1. node-npm

Description:
There are cases where there is no PKG-INFO file in the package's egg-info directory. Instead it exists in the package's root folder. I assume this is the case for those packages which doesn't have binary distribution. The problem rises with SAM CLI when I try to build my lambda app.

Steps to reproduce the issue:

1. Download the following zip: reprosamissue.zip - it contains a sample project
2. Unzip it in a folder.
3. Create a virtual env.
4. Activate virtual env in a terminal
5. Navigate to the project folder
6. cd hello-world
7. pip install -r requirements.txt
8. cd ..
9. sam build -b .\dist

Observed result:

PS C:\Users\pushrbx\PycharmProjects\reprosamissue> sam build --debug -b .\dist
2019-05-16 15:26:05 Using SAM Template at C:\Users\pushrbx\PycharmProjects\reprosamissue\template.yaml
2019-05-16 15:26:05 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-05-16 15:26:05 Changing event name from before-call.apigateway to before-call.api-gateway
2019-05-16 15:26:05 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-05-16 15:26:05 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-05-16 15:26:05 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-05-16 15:26:05 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2019-05-16 15:26:05 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-05-16 15:26:05 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-05-16 15:26:05 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-05-16 15:26:05 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-05-16 15:26:05 Changing event name from before-call.apigateway to before-call.api-gateway
2019-05-16 15:26:05 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-05-16 15:26:05 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-05-16 15:26:05 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-05-16 15:26:05 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2019-05-16 15:26:05 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-05-16 15:26:05 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-05-16 15:26:05 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-05-16 15:26:05 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-05-16 15:26:05 'build' command is called
2019-05-16 15:26:05 No Parameters detected in the template
2019-05-16 15:26:05 2 resources found in the template
2019-05-16 15:26:05 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-05-16 15:26:05 Building resource 'HelloWorldFunction'
2019-05-16 15:26:05 Loading workflow module 'aws_lambda_builders.workflows'
2019-05-16 15:26:05 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-05-16 15:26:05 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-05-16 15:26:05 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-05-16 15:26:05 Running workflow 'PythonPipBuilder'
2019-05-16 15:26:05 Running PythonPipBuilder:ResolveDependencies
2019-05-16 15:26:05 calling pip download -r C:\Users\pushrbx\PycharmProjects\reprosamissue\hello_world\requirements.txt --dest C:\Users\TAMAS~1.SZE\AppData\Local\Temp\tmp2hcw4uih
2019-05-16 15:26:14 PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run
    action.execute()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute
    self.scratch_dir
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages
    scratch_directory, requirements_filepath)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies
    requirements_filename, directory)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 332, in _download_all_dependencies
    in self._osutils.get_directory_contents(directory)}
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 331, in <setcomp>
    deps = {Package(directory, filename) for filename
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 437, in __init__
    self._name, self._version = self._calculate_name_and_version()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 480, in _calculate_name_and_version
    sdist_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 541, in get_package_name_and_version
    pkg_info_filepath = self._generate_egg_info(package_dir)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 524, in _generate_egg_info
    egg_info_dir, info_contents[0], 'PKG-INFO')
IndexError: list index out of range

Build Failed
Error: PythonPipBuilder:ResolveDependencies - list index out of range

Expected result:

C:\Users\pushrbx\PycharmProjects\reprosamissue> sam build -b .\dist
2019-05-16 15:17:06 Building resource 'HelloWorldFunction'
2019-05-16 15:17:06 Running PythonPipBuilder:ResolveDependencies
2019-05-16 15:17:16 Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : dist
Built Template   : dist\template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t dist\template.yaml
[*] Package: sam package --s3-bucket <yourbucket> --template-file dist\template.yaml

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Windows 10
Attempted in a powershell console.
Python version: 3.6 (for the virtualenv)

Possible fix
The following change completely cured the problem:
pushrbx@5449ac3

I'm not clear on the environment requirements to run builds with SAM CLI but this change only worked after I've installed the wheel package in my virtualenv.

I would switch to this anytime (and away from serverless.com bloat) :)

Description:

In my company, we have to use license-maven-plugin-git to check for a company header in all files (it's a company mandate).

Since java workflow is excluding .git (here) when copying it to scratch dir, sam build is failing, because it fails to check for files header.

[INFO] --- license-maven-plugin:3.0:check (default) @ my-lambdas ---
[INFO] Checking licenses...
[WARNING] failure occured while calling class com.mycila.maven.plugin.license.git.CopyrightRangeProvider
java.lang.RuntimeException: Could not compute the year of the last git commit for file mylambda.java
    at com.mycila.maven.plugin.license.git.CopyrightRang[WARNING] failure occured while calling class com.mycila.maven.plugin.license.git.CopyrightRangeProvider

I think we can have at least two approaches to fix:

1. Be able to override EXCLUDED_FILES value to allow .git on java workflow
2. Be able to append custom maven parameters, then I can do -Dlicense.skip=true and use another build command to validate license header.

I do prefer option 1, since I may have only one build command.

Steps to reproduce the issue:

1. Configure a java project with license-maven-plugin-git
2. Try to build it using sam build

Observed result:
sam build is failing since license-maven-plugin-git is unable to get git year.

Expected result:
Be able to build a project using license-maven-plugin-gitby using onlysam build`

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

• java8
• maven 3.6.2

Description:
Need to write an integration test that uses the python_pip packager packager to build Python code.

Hello,

When running sam build npm config optional=false seems to be just ignored whether it is passed through env variables, .npmrc or whatever way to do it. npm config ls -l shows that optional=false is set correctly.

aws --version : aws-cli/1.16.260 Python/3.7.5 Darwin/17.7.0 botocore/1.12.250
node -v : v10.17.0
npm -v : 6.11.3

Description:

I'm proposing to add a new nodejs webpack builder that would be triggered by the presence of a webpack.config.js file.

There are two uses I can see:

1. As a Node developer I want to use TypeScript or Babel to compile my code from a more recent version of Javascript to a version supported by AWS Lambda.
2. As a Node developer I want to use tree shaking to reduce the size of my code.

The builder could be based on the existing nodejs npm builder. It would build the code using a command like npx webpack-cli app.ts -o ../.aws-ts/build/HelloWorldFunction/app.js --config ./webpack.config.js instead of using npm pack. The package.json would probably need to be copied independently.

The Serverless Framework uses the same approach (a serverless-webpack plugin) for TypeScript support through the webpack ts-loader.

Description:
When building aws-xray-sdk, it fails with Error: PythonPipBuilder:ResolveDependencies - {future==0.17.1(sdist), wrapt==1.10.11(sdist)}.

Interestingly enough this works fine on Python 3.6.4.

Steps to reproduce the issue:

1. Add aws-xray-sdk~=2.2.0 to requirements.txt
2. Switch to Python 3.6.2
3. sam build or sam build --use-container

Observed result:
Build should succeed

Expected result:
Build failed with message Error: PythonPipBuilder:ResolveDependencies - {future==0.17.1(sdist), wrapt==1.10.11(sdist)}.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
Python version: 3.6.2
OS: Mac
Also fails on --use-container that builds inside a Docker container simulating Lambda like environment.

Description:
Tracking issue to create the scaffolding necessary to implement core functionality of builders library. Design document is here: https://github.com/awslabs/aws-lambda-builders/blob/master/DESIGN.md#builders-library

Tasks Breakdown:

• Task 1
• ...

Description

When using a Docker image which has the dotnet tool "Amazon.Lambda.Tools" already installed, the command sam build will fail when trying to update the tool. It's something about hardlinks which I'm not knowledgeable enough about Docker to speculate on.

Steps to reproduce

As documented in the example repository I created.

Observed result

2019-08-20 20:28:18 Using SAM Template at /var/task/template.yml
2019-08-20 20:28:18 Changing event name from creating-client-class.iot-data to creating
-client-class.iot-data-plane
2019-08-20 20:28:18 Changing event name from before-call.apigateway to before-call.api-
gateway
2019-08-20 20:28:18 Changing event name from request-created.machinelearning.Predict to
 request-created.machine-learning.Predict
2019-08-20 20:28:18 Changing event name from before-parameter-build.autoscaling.CreateL
aunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-20 20:28:18 Changing event name from before-parameter-build.route53 to before-p
arameter-build.route-53
2019-08-20 20:28:18 Changing event name from request-created.cloudsearchdomain.Search t
o request-created.cloudsearch-domain.Search
2019-08-20 20:28:18 Changing event name from docs.*.autoscaling.CreateLaunchConfigurati
on.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-20 20:28:18 Changing event name from before-parameter-build.cloudsearchdomain.S
earch to before-parameter-build.cloudsearch-domain.Search
2019-08-20 20:28:18 Changing event name from docs.*.cloudsearchdomain.Search.complete-s
ection to docs.*.cloudsearch-domain.Search.complete-section
2019-08-20 20:28:18 Changing event name from before-parameter-build.logs.CreateExportTa
sk to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-20 20:28:18 Changing event name from docs.*.logs.CreateExportTask.complete-sect
ion to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-20 20:28:18 Changing event name from creating-client-class.iot-data to creating
-client-class.iot-data-plane
2019-08-20 20:28:18 Changing event name from before-call.apigateway to before-call.api-
gateway
2019-08-20 20:28:18 Changing event name from request-created.machinelearning.Predict to
 request-created.machine-learning.Predict
2019-08-20 20:28:18 Changing event name from before-parameter-build.autoscaling.CreateL
aunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-20 20:28:18 Changing event name from before-parameter-build.route53 to before-p
arameter-build.route-53
2019-08-20 20:28:18 Changing event name from request-created.cloudsearchdomain.Search t
o request-created.cloudsearch-domain.Search
2019-08-20 20:28:18 Changing event name from docs.*.autoscaling.CreateLaunchConfigurati
on.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-20 20:28:18 Changing event name from before-parameter-build.cloudsearchdomain.S
earch to before-parameter-build.cloudsearch-domain.Search
2019-08-20 20:28:18 Changing event name from docs.*.cloudsearchdomain.Search.complete-s
ection to docs.*.cloudsearch-domain.Search.complete-section
2019-08-20 20:28:18 Changing event name from before-parameter-build.logs.CreateExportTa
sk to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-20 20:28:18 Changing event name from docs.*.logs.CreateExportTask.complete-sect
ion to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-20 20:28:18 'build' command is called
2019-08-20 20:28:18 No Parameters detected in the template
2019-08-20 20:28:18 1 resources found in the template
2019-08-20 20:28:18 Found Serverless function with name='Example' and CodeUri='./src/Ex
ample'
2019-08-20 20:28:18 Building resource 'Example'
2019-08-20 20:28:18 Loading workflow module 'aws_lambda_builders.workflows'
2019-08-20 20:28:18 Registering workflow 'PythonPipBuilder' with capability 'Capability
(language='python', dependency_manager='pip', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'NodejsNpmBuilder' with capability 'Capability
(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'RubyBundlerBuilder' with capability 'Capabili
ty(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'GoDepBuilder' with capability 'Capability(lan
guage='go', dependency_manager='dep', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'GoModulesBuilder' with capability 'Capability
(language='go', dependency_manager='modules', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'JavaGradleWorkflow' with capability 'Capabili
ty(language='java', dependency_manager='gradle', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'JavaMavenWorkflow' with capability 'Capabilit
y(language='java', dependency_manager='maven', application_framework=None)'
2019-08-20 20:28:18 Registering workflow 'DotnetCliPackageBuilder' with capability 'Cap
ability(language='dotnet', dependency_manager='cli-package', application_framework=None
)'
2019-08-20 20:28:18 Found workflow 'DotnetCliPackageBuilder' to support capabilities 'C
apability(language='dotnet', dependency_manager='cli-package', application_framework=No
ne)'
2019-08-20 20:28:18 Running workflow 'DotnetCliPackageBuilder'
2019-08-20 20:28:18 Running DotnetCliPackageBuilder:GlobalToolInstall
2019-08-20 20:28:18 Installing Amazon.Lambda.Tools Global Tool
2019-08-20 20:28:18 executing dotnet: ['dotnet', 'tool', 'install', '-g', 'Amazon.Lambd
a.Tools']
2019-08-20 20:28:18
2019-08-20 20:28:18 Error installing probably due to already installed. Attempt to upda
te to latest version.
2019-08-20 20:28:18 executing dotnet: ['dotnet', 'tool', 'update', '-g', 'Amazon.Lambda
.Tools']
2019-08-20 20:28:19
2019-08-20 20:28:19 DotnetCliPackageBuilder:GlobalToolInstall failed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/workflow.py", line 2
48, in run
    action.execute()
  File "/usr/local/lib/python2.7/site-packages/aws_lambda_builders/workflows/dotnet_cli
package/actions.py", line 42, in execute
    raise ActionFailedError("Error configuring the Amazon.Lambda.Tools .NET Core Global
 Tool: " + str(ex))
ActionFailedError: Error configuring the Amazon.Lambda.Tools .NET Core Global Tool: Dot
net CLI Failed: Tool 'amazon.lambda.tools' failed to update due to the following:
Failed to uninstall tool package 'amazon.lambda.tools': Invalid cross-device link

Build Failed
Error: DotnetCliPackageBuilder:GlobalToolInstall - Error configuring the Amazon.Lambda.
Tools .NET Core Global Tool: Dotnet CLI Failed: Tool 'amazon.lambda.tools' failed to up
date due to the following:
Failed to uninstall tool package 'amazon.lambda.tools': Invalid cross-device link

Expected result

A successful build. Whether that's via a successful update of the tool or via not trying to update the tool (or some other thing?), I have no opinion.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: In the example case, Amazon Linux.
2. sam --version: SAM CLI, version 0.19.0

Add --debug flag to command you are running

Description:

NPM v5.7.0 introduced the npm ci command, intended for use in automated environments. The main benefit, in my eyes, over npm install is that it installs the exact version of packages indicated in the package-lock.json file. If you know things worked on your local machine with those dependencies then with the exact same versions things should continue to work elsewhere. npm ci also increases the speed of the install due to skipping user oriented features of npm install.

My recommendation is that when npm v5.7.0 or above is detected and a package-lock.json file is detected then the npm ci command is used instead of npm install

Description:

Failures to the Go Modules were first seen through Appveyor failures. Builds using v1.11.5 are the last know pass state. It could be a result of how we are checking go versions within the workflow

Steps to reproduce the issue:

1. install go v1.12
2. Try to use go mods builder

Observed result:
See appveyor links above

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Description:

This is necessary to maintain the promise of the builder library that we try to control, as much as possible, where the workflows touches the filesystem

Description:
I can not build Lambda via AWS toolkit (SAM) in the InteliJ. Build fails with an error ValueError: No Maven executable found!, but Maven is already install and the terminal for which mvn returns correct path .../.sdkman/candidates/maven/current/bin/mvn. I'm not a Python developer, so I really don't know, how can I fix this bug. Below attached the log after run the lambda.

Steps to reproduce the issue:

1. Choose YAML template for lambda.
2. Run the Lambda from InteliJ.
3. See result in the "Build Output".

Observed result:
Traceback (most recent call last):
File "/home/linuxbrew/.linuxbrew/bin/sam", line 11, in <module> load_entry_point('aws-sam-cli==0.40.0', 'console_scripts', 'sam')()
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func return ctx.invoke(f, obj, *args, **kwargs)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 96, in wrapped raise exception # pylint: disable=raising-bad-type
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped return_value = func(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 127, in cli mode,
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 192, in do_cli artifacts = builder.build()
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 104, in build lambda_function.runtime)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 195, in _build_function runtime)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 216, in _build_function_in_process mode=self._mode)
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/builder.py", line 115, in build return workflow.run()
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 48, in wrapper exec_paths = binary_path.resolver.exec_paths if not binary_path.path_provided else binary_path.binary_path
File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/java_maven/maven_resolver.py", line 21, in exec_paths raise ValueError("No Maven executable found!")
ValueError: No Maven executable found!

Expected result:
Lambda runs.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
Linux 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description:
To implement Javascript builder workflows, we need to an action that can invoke node build.js on the build.js script that we will write. This is a common pattern for other programming languages too (java+maven, golang dotnet etc). In this task, we will create a new built-in Action that can help run builders implemented in other languages by spawning processes.

Description:
It would be great to have support in the SAM CLI to for go builds with go modules. That would allow SAM projects using go with go modules to have a standardized build process.

Can support be added for using pipenv and Pipfile for resolving dependancies and building instead of the common pip and requirements.txt method.

Description:

Guard against unexpected values when checking the pip version.

Code Location: https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/compat.py#L15

Expected Behavior

More user friendly output message

Motivation

When running sam build commands with the SAM CLI, this is a confusing error to come up against.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Mac and Linux, using pyenv to set up multiple versions (2.7, 3.6, 3.7, 3.8) of python for development and testing.

Description:
SAM CLI's build command has the option to build inside a Docker container - sam build --use-container. When trying to build cryptography library, it fails when building a dependency {pycparser==2.19(sdist)}

Description:
This was reported through aws/aws-toolkit-jetbrains#749

Observed result:

2019-01-15 18:09:52 PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 27, in execute
    self.scratch_dir
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 212, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pycrypto==2.6.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 164, in run
    action.execute()
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 30, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pycrypto==2.6.1(sdist)}
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {pycrypto==2.6.1(sdist)}

Expected result:
A better error message should be shown about what package failed to package, and provide steps to resolve