Missing PKG-INFO in case of python-pip workflow about aws-lambda-builders HOT 27 CLOSED

Same here... any update on the solution?

from aws-lambda-builders.

I'm also hitting this problem.

Specifically, it happens in a Python 3.6 SAM project, when running sam build with --use-container, on the dependency cryptography==3.4.6. When I don't use --use-container, the build works (on macOS 10.15.7).

I dug into it a little bit more and it looks like, when trying to _generate_egg_info in a containerized SAM build for the cryptography 3.4.6 package, the setup.py invocation here ( https://github.com/aws/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/packager.py#L498 ) is failing. Grabbing the stdout and stderr output from the command, it's failing because setuptools_rust is missing. Making sure that setuptools_rust was installed in the container solved the problem.

I'm assuming that I didn't hit the error on macOS because the 'manylinux' package is used instead, thereby skipping the need for Rust compilation.

Since the cryptography package is a big deal, does it make sense to install setuptools_rust in the container image so that trying to _generate_egg_info for this package doesn't fail? I'd submit a pull request but I can't find the Dockerfile for amazon/aws-sam-cli-build-image-python3.6.

Furthermore, it seems like it would be a good idea to check the returncode on the setup.py command and fail or show some debug output if it didn't succeed, since it probably means that we won't be able to locate the PKG-INFO.

from aws-lambda-builders.

Yes, it will.

from aws-lambda-builders.

Closing as this was fixed in Lambda Builders 1.7.0. Feel free to re-open if this is not the case.

from aws-lambda-builders.

same here with googleads package in my virtualenv

from aws-lambda-builders.

Same error here with Amazon's own sagemaker package.

from aws-lambda-builders.

Experiencing the same when using anything that requires cryptography.

from aws-lambda-builders.

same here with gql

from aws-lambda-builders.

Same issue with pendulum

Adding --use-container to my sam build command seems to work

from aws-lambda-builders.

Adding --use-container fixes this temporarily.
However will this issue ever be addressed?

from aws-lambda-builders.

I might try to create a PR when I have free time. 😄

from aws-lambda-builders.

Also Update:
Using Pipenv instead of virtual env for dependencies fixes this issue for me.

from aws-lambda-builders.

Will it be fixed any soon?

from aws-lambda-builders.

This affects dependencies that use PEP518 to specify a non-setuptools build system. E.g. a poetry-based package will not have a setup.py to execute and pull dependency information from: https://github.com/awslabs/aws-lambda-builders/blob/24f0aca6ed41ec2b08333655adbd1433f14cdc9e/aws_lambda_builders/workflows/python_pip/packager.py#L485-L486

from aws-lambda-builders.

Just hit this issue. Switching over to pipenv fixed. Thanks @Nop0x!

from aws-lambda-builders.

Summarizing this thread, some questions for us to understand the scope. If anyone who commented here had a different experience, please do correct me:

1. This problem seems to happen on Windows, and never when using --use-container for building.
2. Moving to pipenv over virtualenv solves the issue.

Are these statements correct for everyone? Especially the first one.

from aws-lambda-builders.

I have the same Error as @pushrbx but when I change to --use-container I receive the following error:

Observed result

"C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" build TelegramUpdateFunction --template C:\Users\user\PycharmProjects\TelegramBot\template.yaml --build-dir C:\Users\user\PycharmProjects\TelegramBot\.aws-sam\build --use-container
Starting Build inside a container
Building resource 'TelegramUpdateFunction'

Fetching lambci/lambda:build-python3.7 Docker container image......
Mounting C:\User\suser\PycharmProjects\TelegramBot\telegram_update as /tmp/samcli/source:ro,delegated inside runtime container

Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - Requirements file not found: /tmp/samcli/source/requirements.txt

As far as I can tell this whole problem occurs once I import the python-telegram-bot library. I am pretty new to python and not so familiar with this virtualenv, pip etc stuff so feel free to give advice 😄 I am coming from java and primary worked on webapps using spring / springboot

I also tried this although don`t really know what wheel. Didn't work, but maybe I did something wrong.

Possible fix
The following change completely cured the problem:
pushrbx/aws-lambda-builders@5449ac3

I'm not clear on the environment requirements to run builds with SAM CLI but this change only worked after I've installed the wheel package in my virtualenv.

I will try the approach to use pipenv, but I first have to get familiar with it.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

I use Pycharm with a virtualenv on Windows.
I have Docker for Windows installed and working.

Update

So now I switched to pipenv. When I run / build it with --use-container it works. Without container I am back at the Error: PythonPipBuilder:ResolveDependencies - list index out of range

from aws-lambda-builders.

Try chalice --debug deploy

For me (chalice 1.13.1, python 3.6.5, windows 10), the issue was in the requirements.txt: I had to downgrade pandas 1.0.0 to 0.24 then it worked!

from aws-lambda-builders.

I'm having this problem and in answer to @awood45 it happens when I use build -u. However, I get a different error.

PS C:\Users\...\project> sam build
Building resource 'LambdaExtractFunction'
Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - list index out of range
PS C:\Users\...\project> sam build -u
Starting Build inside a container
Building resource 'LambdaExtractFunction'

Fetching lambci/lambda:build-python3.8 Docker container image......
Mounting C:\Users\...\project\src as /tmp/samcli/source:ro,delegated inside runtime container

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Requirements file not found: /tmp/samcli/source/requirements.txt
Running PythonPipBuilder:ResolveDependencies

I am using a Python 3.8 virtualenv that was configured using pycharm's Python Interpreter setup.

NINJA EDIT: Just want to pop in and say that this is some kind of cache/runtime issue. I rebooted and started from a clean project (fresh git clone) and it worked fine.

from aws-lambda-builders.

Could someone point me towards how to use pipenv with sam build?

Correct me if this is wrong, but I was under the impression that only requirements.txt is supported with the python-pip workflow? aws/aws-sam-cli#817

from aws-lambda-builders.

We hit this problem too.

• Using Python 3.7 and --use-container we had this problem
• Using Python 3.8 and --use-container it looked like we had a successful build but actually hit another issue outlined here: https://forums.aws.amazon.com/thread.jspa?threadID=314652 (library in the image causes OpenCV to fail. Sigh...)
• Using Python 3.7 without --use-container we had a successful build and were able to run the Lambda.

All builds were on Linux.

requirements.txt looks like this

aws-xray-sdk>=2.4.2
opencv-python-headless==4.4.0.42

from aws-lambda-builders.

I've the same issue with sam build when I import aws-lambda-powertools on Windows 10 and Python 3.8. If I'm using --use-container the build is successful. But --use-container is much slower on my machine, therefore I wanted to find a solution.
I've found a solution for myself, and I want to share it here. I don't know if it solves the complete complexity of this issue but at the moment its working for me.

For the solution I've changed C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py

First I've added a debug log LOG.debug("Generate egg-info for: %s", package_dir) in line 500 to be informed which package raises the error. In my case the future package is the reason fo the error, because as @pushrbx mentioned in the initial post the PKG-INFO file is not part of the egg-info directory after the execution of setuptools and only exists in the package root directory. In fact the complete directory egg-info is empty after setuptools, which then raises the IndexError because info_contents is empty in line 500.

To solve the IndexError, I've edited line 500 to read the PKG-INFO file from the package root directory.

Existing:
pkg_info_path = self._osutils.joinpath(egg_info_dir, info_contents[0], "PKG-INFO")

Edited:

if info_contents:
   pkg_info_path = self._osutils.joinpath(egg_info_dir, info_contents[0], "PKG-INFO")
else:
   pkg_info_path = self._osutils.joinpath(package_dir, "PKG-INFO")

At the moment I've no answer why the egg-info directory is completely empty, so it can be that I only fix the symptom and not the root cause.

from aws-lambda-builders.

I am having the same problem here from Cloud9

2021-03-09 17:08:37,561 | calling pip download -r /home/ec2-user/environment/lambdaproxy/lambdaproxy/lambdaproxy/requirements.txt --dest /tmp/tmpzu7o5ygz
2021-03-09 17:08:40,415 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
File "aws_lambda_builders/workflow.py", line 269, in run
File "aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
File "aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 204, in build_site_packages
File "aws_lambda_builders/workflows/python_pip/packager.py", line 228, in _download_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 311, in _download_all_dependencies
File "aws_lambda_builders/workflows/python_pip/packager.py", line 311, in
File "aws_lambda_builders/workflows/python_pip/packager.py", line 417, in init
File "aws_lambda_builders/workflows/python_pip/packager.py", line 459, in _calculate_name_and_version
File "aws_lambda_builders/workflows/python_pip/packager.py", line 519, in get_package_name_and_version
File "aws_lambda_builders/workflows/python_pip/packager.py", line 502, in _generate_egg_info
IndexError: list index out of range

Build Failed

requirements.txt

aws-xray-sdk
boto3
botocore

from aws-lambda-builders.

I've was having this issue with sam build , I figure it out by:

• Removing any extra blank lines at the end of requirements.txt file
• Assign a version to every package listed on the requirements.txt file

If you are running sam build --use-container with private artifactories make sure you add the URL to the requirements.txt like

--extra-index-url https://artifactory-real-url.com/artifactory/api/pypi/pypi-repos/simple

As it it were any other package.

from aws-lambda-builders.

also experiencing this issue with the deepdiff package

from aws-lambda-builders.

Had same problem with my package dataclass-factory. Packaging it as bdist_wheel isntead of sdist fixed for me

from aws-lambda-builders.

Does #265 completely cure this?

from aws-lambda-builders.