jaymon / herd Goto Github PK

The moved versions should be considered the canonical version from 7-8-2020 onwards:

• utils.NormalizeDict was moved into datatypes
• utils.EnvironParser was moved to captain 4.0.0+
• utils.Extra was moved to captain 4.0.0+

When this is updated to use captain 4.0.0 and datatypes then you can remove the versions and tests that are in herd

https://github.com/aws/chalice - Python Serverless Microframework for AWS.

I'm shocked I hadn't seen this before

It would be cool to have a command that can create a new lambda function, so you could do something like:

$ herd function-write /path/to/filename.py

And it would create a skeleton lambda file, something like:

def handler(event, context):
    """This is the description of the lambda function that will be in amazon"""
    return {
        'statusCode': 200,
        'body': {},
        "isBase64Encoded": False,
        "headers": {}
    }

The problem

Turns out, Python modules with .so libraries installed via pip will probably not work when ran on Lambda. This makes sense but it is also annoying, there are even repos designed to handle this problem by swapping out the original version with a precompiled for lambda version. But relying on 3rd party repos that haven't been updated in years doesn't seem like a viable solution for me.

I can get into the manylinux wheels but I think this is only a partial solution that works with some packages on not others (more research would be needed), this is something Zappa does when figuring out dependencies.

Proposed Solution

I think the solution that might be the least complicated would be to spin up an EC2 box using the same Amazon Linux that Lambda uses, which is currently:

arn:aws:lambda:::awslayer:AmazonLinux1803

Then basically copy over the lambda function file and any local dependencies and then install the other dependencies using pip onto the box, then compile all the dependencies and pull the zip file back down. Then destroy the box. You could even locally cache the dependencies so you wouldn't have to always spin up a new EC2 box every time they deployed (if the dependencies or versions haven't changed).

This solution has a bunch of moving parts but I don't think any of them are that complex or error prone. Biggest issue will be running the remote commands on the EC2 box, I think something like fabric would be overkill, so I would need to look into other solutions.

Search

• aws lambda python so file
• python manylinux wheels - manylinux repo
• python aws lambda and .so files - led to a Docker solution: Python dependencies and AWS Lambda which also seems complicated

Links

• Zappa - Along the same lines I was
• AWS Lambda Python magic