Git Product home page Git Product logo

deprecated-lambda-cfn's Introduction

⚠️ DEPRECATED ⚠️

lambda-cfn

Build Status npm

Quickly create, deploy, and manage AWS Lambda functions via AWS CloudFormation.

Prerequisites

Node.js

Lambda-cfn (or "Lambda CloudFormation") is a Node.js project that runs on AWS Lambda. Node.js v10.x is the current default, Node.js v8.10 is also available. See AWS Lambda execution environment for more information about the Lambda Node.js runtime environment.

S3 buckets

Lambda-cfn uses cfn-config behind the scenes to create and manage CloudFormation stacks. Cfn-config requires two S3 buckets - a config bucket and a template bucket - to work.

Lambda-cfn will look for the following buckets by default if you've set an AWS_ACCOUNT_ID environment variable:

  • Config bucket: cfn-configs-$AWS_ACCOUNT_ID-region
  • Template bucket: cfn-config-templates-$AWS_ACCOUNT_ID-region

For example, if your AWS account ID is 123456789 and you're using the us-west-1 region then lambda-cfn will look for buckets named cfn-configs-123456789-us-west-1and cfn-config-templates-123456789-us-west-1.

If you don't want to use the default naming convention then you'll need to pass both bucket names as arguments for the create, update, and save commands. You can also set a CFN_CONFIG_BUCKET environment variable if you'd prefer.

Installation

First install lamdba-cfn globally so that you can use lambda-cfn utility commands like lambda-cfn init or lambda-cfn create:

npm install -g @mapbox/lambda-cfn

Creating new Lambda functions

After installing lambda-cfn, from within your project directory (e.g. myProject) run the following command:

lambda-cfn init <function name>

For example, running lambda-cfn init myFunction will create the following files and directories:

  • a package.json file in the currently working directory with lambda-cfn as a dependency
  • a myFunction directory, with two files called function.js and function.template.js

This looks like:

myProject/
|-- package.json
|-- myFunction/
  |-- function.js
  |-- function.template.js

You can have multiple function directories with the main project directory (myProject).

myProject/
|-- package.json
|-- myFunction/
  |-- function.js
  |-- function.template.js
|-- myOtherFunction/
  |-- function.js
  |-- function.template.js

function.js

Add the code for your Lambda function to this file. We have already taken care of setting up the handler function for you. Here's what this file looks like:

var lambdaCfn = require('@mapbox/lambda-cfn');

module.exports.fn = function(event, context, callback) {
  // write Lambda function code here

};

function.template.js

Configure your Lambda function in this file. Configuration consists of passing in a JavaScript object to the lambdaCfn.build() function. Lambda-cfn will take care of generating the CloudFormation template for you as part of the deployment process.

Here's what the function.template.js file looks like after running lambda-cfn init. The only required property for deployment is the name of the function - all other properties are optional.

const lambdaCfn = require('@mapbox/lambda-cfn');

module.exports = lambdaCfn.build({
  name: 'myFunction'
});

If you'd like to create a Lambda function that runs on the older Node.js 8.10 runtime with a memory size of 256 MB and a timeout of 120 seconds:

const lambdaCfn = require('@mapbox/lambda-cfn');

module.exports = lambdaCfn.build({
  name: 'myFunction',
  runtime: 'nodejs8.10',
  memorySize: '256',
  timeout: '120'
});

For a full list of available function properties to configure your Lambda function, see the function specification documentation.

Deploying Lambda functions to AWS

First we'll need to zip up the code for our Lambda function and then upload it to S3 before we can deploy it via CloudFormation. We also need to make sure that our project and its functions are within a git repository. Run git init to set this up.

Use of cfn-config

Lambda-cfn uses cfn-config for creating, retrieving, updating, and deleting CloudFormation stacks. It takes the same parameters and values as cfn-config, see the cfn-config CLI usage documentation.

Lambda-cfn will look for the following environment variables if you'd prefer to not pass in flags for each command:

  • CFN_CONFIG_BUCKET for --config-bucket (-c)
  • AWS_REGION for --region (-r)

To get a full list of lambda-cfn commands, run lambda-cfn --help.

Uploading your code to S3

By default lambda-cfn expects your Lambda code to be in the following location on S3:

s3://<your bucket>/<optional prefixes>/<gitsha>.zip

We created a Bash script called upload.sh within lambda-cfn that will do this for you. The script obtains the current gitsha, zips up the code, copies it to S3, then deletes the zip file.

Run this script from the parent directory (myProject or the one with the package.json file). You'll need to provide the path to your S3 bucket and any prefixes, without the trailing slash:

sh upload.sh <bucket>/<optional prefixes>

If I'm using bucket myBucket and the prefix myFunction:

sh upload.sh myBucket/myFunction

This will upload your code to s3://myBucket/myFunction/<gitsha>.zip.

If you'd like to manually upload your code, then zip your entire project directory (myProject or the parent directory containing the package.json file) and copy it to S3 to the location of your choice. You can provide the location of your code during the lambda-cfn create command in the next step.

Creating the CloudFormation stack

To create a new CloudFormation stack for your lambda function, run lambda-cfn create <environment name>. For example, if I want to create to a development environment stack in the default us-east-1 region:

lambda-cfn create dev

This will use the default config and template S3 buckets mentioned in the prerequisites. If you'd like to override these buckets and deploy to us-west-1:

lambda-cfn create dev --region us-west-1 --config-bucket <config bucket> --template-bucket <template bucket>

CloudFormation stacks created by lambda-cfn have the following naming convention:

<project directory name>-<function directory name>-<environment name>

If my project directory is named myProject and I'm creating a stack for the myFunction function, then my stack's CloudFormation name is myProject-myFunction-dev.

CloudFormation stack names must start with a letter and may only contain letters, numbers, or hyphens. Lambda-cfn will warn you if you try to use an invalid name.

Providing parameter values

After running lambda-cfn create, you'll be prompted to provide four values: CodeS3Bucket, CodeS3Prefix, GitSha, and ServiceAlarmEmail. The first two correspond to the S3 bucket and optional prefix to where you uploaded your code. Note: you must leave a trailing forward slash (/) at the end of your prefix. Lambda-cfn will also automatically detect the current gitsha of your project and use that for the filename for the zip file. You are not required to use the gitsha though - feel free to override this with the name of your zip file if you uploaded the file manually:

For example, if I used the upload.sh script to upload the file to s3://myBucket/myFunction/<gitsha>.zip, then I'd use the following information in the parameter prompt:

  • CodeS3Bucket = myBucket
  • CodeS3Prefix = myFunction/
  • GitSha = accept the default value (press enter)

For the service alarm email you can use any email address that you'd like under your control.

Getting information on a CloudFormation stack

Run the lambda-cfn info command. This is the same as the cfn-config info command. Make sure to specify the same region to which you deployed, either via a flag (--region) or via the AWS_REGION environment variable. By default lambda-cfn will look in us-east-1:

lambda-cfn info dev -r us-west-1

Updating a CloudFormation stack

lambda-cfn update dev -r us-west-1

Deleting a CloudFormation stack

lambda-cfn delete dev -r us-west-1

Saving CloudFormation stacks

Saving a CloudFormation stack allows you to reuse the parameter values for later, even if you later delete the stack.

lambda-cfn save dev

How do I contribute?

We're happy you want to contribute! Check out CONTRIBUTING.MD for more information.

Utilities

capitalizeFirst: Capitalize the first word of a string.

splitOnComma: Creates an array from a list of words.

Dispatch Integration

lambda-cfn now supports Mapbox's alert router Dispatch which provides integration service with Github, Slack, and PagerDuty on lambda-cfn version 3.0 or higher. By default, lambda-cfn will have an optional parameter for your dispatch stack SNS Topic ARN. If specified, lambda-cfn will grant the function permission to publish to that SNS Topic.

lib/message.js will route your message to Dispatch is DispatchSnsArn environment variable is set.

How to use it?

Import the utility functions like this:

const lambdaCfn = require('@mapbox/lambda-cfn');

const dispatchMessage = {
    type: 'high-priority',
    pagerDutyServiceId: 'TEST123',
    body: {
      pagerduty: {
        service: 'TEST123',
        title: 'LambdaCfn is awesome!',
        body: 'It routed this message through the power of dispatch!'
      }
    }
};

lambdaCfn.message(dispatchMessage, (err, res) => {
  if (err) console.error(err);
  console.log(res);
});

Questions?

Open new issue in this repo.

deprecated-lambda-cfn's People

Contributors

alulsh avatar elfakyn avatar ianshward avatar ingalls avatar k-mahoney avatar matiskay avatar npeternel avatar pauljlucas avatar rclark avatar tmcw avatar zmully avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

deprecated-lambda-cfn's Issues

Add `statements` to rule spec

@zmully I overlooked this yesterday - a rule definition can add additional IAM policy statements to the IAM role used by a crowsnest stack, but is not documented in the rule spec.

You can assign this to me if you'd rather I add it to the spec.

Cannot find rules when npm linked

When you're developing on lambda-cfn or related, and you npm link lambda-cfn into a patrol stack, rules paths cannot be resolved because the wrong application root is used.

@zmully I believe the start of the problem is here https://github.com/mapbox/lambda-cfn/blob/7e278324d7fe52973f4315b5c9fa65e91f5f0310/lib/lambda-cfn.js#L59 which uses app-root-path here https://github.com/mapbox/lambda-cfn/blob/7e278324d7fe52973f4315b5c9fa65e91f5f0310/lib/lambda-cfn.js#L5 to resolve the root of the application.

There are a few places in lambda-cfn.js that rely on app-root-path. There should be tests to ensure all these cases are handled properly, in addition to properly handling the case around an npm linked copy of lambda-cfn.

webhook API key Output invalid

When a webhook is set to apiKey: true, the key is made available via the template Outputs. Currently, when the Outputs section is added for the key, it is invalid CFN json.

API GW Deployments immutable and do not update

Quirk of API GW and CFN, since deployments are immutable, a new deployment has to be created every time the stack is updated for the latest methods to be deployed. This can be hacked around by randomizing the deployment name. #36

Add docs for utility functions

Lambda-cfn exposes two utility functions - splitOnComma and message - in https://github.com/mapbox/lambda-cfn/blob/14015e33f2ca0ec87bffa1866d5f61ab0f9d0ba6/index.js#L4-L5 that are useful when working with lambda-cfn based projects such as Patrol. For more context, you can see both of these functions being used in the function.js file for the cloudTrail rule in /patrol-rules-aws.

We should write brief docs on these functions in this repo, including why they might be useful and how to use them. Unless you're closely looking at the index.js file you wouldn't know about these functions, and it's not clear from the message() function's name what it does.

/cc @zmully @ianshward

Rule deletion

Per @zmully

The lambda-cfn-rules.js needs a --delete option to remove the ruleset if the main Crowsnest stack is deleted.

lambda-cfn upload command

We currently provide an upload.sh script to help people zip and upload code from their local machines to S3.

We could turn this into a more powerful lambda-cfn upload command to make this even easier for users. This command could also run automatically before running any lambda-cfn create or lambda-cfn update commands since it's common to forget to re-upload any changed code.

No urgency in working on this ticket - we can save this work for a later release. Leave any thoughts on this command and how it'd fit with other lambda-cfn commands in this ticket.

/cc @zmully @ianshward

test/package.json is a required file for all projects using lambda-cfn

I'm getting the following error when I run tests using a project with lambda-cfn as a devdependency:

module.js:327
    throw err;
    ^

Error: Cannot find module '/../test/package.json'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/../node_modules/lambda-cfn/lib/lambda-cfn.js:47:10)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)

Going through the stack trace, this is caused by the following code in lambda-cfn.js at line 45:

var pkgs;
if (process.env.NODE_ENV == 'test') {
    pkgs = require(path.join(root,'test/package.json'));
} else {
    pkgs = require(path.join(root,'package.json'));
}

It seems this code is needed for lambda-cfn tests to run. But because of this, it requires test/package.json to exist to all projects using this package - which shouldn't be the case.

Add helper to generate namespaced params

We need to provide a helper function to generate the namespaced parameter name so it can be used as a statement reference.

See mapbox/DEPRECATED-patrol-rules-aws#80

  parameters: {
    deviceHistory: {
      Type: 'String',
      Description: 'ARN of S3 bucket for storing a list of known devices'
    }
  },
  statements: [
    {
      Effect: 'Allow',
      Action: [
        's3:GetObject',
        's3:ListBucket',
        's3:PutObject'
      ],
      Resource: { Ref: 'deviceHistory' }
    }
  ],

Ref to deviceHistory will not work in this case

cc: @zmully

Better parameter encryption logic

To encrypt parameters, you have to pass -k and this is really easy to forget to do. Since this is passed through to cfn-config which actually implements this behavior it will be difficult change.

Some thoughts:

  • lambda-cfn defaults to passing just -k to cfn-config and we add another option -nok or something to override that default. If you pass -k <kmsArn> it gets passed wholesale to cfn-config
    • -k with no arn defaults to the kms id alias/cloudformation, what happens if that doesn't exist?

Use 6.10.3 Node runtime, not 6.10.2

I noticed that we're using the Node.js 6.10.2 runtime in the package.json https://github.com/mapbox/lambda-cfn/blob/14015e33f2ca0ec87bffa1866d5f61ab0f9d0ba6/package.json#L7 as well as in the Travis tests https://github.com/mapbox/lambda-cfn/blob/master/.travis.yml#L5.

The AWS Lambda Node.js 6x runtime is 6.10.3 and we say this in the lambda-cfn docs:

Lambda-cfn (or "Lambda CloudFormation") is a Node.js project that runs on Amazon Web Service's supported Node runtimes, currently Node.js v4.3.2 and v6.10.3.

This is a really minor issue. It's only a minor version difference and I don't see anything critical or important in the 6.10.3 changelog. Also, this is only to ensure consistency when developing locally and testing on Travis so that you're using the same version of Node that's in production. Even though we specified v6.10.2 in the package.json any lambda-cfn code is going to run on 6.10.3 on AWS anyway.

We should update this in lambda-cfn v2.1 or 2.0.1 and also make sure Patrol projects are using 6.10.3 as well.

/cc @zmully

Reference other resources in the template

Given the following example:

var lambdaCfn = require('lambda-cfn');

module.exports = lambdaCfn(
  [
    'myHandler.js',
  ],
  {
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "my stack",
    "Resources": {
       "AutoScalingGroup": { ... }
    }
  }
);

How would I go about referencing AutoScalingGroup inside my rule configuration? Specifically, I'd like make the physical id of AutoScalingGroup available to the function. I'm currently only seeing a way to do this with a parameter.

`init` command for new rules

As part of the lambda-cfn refactor to 1 rule 1 stack, a convenience function for new rules should initialize and create skeleton files to speed rule development.

Let's keep this super simple for this first go round, the command should assume you're writing a new CloudWatch Event based rule.

lambda-cfn init <rulename> should:

  1. Be run from a top level parent directory, all rules will be sub directories of this directory
  2. Check for a package.json in the parent directory, create it if necessary
  3. Add/update lambda-cfn as dependency in the package.json
  4. Check the rulename (no spaces, no underscores, anything else not legal in a Cloudformation stack name, must contain only letters, numbers, dashes and start with an alpha character. )
  5. Check and create a sub directory for the rule ./<rulename>
  6. Create function.js and function.template.js file skeletons (contents TBD)

Refactor lambda-cfn to one rule one stack

Currently Lambda-cfn support multiple rules, in multiple repositories in a single stack. This makes deployment and maintenance a headache as well as being a hinderance to new users. Moving to a one rule one stack architecture will simplify the lambda-cfn codebase as well as improve usability.

  • support existing lambda-cfn rule functionality
  • split lambda code from the lambda Cloudformation template
    • function.js: contains the actual lambda function code
    • function.template.js: Cloudformation template

Check that key on S3 exists before attempting deploy

If you run lambda-cfn update <stack> or lambda-cfn create <stack> and the key for the Lambda code on S3 has not been uploaded yet (either manually or as part of an automated build + deploy system) then lambda-cfn will error during the CloudFormation deploy process (and not cancel the deploy). This means a user will have gone through the CloudFormation parameter input process and deploy confirmation process only to have a failed deploy at the final step during the CloudFormation deloy. This process takes a while and can be frustrating.

To make this module a little more user friendly we could have lambda-cfn first check that the code exists on S3 after providing parameter values (in order to obtain the location on S3) but before the final CloudFormation deploy.

/cc @ianshward @zmully

Reducing the effects of changing the function name

It is not obvious the consequences of updating the function name specified in the function.template.js for a running function. Because this name is used within the CloudFormation template as a prefix for resources, changing the function name also changes the names these resources, and CloudFormation treats the renaming as a requests for new resources. It'll delete all old resources and create new ones, and many of the new resources will have different ARNs and URLs (like SNS topics, API Gateway URLs). This can wreak havoc if there are external dependencies configured with the old values.

With the one rule, one stack architecture, I believe it may be possible to remove the function name prefixing, and remove the requirement for a function name declaration within the template entirely. This would prevent a mistaken name change from drastically altering running stack.

API Gateway rules with api key require usage plan

If an apikey is requested for a gateway rule, a corresponding usage plan must be associated with the key. Right now, lambda-cfn does not create a plan and so any rule using API-gateway will not respond to REST api requests until one is associated. Initially, we can make this simple and just create a default plan, with no throttle and no quota. In the future these should be parameterized.

`deploy` command

For the 1 rule 1 stack refactor, we're moving from one repository containing multiple rules and creating a single stack for all rules to one repository with multiple rules each which gets its own stack.

For the initial command, wrap cfn-config with sane defaults to create a stack for a single rule, so

~/patrol-rules-aws/rule$ lambda-cfn deploy <environment> would run cfn-config with sane defaults, like:

cfn-config create rule-<production> function.template.js -n patrol-rules-aws -r us-east-1 -c cfn-config-${AWS_ACCOUNT_ID}-us-east-1 -k kmsKey​

Allow defining additional CloudWatch alarms

Right now only an Errors and NoInvocations alarm are defined for all lambdas. It will be easy to let a rule define their own alarms by specifying some properties the rule's in module.exports.config.

Support API-Gateway for webhooks

For external webhooks, currently the only way to wire them into a lambda is via a service like Zapier to proxy the hook to the rule SNS topic. Instead of the extra hop and external service requirement, the webhooks could call the lambda directly via API-Gateway. There is no CFN support for API-Gateway, but because we've already a dependency on an external script to create the CWE rules, modifying it to also support API-Gateway shouldn't be difficult.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.