99x / serverless-dynamodb-local Goto Github PK

After running sls dynamodb table -n

When running sls dynamodb table -c , folder not exists gives error.

When quitting a serverless offline session, it looks like the plugin attempts to restart DyanmoDB Local. This hangs the process indefinitely.

Console output:

^CServerless: Serverless Halting...
DynamoDB Local failed to start with code 130

serverless.yml:

plugins:
  - serverless-offline
  - serverless-dynamodb-local

custom:
  dynamodb:
    start:
      port: 8000
      inMemory: true
      migrate: true

Version 0.2.19

Write unit test to test the code using chai and mocha
Guide https://www.codementor.io/nodejs/tutorial/unit-testing-nodejs-tdd-mocha-sinon

If I try to seed something like this:

{
 "name":"test",
 "email":"[email protected]",
 "salt": {
     "type":"Buffer",
     "data":[118,71,177,248,110,54,84,149,174,240,32,125,83,151,113,196]
 },
 "encryptedPassword":"LCmT7j986FbGDHdX92Y0OfeQr0bfozI4XbCwDeLuEl/RHTF/Q==",
 "id":"6c56f4e0-1e29-11e7-b5a1-d9397b6b601c",
}

saltwon't be seeded as Buffer value but as json object.

The DynamoDB tooling you guys have written is very useful, but it's spread out over half a dozen packages, and in my experience projects like that can be hard to keep in sync. The monorepo style has been used a lot recently, and it may be a great fit here. You still maintain package-level modularity for the individual tools, but you can pull development tasks (linting, testing), to the top level project, test everything as a complete system, and automate npm-linking of inter-package dependencies.

Implement proper list of log messages with correct feedback to the user

• When dynamodb is installing
• When running, to show the current state

It seems that running migrations via sls dynamodb executeAll runs all files within /dynamodb regardless of whether they are .json extensions or not.

It may be useful to ignore non-json files.

Spec:

• serverless v1
• serverless-dynamodb-local v1

Steps to Reproduce:

• create migration file using sls dynamodb create -n <filename>
• open file using VIM or create .<filename>.json.swp
• run sls dynamodb executeAll

If we have dynamodb-local running as a docker container and compose the services with docker-compose, we'd still like to use this package for the migrate functionality.
However, in a docker-compose scenario, dynamodb-local runs on a separate host and for this to work, this package would need to expose a --host option.

Is this something you'd consider exposing?

The NPM Release doesn't include Seed functionality.

I think the community is shifting towards serverless v1.0. Any support on that or do we have to use the native dynamodb local by aws?

I followed the instructions to install plugin, declare in "plugins" section of serverless.yml, and install with sls dynamodb install.
when i run sls dynamodb start I get the error in the title, at /Users/overlord/Dev/Trails4.0/serverless/node_modules/serverless-dynamodb-local/index.js:211:45

here's what that line looks like in context:
startHandler() { let self = this, options = this.options; return new BbPromise(function(resolve) { let config = self.service.custom.dynamodb, options = _.merge({ sharedDb: self.options.sharedDb || true }, self.options, config && config.start

Any ideas on how to make this work?

Create glitter chat room for the project and integrate it with github

Gets an error when running commands other than sls dynamodb start for the first time.
This can be solved by having sls dynamodb install command (which is mandatory) or run the initial installation for all the scripts if dynamodb local is not installed

First implementation is to support all the optional parameters to be given as parameters when executing serverless commands related with the plugin. However it is difficult to type and run if there are multiple parameters are set. This task is to provide the capability for users to implement support to configure these parameters also in s-project.js custom configuration field provided for plugins

Dynamodb local download percentage goes beyond 100% when running sls dynamodb start for the first time.

Running the sls dynamodb install command downloads and installs the dynamodb-local every time even if it is already installed.
It would be nice to have a parameter ( --no-reinstall ?) , in which case the install command would first check if dynamodb-local is installed, and if it is return straight away.

This would help usage of serverless-dynamodb-local in different automation scenarios.

Could this plugin switch to relying on docker or better yet docker-compose so I would have to have blobal JDK installation for DynamoDB?

Currently only the sls dynamodb launch -p 8000 is only implemented (Only port parameter). Implement other parameter support for launch operation. The complete list of parameters can be found here

Add a lint npm script; there's a .jshint file here, but it's not being used. It may be worth looking into a strict eslint rule-set that makes sense for LTS versions of NodeJS (4.x, 6.x).

Some options may be:
https://github.com/thenativeweb/eslint-config-es
https://www.npmjs.com/package/eslint-config-airbnb

etc..

The airbnb config, while great, aggressively suggests bleeding edge features that need transpilation support from babel, so it may not be a great fit here.

Implement support to store table creation scripts so that, when dynamodb starts it is possible to run the commands and create the database table automatically

Note: It will require another configuration parameter to be added to plugins custom configuration area which is added in s-project.json file, to get the table creation scripts root location, and a template to define the data so that it can be run individually or as a batch with

sls dynamodb tables init -n 'name'
sls dynamodb tables init --a (which means create all)

and to create table creation template
sls dynamodb tables create -n 'name' (name of the function)

Integrate Travis CI to run tests automatically on sending pull requests

A new NPM version needs to be released. The version installed via NPM doesn't include the dummy-credentials and offline support, which results in an error like this when trying to run:

sls dynamodb executeAll
{ TimeoutError: Missing credentials in config
    at ClientRequest.<anonymous> (/Users/invisibletotoro/Documents/Development/GitHub/StashCloud/Stash/node_modules/aws-sdk/lib/http/node.js:56:34)
    at ClientRequest.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:185:7)
    at Socket.emitTimeout (_http_client.js:620:10)
    at Socket.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:339:8)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
  message: 'Missing credentials in config',
  code: 'CredentialsError',
  time: 2017-01-18T04:05:18.168Z,
  retryable: true,
  originalError: 
   { message: 'Could not load credentials from any providers',
     code: 'CredentialsError',
     time: 2017-01-18T04:05:18.168Z,
     retryable: true,
     originalError: 
      { message: 'Connection timed out after 1000ms',
        code: 'TimeoutError',
        time: 2017-01-18T04:05:18.166Z,
        retryable: true } } }

Currently inside the dynamodb/* plugin code, it uses 'Q' promises (require('q')) to include Promises to the code. Since serverless is already using 'blubird' for Promises which is more capable than 'Q' and to keep the uniformity of promises, this tasks scope is to remove 'Q' promises and replace with 'bluebird' promises and remove, the dependency for 'Q' in package.json

Replace dynamodb handling core functionality with dynamodb-localhost library
https://github.com/99xt/dynamodb-localhost

Also provide a different command for installing dynamodb (Which is a one time operation)

Hey!

So, somehow my version of dynamoDB local is now missing the executeALL function? It's also refusing to load the migrations when I start the utility.

custom:
  dynamodb:
    start:
      port: 8001
      inMemory: true
      migration: true
    migration:
      table_prefix: ${opt:stage}
      dir: ./dynamolocal/migrations

The stage variable is set to local.

Help!

Replace console.log with serverless SCLI logs
e.g
SCli = require(S.getServerlessPath('utils/cli'));
SCli.log('Successfully pruned the following functions in the following regions: ');

Implement support to run the table scripts, and seeds to a dynamodb in a AWS region so that the same table creation scripts and seeds can be used to update a AWS dynamodb database running inside AWS

sls dynamodb tables aws-init(other options)
sls dynamodb seeds aws-run(other options)

I think dbPath property needs to be set to default working_direcetory.

As a feature the seed should be exportable from the current database.

Hello,

Is there anyway to append env variables to the table names in migrations?

Thanks,
Cory

I installed the 1.0 branch following instructions, but saw this error when trying to run sls dyanmodb start

  Type Error ---------------------------------------------

     Cannot read property 'dynamodb' of undefined

     For debugging logs, run again after setting SLS_DEBUG env var.

Debugging shows it comes from this line. I was able to get past the error by adding this to my serverless.yml file

custom:
  dynamodb:

I'm a serverless noob - is it just a matter of updating the 1.0 branch README to reference serverless.yml instead of s-project.json?

Wouldn't it be better if this plugin would work with Resources section from Serverless v.1 ?

That way I wouldn't have to define them twice in yml and in json for this plugin.

Currently only sls dynamodb launch is only implemented. Implement the following commands

• stop
• relaunch
  Note: the basic logic for these functions are already implemented in dynamodb/db.js. The need is to create custom actions in root/index.js and call the relevant functions

I get the following errors when trying to install dynamodb on my local machine and my travis builds.

Local Machine:

Error: Error getting DynamoDb local latest tar.gz location: 200
    at ClientRequest.<anonymous> (/Users/tony/git/pathway-server/node_modules/dynamodb-localhost/dynamodb/installer.js:14:26)
    at ClientRequest.g (events.js:291:16)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:474:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)
    at Socket.socketOnData (_http_client.js:363:20)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:548:20)
/Users/tony/git/pathway-server/node_modules/dynamodb-localhost/dynamodb/installer.js:43
                    throw Error("Error in downloading Dynamodb local " + err);
                    ^

Error: Error in downloading Dynamodb local Error: connect ECONNREFUSED 127.0.0.1:80
    at Error (native)
    at ClientRequest.<anonymous> (/Users/tony/git/pathway-server/node_modules/dynamodb-localhost/dynamodb/installer.js:43:27)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1276:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

Travis-CI:

Error: Error in downloading Dynamodb local Error: connect ECONNREFUSED 127.0.0.1:80
    at Error (native)
    at ClientRequest.<anonymous> (/home/travis/build/ProlifiqSoftware/pathway-server/node_modules/serverless-dynamodb-local/node_modules/dynamodb-localhost/dynamodb/installer.js:43:27)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at Socket.socketErrorListener (_http_client.js:256:9)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1255:8)
    at nextTickCallbackWith2Args (node.js:437:9)
    at process._tickDomainCallback (node.js:392:17)

The seeds field on a provision file cannot be omitted or else you get this error:

Unhandled rejection TypeError: Cannot read property 'map' of undefined

You cannot create an empty array or else you get this error:

ValidationException: The batch write request list for a table cannot be null or empty: products

Suggest code checks for a seeds array before running dynamodb-migrations on https://github.com/99xt/serverless-dynamodb-local/blob/master/index.js#L193

Currently the plugin code is full of, auto generated code form the plugin boiler plate. These unnecessary code needs to be removed and cleaned up from the plugin code

When I ran sls dynamodb install an error message appeared as shown below.

Full stack trace:

[Error: Error getting DynamoDb local latest tar.gz location: 200]
/home/vagrant/aws-serverless-prototype/node_modules/dynamodb-localhost/dynamodb/installer.js:43
                    throw Error("Error in downloading Dynamodb local " + err);
                    ^

Error: Error in downloading Dynamodb local Error: connect ECONNREFUSED 127.0.0.1:80
    at Error (native)
    at ClientRequest.<anonymous> (/home/vagrant/aws-serverless-prototype/node_modules/dynamodb-localhost/dynamodb/installer.js:43:27)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at Socket.socketErrorListener (_http_client.js:269:9)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1269:8)
    at nextTickCallbackWith2Args (node.js:458:9)
    at process._tickDomainCallback (node.js:413:17)

My operating environment is as follows.

• HostOS
  • MacOS Sierra 10.12.3
• GuestOS(Vagrant 1.8.7 + VirtualBOX 5.1.12)
  • CentOS Linux release 7.3.1611 (Core)
• Node.js(v4.7.2)
• serverless(1.5.1)
• serverless-dynamodb-local(0.2.16)

I did a bit of node_modules/dynamodb-localhost/dynamodb/installer.js debugging.

I could download from the URL stored in the variable "downloadUrl" on line 11 using curl.

$ curl -L -O -kv http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.tar.gz
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to dynamodb-local.s3-website-us-west-2.amazonaws.com port 80 (#0)
*   Trying 52.218.144.91...
* Connected to dynamodb-local.s3-website-us-west-2.amazonaws.com (52.218.144.91) port 80 (#0)
> GET /dynamodb_local_latest.tar.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dynamodb-local.s3-website-us-west-2.amazonaws.com
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: I1DQB9usvczdpC6MXBB09f7rnkCHHEei4+QLjsI21YxluUraUfpYfU80ooIOWsadH3E7eKm+/z4=
< x-amz-request-id: 49C3FA2E3E3AADD0
< Date: Sun, 29 Jan 2017 12:25:40 GMT
< Last-Modified: Wed, 18 Jan 2017 18:26:45 GMT
< x-amz-version-id: wRzlbckhjnZIm0CVgcmTkkgmHBAB9l_U
< ETag: "c7c1bacf57b840022927f67d1a4b8e46"
< Content-Type: application/x-gzip
< Content-Length: 44203144
< Server: AmazonS3
<
{ [data not shown]
100 42.1M  100 42.1M    0     0  6133k      0  0:00:07  0:00:07 --:--:-- 8229k
* Connection #0 to host dynamodb-local.s3-website-us-west-2.amazonaws.com left intact

However, the contents of response.headers.location passed to http.get () on line 16 was undefined.

We need support for multiple profiles, currently it is using the default profile.

Stage used to be prepended to the table name, since the latest NPM update this is has changed:

Input:

sls dynamodb start -s local

Old output:

Dynamodb Local Started, Visit: http://localhost:8000/shell

Table creation completed for table: local-databaseName

New output:

Dynamodb Local Started, Visit: http://localhost:8000/shell

Table creation completed for table: -databaseName

Is it intended or a bug?

If intended how do I set a stage in my dynamodb names?

Implement support to store data seed scripts so that, when dynamodb starts it is possible to run the commands and seed the database table automatically

Note: It will require another configuration parameter to be added to plugins custom configuration area which is added in s-project.json file, to get the seed scripts root location, and a template to define the data so that it can be run individually or as a batch with

sls dynamodb table -n <table_name> // Create template
sls dynamodb table -c // Create the table and run seeds

Write documentation in Readme file to guide the users in configuring the plugin and available commands and options

Replace dynamodb migration handling functionality with dynamodb-migrations library
https://github.com/99xt/dynamodb-migrations

After building the basic plugin, add the plugin to npm registry, so that others can install it using
npm install serverless-dynamodb-local

Implement configuration options to set dynamodb local download path. currently its inside a variable of a function of dynamodb/db.js and it is not configurable. This task is to provide a configuration option in s-project.json where, a user can override the default download path if needed to use any version of dynamodb local plugin

The current code state is not published to NPM. It took me a while to figure out why tables where not being created 😕 Ends up that the published version is out of date. Could you update to NPM please?

Provide credit to the original dynamodb plugin writer

Console output:

>sls dynamodb start
Dynamodb Local Started, Visit: http://localhost:8000/shell

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger
        at com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer.<clinit>(DynamoDBProxyServer.java:36)
        at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServer(ServerRunner.java:123)
        at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServerFromCommandLineArgs(ServerRunner.java:119)
        at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.main(ServerRunner.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Logger
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 4 more
DynamoDB Local failed to start with code 1

Versions:

>sls --v
1.5.1

>java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

>node -v
v7.2.0

Package.json:

   "serverless-dynamodb-local": "^0.2.16",
    "serverless-offline": "^3.8.3"

serverless.yaml

service: aws-nodejs-nexit

provider:
  name: aws
  runtime: nodejs4.3
functions:
  hello:
    handler: src/handler.hello
    events:
      - http:
          path: hello
          method: get
plugins:
  - serverless-offline
  - serverless-dynamodb-local

custom:
  dynamodb:
    start:
      port: 8000
      inMemory: true
      migration: true
    migration:
      dir: ./offline/migrations

Hi,

it took me a while to realize that the index.js in https://registry.npmjs.org/serverless-dynamodb-local/-/serverless-dynamodb-local-0.2.19.tgz is not the same as the one on github https://github.com/99xt/serverless-dynamodb-local/blob/v1/index.js
would it be possible to update npm?

thanks