Git Product home page Git Product logo

dynode's Introduction

dynode Build Status

node.js client for working with Amazon's DynamoDB service.

Urgent

A bug was discovered generating aws request starting on or over the 10th month of the year. This causes all requests to fail starting on October 1st, more information can be found here. Version 0.6.1 has been released fixing this issue. Anyone currently running 0.6.0 should upgrade immediately.

Installation

Installing npm (node package manager)

  $ curl http://npmjs.org/install.sh | sh

Installing dynode

  $ [sudo] npm install dynode

Motivation

Dynode is designed to be a simple and easy way to work with Amazon's DynamoDB service. Amazon's http api is complicated and non obvious how to interact with it. This client aims to offer a simplified more obvious way of working with DynamoDB, but without getting in your way or limiting what you can do with DynamoDB.

Usage

There are two different ways to use dynode: directly via the default dynamoDB client, or by instantiating your own client. The former is merely intended to be a convenient shared client to use throughout your application if you so choose.

Using the Default DynamoDB Client

The default client is accessible through the dynode module directly. Any method that you could call on an instance of a client is available on the default client:

  var dynode = require('dynode');
  // When using the default client you must first give it auth credentials
  dynode.auth({accessKeyId: "AWSAccessKey", secretAccessKey: "SecretAccessKey"});

  dynode.createTable("NewTable", console.log);
  dynode.listTables(console.log);

Instantiating your own DynamoDB Client

If you would prefer to manage your own client, potentially with different auth params:

  var client = new (dynode.Client)({
    accessKeyId: "AWSAccessKey", secretAccessKey: "SecretAccessKey"
  });

Region support

dynode supports all available DynamoDB regions. By default dynode will connect to the us-east-1 region. To connect to a different region pass in the region option when creating a client, for example:

  // connect to the US West (Northern California) Region
  var client = new (dynode.Client)({
    region: "us-west-1"
    accessKeyId: "AWSAccessKey", 
    secretAccessKey: "SecretAccessKey"
  });

The default client can also connect to any region

  // connect to the Asia Pacific (Tokyo) Region
  dynode.auth({
    region: "ap-northeast-1"
    accessKeyId: "AWSAccessKey", 
    secretAccessKey: "SecretAccessKey"
  });

Callback Signature

Callbacks return (error, [results], meta) where results are the returned data and meta is the extra information returned by DynamoDB

API Documentation

Auth

Before you can perform any operations on DynamoDB you need to provide your Amazon credentials.

  dynode.auth({accessKeyId: "AWSAccessKey", secretAccessKey: "SecretAccessKey"});

Table Name Prefix

dynode client takes an optional tableNamePrefix in order to support running in different environments such as dev, testing, production

  var client = new (dynode.Client)({
    accessKeyId: "AWSAccessKey", secretAccessKey: "SecretAccessKey", tableNamePrefix: "Dev_"
  });

Now all operations will be performed against tables starting with that prefix, for example

  client.createTable("NewTable", console.log); // will create table named 'Dev_NewTable'

HTTPS

To use HTTPS for connecting to DynamoDB pass in the https option, by default dynode will use HTTP

  dynode.auth({https: true, accessKeyId: "AWSAccessKey", secretAccessKey: "SecretAccessKey"});

Create Table

The CreateTable operation adds a new table to your account. For more info see here

By default createTable will create the given table with a primary key of id : String, a read capacity of 10 and write capacity of 5.

  dynode.createTable("ExampleTable", console.log);

createTable accepts an options hash to override any of the table creation defaults.

  var opts = {read: 20, write: 25, hash: {name: String}, range: {age: Number}};
  dynode.createTable("ExampleTable", opts, console.log);

List Tables

Returns an array of all the tables associated with the current account. For more info see here

By default listTables will list all of your DynamoDB tables.

  dynode.listTables(console.log);

You can also pass in options to filter which tables to list. See Amazon's docs for more info

  dynode.listTables({Limit: 3, ExclusiveStartTableName: "ExampleTable"}, console.log);

Describe Table

Returns information about the table, including the current status of the table, the primary key schema and when the table was created. For more info see here

  dynode.describeTable("ExampleTable", function (error, info));

Update Table

Updates the provisioned throughput for the given table. For more info see here

  dynode.updateTable("ExampleTable", {read: 15, write: 10}, console.log);

Delete Table

Deletes a table and all of its items. For more info see here

  dynode.deleteTable("ExampleTable", console.log);

Put Item

Creates a new item, or replaces an old item with a new item (including all the attributes). For more info see here

  dynode.putItem("ExampleTable", {name : "Foo", age: 80, baz : ["a", "b", "c"], nums: [1,2,3]}, console.log);

You can also pass in any option that Amazon accepts.

  var item = {name : "Bob"};
  var options = {ReturnValues:"ReturnValuesConstant", Expected :{"age":{"Value": {"N":"42"},{"Exists":Boolean}}}};

  dynode.putItem("ExampleTable", item, options, console.log);

Update Item

Edits an existing item's attributes. For more info see here

By default all operations will be a PUT, which will add or replace the attribute with the new value.

  dynode.updateItem("ExampleTable", "ItemHashKey", {name: "Ryan"}, console.log);

updateItem also accepts a key object to specify which item to update.

  dynode.updateItem("ExampleTable", {hash: "Key", range: 22}, {name: "Ryan"}, console.log);

Perform specific action to perform for the given update

  var updates = {nums: {'delete' : [5]}, age: {add: 2}};

  dynode.updateItem("ExampleTable", "ItemsHashKey", updates, console.log);

Delete the attribute from an existing item

  var updates = {age: {'Action' : 'DELETE'}};

  dynode.updateItem("ExampleTable", "ItemsHashKey", updates, console.log);

DynamoDB doesn't allow empty strings or empty sets updateItem will delete attributes when passing in null, empty string, or empty array

  var updates = {age: null, nums: [], fullname: ''};

  dynode.updateItem("ExampleTable", "ItemsHashKey", updates, console.log);

updateItem accepts options which lets you pass in any option that Amazon accepts.

  var opts = {ReturnValues : "ReturnValuesConstant", Expected :{"status":{"Value":{"S":"offline"}}}};

  dynode.updateItem("ExampleTable", "TheKey", {name: "Ryan"}, opts, console.log);

Get Item

The getItem operation returns a set of Attributes for an item that matches the primary key. For more info see here

  dynode.getItem("ExampleTable", "TheHashKey", console.log);

getItem also accepts a key object to specify which item to get.

  dynode.getItem("ExampleTable", {hash: "TheHashKey", range: 123}, console.log);

getItem accepts any option that Amazon accepts.

  var opts = {AttributesToGet: ["status","friends"], ConsistentRead : true};

  dynode.getItem("ExampleTable", "TheHashKey", opts, console.log);

Delete Item

Deletes a single item in a table by primary key. For more info see here

  dynode.deleteItem("ExampleTable", "TheHashKey", console.log);

deleteItem also accepts a key object to specify which item to delete.

  dynode.deleteItem("ExampleTable", {hash: "TheHashKey", range: 123}, console.log);

deleteItem accepts any option that Amazon accepts.

  var opts = {ReturnValues : "ALL_OLD"};

  dynode.deleteItem("ExampleTable", "TheHashKey", opts, console.log);

Query

A Query operation gets the values of one or more items and their attributes by primary key. For more info see here

  dynode.query("ExampleTable", "TheHashKey", console.log);

query accepts any option that Amazon accepts.

  var opts = {RangeKeyCondition: {AttributeValueList :[{"N":"AttributeValue2"}],"ComparisonOperator":"GT"}};

  dynode.query("ExampleTable", "TheHashKey", opts, console.log);

Scan

The Scan operation returns one or more items and its attributes by performing a full scan of a table. For more info see here

  dynode.scan("ExampleTable", console.log);

scan accepts any option that Amazon accepts.

  var opts = {
    Limit: 5,
    ScanFilter : {"AttributeName1":{"AttributeValueList":[{"S":"AttributeValue"}],"ComparisonOperator":"EQ"}},
    AttributesToGet : ["AttributeName1", "AttributeName2", "AttributeName3"]
  };

  dynode.scan("ExampleTable", opts, console.log);

Batch Get Item

The BatchGetItem operation returns the attributes for multiple items from multiple tables using their primary keys. For more info see here

  var query = {
    "ExampleTable": {keys:[{hash: "someKey"}, {hash: "someKey2"}]},
    "AnotherTable": {keys:[{hash: "anotherKey", range: 123}]}
  }

  dynode.batchGetItem(query, console.log);

batchGetItem accepts any option that Amazon accepts.

  var filter = {
    "ExampleTable": {keys:[{hash: "someKey"}], AttributesToGet :["name", "age"]},
    "AnotherTable": {keys:[{hash: "anotherKey", range: 123}], AttributesToGet :["brand", "price"]}
  }

  dynode.batchGetItem(filter, console.log);

Batch Write Item

The BatchWriteItem operation This operation enables you to put or delete several items across multiple tables in a single API call. For more info see here

  var writes = {
    "BatchTable": [
      {put : {id : "foo", name: "bar"}},
      {del : "hash-key"}
    ],
    "AnotherTable": [
      {del : {hash: "somekey", range: "foo"}},
      {del : {hash: "another key", range: "moar foo"}}
    ]
  };

  dynode.batchWriteItem(writes, console.log);

Truncate

The Truncate operation will scan all items currently in the given table and remove them one by one. This has the potential to use up your write capacity, so use this call with care. note - This api is not provided directly by DynamoDB.

  var options = {
    throughputPercent : 0.5 // attempt to only consume 50% of write capacity, defaults to 100%
  };

  dynode.truncate("ExampleTable", options, console.log);

Tests

All tests are written with mocha and should be run with make:

  $ make test

License: Apache 2.0

dynode's People

Contributors

dickhardt avatar domenic avatar sharvil avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

dynode's Issues

getBatchItem not sniffing data types correctly

Getting the following error:

{ name: 'AmazonError',
type: 'ValidationException',
serviceName: 'com.amazon.coral.validate',
message: 'One or more parameter values were invalid: Mismatching attribute types between location and schema',
statusCode: 400,
retry: [Getter] }

Basically it came down to that the range selector I was using was being passed through as a string instead of a number. This is even though I am chaining queries, having just gotten the range selector out of another table and am immediately using it in the query that is failing. The way I was able to fix it was by casting it from a string into a number (the way Amazon expects it to come across):

results.Items.forEach(function(element, index, array){
    console.log(element);

    element.events.SS.forEach(function(ielement, iindex, iarray){
        var batchVars = {"events": {keys:[ {hash: ielement, range: parseInt(element.timestamp.N)} ]}};
        dynode.batchGetItem(batchVars, events.debugOutput);
    }, query);
}, query);

Here's a gist with a lot more context:
https://gist.github.com/2502794

I know this code is probably all crap, I'm a complete newb to node.js & dynamoDB (not to mention dynode).

Thanks!

Issues with date starting on month 10

Starting from 2012-10-01 we get the following error:

error: dynode.batchGetItem returned error: AmazonError - BatchGetItem 400 IncompleteSignatureException: Date must be in ISO-8601 'basic format'. Got '202201T000748Z'. See http://en.wikipedia.org/wiki/ISO_8601

Please note that the date is 202201 instead of 20121001. This is a variable type isue - number vs string - in aws-signer.js lines 65-70 and line 90.

To resolve, change line 90 to:

return n < 10 ? '0'+n : String(n);

EU region

Hello

I'm trying to use the EU region but the table keeps getting created in the US-EAST region.

My code is this:

var dynode = require('dynode');
dynode.auth({
region: "eu-west-1",
accessKeyId: "xxxxxx",
secretAccessKey: "xxxxxx"
});

dynode.createTable("CardTable", console.log);

output:

null { TableDescription:
{ CreationDateTime: 1342962562.553,
KeySchema: { HashKeyElement: [Object] },
ProvisionedThroughput: { ReadCapacityUnits: 10, WriteCapacityUnits: 5 },
TableName: 'CardTable',
TableStatus: 'CREATING' } }

Then when I try to use:

dynode.putItem("CardTable", {id: "1",card: 1, suit: "H"}, console.log);

And I go to the AWS console, I see it's in the US table.

I've also tried: dynamodb.eu-west-1.amazonaws.com

Thanks

Sam

can't install with npm

Hello there!

First of all, thanks for creating this module! I'm anxious to get started testing it out : )

However, when, I try to install with npm (following your instructions here on github), the dynode directory doesn't contain an index.js (or any js file).

[servername@ip-10-117-34-116 wsnode]$ cat /proc/version
Linux version 2.6.35.14-106.49.amzn1.x86_64 ([email protected]) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Fri Dec 2 18:19:57 UTC 2011
[servername@ip-10-117-34-116 wsnode]$ nvm ls
v0.6.4
current:        v0.6.4
default -> v0.6.4
[servername@ip-10-117-34-116 wsnode]$ npm --unicode=false ls
[email protected] /home/servername/wsnode
+-- [email protected]  extraneous
â +-- [email protected]
â +-- [email protected]
â +-- [email protected]
â â +-- [email protected]
â â +-- [email protected]
â â +-- [email protected]
â â `-- [email protected]
â `-- [email protected]
â   `-- [email protected]
+-- [email protected]
â +-- [email protected]
â â `-- [email protected]
â +-- [email protected]
â +-- [email protected]
â `-- [email protected]
+-- [email protected]  extraneous
+-- [email protected]
+-- [email protected]  extraneous
â `-- [email protected]
â   +-- [email protected]
â   `-- [email protected]
`-- [email protected]  extraneous
  +-- [email protected]
  +-- [email protected]
  `-- [email protected]
    +-- [email protected]
    +-- [email protected]
    `-- [email protected]
[servername@ip-10-117-34-116 wsnode]$ ll node_modules/dynode/
total 44
drwxr-xr-x 3 servername servername  4096 Feb 27 23:34 lib
-rw-r--r-- 1 servername servername 10212 Jan 27 05:35 LICENSE
-rw-r--r-- 1 servername servername   756 Feb  5 18:35 Makefile
drwxrwxr-x 6 servername servername  4096 Feb 27 23:34 node_modules
-rw-r--r-- 1 servername servername   742 Feb  7 21:39 package.json
-rw-r--r-- 1 servername servername  9999 Feb  5 18:25 README.md
drwxr-xr-x 4 servername servername  4096 Feb 27 23:34 test

Any thoughts on what I'm messing up?

InvalidSignatureException when inserting a unicode string?

This is the AmazonError message that comes back when inserting a simple unicode string using the putItem method.

"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."

Any ideas?

deleteTable callback returns before table is deleted

I'm using this route as a way to nuke my database. The idea is to delete the table and when done, recreate it.

server.get('/fixture/nukedata', function (req, res) {
dynode.deleteTable('CardTable', function(a, b) {
console.log(a, b);
dynode.createTable("CardTable", console.log);
});
});

I get an error message saying: "Attempt to change a resouce which is still in use: Table is being deleted: CardTable"

Is this dynode returning too early or is it that Amazon's response doesn't is async?

Error: Unsupported type: BOOL

Why is a simple
client.scan("Employee", console.log);

giving me the error:
/src/node_modules/dynode/lib/dynode/types.js:118
throw new Error("Unsupported type: "+ type);
^
Error: Unsupported type: BOOL

My dynamoDB JSON looks like that:
{ "isPresent": { "BOOL": false }, "name": { "S": "Some Name" } }

add a tableNamePrefix option

add in an option to provide a table name prefix in order to support different environments such as dev, testing and production

Can't pass nested JSON Object

I noticed when you want to pass a nested JSON object with put item (also update item), it insert the object into the database as [object, object].

I haven't started looking into your code in to mush depth yet, but as aws can support nested json objects, it would worth looking into.

Example
Trying to insert the following object into a new table won't give any warnings

var object = {
	"id": "id",
	"num": 1,
	"str": "string",
	"Obj": {
		"num": 1,
		"str": "string"
	}
};
dynode.putItem("NewTable", object);

However amazon will show it inserted as:

{
  "id": "id",
  "num": 1,
  "Obj": "[object Object]",
  "str": "string"
}

How to add variable in number set?

I need to store my dialogs id in dialogs column. But I don't know how to set Action: ADD for adding values to existing data in number set. By default set is PUT whitch override whole NS Need help.

My attempts

var userDialog = {
            dialogs:[Number(dialog.id)]
};
dynode.updateItem('userDialogs', 1, userDialog, {"Action":"ADD"}, function (err, res) {
      if (err) {
          console.log(err);
           return false;
      }
     console.log(res);
});

My db structure

id Number ( Hash key)
dialogs Number Set

ps Sorry for english

IAM role issue

I'm using dynode with my instances and works well when credentials are passed during configuration.

How should I make it work without passing access key and secret? AWS NodeJS SDK provides support for this where it looks for IAM role credentials when not provided. Do you suggest a cleaner approach?

Callback is not being called on any functions

I am not sure if this is an issue with just my setup or has something to do with something else. I've gotten the module installs, i can reference it accordingly.

However, when I call a function from the library I cant seem to get anything to print out or the callback to ever get called. It seems to never get to the 'self.request.send' callback in _request of client.js.

I also cant seem to get any kind of error printed or anything like that either. Any ideas?

I'm literally just doing:

client.listTables(callbackFunction);

Use AWS Signature v4 to reduce # of requests

I got this email this morning.

Dear Amazon DynamoDB Customer,

We are happy to announce that Amazon DynamoDB now supports the AWS Signature V4 request signing protocol. With Signature V4, you can make requests to DynamoDB that are signed by your regular AWS User or Account credentials, meaning that your application no longer needs to obtain temporary credentials from the AWS Security Token Service prior to making requests to DynamoDB. This improves application startup time, and removes the need to do complex temporary credential caching logic in applications with short-lived process lifetimes, which is common in PHP applications.

This seems like it would help a lot with the number of requests that Dynode makes to DynamoDB. Are you working on integrating this by now, perchance? If not, we'll try to take a crack at it---any pointers to the appropriate code areas to modify would be appreciated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.