• README.md
• GoDoc

Instead of erroring out immediately, operations that fail due to ExpiredTokenException get retried over and over.

You can reproduce this by using credentials obtained with aws sts get-session-token --duration-seconds 900 instead of a long-lived API key, then waiting for them to expire.

The error response at the HTTP level looks like this (as logged by db := dynamo.New(awsSession, &aws.Config{Region: &region, LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody)})):

-----------------------------------------------------
2018/03/21 19:06:17 DEBUG: Response dynamodb/Scan Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Content-Length: 125
Connection: keep-alive
Content-Type: application/x-amz-json-1.0
Date: Thu, 22 Mar 2018 02:06:17 GMT
Server: Server
X-Amz-Crc32: 2167418451
X-Amzn-Requestid: RCI9REDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTED


-----------------------------------------------------
2018/03/21 19:06:17 {"__type":"com.amazon.coral.service#ExpiredTokenException","message":"The security token included in the request is expired"}

Have these operations return a dynamo.Iter.

• Scan
• Query
• BatchGet

Add releases to project as defined here https://semver.org/ to allow for semantic versioning when using vendoring tools such as glide

First

• Marshal tests
• Unmarshal tests

Later...

• Integration tests...

Allow the use of multiple segments in a Scan operation.
See: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScanParallelScan

Shouldn't be too hard... although figuring out how to wrangle Iter and parallel scans may be difficult.

If I have a table in DynamoDB, say Animals, where I insert both cats and dogs, is there a best practice way of fetching both types, at once, into an Animal list?

For Example in a batched query, I just want to hydrate all of the animals, but ensure all their properties are initialized. I know the following code won't work (how could it, without knowing what animal type you're trying to populate into), but am unsure of the best-practice way of doing this.

var ls []Animal
err := t.Batch("Id").Get(k...).All(&ls)

For reference, there is a property on every animal, called AnimalType that was meant to be used as a differentiator. Do I first have to batch query to get the types, and then somehow populate my animals from there?

Decoding is easy, but encoding is kind of hard...

Maybe a struct tag

type Thing struct {
    MySet []int `dynamo:",set"
}

Could also use a type with special marshaling rules like:

type Set interface{}

but then dealing with dynamo.Sets would be really annoying.

Aws sdk has their own implementation of the struct tags contained in this library suggest refactoring this library to make use of their struct tags to stay more inline with the official release

See:
https://godoc.org/github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute

Hello,

I am new to Dynamo and Go so excuse if I made obvious mistake.

This is my struct:

type Account struct {
	ID            string `dynamodbav:"id"`
	Identificator string `dynamodbav:"identificator"`

	Password   *string   `dynamo:"password"`
	Salt       *string   `dynamo:"salt"`
	Type       *int      `dynamo:"type"`
	Level      *int      `dynamo:"level"`
	ExpiryDate time.Time `dynamo:"expiry_date"`
	CreatedAt  time.Time `dynamo:"created_at"`
	UpdateAt   time.Time `dynamo:"updated_at"`
	QrKey      *string   `dynamo:"qr_key"`
}

Table:

{
    "TableDescription": {
        "ProvisionedThroughput": {
            "WriteCapacityUnits": 5,
            "ReadCapacityUnits": 5,
            "LastIncreaseDateTime": 0.0,
            "LastDecreaseDateTime": 0.0,
            "NumberOfDecreasesToday": 0
        },
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            },
            {
                "AttributeName": "identificator",
                "AttributeType": "S"
            }
        ],
        "TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/accounts",
        "ItemCount": 0,
        "KeySchema": [
            {
                "KeyType": "HASH",
                "AttributeName": "id"
            },
            {
                "KeyType": "RANGE",
                "AttributeName": "identificator"
            }
        ],
        "TableSizeBytes": 0,
        "TableName": "accounts",
        "CreationDateTime": 1525168799.569,
        "TableStatus": "ACTIVE"
    }
}

// I am testing local
	db := dynamo.New(session.New(), &aws.Config{Endpoint: aws.String("http://172.18.0.2:8000")})
	table := db.Table(account.Table)

	// put item
	idmid := generator.SecureString(50) // simple  string generator
	w := account.Account{ID: idmid, Identificator: "Idnetiti", CreatedAt: time.Now(), UpdateAt: time.Now(), ExpiryDate:time.Now()}
	err := table.Put(w).Run()

I am getting

ValidationException: One of the required keys was not given a value
        status code: 400, request id: 73fa1d70-3fdb-45d1-854f-ea0bb4c9f6eb

It would be helpful in situations to allow raw access to the queries and commands being sent to dynamo without an interceptor layer at the dynamodbiface layer. For instance if I want to tweak a query and find it more fluent with raw access to the query by calling Table.Get().QueryInput() and then feed that back into All or One or whatever. This would alleviate situations where the library doesn't support a feature I need while I work to get those features into the library.

Hi! I was trying to use this library to store/read some tweets on dynamo db but it fails when it tries to unmarshal a tweet:

2017/06/07 21:17:36 dynamo: can't unmarshal to type: twitter.Indices ([0 0])

The twitter.Indices type is this:

// Indices represent the start and end offsets within text.
type Indices [2]int

I tried to have a look at the code to make a PR but I'm still a newbie with go. I was wondering if it is possible to add support for arrays? :)

Would it be possible to add context, enabling cancellations. go-aws-sdk supports this, and in case when the dynamodb servers are unavailable or latency is high, this would be useful.

We could add a context argument to Run, e.g. RunWithContext(ctx) - would you be open to a patch?

Hi, here is the code snipped:

var records []map[string]interface{}

err := c.Scan().Filter("$ = ?", "email", "[email protected]").Consistent(true).All(&records)
if err != nil {
	return err
}

...

I am getting the following error:

"panic: reflect: call of reflect.Value.Set on zero Value\npanic(0x9c45a0, 0xc4202abb20)\n\t/usr/local/go/src/runtime/panic.go:489 +0x2cf\nreflect.flag.mustBeExported(0x0)\n\t/usr/local/go/src/reflect/value.go:209 +0x145\nreflect.Value.Set(0x9bd700, 0xc4202b8e90, 0x194, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/reflect/value.go:1346 +0x40\ngithub.com/guregu/dynamo.unmarshalReflect(0xc4204039a0, 0x9bd700, 0xc4202b8e90, 0x194, 0xc4202b8e90, 0x194)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:223 +0x1e5d\ngithub.com/guregu/dynamo.unmarshalItem(0xc4202d49c0, 0x97eb60, 0xc42009a0a8, 0x97eb60, 0xc42009a0a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:362 +0x826\ngithub.com/guregu/dynamo.unmarshalAppend(0xc4202d49c0, 0x974260, 0xc4202a5720, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:380 +0x1d8\ngithub.com/guregu/dynamo.(*scanIter).NextWithContext(0xc4201b8c18, 0x7f158d224498, 0xc42020db00, 0x974260, 0xc4202a5720, 0xc4201b8c58)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:255 +0x2d3\ngithub.com/guregu/dynamo.(*Scan).AllWithLastEvaluatedKeyContext(0xc420402320, 0x7f158d224498, 0xc42020db00, 0x974260, 0xc4202a5720, 0x1, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:136 +0xaf\ngithub.com/guregu/dynamo.(*Scan).All(0xc420402320, 0x974260, 0xc4202a5720, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:110 +0x9e\ngithub.com/JormungandrK/backends.(*DynamoCollection).GetOne(0xc42010a028, 0xc4202d4120, 0x97b1e0, 0xc4202b8190, 0xc4200648a8, 0xc4200a5328)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:168 +0x5d2\ngithub.com/JormungandrK/backends.(*DynamoCollection).DeleteOne(0xc42010a028, 0xc4202d4120, 0xc4201b9020, 0xc4200a53a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:310 +0x12f\ngithub.com/JormungandrK/backends.(*DynamoCollection).Save(0xc42010a028, 0x9c0b80, 0xc420435260, 0x0, 0xc420318600, 0x3c, 0x3c, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:258 +0xce2\nmain.(*UserController).Create(0xc42020ad20, 0xc4202a2ff0, 0xc4202b20a0, 0xc420198e00)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/user-microservice/user.go:82 +0x106\ngithub.com/JormungandrK/user-microservice/app.MountUserController.func1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0xc42004b2b0, 0xc42004b2a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/user-microservice/app/controllers.go:92 +0x11f\ngithub.com/goadesign/goa.(*Controller).MuxHandler.func1.1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x2, 0x2)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/service.go:309 +0xaa\ngithub.com/goadesign/goa/middleware.Recover.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/recover.go:37 +0xac\ngithub.com/goadesign/goa/middleware.ErrorHandler.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x2, 0x2)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/error_handler.go:21 +0x90\ngithub.com/goadesign/goa/middleware.LogRequest.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0xc42025deb0, 0xd62d60)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/log_request.go:75 +0xe50\ngithub.com/goadesign/goa/middleware.RequestIDWithHeaderAndLengthLimit.func1.1(0xd62d60, 0xc4202a2e10, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x0, 0xd62d60)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/request_id.go:63 +0x177\ngithub.com/goadesign/goa.(*Controller).MuxHandler.func1(0xd62360, 0xc4202f8620, 0xc42022d300, 0xc4202a2d80)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/service.go:342 +0x1f1\ngithub.com/goadesign/goa.(*mux).Handle.func1(0xd62360, 0xc4202f8620, 0xc42022d300, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/mux.go:69 +0x1b3\ngithub.com/dimfeld/httptreemux.(*TreeMux).ServeLookupResult(0xc420112000, 0xd62360, 0xc4202f8620, 0xc42022d300, 0xc8, 0xc4202a0820, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/dimfeld/httptreemux/router.go:245 +0x13d\ngithub.com/dimfeld/httptreemux.(*TreeMux).ServeHTTP(0xc420112000, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/dimfeld/httptreemux/router.go:266 +0xef\ngithub.com/goadesign/goa.(*mux).ServeHTTP(0xc420106670, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/mux.go:100 +0x4c\nnet/http.serverHandler.ServeHTTP(0xc4201591e0, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/usr/local/go/src/net/http/server.go:2568 +0x92\nnet/http.(*conn).serve(0xc4203508c0, 0xd62ca0, 0xc420318100)\n\t/usr/local/go/src/net/http/server.go:1825 +0x612\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2668 +0x2ce\n"

I guess I am doing something wrong.

Thanks in advance.

First of all, thanks for putting this great library together - saves a lot of time!

I have a question on the error handling. Most of the times the errors returned are the original AWS errors but this behaviour is inconsistent. For example for queries the errors are hidden behind custom ones: https://github.com/guregu/dynamo/blob/master/query.go#L42-L44.

Is there a reason for this? Would it make sense to always return the original AWS error messages so that the clients can decide how to deal with them and the error handling is consistent ?

Hello,
I have columns with complex objects in json.

Is it possible to serialize and deserialize these columns directly?

Read and write operations return a consumed capacity field that describes how much of the table capacity has been consumed during the operation. This can be useful in scenarios where the operations need to be throttled.

Suggestion is to add a method to all table operations called ConsumedCapacity where the client can pass a variable to store the output in.

var dog dogs.Dog
var consumed dynamodb.ConsumedCapacity
err := t.Get("OwnerID", owner.ID).
	Index(dogs.OwnerIndex).
	Range("Name", dynamo.Equal, name).
	ConsumedCapacity(&consumed).
	One(&dog)

Because aws/aws-sdk-go has breaking change about Configration, I often be encountered a conflict import lib error.

like this

script/try_dynamo.go:24: cannot use session.New() (type *session.Session) as type "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".ConfigProvider in argument to dynamo.New:
        *session.Session does not implement "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".ConfigProvider (wrong type for ClientConfig method)
                have ClientConfig(string, ...*"github.com/aws/aws-sdk-go/aws".Config) "github.com/aws/aws-sdk-go/aws/client".Config
                want ClientConfig(string, ...*"github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws".Config) "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".Config

Would you use a library versioning solution e.g. dep?

thank you;

This prevents mocking or supplying a different dynamo object.

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html

Query - Returns one or more items from a table or a secondary index. You must provide a specific hash key value. You can narrow the scope of the query using comparison operators against a range key value, or on the index key. Query supports either eventual or strong consistency. A single response has a size limit of 1 MB.

I have basic support like

var result []anything
err := table.Get("keyname", value).All(&result)

• getting all items
• limit
• specifying consistency
• projection expressions and stuff

Maps and structs in structs aren't supported by the automatic marshaler right now. You can use them by implementing dynamo.Marshaler if you're desperate.

• unmarshaling to structs
• unmarshaling to map[string]<something>
• marshaling structs
• marshaling maps

map[string]interface{} is at #8

probably

var thing anything
err := table.Get(keyname, keyvalue).One(&thing)

with further filtering possibilities via method chaining or something

DynamoDB doesn't like empty strings: would it be ok to have omitempty behavior for all strings? Or are there people who rely on DynamoDB complaining about empty strings to keep track of their data?

I think I will go with the automatic omission path for now.

@guregu So I'm still trying to track down the cause of this, but I have the following code:

t := db.db.Table("Location")
	k := make([]dynamo.Keyed, len(keys))
	for i, key := range keys {
		k[i] = dynamo.Keys{key, nil}
	}
	var ls []Location
	err := t.Batch("Id").Get(k...).All(&ls)
	retVal := make([]LocationDataResult, len(keys))
	for i := range keys {
		if i < len(ls) {
			retVal[i] = LocationDataResult{&ls[i], err}
		} else {
			retVal[i] = LocationDataResult{Error:err}
			//TODO--If error is nil, we say the object wasn't found
		}
	}
	return retVal

And when I pass in 2 keys, where one can be "found" and the other can't (as in, there's a single object in my DB table, and only one of those keys corresponds to it), I receive back 2 copies of the one "valid" object. I would expect to get back only one result?

I do see that the Keys are created correctly, and are getting appended onto the BatchGet correctly. That's where I'm at right now, and will continue to investigate further.

https://godoc.org/gopkg.in/aws/aws-sdk-go.v0/service/dynamodb#DynamoDB.DescribeTable

It'd be nice if there's a streamlined implementation of the TTL feature for tables similar to what is present in the AWS Go SDK

Go documentation for the update operation:
https://docs.aws.amazon.com/sdk-for-go/api/service/dynamodb/#DynamoDB.UpdateTimeToLive

AWS Developer's guide for the TTL feature:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html

Request syntax for the operation in the API:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTimeToLive.html

Hello,

Is it possible to use attribute_not_exists for composite key? Example:

err = c.Table.Put(av).If("attribute_not_exists($) AND attribute_not_exists($)", hashKey, rangeKey).Run()

Thanks.

If there's a reserved keyword in Project (e.g. "Name") the request would fail.

// Project limits the result attributes to the given paths.
func (s *Scan) Project(paths ...string) *Scan {
	expr, err := s.subExpr(strings.Join(paths, ", "), nil)
	s.setError(err)
	s.projection = expr
	return s
}

Maybe it should use expression attribute names for the paths?

Needs:

• A longer package description.
• Examples
• Details about how Go types and DynamoDB types are handled

Seems like Amazon changed the name of these keys out of nowhere.
Not really sure if I want to change them here as well.
.Range(name, operator, value) wouldn't make any sense as .Sort()

Error: dynamo: unmarshal int: expected N to be non-nil

using struct with
varname int dynamo:"varname,omitempty"

Dynamo search shows "varname Null : true" when using "Iter.Next(&struct)"

I'm not finding a way to continue without this failing out.

Hello,
I have a struct where i have a numerical representation of various skills, but i get the error (at the bottom). Can you not use map[string]int as a set???

type Talent struct {
  Id              string         `json:"id"`
  Name            string         `json:"name"`
  LinkedInProfile string         `json:"linkedin_profile"`
  Skills          map[string]int `json:"skills" dynamo:",set"`
  CreatedAt       string         `json:"created_at"`
}

dynamo: cannot marshal type map[string]int into a set

Issue

I have a table with a nested customized field map like

{
  "fields": {
    "custom_field_a": {"S": "blah blah blah"},
    "custom_field_b": {"S": "blah blah blah"},
  }
}

when custom_field_a's name is reserved, say "string" or "all", I get following exception:

ValidationException: Invalid UpdateExpression: Syntax error; token: "string", near: ".string ="
	status code: 400, request id: 33bd85d4-ade8-42c7-86b6-fc688f8692d8

Wondering if anyone has gotten dynamo to work on Dynamodb local? If so, does anyone have an example of how to get it all setup??

• CreateTable
• DeleteTable
• DescribeTable
• UpdateTable

• Timeout and retry options
• Ability to turn parsed expression cache off

For instance, take following schema:

{
   id: 1,
   meta: {
      distance: 10   
   }
}

How do I scan/query with a filter: distance > 1.

Here's what I have tried:

t.Scan().Filter("'meta.distance' > ?", 1)

hi @guregu , I'm trying to store a complex structure into dynamodb. here is an overview of what i'm trying to achieve:

type Event struct {
	Events []EventElement `dynamo:"events,set"`
	ID     string         `dynamo:"id"`
}

type EventElement struct {
	ID       string `dynamo:"string"`
	Items        []Item `dynamo:"items,set,omitempty"`
}

type Item struct {
	ID    string `dynamo:"id"`
	Price Price  `dynamo:"price"`
}

type Price struct {
	Amount   int    `dynamo:"amount"`
	Currency string `dynamo:"currency"`
}

You got the idea, i want to store Event which have many EventElements, which have many Items... etc.
How can i map such complex structure?
Thank you

For 'async' APIs, add a Wait method that blocks until e.g. the table's status is active.

will involve some annoying type inference

It would be beneficial to be able to trace DynamoDB calls with X-Ray, as it provides a hell lot of useful insights into how your code is performing. We're using X-Ray with newly announced Go support for Lambda's, and with this library hiding DynamoDB client initialization it's difficult to wrap it up easily without modifying the code.

My suggestion would be to either made a different implementation to initialize the client or to provide a constructor that will take initialized DynamoDB client as a parameter (maybe there is one, but I couldn't find it)

I came into a scenario where I had an object that contained a property for which I was trying to unmarshal (dynamoDB -> struct) into an incorrect type. In my case, in the dynamoDB table, it was of type map (fat-fingered data entry earlier), but was defined as a slice within the code.

This was fairly hard to track down because of the error message I received from decode.go:

dynamo: unmarshal slice: B, L, BS, SS, NS are nil

I see where this is coming from within the decode file; is there anyway to improve this error, as to be more explicit? I feel like these mismatched types is a fairly common error.

Implement the UpdateTimeToLive API: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTimeToLive.html

Hi, how I can use this library to throw an error if item does not exist?
Thanks in advance.

• BatchGetItem
• BatchWriteItem
• Chunk requests (Get: 100, Write: 25)

Is there a way to avoid having a error if table already exists ?

"ResourceInUseException: Cannot create preexisting table\n\tstatus code: 400, request id: 5cffe283-a803-4b6b-a0b8-220a30778498"

Thank you

I have a situation where I need to put a Hash only GSI on a column, and also a GSI with the same column as a range key eg:

type Record struct {
OwnerID string dynamo:"o,hash"
ID string dynamo:"i,range" index:"IDX-id,hash" index:"IDX-pd,range"
PD string dymamo:"pd" index:"IDX-pd,hash"
}

However the tags in golang seem to collide when you have multiples of the same name, so the hash on IDX-id gets clobbered. If this is in fact what's happening, exposing the index apis seems like a good short term, and exposing a multi-index syntax seems like a good long term.

Let me know if I'm doing this wrong.

Make table.Put(...) more like table.Update(...)

Hello,

How to use this library for pagination? I want to set limit and offset plus if it is possible to order the results by some attribute.

Thanks in advance.

Hi, how can I use this library to expose the lastEvaluatedKey to the client and let the client incrementally paginate through? thanks a lot for your help