guregu / dynamo Goto Github PK
View Code? Open in Web Editor NEWexpressive DynamoDB library for Go
License: Other
expressive DynamoDB library for Go
License: Other
Instead of erroring out immediately, operations that fail due to ExpiredTokenException
get retried over and over.
You can reproduce this by using credentials obtained with aws sts get-session-token --duration-seconds 900
instead of a long-lived API key, then waiting for them to expire.
The error response at the HTTP level looks like this (as logged by db := dynamo.New(awsSession, &aws.Config{Region: ®ion, LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody)})
):
-----------------------------------------------------
2018/03/21 19:06:17 DEBUG: Response dynamodb/Scan Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Content-Length: 125
Connection: keep-alive
Content-Type: application/x-amz-json-1.0
Date: Thu, 22 Mar 2018 02:06:17 GMT
Server: Server
X-Amz-Crc32: 2167418451
X-Amzn-Requestid: RCI9REDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTED
-----------------------------------------------------
2018/03/21 19:06:17 {"__type":"com.amazon.coral.service#ExpiredTokenException","message":"The security token included in the request is expired"}
Have these operations return a dynamo.Iter
.
Add releases to project as defined here https://semver.org/ to allow for semantic versioning when using vendoring tools such as glide
Allow the use of multiple segments in a Scan operation.
See: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#QueryAndScanParallelScan
Shouldn't be too hard... although figuring out how to wrangle Iter and parallel scans may be difficult.
If I have a table in DynamoDB, say Animals, where I insert both cats and dogs, is there a best practice way of fetching both types, at once, into an Animal list?
For Example in a batched query, I just want to hydrate all of the animals, but ensure all their properties are initialized. I know the following code won't work (how could it, without knowing what animal type you're trying to populate into), but am unsure of the best-practice way of doing this.
var ls []Animal
err := t.Batch("Id").Get(k...).All(&ls)
For reference, there is a property on every animal, called AnimalType that was meant to be used as a differentiator. Do I first have to batch query to get the types, and then somehow populate my animals from there?
Decoding is easy, but encoding is kind of hard...
Maybe a struct tag
type Thing struct {
MySet []int `dynamo:",set"
}
Could also use a type with special marshaling rules like:
type Set interface{}
but then dealing with dynamo.Sets would be really annoying.
Aws sdk has their own implementation of the struct tags contained in this library suggest refactoring this library to make use of their struct tags to stay more inline with the official release
See:
https://godoc.org/github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute
Hello,
I am new to Dynamo and Go so excuse if I made obvious mistake.
This is my struct:
type Account struct {
ID string `dynamodbav:"id"`
Identificator string `dynamodbav:"identificator"`
Password *string `dynamo:"password"`
Salt *string `dynamo:"salt"`
Type *int `dynamo:"type"`
Level *int `dynamo:"level"`
ExpiryDate time.Time `dynamo:"expiry_date"`
CreatedAt time.Time `dynamo:"created_at"`
UpdateAt time.Time `dynamo:"updated_at"`
QrKey *string `dynamo:"qr_key"`
}
Table:
{
"TableDescription": {
"ProvisionedThroughput": {
"WriteCapacityUnits": 5,
"ReadCapacityUnits": 5,
"LastIncreaseDateTime": 0.0,
"LastDecreaseDateTime": 0.0,
"NumberOfDecreasesToday": 0
},
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
},
{
"AttributeName": "identificator",
"AttributeType": "S"
}
],
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/accounts",
"ItemCount": 0,
"KeySchema": [
{
"KeyType": "HASH",
"AttributeName": "id"
},
{
"KeyType": "RANGE",
"AttributeName": "identificator"
}
],
"TableSizeBytes": 0,
"TableName": "accounts",
"CreationDateTime": 1525168799.569,
"TableStatus": "ACTIVE"
}
}
// I am testing local
db := dynamo.New(session.New(), &aws.Config{Endpoint: aws.String("http://172.18.0.2:8000")})
table := db.Table(account.Table)
// put item
idmid := generator.SecureString(50) // simple string generator
w := account.Account{ID: idmid, Identificator: "Idnetiti", CreatedAt: time.Now(), UpdateAt: time.Now(), ExpiryDate:time.Now()}
err := table.Put(w).Run()
I am getting
ValidationException: One of the required keys was not given a value
status code: 400, request id: 73fa1d70-3fdb-45d1-854f-ea0bb4c9f6eb
It would be helpful in situations to allow raw access to the queries and commands being sent to dynamo without an interceptor layer at the dynamodbiface layer. For instance if I want to tweak a query and find it more fluent with raw access to the query by calling Table.Get().QueryInput() and then feed that back into All or One or whatever. This would alleviate situations where the library doesn't support a feature I need while I work to get those features into the library.
Hi! I was trying to use this library to store/read some tweets on dynamo db but it fails when it tries to unmarshal a tweet:
2017/06/07 21:17:36 dynamo: can't unmarshal to type: twitter.Indices ([0 0])
The twitter.Indices type is this:
// Indices represent the start and end offsets within text.
type Indices [2]int
I tried to have a look at the code to make a PR but I'm still a newbie with go. I was wondering if it is possible to add support for arrays? :)
Would it be possible to add context, enabling cancellations. go-aws-sdk supports this, and in case when the dynamodb servers are unavailable or latency is high, this would be useful.
We could add a context argument to Run, e.g. RunWithContext(ctx) - would you be open to a patch?
Hi, here is the code snipped:
var records []map[string]interface{}
err := c.Scan().Filter("$ = ?", "email", "[email protected]").Consistent(true).All(&records)
if err != nil {
return err
}
...
I am getting the following error:
"panic: reflect: call of reflect.Value.Set on zero Value\npanic(0x9c45a0, 0xc4202abb20)\n\t/usr/local/go/src/runtime/panic.go:489 +0x2cf\nreflect.flag.mustBeExported(0x0)\n\t/usr/local/go/src/reflect/value.go:209 +0x145\nreflect.Value.Set(0x9bd700, 0xc4202b8e90, 0x194, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/reflect/value.go:1346 +0x40\ngithub.com/guregu/dynamo.unmarshalReflect(0xc4204039a0, 0x9bd700, 0xc4202b8e90, 0x194, 0xc4202b8e90, 0x194)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:223 +0x1e5d\ngithub.com/guregu/dynamo.unmarshalItem(0xc4202d49c0, 0x97eb60, 0xc42009a0a8, 0x97eb60, 0xc42009a0a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:362 +0x826\ngithub.com/guregu/dynamo.unmarshalAppend(0xc4202d49c0, 0x974260, 0xc4202a5720, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/decode.go:380 +0x1d8\ngithub.com/guregu/dynamo.(*scanIter).NextWithContext(0xc4201b8c18, 0x7f158d224498, 0xc42020db00, 0x974260, 0xc4202a5720, 0xc4201b8c58)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:255 +0x2d3\ngithub.com/guregu/dynamo.(*Scan).AllWithLastEvaluatedKeyContext(0xc420402320, 0x7f158d224498, 0xc42020db00, 0x974260, 0xc4202a5720, 0x1, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:136 +0xaf\ngithub.com/guregu/dynamo.(*Scan).All(0xc420402320, 0x974260, 0xc4202a5720, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/guregu/dynamo/scan.go:110 +0x9e\ngithub.com/JormungandrK/backends.(*DynamoCollection).GetOne(0xc42010a028, 0xc4202d4120, 0x97b1e0, 0xc4202b8190, 0xc4200648a8, 0xc4200a5328)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:168 +0x5d2\ngithub.com/JormungandrK/backends.(*DynamoCollection).DeleteOne(0xc42010a028, 0xc4202d4120, 0xc4201b9020, 0xc4200a53a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:310 +0x12f\ngithub.com/JormungandrK/backends.(*DynamoCollection).Save(0xc42010a028, 0x9c0b80, 0xc420435260, 0x0, 0xc420318600, 0x3c, 0x3c, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/backends/dynamodb.go:258 +0xce2\nmain.(*UserController).Create(0xc42020ad20, 0xc4202a2ff0, 0xc4202b20a0, 0xc420198e00)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/user-microservice/user.go:82 +0x106\ngithub.com/JormungandrK/user-microservice/app.MountUserController.func1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0xc42004b2b0, 0xc42004b2a8)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/JormungandrK/user-microservice/app/controllers.go:92 +0x11f\ngithub.com/goadesign/goa.(*Controller).MuxHandler.func1.1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x2, 0x2)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/service.go:309 +0xaa\ngithub.com/goadesign/goa/middleware.Recover.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/recover.go:37 +0xac\ngithub.com/goadesign/goa/middleware.ErrorHandler.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x2, 0x2)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/error_handler.go:21 +0x90\ngithub.com/goadesign/goa/middleware.LogRequest.func1.1(0xd62d60, 0xc4202a2f60, 0xd62020, 0xc4203183c0, 0xc42022d300, 0xc42025deb0, 0xd62d60)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/log_request.go:75 +0xe50\ngithub.com/goadesign/goa/middleware.RequestIDWithHeaderAndLengthLimit.func1.1(0xd62d60, 0xc4202a2e10, 0xd62020, 0xc4203183c0, 0xc42022d300, 0x0, 0xd62d60)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/middleware/request_id.go:63 +0x177\ngithub.com/goadesign/goa.(*Controller).MuxHandler.func1(0xd62360, 0xc4202f8620, 0xc42022d300, 0xc4202a2d80)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/service.go:342 +0x1f1\ngithub.com/goadesign/goa.(*mux).Handle.func1(0xd62360, 0xc4202f8620, 0xc42022d300, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/mux.go:69 +0x1b3\ngithub.com/dimfeld/httptreemux.(*TreeMux).ServeLookupResult(0xc420112000, 0xd62360, 0xc4202f8620, 0xc42022d300, 0xc8, 0xc4202a0820, 0x0, 0x0)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/dimfeld/httptreemux/router.go:245 +0x13d\ngithub.com/dimfeld/httptreemux.(*TreeMux).ServeHTTP(0xc420112000, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/dimfeld/httptreemux/router.go:266 +0xef\ngithub.com/goadesign/goa.(*mux).ServeHTTP(0xc420106670, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/home/vladimir/projects/jormungandr/microservices/src/github.com/goadesign/goa/mux.go:100 +0x4c\nnet/http.serverHandler.ServeHTTP(0xc4201591e0, 0xd62360, 0xc4202f8620, 0xc42022d300)\n\t/usr/local/go/src/net/http/server.go:2568 +0x92\nnet/http.(*conn).serve(0xc4203508c0, 0xd62ca0, 0xc420318100)\n\t/usr/local/go/src/net/http/server.go:1825 +0x612\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2668 +0x2ce\n"
I guess I am doing something wrong.
Thanks in advance.
First of all, thanks for putting this great library together - saves a lot of time!
I have a question on the error handling. Most of the times the errors returned are the original AWS errors but this behaviour is inconsistent. For example for queries the errors are hidden behind custom ones: https://github.com/guregu/dynamo/blob/master/query.go#L42-L44.
Is there a reason for this? Would it make sense to always return the original AWS error messages so that the clients can decide how to deal with them and the error handling is consistent ?
Read and write operations return a consumed capacity field that describes how much of the table capacity has been consumed during the operation. This can be useful in scenarios where the operations need to be throttled.
Suggestion is to add a method to all table operations called ConsumedCapacity
where the client can pass a variable to store the output in.
var dog dogs.Dog
var consumed dynamodb.ConsumedCapacity
err := t.Get("OwnerID", owner.ID).
Index(dogs.OwnerIndex).
Range("Name", dynamo.Equal, name).
ConsumedCapacity(&consumed).
One(&dog)
Because aws/aws-sdk-go has breaking change about Configration, I often be encountered a conflict import lib error.
like this
script/try_dynamo.go:24: cannot use session.New() (type *session.Session) as type "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".ConfigProvider in argument to dynamo.New:
*session.Session does not implement "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".ConfigProvider (wrong type for ClientConfig method)
have ClientConfig(string, ...*"github.com/aws/aws-sdk-go/aws".Config) "github.com/aws/aws-sdk-go/aws/client".Config
want ClientConfig(string, ...*"github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws".Config) "github.com/guregu/dynamo/vendor/github.com/aws/aws-sdk-go/aws/client".Config
Would you use a library versioning solution e.g. dep?
thank you;
This prevents mocking or supplying a different dynamo object.
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
Query - Returns one or more items from a table or a secondary index. You must provide a specific hash key value. You can narrow the scope of the query using comparison operators against a range key value, or on the index key. Query supports either eventual or strong consistency. A single response has a size limit of 1 MB.
I have basic support like
var result []anything
err := table.Get("keyname", value).All(&result)
Maps and structs in structs aren't supported by the automatic marshaler right now. You can use them by implementing dynamo.Marshaler if you're desperate.
map[string]<something>
map[string]interface{}
is at #8
probably
var thing anything
err := table.Get(keyname, keyvalue).One(&thing)
with further filtering possibilities via method chaining or something
DynamoDB doesn't like empty strings: would it be ok to have omitempty behavior for all strings? Or are there people who rely on DynamoDB complaining about empty strings to keep track of their data?
I think I will go with the automatic omission path for now.
@guregu So I'm still trying to track down the cause of this, but I have the following code:
t := db.db.Table("Location")
k := make([]dynamo.Keyed, len(keys))
for i, key := range keys {
k[i] = dynamo.Keys{key, nil}
}
var ls []Location
err := t.Batch("Id").Get(k...).All(&ls)
retVal := make([]LocationDataResult, len(keys))
for i := range keys {
if i < len(ls) {
retVal[i] = LocationDataResult{&ls[i], err}
} else {
retVal[i] = LocationDataResult{Error:err}
//TODO--If error is nil, we say the object wasn't found
}
}
return retVal
And when I pass in 2 keys, where one can be "found" and the other can't (as in, there's a single object in my DB table, and only one of those keys corresponds to it), I receive back 2 copies of the one "valid" object. I would expect to get back only one result?
I do see that the Keys are created correctly, and are getting appended onto the BatchGet correctly. That's where I'm at right now, and will continue to investigate further.
It'd be nice if there's a streamlined implementation of the TTL feature for tables similar to what is present in the AWS Go SDK
Go documentation for the update operation:
https://docs.aws.amazon.com/sdk-for-go/api/service/dynamodb/#DynamoDB.UpdateTimeToLive
AWS Developer's guide for the TTL feature:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html
Request syntax for the operation in the API:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTimeToLive.html
Hello,
Is it possible to use attribute_not_exists for composite key? Example:
err = c.Table.Put(av).If("attribute_not_exists($) AND attribute_not_exists($)", hashKey, rangeKey).Run()
Thanks.
If there's a reserved keyword in Project
(e.g. "Name") the request would fail.
// Project limits the result attributes to the given paths.
func (s *Scan) Project(paths ...string) *Scan {
expr, err := s.subExpr(strings.Join(paths, ", "), nil)
s.setError(err)
s.projection = expr
return s
}
Maybe it should use expression attribute names for the paths
?
Needs:
Seems like Amazon changed the name of these keys out of nowhere.
Not really sure if I want to change them here as well.
.Range(name, operator, value)
wouldn't make any sense as .Sort()
Error: dynamo: unmarshal int: expected N to be non-nil
using struct with
varname int dynamo:"varname,omitempty"
Dynamo search shows "varname Null : true" when using "Iter.Next(&struct)"
I'm not finding a way to continue without this failing out.
Hello,
I have a struct where i have a numerical representation of various skills, but i get the error (at the bottom). Can you not use map[string]int
as a set???
type Talent struct {
Id string `json:"id"`
Name string `json:"name"`
LinkedInProfile string `json:"linkedin_profile"`
Skills map[string]int `json:"skills" dynamo:",set"`
CreatedAt string `json:"created_at"`
}
dynamo: cannot marshal type map[string]int into a set
I have a table with a nested customized field map like
{
"fields": {
"custom_field_a": {"S": "blah blah blah"},
"custom_field_b": {"S": "blah blah blah"},
}
}
when custom_field_a
's name is reserved, say "string" or "all", I get following exception:
ValidationException: Invalid UpdateExpression: Syntax error; token: "string", near: ".string ="
status code: 400, request id: 33bd85d4-ade8-42c7-86b6-fc688f8692d8
Wondering if anyone has gotten dynamo to work on Dynamodb local
? If so, does anyone have an example of how to get it all setup??
For instance, take following schema:
{
id: 1,
meta: {
distance: 10
}
}
How do I scan/query with a filter: distance > 1
.
Here's what I have tried:
t.Scan().Filter("'meta.distance' > ?", 1)
hi @guregu , I'm trying to store a complex structure into dynamodb. here is an overview of what i'm trying to achieve:
type Event struct {
Events []EventElement `dynamo:"events,set"`
ID string `dynamo:"id"`
}
type EventElement struct {
ID string `dynamo:"string"`
Items []Item `dynamo:"items,set,omitempty"`
}
type Item struct {
ID string `dynamo:"id"`
Price Price `dynamo:"price"`
}
type Price struct {
Amount int `dynamo:"amount"`
Currency string `dynamo:"currency"`
}
You got the idea, i want to store Event
which have many EventElement
s, which have many Item
s... etc.
How can i map such complex structure?
Thank you
For 'async' APIs, add a Wait method that blocks until e.g. the table's status is active.
will involve some annoying type inference
It would be beneficial to be able to trace DynamoDB calls with X-Ray, as it provides a hell lot of useful insights into how your code is performing. We're using X-Ray with newly announced Go support for Lambda's, and with this library hiding DynamoDB client initialization it's difficult to wrap it up easily without modifying the code.
My suggestion would be to either made a different implementation to initialize the client or to provide a constructor that will take initialized DynamoDB client as a parameter (maybe there is one, but I couldn't find it)
I came into a scenario where I had an object that contained a property for which I was trying to unmarshal (dynamoDB -> struct) into an incorrect type. In my case, in the dynamoDB table, it was of type map (fat-fingered data entry earlier), but was defined as a slice within the code.
This was fairly hard to track down because of the error message I received from decode.go:
dynamo: unmarshal slice: B, L, BS, SS, NS are nil
I see where this is coming from within the decode file; is there anyway to improve this error, as to be more explicit? I feel like these mismatched types is a fairly common error.
Implement the UpdateTimeToLive API: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTimeToLive.html
Hi, how I can use this library to throw an error if item does not exist?
Thanks in advance.
Is there a way to avoid having a error if table already exists ?
"ResourceInUseException: Cannot create preexisting table\n\tstatus code: 400, request id: 5cffe283-a803-4b6b-a0b8-220a30778498"
Thank you
I have a situation where I need to put a Hash only GSI on a column, and also a GSI with the same column as a range key eg:
type Record struct {
OwnerID string dynamo:"o,hash"
ID string dynamo:"i,range" index:"IDX-id,hash" index:"IDX-pd,range"
PD string dymamo:"pd" index:"IDX-pd,hash"
}
However the tags in golang seem to collide when you have multiples of the same name, so the hash on IDX-id gets clobbered. If this is in fact what's happening, exposing the index apis seems like a good short term, and exposing a multi-index syntax seems like a good long term.
Let me know if I'm doing this wrong.
Make table.Put(...)
more like table.Update(...)
Hello,
How to use this library for pagination? I want to set limit and offset plus if it is possible to order the results by some attribute.
Thanks in advance.
Hi, how can I use this library to expose the lastEvaluatedKey to the client and let the client incrementally paginate through? thanks a lot for your help
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.