rustyconover / amazon-dynamodb Goto Github PK

Having an an issue with scan using LastEvaluatedKey with keys of type N when passed as a variable to the scan function.

E.g.
$lek_id = 10;
then scan:
ExclusiveStartKey => {
id => $lek_id
}
fails for me with because the svref_2object check in _encode_type_and_value sets id to type S rather than N.
this works but seems like an ugly work-around:

ExclusiveStartKey => {
id => $lek_id+0
}

Hi, I wouldn't be surprised if this is not an issue or I'm being a bit stupid, but I'm having some trouble using global secondary indices and the get_item method (or batch_get_item). According to
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html#GSI.Using the query should specify an IndexName parameter (kind of like 02-tables-gsi.t). Unfortunately, when I look at the code for Amazon::DynamoDB the IndexName parameter is not used in the _make_payload function. If I am missing something, please let me know and excuse my stupidity.

Thanks.

Looks like this module may be a bit behind on API support.

The thing I found is that GetItem now takes a ProjectionExpression which replaces the "legacy" AttributesToGet. Any idea if there were a large round of changes since the 20120810 API, or just minor tweaks? AttributesToGet still works, but ProjectionExpression is more powerful and it would be nice to support.

Also, shouldn't the Items in set_item support hashref values? Right now the type constraint only allows scalar, scalar ref, and array ref values via the AttributeValueType type. I wonder if this is also a new API feature, I don't know how recently DynamoDB added complex data types.

Amazon::DynamoDB uses Kavorka, which fails on perl 5.22 currently. Would you be willing to accept a patch that rewrites the kavorka stuff in pure perl or some more stable modules?

Seems to me that directly depending on Mojo is an unnecessary requirement. Since I don't use Mojo, I really don't want to be installing that giant package everywhere I use Amazon::DynamoDB.

Perhaps move Amazon::DynamoDB::MojoUA to a separate distribution?

I'm just trying to get the test suite running and several tests are failing. I'm fixing a couple which I'll submit as a pull request, but here's the ones that I'm hoping to not have to fix myself:

perl -Ilib t/03-simple-get-put.t
1..84
ok 1 - Create request was completed
ok 2 - Created table is ready
ok 3 - put_item completed successfully
ok 4 - Results of put_item with no attributes returned didn't return any.
ok 5 - put_item completed successfully
ok 6 - Results of put_item with no attributes returned didn't return any.
ok 7 - get_item completed ok
ok 8 - an item was retrieved
ok 9 - Email matches
ok 10 - Name matches
Can't use string ("Rusty Conover") as a SCALAR ref while "strict refs" in use at t/03-simple-get-put.t line 83.
# Looks like you planned 84 tests but ran 10.
# Looks like your test exited with 255 just after 10.

perl -Ilib t/04-batch-get.t
1..4869
ok 1 - Create request was completed
ok 2 - Created table is ready
ok 3 - Saved test item
ok 4 - Table name matches for batch get
ok 5 - Retrieved test record successfully
ok 6 - Batch get was successfully completed
ok 7 - Saved test item - 0 of 800
ok 8 - Saved test item - 1 of 800
ok 9 - Saved test item - 2 of 800
...
ok 805 - Saved test item - 798 of 800
ok 806 - Saved test item - 799 of 800
ok 807 - Saved test item - 800 of 800
Unknown named parameter: Limit at t/04-batch-get.t line 100.
# Looks like you planned 4869 tests but ran 807.
# Looks like your test exited with 255 just after 807.

perl -Ilib t/06-scan.t
1..40
ok 1 - Create request was completed
ok 2 - Created table is ready
ok 3 - Batch write item successfully completed with 10 written
ok 4 - Scan for count completed
ok 5 - Scan count is right
ok 6 - Scanned count is right
ok 7 - Scan for count completed
ok 8 - Scan count is right
ok 9 - Scanned count is right (should be all records)
ok 10 - Scan completed successfully.
not ok 11 - Equal number of items retrieved from table as were put
#   Failed test 'Equal number of items retrieved from table as were put'
#   at t/06-scan.t line 102.
#          got: '3'
#     expected: '10'
Bail out!  Test failed.  BAIL OUT!.

in my case pagination result is wrong.. it is fetching data from table , lets take limit as 3 then it will return 3 items and on that it will apply filter so items are less than 3.. but my page size is 3.. pagination should be apply on filtered data

Please add [MetaJSON] to dist.ini to populate a META.json file in the distribution... modern toolchain code is very sad having to work with a 1.4 meta file when 2.0 is available (e.g. which separates build and test prereqs into separate phases). thanks!

We use STS to get temporary session tokens for added security. I couldn't find a way to use the STS tokens with Amazon::DynamoDB directly, so I modified lib/Amazon/DynamoDB/20120810.pm by adding a couple of lines that allow a 'session_token' parameter to be passed:

 892     if ($self->{use_iam_role}) {
 893         my $creds = VM::EC2::Security::CredentialCache->get();
 894         defined($creds) || die("Unable to retrieve IAM role credentials");
 895         $self->{access_key} = $creds->accessKeyId;
 896         $self->{secret_key} = $creds->secretAccessKey;
 897         $req->header('x-amz-security-token' => $creds->sessionToken);
 898     }
 899     elsif ($self->{session_token}) {
 900         $req->header('x-amz-security-token' => $self->{'session_token'});
 901     }

I wonder if this change could, perhaps, be included in the official release of Amazon::DynamoDB

Thanks,
-- Robert Watkins

NB: I tried to submit a pull request but my evident lack of Git knowledge got me a 403. In case it helps, here's the history of my attempt:

$ git clone https://github.com/rustyconover/Amazon-DynamoDB.git Amazon-DynamoDB

$ cd Amazon-DynamoDB/

$ git checkout -b sts
Switched to a new branch 'sts'

$ vi Amazon-DynamoDB/lib/Amazon/DynamoDB/20120810.pm

$ git status
On branch sts
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

modified:   lib/Amazon/DynamoDB/20120810.pm

no changes added to commit (use "git add" and/or "git commit -a")

$ git commit -a -m 'added session-token parameter to allow token refresh using token from STS'
[sts fa53e81] added session-token parameter to allow token refresh using token from STS
1 file changed, 3 insertions(+)

$ git push origin sts
remote: Permission to rustyconover/Amazon-DynamoDB.git denied to rwatkinsbc.
fatal: unable to access 'https://github.com/rustyconover/Amazon-DynamoDB.git/': The requested URL returned error: 403

Looks like 0.35 was made from a unpushed commit 09f5f3360f685a71f9f2a4fb7e8f2e2d6e6cc6db and tags have not been pushed since 0.26.

I have a lambda function listening to Dynamodb streams, If i create an article like this

{

  "cities": {
    "cityArr": [
      {
        "city-01": "single category"
      }
    ]
  },
  "id": "article-cffb0d57a1314e718e85f23f4d3791a7",
  "tags": {
    "tagsArr": [
      {
        "category-b1a9ce9206064186bd9a56d11e81d2a6": "burger"
      }
    ]
  }
}

Lambda which captures the events prints newImage like this in TOKYO Region

{
    "cities": {
        "M": {
            "cityArr": {
                "L": [
                    {
                        "M": {
                            "city-01": {
                                "S": "single category"
                            }
                        }
                    }
                ]
            }
        }
    },
    "tags": {
        "M": {
            "tagsArr": {
                "L": [
                    {
                        "M": {
                            "category-b1a9ce9206064186bd9a56d11e81d2a6": {
                                "S": "burger"
                            }
                        }
                    }
                ]
            }
        }
    },
    "id": {
        "S": "article-cffb0d57a1314e718e85f23f4d3791a7"
    }
}

Now If i save same item in SINGAPORE Region newImage is like this

{
    "cities": {
        "M": {
            "cityArr": {
                "M": {
                    "city-01": {
                        "S": "single category"
                    }
                }
            }
        }
    },
    "tags": {
        "M": {
            "tagsArr": {
                "M": {
                    "category-b1a9ce9206064186bd9a56d11e81d2a6": {
                        "S": "burger"
                    }
                }
            }
        }
    },
    "id": {
        "S": "article-cffb0d57a1314e718e85f23f4d3791a7"
    }
}

It should come inside an array 'L' key{array} but it is converting to object, If i put more than two objects inside my array it works fine. Its happening in Singapore Region

Stop the iteration early.

Trying to call put_item on an item with an undef value, e.g.:

$ddb->put_item(
   TableName => $table_name,
   Item => {
     name => 'Test Name'
     email => undef,
   },
   ReturnValues => 'ALL_OLD');

throws the error:

...did not pass type constraint "ItemType"...
    "ItemType" is a subtype of "Map[AttributeNameType,AttributeValueType]"
    Reference {"__CLASS__" => "CE::LeadRequestTransaction::State","_comple...} did not pass type constraint "Map[AttributeNameType,AttributeValueType]"
    "Map[AttributeNameType,AttributeValueType]" constrains each value in the hash with "AttributeValueType"
    "AttributeValueType" is a subtype of "Defined"
    Undef did not pass type constraint "Defined" (in $_->{"clickouts_only"})
    "Defined" is defined as: (defined($_))

You should be able to store these values using DynamoDB's NULL data type:

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html