cognitect-labs / aws-api Goto Github PK
View Code? Open in Web Editor NEWAWS, data driven
License: Apache License 2.0
aws-api's People
Stargazers
Watchers
Forkers
timotheosh darongmean clojure-pirates extremenelson zz rastandy dept24c hkrishnan xtech2k dfcarpenter yummly unicornlaunching jcolechanged levand gzmask janag tramzee tank157 whitepages randylien clojure-land kigiri laingit davezen1 fedreg csm willmruzek claj khmelevskii mkurtak rttomlinson makinoshi jeroenvandijk ezand arohner computesoftware zeniten youplemob nikaxi eoliphan griffinbank sergey-latacora agstubbs lgouger newbision galdolber djblue lukaszkorecki korrawit450 kimchhickey 00mjk kenji holocene-systems agentfunk legkvla fierycod karlwilliamstide ywsfay skuttleman stuartrexking grzm thomaswhitcomb dancmeyers mtwomey chance-latacora kmbotts davidyang digash prepor waffletower-reify singhabhishek4u owen241 the-alchemist mitep arraykis thayannevls lowecg helins practicalli-johnny jonaskriks samkenxstream borisabramovich86 burbma diegogub jmglov kevin-ewing green-labs wevre rinconjc insilicaaws-api's Issues
Signature error when attempting to use STS
Hello, I'm getting the following signature error
{:ErrorResponse {:Error {:Type "Sender",
:Code "SignatureDoesNotMatch",
:Message "Credential should be scoped to a valid region, not 'ap-southeast-2'. "}},
:ErrorResponseAttrs {:xmlns "https://sts.amazonaws.com/doc/2011-06-15/"},
:cognitect.anomalies/category :cognitect.anomalies/forbidden}When calling
(aws/invoke sts {:op :AssumeRole :request {:RoleArn arn :RoleSessionName "REPL"}})For reference other APIs work fine
(aws/invoke ec2 {:op :DescribeInstances})Deps
{:deps {org.clojure/clojure {:mvn/version "1.10.0"}
com.cognitect.aws/api {:mvn/version "0.8.158"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.467"}
com.cognitect.aws/sts {:mvn/version "669.2.364.0"}
com.cognitect.aws/ec2 {:mvn/version "681.2.373.0"}}}prefer aws methods as functions
This is great, but I'd like to be able to import a namespace of aws service functions. Playing with ssm, I ended up writing code like this to get there:
(def ssm-client (aws/client {:api :ssm}))
(def ssm-operation (partial aws/invoke ssm-client))
(defn ssm [op request]
(ssm-operation {:op op :request request}))
(def ssm-send-command (partial ssm :SendCommand))
(def ssm-list-commands (partial ssm :ListCommands))
(def ssm-list-command-invocations (partial ssm :ListCommandInvocations))
(def ssm-get-command-invocation (partial ssm :GetCommandInvocation))
This gives me functions that take the :request value as argument and allow my written-for-Amazonica pagination helper functions to work.*
What do you think? Lots of room for helper functions unwrapping nested paginated results, too.
Today I wrote into a let..
responses (into [] (flatten (for [p response-pages] (for [r (:CommandInvocations p)] r))))
ec2 reports "could not find operation"
;; com.cognitect.aws/api {:mvn/version "0.8.99"}
;; com.cognitect.aws/endpoints {:mvn/version "1.1.11.451"}
;; com.cognitect.aws/ec2 {:mvn/version "657.2.361.0"}
(def client (aws/client {:api :ec2}))
(aws/invoke client {:op :DescribeInstances
:request
{:MaxResults 10}})
{:Response {:Errors {:Error {:Code "InvalidAction", :Message "The action :DescribeInstances is not valid for this web service."}}, :RequestID "ccb54665-08a1-41a6-a908-52962cd5a2a0"}, :cognitect.anomalies/category :cognitect.anomalies/incorrect} ```
Handling integers in request params
From the (aws/ops) output, I expected :FromPort and :ToPort to require a number, but I got a casting error: “java.lang.Long cannot be cast to java.lang.String”.
(select-keys (get-in (aws/ops @ec2-client)
[:AuthorizeSecurityGroupIngress :request])
[:FromPort :ToPort])
{:FromPort integer, :ToPort integer}(aws/invoke @ec2-client {:op :AuthorizeSecurityGroupIngress
:request {:CidrIp cidr-ip
:FromPort 22
:GroupId security-group-id
:IpProtocol "tcp"
:ToPort 22}})Passing in strings works fine:
(aws/invoke @ec2-client {:op :AuthorizeSecurityGroupIngress
:request {:CidrIp cidr-ip
:FromPort "22"
:GroupId security-group-id
:IpProtocol "tcp"
:ToPort "22"}})I’m using the following versions:
[com.cognitect.aws/api "0.8.149"]
[com.cognitect.aws/endpoints "1.1.11.462"]
[com.cognitect.aws/ec2 "681.2.373.0"]
[com.cognitect.aws/ecs "668.2.364.0"]Should we always pass in strings as request parameters?
resourcegroupstaggingapi reports "nodename nor servname provided"
;; com.cognitect.aws/api {:mvn/version "0.8.99"}
;; com.cognitect.aws/endpoints {:mvn/version "1.1.11.451"}
;; com.cognitect.aws/resourcegroupstaggingapi {:mvn/version "657.2.352.0"}
(def client (aws/client {:api :resourcegroupstaggingapi}))
(aws/invoke client {:op :GetResources
:request
{:limit 100
:TagFilters [{"datomic:system"
["devdata"]}]}})
=> {:cognitect.anomalies/category :cognitect.anomalies/not-found, :cognitect.anomalies/message "resourcegroupstaggingapi.us-east-1.amazonaws.com: nodename nor servname provided, or not known", :cognitect.http-client/meta {:op :GetResources, :request {:limit 100, :TagFilters [{"datomic:system" ["devdata"]}]}}}
Using local endpoints for testing
A question, and possibly a feature request:
tl;dr Is there a "correct" way to substitute a local endpoint for an AWS hosted endpoint?
I work on applications which make use of several AWS products - for our CI tests, we use docker-compose with services which are (for our purposes) functionally identical behaviours. The SDKs we currently use all allow us to configure endpoint URLs directly. I cannot see an obvious hook for this here. I looked at first at simply shadowing the endpoints.edn file, but it isn't obvious to me where that comes from, or lives, or is supposed to look like. So before pursuing that line, I figured I find out if there's a simpler way.
If this is not currently a supported use case, I would like to request it be added.
Thanks for your work on this @dchelimsky
Illegal reflective access warnings when doing through the default steps in readme with java11
Environment Ubuntu 18.10 box:
$ uname -a
Linux martin-nitro 4.18.0-12-generic #13-Ubuntu SMP Wed Nov 14 15:17:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Ubuntu-2ubuntu1)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Ubuntu-2ubuntu1, mixed mode, sharing)
$ lein --version
Leiningen 2.8.1 on Java 11.0.1 OpenJDK 64-Bit Server VM
#obviously keeping dummy usernames in default aws config for start
$ cat ~/.aws/credentials
[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
region=eu-west-1
Steps followed :
Creating a new app with leiningen
lein new app aws-api-testbed
Modifying only the dependencies in project file to contain the latest versions, after edit looks like this
$ cat project.clj
(defproject aws-api-testbed "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [
[org.clojure/clojure "1.10.0-RC4"]
[org.clojure/core.async "0.4.490"]
[com.cognitect.aws/api "0.8.122"]
[com.cognitect.aws/endpoints "1.1.11.462"]
[com.cognitect.aws/s3 "680.2.370.0"]
]
:main ^:skip-aot aws-api-testbed.core
:target-path "target/%s"
:profiles {:uberjar {:aot :all}})
Now executing with lein repl:
$ lein repl
nREPL server started on port 43883 on host 127.0.0.1 - nrepl://127.0.0.1:43883
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.10.0-RC4
OpenJDK 64-Bit Server VM 11.0.1+13-Ubuntu-2ubuntu1
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
aws-api-testbed.core=> (require '[cognitect.aws.client.api :as aws])
nil
aws-api-testbed.core=> (def s3-client (aws/client {:api :s3}))
2018-12-08 18:27:08.035:INFO::async-dispatch-1: Logging initialized @12589ms
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.eclipse.jetty.util.BufferUtil (file:/home/martin/.m2/repository/org/eclipse/jetty/jetty-util/9.3.7.v20160115/jetty-util-9.3.7.v20160115.jar) to field java.nio.MappedByteBuffer.fd
WARNING: Please consider reporting this to the maintainers of org.eclipse.jetty.util.BufferUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Syntax error (ExceptionInfo) compiling at (form-init14772803263696788523.clj:1:16).
No region found by any region provider.
aws-api-testbed.core=>
Just maybe the final error has some reason for the illegal reflective access ?
Creating and deleting a Kinesis stream returns an error despite success
Despite receiving an error after invoking this, the stream has actually been created. The same is seen when invoking :DeleteStream.
(aws/invoke kc {:op :CreateStream
:request {:StreamName stream-name
:ShardCount 1}})
#> {:cognitect.anomalies/category :cogniect.anomalies/fault, :cognitect.aws.client/throwable #error {
:cause nil
:via
[{:type java.lang.NullPointerException
:message nil
:at [java.io.StringReader <init> "StringReader.java" 50]}]
:trace
[[java.io.StringReader <init> "StringReader.java" 50]
[clojure.data.json$read_str invokeStatic "json.clj" 278]
[clojure.data.json$read_str doInvoke "json.clj" 274]
[clojure.lang.RestFn invoke "RestFn.java" 439]
[cognitect.aws.shape$json_parse invokeStatic "shape.clj" 117]
[cognitect.aws.shape$json_parse invoke "shape.clj" 114]
[cognitect.aws.protocols.json$eval15788$fn__15791 invoke "json.clj" 48]
[clojure.lang.MultiFn invoke "MultiFn.java" 239]
[cognitect.aws.client$handle_http_response invokeStatic "client.clj" 37]
[cognitect.aws.client$handle_http_response invoke "client.clj" 31]
[cognitect.aws.client$send_request$fn__13875 invoke "client.clj" 56]
[clojure.core$map$fn__5847$fn__5848 invoke "core.clj" 2742]
[clojure.core.async.impl.channels$chan$fn__1496 invoke "channels.clj" 300]
[clojure.core.async.impl.channels.ManyToManyChannel put_BANG_ "channels.clj" 83]
[clojure.core.async$put_BANG_ invokeStatic "async.clj" 165]
[clojure.core.async$put_BANG_ invoke "async.clj" 158]
[cognitect.http_client.Client$fn$reify__13176 onComplete "http_client.clj" 236]
[org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 193]
[org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 185]
[org.eclipse.jetty.client.HttpReceiver terminateResponse "HttpReceiver.java" 454]
[org.eclipse.jetty.client.HttpReceiver responseSuccess "HttpReceiver.java" 401]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP messageComplete "HttpReceiverOverHTTP.java" 268]
[org.eclipse.jetty.http.HttpParser parseHeaders "HttpParser.java" 957]
[org.eclipse.jetty.http.HttpParser parseNext "HttpParser.java" 1188]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP parse "HttpReceiverOverHTTP.java" 158]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP process "HttpReceiverOverHTTP.java" 119]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP receive "HttpReceiverOverHTTP.java" 69]
[org.eclipse.jetty.client.http.HttpChannelOverHTTP receive "HttpChannelOverHTTP.java" 90]
[org.eclipse.jetty.client.http.HttpConnectionOverHTTP onFillable "HttpConnectionOverHTTP.java" 113]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded "AbstractConnection.java" 273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.ssl.SslConnection onFillable "SslConnection.java" 197]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded "AbstractConnection.java" 273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.SelectChannelEndPoint$2 run "SelectChannelEndPoint.java" 75]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume produceAndRun "ExecuteProduceConsume.java" 213]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume run "ExecuteProduceConsume.java" 147]
[org.eclipse.jetty.util.thread.QueuedThreadPool runJob "QueuedThreadPool.java" 654]
[org.eclipse.jetty.util.thread.QueuedThreadPool$3 run "QueuedThreadPool.java" 572]
[java.lang.Thread run "Thread.java" 748]]}}```
Missing a consistent way to indicate errors
There is no consistent way to check for errors in aws-api responses. There is an :Errors key on some, but not all operations. There is a :cognitect.anomalies/category key whenever (>= status 400), but some ops return 200 even when there are errors. Specs for most ops include a reference to a description of errors in the specs, but about 10% don't, and of those that do, about 90% don't refer to shapes that are actually described.
Container credentials provider doesn't discover ECS Fargate creds
aws-api/src/cognitect/aws/credentials.clj
Lines 220 to 231 in c43cc45
ECS should pull credentials from a different endpoint than EC2 (169.254.170.2 vs 169.254.169.254 respectively)
Deserialize HAL responses
Seems that API Gateway responses are not deserialized currently. I don't know the proper place/way to do deserialization, but here might be a pointer do get the embedded data of the HAL response: https://github.com/portkey-cloud/aws-clj-sdk/blob/master/src/portkey/aws.clj#L182-L197.
Invalid 'Location' header: null
; :op :ListBuckets worked fine, when I went to use :ListObjects I obtained the following;
(aws/invoke s3-client {:op :ListObjects :request {:Bucket "bucket-name"}})
=>
{:cognitect.anomalies/category :cognitect.anomalies/fault,
:cognitect.anomalies/message "Invalid 'Location' header: null",
:cognitect.http-client/throwable #error{:cause "Invalid 'Location' header: null",
:via [{:type org.eclipse.jetty.client.HttpResponseException,
:message "Invalid 'Location' header: null",
:at [org.eclipse.jetty.client.HttpRedirector
redirect
"HttpRedirector.java"
166]}],
:trace [[org.eclipse.jetty.client.HttpRedirector
redirect
"HttpRedirector.java"
166]
[org.eclipse.jetty.client.RedirectProtocolHandler
onComplete
"RedirectProtocolHandler.java"
63]
[org.eclipse.jetty.client.ResponseNotifier
notifyComplete
"ResponseNotifier.java"
193]
[org.eclipse.jetty.client.ResponseNotifier
notifyComplete
"ResponseNotifier.java"
185]
[org.eclipse.jetty.client.HttpReceiver
terminateResponse
"HttpReceiver.java"
454]
[org.eclipse.jetty.client.HttpReceiver
responseSuccess
"HttpReceiver.java"
401]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
messageComplete
"HttpReceiverOverHTTP.java"
268]
[org.eclipse.jetty.http.HttpParser parseContent "HttpParser.java" 1470]
[org.eclipse.jetty.http.HttpParser parseNext "HttpParser.java" 1203]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
parse
"HttpReceiverOverHTTP.java"
158]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
process
"HttpReceiverOverHTTP.java"
119]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
receive
"HttpReceiverOverHTTP.java"
69]
[org.eclipse.jetty.client.http.HttpChannelOverHTTP
receive
"HttpChannelOverHTTP.java"
90]
[org.eclipse.jetty.client.http.HttpConnectionOverHTTP
onFillable
"HttpConnectionOverHTTP.java"
113]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback
succeeded
"AbstractConnection.java"
273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.ssl.SslConnection
onFillable
"SslConnection.java"
197]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback
succeeded
"AbstractConnection.java"
273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.SelectChannelEndPoint$2
run
"SelectChannelEndPoint.java"
75]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
produceAndRun
"ExecuteProduceConsume.java"
213]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
run
"ExecuteProduceConsume.java"
147]
[org.eclipse.jetty.util.thread.QueuedThreadPool
runJob
"QueuedThreadPool.java"
654]
[org.eclipse.jetty.util.thread.QueuedThreadPool$3
run
"QueuedThreadPool.java"
572]
[java.lang.Thread run "Thread.java" 748]]},
:cognitect.http-client/meta {:op :ListObjects, :request {:Bucket "bucket-name"}}}
Unable to decode KMS encrypted value stored as a string
(require '[cognitect.aws.client.api :as aws])
(def kms (aws/client {:api :kms}))
(aws/validate-requests kms true) ; this really should be in the README while we're here!
(def master-key "your-aws-kms-key")
; works fine, outputs "hi"
(def ctb (:CiphertextBlob (aws/invoke kms {:op :Encrypt
:request {:KeyId master-key
:Plaintext (.getBytes "hi")}})))
(slurp (:Plaintext (aws/invoke kms {:op :Decrypt
:request {:CiphertextBlob ctb}})))
; However, this doesn't work, giving a `{"__type" "InvalidCiphertextException", :cognitect.anomalies/category :cognitect.anomalies/incorrect}` error:
(def bla (-> (aws/invoke kms {:op :Encrypt
:request {:KeyId master-key
:Plaintext (.getBytes "hi")}})
:CiphertextBlob
slurp))
(aws/invoke kms {:op :Decrypt
:request {:CiphertextBlob (io/input-stream (.getBytes bla))}})README claims Clojure >= 1.9.0 is required, but it's actually Clojure >= 1.10-beta2
Datafiable was introduced in 1.10-beta2. I guess once 1.10 is released it'll be easier to just say that :)
make com.cognitect/http-client source available
Hi,
First of all: thanks for the project.
One of the deps listed is available on maven and seems to be Apache 2 licensed but the sources are not available anywhere as far as I can see. Would it be possible to publish these?
https://search.maven.org/artifact/com.cognitect/http-client/0.1.87/jar
Crash when date format is Double
Hi all.
Im working with this lib for cognito-idp and I have encountered an exception when parsing the aws response:
java.lang.Double cannot be cast to java.lang.CharSequence
[clojure.core$re_matcher invokeStatic "core.clj" 4849]
[clojure.core$re_matches invokeStatic "core.clj" 4886]
[clojure.core$re_matches invoke "core.clj" 4886]
[cognitect.aws.shape$parse_date invokeStatic "shape.clj" 86]
[cognitect.aws.shape$parse_date invoke "shape.clj" 79]
[cognitect.aws.shape$eval26825$fn__26826 invoke "shape.clj" 150]
[clojure.lang.MultiFn invoke "MultiFn.java" 234]
After some debugging I saw that the response of the aws request looks like this:
;;Requesting a list group of a user pool, This throws the error above
(aws/invoke cognito {:op :ListGroups :request {:UserPoolId "my-user-pool-id"}})
;; Added this for debugging
(def http-meta (slurp (:body (:http-response (meta (aws/invoke cognito {:op :ListGroups :request {:UserPoolId "my-user-pool-id"}}))))))
(def response (json/parse-string http-meta true))
;; the response looks like:
{:Groups
[{:CreationDate 1.547453349895E9,
:Description "My App",
:GroupName "Group 1",
:LastModifiedDate 1.547453349895E9,
:UserPoolId "my-user-pool-id"}]}
As you can see the format of CreationDate and LastModifiedDate is double
this causes to crash in the shape/parse-data.
adding a (double? data) (java.util.Date. (* 1000 (long data))) is fixing the issue
if more info is needed let me know,
Thanks ahead
Latest realease link broken?
The readme has a link to the following and gives me a 404 back https://github.com/cognitect-labs/aws-api/blob/master/doc/latest-releases.md
Support response streaming
A couple of people have asked me if/when we'll support streaming for blob types. We do accept and return InputStreams, but the underlying http-client does not.
:MessageSystemAttributeName spec uses strings, and the result is keywords
(s/explain :cognitect.aws.sqs/ReceiveMessageResult
(aws/invoke sqs {:op :ReceiveMessage
:request (merge queue-url {:AttributeNames ["All"]})}))Fails spec with:
In: [:Messages 0 :Attributes :SenderId 0] val: :SenderId fails spec: :cognitect.aws.sqs/MessageSystemAttributeName at: [:Messages :Attributes 0] predicate: #{"SentTimestamp" "SenderId" "SequenceNumber" "ApproximateReceiveCount" "MessageGroupId" "ApproximateFirstReceiveTimestamp" "MessageDeduplicationId"}
In: [:Messages 0 :Attributes :ApproximateFirstReceiveTimestamp 0] val: :ApproximateFirstReceiveTimestamp fails spec: :cognitect.aws.sqs/MessageSystemAttributeName at: [:Messages :Attributes 0] predicate: #{"SentTimestamp" "SenderId" "SequenceNumber" "ApproximateReceiveCount" "MessageGroupId" "ApproximateFirstReceiveTimestamp" "MessageDeduplicationId"}
In: [:Messages 0 :Attributes :ApproximateReceiveCount 0] val: :ApproximateReceiveCount fails spec: :cognitect.aws.sqs/MessageSystemAttributeName at: [:Messages :Attributes 0] predicate: #{"SentTimestamp" "SenderId" "SequenceNumber" "ApproximateReceiveCount" "MessageGroupId" "ApproximateFirstReceiveTimestamp" "MessageDeduplicationId"}
In: [:Messages 0 :Attributes :SentTimestamp 0] val: :SentTimestamp fails spec: :cognitect.aws.sqs/MessageSystemAttributeName at: [:Messages :Attributes 0] predicate: #{"SentTimestamp" "SenderId" "SequenceNumber" "ApproximateReceiveCount" "MessageGroupId" "ApproximateFirstReceiveTimestamp" "MessageDeduplicationId"}The received message has this structure:
{:Messages
[{:MessageId "xxx",
:ReceiptHandle "xxx",
:MD5OfBody "0a997fa7fd98d3850f5ce76c943afe7c",
:Body "xxx",
:Attributes
{:SenderId "xxx",
:ApproximateFirstReceiveTimestamp "1548363122352",
:ApproximateReceiveCount "11",
:SentTimestamp "1548363122352"}}]}Dynamodb transaction API methods missing
The dynamodb client is missing the new Transact methods that were added recently:
https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-transactions/
I'm assuming this is due to needing to re-run the generators, however, I wanted to create an issue since there are actually 2 versions of the dynamodb descriptors: 2011-12-05 and 2012-08-10 - it doesn't look like the version with the Transaction methods is getting generated.
I want to be able to require every AWS API
Right now, with Amazonica, I do this by depending on com.amazonaws/aws-java-sdk, which pulls everything in. aws-api does not appear to have a similar artifact.
The second problem is being able to load them. Right now, with amazonica, I do this by walking the classpath and looking for namespaces. You probably don't want in this library itself, but just to make sure that's something that third parties can do, each maven artifact for a service must define a namespace that has some kind of predictable name., Looks like that's going to be cognitect.aws.some-service.specs? And I can count on (keyword :some-service) being a valid name I can use to create a client?
Documentation / Guidance on handling errors
I'm currently doing something like this:
(defn invoke
"Invokes the AWS request, converting any Error's into exceptions."
[client req]
(let [response (aws/invoke client req)]
(if (:Error response)
(throw (ex-info "Error invoking AWS service" response)) ;; if you want an exception out
response)))I'm assuming checking the existence of the :Error key is the right thing to do, but I've not seen it documented anywhere, so it would be nice to know that it can be consistently relied on for every request.
Invoke on cloudsearchdomain service gets SignatureDoesNotMatch error
Steps to reproduce:
(def cs (aws/client {:api :cloudsearch}))
(def domain-name "<insert domain here>")
(def domain (-> cs
(aws/invoke {:op :DescribeDomains :request {:DomainNames [domain-name]}})
:DomainStatusList
first))
(def search-endpoint (-> domain :SearchService :Endpoint))
(def search-client (aws/client {:api :cloudsearchdomain :endpoint-override search-endpoint}))
> (aws/invoke search-client {:op :Search :request {:query "*:*" :queryParser "lucene" :size 1}})
{"__type" "#SignatureDoesNotMatch",
"error"
{"message"
"[*Deprecated*: Use the outer message field] The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."},
"message"
"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.",
:cognitect.anomalies/category :cognitect.anomalies/forbidden}
Per suggestion from @Ghadi on Slack, I added explicit :region values to the client calls with no effect.
endpoint resolution for appsync data plane needs special casing
Per https://docs.aws.amazon.com/general/latest/gr/rande.html, the endpoint format for AppSync Data Plane is <unique-id>.appsync-api.<region>.amazonaws.com. We have no support for prefixing with the unique-id.
Add support for CLJS
[S3] :DeleteObjects Missing required header Content-MD5
When calling :DeleteObjects against S3:
(aws/invoke client {:op :DeleteObjects, :request {:Delete {:Objects [{:Key "annual/my-file"}]}, :Bucket "test-bucket"}}
I get the following error:
1. Unhandled clojure.lang.ExceptionInfo
Error invoking AWS service
{:Error
{:Code "InvalidRequest",
:Message "Missing required header for this request: Content-MD5",
:RequestId "63DDF59BD7350E9C",
:HostId
"BjCzOVMPQIw912la2lnv/FuJE64YeP2nbFtwx58ei9KKFEfIa2kv96JkqAS0vIdw6LaT3anuLAk="},
:cognitect.anomalies/category :cognitect.anomalies/incorrect}
s3.clj: 36 data-loader.concerns.s3/invoke
s3.clj: 31 data-loader.concerns.s3/invoke
REPL: 87 data-loader.pipelines.annual.annual-test/eval33563
REPL: 85 data-loader.pipelines.annual.annual-test/eval33563
Compiler.java: 7176 clojure.lang.Compiler/eval
Compiler.java: 7131 clojure.lang.Compiler/eval
core.clj: 3214 clojure.core/eval
core.clj: 3210 clojure.core/eval
main.clj: 414 clojure.main/repl/read-eval-print/fn
main.clj: 414 clojure.main/repl/read-eval-print
main.clj: 435 clojure.main/repl/fn
main.clj: 435 clojure.main/repl
main.clj: 345 clojure.main/repl
RestFn.java: 137 clojure.lang.RestFn/applyTo
core.clj: 665 clojure.core/apply
core.clj: 660 clojure.core/apply
regrow.clj: 18 refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 83 nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 665 clojure.core/apply
core.clj: 1973 clojure.core/with-bindings*
core.clj: 1973 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 81 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 50 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 221 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
interruptible_eval.clj: 189 nrepl.middleware.interruptible-eval/run-next/fn
AFn.java: 22 clojure.lang.AFn/run
ThreadPoolExecutor.java: 1149 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 624 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 748 java.lang.Thread/run
DeleteObject and ListBucket both work with my permission set. aws/validate-requests does not report a problem with the request.
Missing region ap-northeast-3 - Asia Pacific (Osaka-Local) in endpoints
The endpoints.edn is missing info on the ap-northeast-3 region.
Add support for the new WebSocket features in API Gateway
Amazon recently announced WebSocket support for API Gateway. It looks to be a two part update according to the aws-sdk-js descriptor files:
apigatewayv2-2018-11-29apigatewaymanagementapi-2018-11-29
I'm eager to get a conversation going on how this could be incorporated into the aws-api. WebSocket + Datomic Ions could be a very powerful architecture.
Potential streams-dynamodb vs dynamodb signature issue
Example code that fails:
(comment
(def ddbs-client (aws/client {:api :streams-dynamodb
:region :eu-west-1}))
(aws/invoke ddbs-client {:op :DescribeStream
:request {:StreamArn example-arn}})
;; => {"__type" "com.amazon.coral.service#InvalidSignatureException",
;; "message" "Credential should be scoped to correct service: 'dynamodb'. ",
;; :cognitect.anomalies/category :cognitect.anomalies/incorrect}
)
Environment:
{:deps {org.clojure/clojure {:mvn/version "1.10.0-RC2"}
org.clojure/core.async {:mvn/version "0.4.490"}
com.cognitect.aws/api {:mvn/version "0.8.155"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.462"}
com.cognitect.aws/kinesis {:mvn/version "668.2.357.0"}
com.cognitect.aws/dynamodb {:mvn/version "675.2.365.0"}
com.cognitect.aws/streams-dynamodb {:mvn/version "669.2.364.0"}}}
Note: I'm using credentials for default profile from my .aws/credentials which I have confirmed are working for other API usage e.g
(comment
(def ddb-client (aws/client {:api :dynamodb
:region :eu-west-1}))
(aws/invoke ddb-client {:op :DescribeTable
:request {:TableName :secret-table-name}})
;; Succeeds
)
ApiGateway GetRestApis returns `[]` even when there are results
I have one API defined but I can't get it through aws/client
(def apigw (aws/client {:api :apigateway}))
(aws/invoke apigw {:op :GetRestApis})gives {}
but in
(-> *1
meta
(get-in [:http-response :body])
slurp
cheshire.core/parse-string)I can see my api (shortend and with identifiers replaced)
{"_links" {"curies" [,,,],
"self" {"href" "/restapis"},
"item" {"href" "/restapis/<removed>"},
"restapi:by-id" {"href" "/restapis/{restapi_id}", "templated" true},
"restapi:create" {"href" "/restapis"},
"restapi:put" {"href" "/restapis/{restapi_id}?failonwarnings=false{&mode}", "templated" true}},
"_embedded" {"item" {"_links" {,,,},
"apiKeySource" "HEADER",
"binaryMediaTypes" "*/*",
"createdDate" "2018-12-21T12:19:45Z",
"description" "<removed>",
"endpointConfiguration" {"types" "REGIONAL", "ipv6" false},
"id" "<removed>",
"name" "<removed>"}}}I noticed that the key in "_embedded" is called item and not items as I expected. The value is also an object, and not a list. If I define another API, the value of item turns into an array, but the return from invoke is still {}.
The cli gives me the api
aws apigateway get-rest-apis{
"items": [
{
"apiKeySource": "HEADER",
"description": "<removed>",
"endpointConfiguration": {
"types": [
"REGIONAL"
]
},
"createdDate": 1545394785,
"binaryMediaTypes": [
"*/*"
],
"id": "<removed>",
"name": "<removed>"
}
]
}chained credentials provider is very noisy during fall-through
When using the default chained credentials provider, every failure is logged as an error.
aws-api/src/cognitect/aws/credentials.clj
Line 99 in db3e9e0
I was surprised to see multiple SEVERE: Error fetching credentials from log lines when running from the command line even when the application was running as expected.
Given the purpose of using a chained provider is to carry on and try subsequent providers, I'd argue this is not an error, but rather something less significant. I typically use ERROR-level logging when I want logging to be pretty much silent when behaving as expected. I'd suggest at most log/warn, and perhaps even log/info or log/debug
Support Clojure 1.9
The dependency on Datafiable forces a dependency on Clojure 1.10. Moving that dependency to metadata would support using Clojure 1.9 and still support datafy when using Clojure 1.10.
Invalid config format with ADFS + multiple profiles
user=> (require '[cognitect.aws.client.api :as aws])
nil
user=> (def c (aws/client {:api :rds}))
Jan 24, 2019 8:23:20 AM clojure.tools.logging$eval14992$fn__14995 invoke
SEVERE: Unable to fetch region from the AWS config file /Users/r627543/.aws/config
clojure.lang.ExceptionInfo: Invalid format in config {:file #object[java.io.File 0x3a5244a1 "/Users/r627543/.aws/config"]}
at cognitect.aws.config$parse$fn__15036.invoke(config.clj:85)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
at clojure.core$reduce.invokeStatic(core.clj:6827)
at clojure.core$reduce.invoke(core.clj:6810)
at cognitect.aws.config$parse.invokeStatic(config.clj:69)
at cognitect.aws.config$parse.invoke(config.clj:63)
at cognitect.aws.region$profile_region_provider$reify__15667.fetch(region.clj:96)
at cognitect.aws.region$eval15640$fn__15641$G__15632__15643.invoke(region.clj:22)
at cognitect.aws.region$eval15640$fn__15641$G__15631__15646.invoke(region.clj:22)
at clojure.core$some.invokeStatic(core.clj:2701)
at clojure.core$some.invoke(core.clj:2692)
at cognitect.aws.region$chain_region_provider$reify__15657.fetch(region.clj:41)
at cognitect.aws.client.api$client.invokeStatic(api.clj:59)
at cognitect.aws.client.api$client.invoke(api.clj:29)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
The offending config file:
$ cat ~/.aws/config
[profile dev]
region = eu-central-1
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::767457873974:role/chp-developer-dev
adfs_config.adfs_host = <redacted>
adfs_config.adfs_user = <redacted>
adfs_config.session_duration = 3600
[default]
region = eu-central-1
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::395127396906:role/chp-developer-poc
adfs_config.adfs_host = <redacted>
adfs_config.adfs_user = <redacted>
adfs_config.session_duration = 3600
[profile poc]
region = eu-central-1
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::850374662971:role/chp-developer-uat
adfs_config.adfs_host = <redacted>
adfs_config.adfs_user = <redacted>
adfs_config.session_duration = 3600
[profile uat]
region = us-west-2
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::850374662971:role/chp-developer-uat
adfs_config.adfs_host = <redacted>
adfs_config.adfs_user = None
adfs_config.session_duration = 3600
[profile sit]
region = us-west-2
output = json
adfs_config.ssl_verification = True
adfs_config.role_arn = arn:aws:iam::647507038414:role/chp-developer-sit
adfs_config.adfs_host = <redacted>
adfs_config.adfs_user = None
adfs_config.session_duration = 3600
Decouple generating of the request from sending it
When I've been using java SDK, once I had to issue an AWS request via ssh from the remote host. That was terribly hard to generate and sign that request. And the only reason for doing that is that SDK doesn't expose the HTTP request itself.
So please decouple generating request and response parsing from sending/receiving it from http client.
s3-control.us-east-1.amazonaws.com: nodename nor servname provided, or not known
(aws/invoke c {:op :GetPublicAccessBlock
:AccountId "***"})
{:cognitect.anomalies/category :cognitect.anomalies/not-found,
:cognitect.anomalies/message
"s3-control.us-east-1.amazonaws.com: nodename nor servname provided, or not known",
:cognitect.http-client/meta
{:op :GetPublicAccessBlock, :AccountId "***"}}
[S3] Validated invoke request :PutObject spec failure
Reproduce Undesirable Behavior
(def s3-client (aws/client {:api :s3}))
(aws/validate-requests s3-client true)
(def valid-spec-request-map {:Bucket "any-bucket-here"
:Key "spec-says-ok.txt"
:Body (.getBytes "Looks good to me")})
(def invalid-spec-request-map {:Bucket "any-bucket-here"
:Key "spec-says-no-no.txt"
:Body "I'm hungry feed me bytes"})
(s/explain (aws/request-spec s3-client :PutObject) valid-spec-request-map) ; => Success!
(s/explain (aws/request-spec s3-client :PutObject) invalid-spec-request-map) ; => "I'm hungry feed me bytes" - failed: bytes? in: [:Body] at: [:Body] spec: :cognitect.aws.s3/Body
;; ---
; {:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.aws.client/throwable #error {
; :cause "[B cannot be cast to java.lang.String"
; :via ...
(aws/invoke s3-client {:op :PutObject
:request valid-spec-request-map})
;; ---
; {:clojure.spec.alpha/problems ({:path [:Body],
; :pred clojure.core/bytes?,
; :val "I'm hungry feed me bytes",
; :via [:cognitect.aws.s3/PutObjectRequest :cognitect.aws.s3/Body],
; :in [:Body]}),
; :clojure.spec.alpha/spec :cognitect.aws.s3/PutObjectRequest,
; :clojure.spec.alpha/value {:Bucket "oddity-1541461865",
; :Key "spec-says-no-no.txt",
; :Body "I'm hungry feed me bytes"},
; :cognitect.anomalies/category :cognitect.anomalies/incorrect}
(aws/invoke s3-client {:op :PutObject
:request invalid-spec-request-map})
(aws/validate-requests s3-client false)
;; ---
; Successful request
(aws/invoke s3-client {:op :PutObject
:request invalid-spec-request-map}))Probable Cause
Validation occurs in the invoke function before the transformation. Transformation happens when the request is built prior to sending.
Problem parsing the credentials file
Hello
I have a problem when using your library with my generated session token.
The parse function in the cognitect.aws.config namespace does not seem to parse correctly when the session token ends with an equals sign (=) which is the case with our credentions provider.
Exception on a successful call to :DeleteSecurityGroup
I’ve run into an exception when deleting a security group with the EC2 client:
(aws/invoke @ec2-client {:op :DeleteSecurityGroup
:request {:GroupId "sg-096a9d70a938d6ee5"}})The security group is successfully deleted, but this is the return value I get back:
{:cognitect.anomalies/category :cognitect.anomalies/fault,
:cognitect.aws.client/throwable #error{:cause "No method in multimethod 'xml-parse*' for dispatch value: null",
:via [{:type java.lang.IllegalArgumentException,
:message "No method in multimethod 'xml-parse*' for dispatch value: null",
:at [clojure.lang.MultiFn getFn "MultiFn.java" 156]}],
:trace [[clojure.lang.MultiFn getFn "MultiFn.java" 156]
[clojure.lang.MultiFn invoke "MultiFn.java" 233]
[cognitect.aws.shape$xml_parse invokeStatic "shape.clj" 167]
[cognitect.aws.protocols.ec2$eval1775$fn__1778 invoke "ec2.clj" 71]
[clojure.lang.MultiFn invoke "MultiFn.java" 239]
[cognitect.aws.client$handle_http_response invokeStatic "client.clj" 37]
[cognitect.aws.client$send_request$fn__8896 invoke "client.clj" 55]
[clojure.core$map$fn__5847$fn__5848 invoke "core.clj" 2742]
[clojure.core.async.impl.channels$chan$fn__565
invoke
"channels.clj"
300]
[clojure.core.async.impl.channels.ManyToManyChannel
put_BANG_
"channels.clj"
83]
[clojure.core.async$put_BANG_ invokeStatic "async.clj" 165]
[cognitect.http_client.Client$fn$reify__8165
onComplete
"http_client.clj"
233]
[org.eclipse.jetty.client.ResponseNotifier
notifyComplete
"ResponseNotifier.java"
193]
[org.eclipse.jetty.client.ResponseNotifier
notifyComplete
"ResponseNotifier.java"
185]
[org.eclipse.jetty.client.HttpReceiver
terminateResponse
"HttpReceiver.java"
454]
[org.eclipse.jetty.client.HttpReceiver
responseSuccess
"HttpReceiver.java"
401]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
messageComplete
"HttpReceiverOverHTTP.java"
268]
[org.eclipse.jetty.http.HttpParser parseContent "HttpParser.java" 1386]
[org.eclipse.jetty.http.HttpParser parseNext "HttpParser.java" 1203]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
parse
"HttpReceiverOverHTTP.java"
158]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
process
"HttpReceiverOverHTTP.java"
119]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP
receive
"HttpReceiverOverHTTP.java"
69]
[org.eclipse.jetty.client.http.HttpChannelOverHTTP
receive
"HttpChannelOverHTTP.java"
90]
[org.eclipse.jetty.client.http.HttpConnectionOverHTTP
onFillable
"HttpConnectionOverHTTP.java"
113]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback
succeeded
"AbstractConnection.java"
273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.ssl.SslConnection
onFillable
"SslConnection.java"
197]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback
succeeded
"AbstractConnection.java"
273]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 95]
[org.eclipse.jetty.io.SelectChannelEndPoint$2
run
"SelectChannelEndPoint.java"
75]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
produceAndRun
"ExecuteProduceConsume.java"
213]
[org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
run
"ExecuteProduceConsume.java"
147]
[org.eclipse.jetty.util.thread.QueuedThreadPool
runJob
"QueuedThreadPool.java"
654]
[org.eclipse.jetty.util.thread.QueuedThreadPool$3
run
"QueuedThreadPool.java"
572]
[java.lang.Thread run "Thread.java" 745]]}}Running the command via the CLI results in no output, so I’m assuming that’s the norm.
But the docs claim otherwise:
<DeleteSecurityGroupResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>59dbff89-35bd-4eac-99ed-be587EXAMPLE</requestId>
<return>true</return>
</DeleteSecurityGroupResponse>I'm using the following versions:
[com.cognitect.aws/api "0.8.146"]
[com.cognitect.aws/endpoints "1.1.11.462"]
[com.cognitect.aws/ec2 "681.2.373.0"]
[com.cognitect.aws/ecs "668.2.364.0"]ApiGateway GetRestApis returns empty maps when there's only one api
Following up on #35, when I removed the api definition I made for testing, leaving only one api defined, the call to GetRestApis stopped working.
With two apis defined:
(aws/invoke apigw {:op :GetRestApis})
=>
{:items [{:endpointConfiguration {:types [\R \E \G \I \O \N \A \L]},
:name "test",
:apiKeySource "HEADER",
:id "<removed>",
:createdDate #inst"2019-01-21T17:16:52.000-00:00"}
{:endpointConfiguration {:types [\R \E \G \I \O \N \A \L]},
:name "<removed>",
:apiKeySource "HEADER",
:binaryMediaTypes [\* \/ \*],
:id "<removed>",
:createdDate #inst"2018-12-21T12:19:45.000-00:00",
:version "2019-01-17T15:45:00Z"}]}With only one:
(aws/invoke apigw {:op :GetRestApis})
=>
{:items [{} {} {} {} {} {} {}]}The api is in the request body, but it's not wrapped in a vector as it is when there are more than one api's defined.
Support timeouts on http calls
Currently, cognitect.http-client supports timeouts through :cognitect.http-client/timeout-msec but it is not possible to pass this option through aws-api.
Support for CloudWatch (and CloudWatchEvents)?
I think I must be missing it, but I don’t see Cloudwatch among the releases for the cognitect aws client api. https://github.com/cognitect-labs/aws-api/blob/master/latest-releases.edn
I'm looking for something that would correspond to the following:
How do I generate a pre-signed URL?
The SDKs include support for generating presigned URLs in S3 clients, but the s3 service description does not include an op for it. We need, at the very least, an answer to the question "how do I generate a pre-signed URL", if not a solution for it.
Creating ec2 client fails with "RuntimeException Map literal must contain an even number of forms"
With deps.edn containing
{:deps {com.cognitect.aws/api {:mvn/version "0.8.223"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.481"}
com.cognitect.aws/ec2 {:mvn/version "697.2.391.0"}}}
Then from Clojure cli:
Clojure 1.9.0
user=> (require '[cognitect.aws.client.api :as aws])
nil
user=> (def ec2 (aws/client {:api :ec2}))
RuntimeException Map literal must contain an even number of forms clojure.lang.Util.runtimeException (Util.java:221)
I don't get this with s3. I've tried Clojure 1.10.0 and 1.10.0-RC5 with the same results.
No method in multimethod 'sign-http-request' for dispatch value: v2
aws-api does not explicitly support SignatureVersion v2, and there are two services that use it:
({:uid "sdb-2009-04-15", :signatureVersion "v2"}
{:uid "importexport-2010-06-01", :signatureVersion "v2"})
This means that sdb and importexport requests are met with client side errors.
Nested configuration values are not supported
As described in the CLI documentation, some configuration values can have nested values. As a result a line like the following in a .aws/config file will result in a NullPointerException when being parsed:
[default]
s3 =
signature_version = s3v4
SQS ReceiptHandles are not properly encoded when calling DeleteMessage
Calling the SQS DeleteMessage operation with the receipt handle string given in a ReceiveMessage response fails with a ReceiptHandleIsInvalid error from AWS. Looking at the error message, it appears that the library is replacing the + signs in the handle with spaces.
user=> (aws/invoke sqs {:op :ReceiveMessage :request {:QueueUrl url}})
{:Messages
[{:MessageId "c61ea1fa-4371-435b-9af6-d867b83747a5",
:ReceiptHandle
"AQEBPwr8VzRpb0QmLfk6fTKH+op2XVD6TmuliAz82dYxsKncTfq2s+pPcPPlE7/ytEE1pfOe40Qw9qTZWd2sdOOlXaV8v2TcLEr35nOGlZlS6g7O1FfEKvMGMI6asvMSRfwZOvf156lpgWiYiUHerUVlK3bpnZ9ouS+lEkzLLDwTjrUsmzTxoul0cvA61B8Z1wzwCNvlRtkAMOelAGGmPBAH+HS+Rlv8FWWIap0cvcdqoy+42pVb+5fQjQXtw/Qg6n581UZw8iQlacahgkcKF/n76Qp9TdFbzCaS07sdLlRwKpGpmimKuYCxELXysUn1C4OVJFWuffQLCInuEXbTzqQxNd+CIIW1iQKr/+vu9jWHqbmj6CrzpQ/FhBHfopYKE7zbn25r6f2+9l3tRXvWbZgF04L3Ryn3Zf5APZhZDgsLUl4=",
:MD5OfBody "136d4c574bd4e7b45bc0c19d91cbb57d",
:Body "......"}]}
user=> (aws/invoke sqs {:op :DeleteMessage :request {:QueueUrl url :ReceiptHandle (:ReceiptHandle (first (:Messages *1)))}})
{:ErrorResponse
{:Error
{:Type "Sender",
:Code "ReceiptHandleIsInvalid",
:Message
"The input receipt handle \"AQEBPwr8VzRpb0QmLfk6fTKH op2XVD6TmuliAz82dYxsKncTfq2s pPcPPlE7/ytEE1pfOe40Qw9qTZWd2sdOOlXaV8v2TcLEr35nOGlZlS6g7O1FfEKvMGMI6asvMSRfwZOvf156lpgWiYiUHerUVlK3bpnZ9ouS lEkzLLDwTjrUsmzTxoul0cvA61B8Z1wzwCNvlRtkAMOelAGGmPBAH HS Rlv8FWWIap0cvcdqoy 42pVb 5fQjQXtw/Qg6n581UZw8iQlacahgkcKF/n76Qp9TdFbzCaS07sdLlRwKpGpmimKuYCxELXysUn1C4OVJFWuffQLCInuEXbTzqQxNd CIIW1iQKr/ vu9jWHqbmj6CrzpQ/FhBHfopYKE7zbn25r6f2 9l3tRXvWbZgF04L3Ryn3Zf5APZhZDgsLUl4=\" is not a valid receipt handle.",
:Detail nil},
:RequestId "8e8e70d1-a822-5101-bb8f-245babb6f7c8"},
:ErrorResponseAttrs
{:xmlns "http://queue.amazonaws.com/doc/2012-11-05/"},
:cognitect.anomalies/category :cognitect.anomalies/not-found}(By the way, thanks for this library! I have high hopes for it!)
Add support for a making aws/client with map of credential keys.
I would like to add support to instantiate a client with a map of credentials rather than setting environment variables.
Currently to work around this I do the following:
(require '[cognitect.aws.credentials :refer [CredentialsProvider]]
'[cognitect.aws.client.api :as aws])
(defn map->credentials-provider
[{:keys [access-key-id secret-access-key]}]
(reify CredentialsProvider
(fetch [_]
{:aws/access-key-id access-key-id
:aws/secret-access-key secret-access-key})))
(def s3-client (aws/client {:api :s3
:region "us-east-1"
:credentials (map->credentials-provider {:access-key-id access-key-id
:secret-access-key secret-access-key})}))
I would like to instead do the following:
(def s3-client (aws/client {:api s3 :region "us-east-1"
:credentials {:access-key-id access-key-id
:secret-access-key secret-access-key}}))
Maybe one way to do this is in aws/client with the function provided above:
{:credentials (credentials/auto-refreshing-credentials
(or (when (map? credentials-provider)
(credentials/map->credentials-provider credentials-provider))
credentials-provider
(credentials/default-credentials-provider)))}
Thanks!
Credentials are not being refreshed because they don't return a TTL
I learned after seeing my Datomic Ions fail intermittently that apparently:
(def s3 (aws/client {:api :s3}))
creates some sort of long-lived connection that needs to be refreshed.
I'm trying to figure out how to use the client now in a service where I need to reach out to S3 on each invocation.
Should I create a new client every invoke? Will that consume too many resources?
Should I try and manage this connection in some way and re-create only if it drops?
How should I manage the lifecycle of it?
Thanks!
Inconsistency in SQS ReceiveMessage API - Cannot directly specify "SentTimestamp" as an attribute name
(aws/invoke sqs {:op :ReceiveMessage
:request (merge queue-url
{:WaitTimeSeconds 1
;; Below fails per the spec
;; :AttributeNames ["ApproximateReceiveCount" "SentTimestamp"]
;; Can get to those using "All"
:AttributeNames ["All"]})})The documentation is inconsistent: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
Under AttributeName.N:
"These attributes include" describes attributes that are not listed in "Valid Values"
Credentials provider chain not respecting AWS_PROFILE
When specifying an AWS_PROFILE environment variable using a named profile the API uses the 'default' profile, not the named profile in the envar.
Providing the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY environment variables explicitly behaves as expected.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.