aristidb / aws Goto Github PK
View Code? Open in Web Editor NEWAmazon Web Services for Haskell
License: BSD 3-Clause "New" or "Revised" License
Amazon Web Services for Haskell
License: BSD 3-Clause "New" or "Revised" License
AWS have announce Elastic Transcoder.
I am expecting to build out an interface to this service fairly soon for a project, and I thought it would be a good idea to check in here to see if you are already doing this or would be interested in extending the AWS package to cover this. (I will of course be very happy to feed the code back into the AWS package.)
To prevent scripts from hammering Amazon's servers and getting rate limited server-side, add rate limiting here.
If there is an existing Haskell package for rate limiting, consider using it.
I can see we have a SNS submodule to handle SES specific events, but not a general one. Is there any blocker for that or just it haven't been thought yet?
I might suspect SNS by nature is fairly domain-specific and might be unfeasible to come up with a general module, is this the reason?
Thanks!
Alfredo
Hey!
There are some headers that I'm setting on PutObject that simply don't work: poContentType
, poCacheControl
and poExpires
. If I'm reading the code correctly, that's becuase these headers are actually being sent in the query, not as headers. So AWS simply ignore them. Is that correct?
Cheers!
I am uncertain how to work with versioned objects using this library. Shall I use the s3QQuery
field of S3Query
? It seems not possible to do it through the interface provided by GetBucket
.
Use classy lenses for the record fields of requests and responses, that should also make "subtyping" of requests more convenient.
We might also be able to use it for the Response type.
This is a big change, I guess.
S3Configuration
carries a phantom type of the request type. I am storing my S3Configuration
in a bigger Env
data type, and it's fine if I always do NormalQuery
s. However, I need to do both types of queries, so I end up having to do:
let s3' = case s3 of
S3.S3Configuration a b c d e f -> S3.S3Configuration a b c d e f
It would be nice if aws
provided this inside the library itself.
As far as EC2 api support is concerned, does this package only retrieve metadata for running EC2 instances? I'd like to be able to create, start, stop and terminate EC2 instances, and add tags to them. Wanted to be sure these things weren't supported before I attempt to submit a patch.
If you look at s3SignQuery in Aws/S3/Core.hs
s3SignQuery :: S3Query -> S3Configuration qt -> SignatureData -> SignedQuery
s3SignQuery S3Query{..} S3Configuration{..} SignatureData{..}
= SignedQuery {
sqMethod = s3QMethod
, sqProtocol = s3Protocol
, sqHost = B.intercalate "." $ catMaybes host
, sqPort = s3Port
, sqPath = mconcat $ catMaybes path
, sqQuery = sortedSubresources ++ s3QQuery ++ authQuery :: HTTP.Query
, sqDate = Just signatureTime
, sqAuthorization = authorization
, sqContentType = s3QContentType
, sqContentMd5 = s3QContentMd5
, sqAmzHeaders = amzHeaders
, sqOtherHeaders = s3QOtherHeaders
, sqBody = s3QRequestBody
, sqStringToSign = stringToSign
}
You'll see that
sqQuery = sortedSubresources ++ s3QQuery ++ authQuery :: HTTP.Query
differs from its form in the stringToSign
HTTP.renderQueryBuilder True sortedSubresources
This hasn't been an issue because all of the commands that can set the s3QQuery field
to something other than an empty list do not do so in practice.
Example: in GetObject
s3QQuery = HTTP.toQuery [
("response-content-type" :: B8.ByteString,) <$> goResponseContentType
, ("response-content-language",) <$> goResponseContentLanguage
, ("response-expires",) <$> goResponseExpires
, ("response-cache-control",) <$> goResponseCacheControl
, ("response-content-disposition",) <$> goResponseContentDisposition
, ("response-content-encoding",) <$> goResponseContentEncoding
]
In practice, people must use getObject
getObject :: Bucket -> T.Text -> GetObject
getObject b o = GetObject b o Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing
I ran into this issue writing DeleteObjects. Putting "delete" in the s3QQuery doesn't work. Making the obvious fix to the stringToSign process broke another command, (GetBucket?), so I just put it in s3QSubresources as a hack.
Test case that should generate a 403 (slightly modified version of the GetObject example):
{-# LANGUAGE OverloadedStrings #-}
import qualified Aws
import qualified Aws.S3 as S3
import qualified Aws.S3.Commands.GetObject as G
import Data.Conduit (($$+-))
import Data.Conduit.Binary (sinkFile)
import Network.HTTP.Conduit (withManager, responseBody)
main :: IO ()
main = do
{- Set up AWS credentials and the default configuration. -}
cfg <- Aws.baseConfiguration
let s3cfg = Aws.defServiceConfig :: S3.S3Configuration Aws.NormalQuery
{- Set up a ResourceT region with an available HTTP manager. -}
withManager $ \mgr -> do
{- Create a request object with S3.getObject and run the request with pureAws. -}
S3.GetObjectResponse { S3.gorResponse = rsp } <-
Aws.pureAws cfg s3cfg mgr $
(S3.getObject "haskell-aws" "cloud-remote.pdf") { G.goResponseCacheControl = Just "300" }
{- Save the response to a file. -}
responseBody rsp $$+- sinkFile "cloud-remote.pdf"
It would be great to have DevPay support in this library. It changes a bit REST requests adding product key and user token. Have you considered adding it i near future?
I fused a s3 source to s3 and it worked, but as soon as I did some processing per line, the socket would vanish. S3 also should be given a content length, which a lazy source can't provide.
For these reasons, I believe put object should only take a strict byte string, and encode the content length from it.
My program already stores AWS credentials in its own way, so I need a way to get from ByteString -> ByteString -> Credentials
It's certainly possible to write that in my program, but then I have to worry about the v4SigningKeys and iamToken fields, which seem like implementation details that could change. And one of them, being an IORef, I'm wary of constructing on my own, since making an IORef that's empty when it should be full can lead to problems, for example.
So, I suggest providing in aws:
genCredentials :: ByteString -> ByteString -> IO Credentials
Using that might also simplify the implementation of loadCredentials*
Aws/DynamoDb/Commands/Table.hs:42:18:
Could not find module `GHC.Generics'
It is a member of the hidden package `ghc-prim'.
Perhaps you need to add `ghc-prim' to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
With the release of conduit 0.3., it looks like the conduit dependency is out of date and requires more than a trivial version bump.
Unsure if this is s3 or the aws pkg. It says this operation isn't Implemented. Was wondering if it's because I'm using a NormalQuery for my S3Configuration when doing a PUT. Any ideas are appreciated.
uploadToCompletedBucket :: Worker ()
uploadToCompletedBucket = do
prnt "uploading"
(acid, cfg, QMsg{..}) <- ask
let s3cfg = Aws.defServiceConfig :: S3.S3Configuration Aws.NormalQuery
zipSize <- io $ sz (T.unpack qId ++ ".zip")
io $ withManager $ \mgr ->
do S3.PutObjectResponse { porVersionId = result } <-
Aws.pureAws cfg s3cfg mgr $ S3.putObject stcompleted qId $
RequestBodySource (fromIntegral zipSize) (source qId)
io $ print result
prnt "Put object onto completed bucket"
where
source :: MonadResource m => T.Text -> Source m Builder
source qId = CB.sourceFile (output ++ "/" ++ T.unpack qId <> ".zip") $= CL.map fromByteString
sz file = fileSize <$> getFileStatus file
And the corresponding error.
S3Error {s3StatusCode = Status {statusCode = 501,statusMessage = "Not Implemented"},
s3ErrorCode = "NotImplemented",
s3ErrorMessage = "A header you provided implies functionality that is not implemented",
s3ErrorResource = Nothing,
s3ErrorHostId = Just "KF8FwxnEalEL4sgPQhFcoxmu3q2rdOhz1mgG0+S7E/yz1kWvZPX5Hy2S0ILwwufc",
s3ErrorAccessKeyId = Nothing,
s3ErrorStringToSign = Nothing
}
*Aws> runAws (aws $ putAttributes "ßß" [addAttribute "ö" "ü"] "MyDomain") bc
AWS Debug: String to sign - "POST\nsdb.amazonaws.com\n/\nAWSAccessKeyId=AKIAIOJF6QL7WAZUCBBA&Action=PutAttributes&Attribute.1.Name=%C3%B6&Attribute.1.Value=%C3%BC&DomainName=MyDomain&ItemName=%C3%9F%C3%9F&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-01-31T14%3A19%3A06&Version=2009-04-15"
*** Exception: SdbError {sdbStatusCode = 403, sdbErrorCode = SignatureDoesNotMatch, sdbErrorMessage = "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.", sdbErrorMetadata = FromSdbMetadata (SdbMetadata {requestId = "4ab63c9c-5d2e-0bfd-f6eb-a544555e63f4", boxUsage = Nothing})}
where bc uses a POST configuration
*Aws> runAws (aws $ putAttributes "ßß" [addAttribute "ö" "ü"] "MyDomain") c
AWS Debug: String to sign - "GET\nsdb.amazonaws.com\n/\nAWSAccessKeyId=AKIAIOJF6QL7WAZUCBBA&Action=PutAttributes&Attribute.1.Name=%C3%B6&Attribute.1.Value=%C3%BC&DomainName=MyDomain&ItemName=%C3%9F%C3%9F&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-01-31T14%3A18%3A57&Version=2009-04-15"
SdbResponse {fromSdbResponse = PutAttributesResponse, sdbResponseMetadata = SdbMetadata {requestId = "80dcad41-8e89-c3f9-db94-09190487ff87", boxUsage = Just "0.0000219909"}}
here with GET configuration
http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUTlifecycle.html
Would be really nice to have. I can try to take a stab at it myself.
Here's the output:
$ cabal install aws
...
...
...
Building aws-0.9...
Preprocessing library aws-0.9...
[ 1 of 64] Compiling Aws.Ec2.InstanceMetadata ( Aws/Ec2/InstanceMetadata.hs, dist/build/Aws/Ec2/InstanceMetadata.o )
[ 2 of 64] Compiling Aws.Core ( Aws/Core.hs, dist/build/Aws/Core.o )
Aws/Core.hs:379:14:
No instance for (Data.Default.Default HTTP.Request)
arising from a use of `def'
Possible fix:
add an instance declaration for (Data.Default.Default HTTP.Request)
In the expression: def
In the second argument of `($)', namely
`def
{HTTP.method = httpMethod sqMethod,
HTTP.secure = case sqProtocol of {
HTTP -> False
HTTPS -> True },
HTTP.host = sqHost, HTTP.port = sqPort, HTTP.path = sqPath,
HTTP.queryString = HTTP.renderQuery False sqQuery,
HTTP.requestHeaders = catMaybes
[checkDate (\ d -> ("Date", fmtRfc822Time d)) sqDate,
fmap (\ c -> ("Content-Type", c)) contentType,
fmap
(\ md5 -> ("Content-MD5", Base64.encode $ toBytes md5))
sqContentMd5,
....]
++ sqAmzHeaders ++ sqOtherHeaders,
HTTP.requestBody = case sqMethod of {
PostQuery
-> HTTP.RequestBodyLBS . Blaze.toLazyByteString
$ HTTP.renderQueryBuilder False sqQuery
_ -> case sqBody of {
Nothing -> HTTP.RequestBodyBuilder 0 mempty
Just x -> x } },
HTTP.decompress = HTTP.alwaysDecompress,
HTTP.checkStatus = \ _ _ _ -> Nothing}'
In a stmt of a 'do' block:
return
$ def
{HTTP.method = httpMethod sqMethod,
HTTP.secure = case sqProtocol of {
HTTP -> False
HTTPS -> True },
HTTP.host = sqHost, HTTP.port = sqPort, HTTP.path = sqPath,
HTTP.queryString = HTTP.renderQuery False sqQuery,
HTTP.requestHeaders = catMaybes
[checkDate (\ d -> ("Date", fmtRfc822Time d)) sqDate,
fmap (\ c -> ("Content-Type", c)) contentType,
fmap
(\ md5 -> ("Content-MD5", Base64.encode $ toBytes md5))
sqContentMd5,
....]
++ sqAmzHeaders ++ sqOtherHeaders,
HTTP.requestBody = case sqMethod of {
PostQuery
-> HTTP.RequestBodyLBS . Blaze.toLazyByteString
$ HTTP.renderQueryBuilder False sqQuery
_ -> case sqBody of {
Nothing -> HTTP.RequestBodyBuilder 0 mempty
Just x -> x } },
HTTP.decompress = HTTP.alwaysDecompress,
HTTP.checkStatus = \ _ _ _ -> Nothing}
Failed to install aws-0.9
cabal: Error: some packages failed to install:
aws-0.9 failed during the building phase. The exception was:
ExitFailure 1
When I started working on aws a while ago, I wanted to put everything in a monad. This plan was scrapped fairly quickly, however now I have plans to have another go, albeit with different motivations. Two things first:
So why do I want this?
Originally, I just wanted to avoid giving the configuration / service configuration / HTTP manager parameters every time. But that isn't really a good reason IMHO.
I have recently started adding the first "complex" functionality to aws, inspired by @larskuhtz, specifically the IteratedTransaction functions awsIteratedSource and awsIteratedList. They just hammer out the HTTP requests and that's that. But what if we want to do things like:
Or perhaps fancier things that I haven't thought of yet.
I think there might be two components to this:
It would also have to be a transformer so there can be actions in some underlying monad.
And of course it would have pluggable interpreters, which can then implement request retry / rate limiting / ....
That's pretty cool, I want that!
We have been getting an HTTP 403 error back from AWS because the API signature does not match. We have discovered that the problem happens on ghc-7.6.1 but not on ghc-7.0.4, and that it happens for getObject when any of the GetObject Maybe fields (goVersionId, goResponseContentType, goResponseContentLanguage, goResponseExpires, goResponseCacheControl, goResponseContentDisposition, goResponseContentEncoding) are set to Just.
I'm not sure, but it seems like s3SignQuery is not taking into account all the parameters that the signing should be based on. I have no idea why this is working on ghc-7.0.4 and not ghc-7.6.1.
Uploading an object to S3 fails about half the time (with TlsException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))). I suspect a bug in the haskell tls package. It wouldn't be the first one that I came across.
I got an error while compiling your library. It does not work with exceptions >= 0.4. Please update library to use newest exceptions.
Preprocessing library aws-0.8.5...
[ 1 of 64] Compiling Aws.Ec2.InstanceMetadata ( Aws/Ec2/InstanceMetadata.hs, dist/dist-sandbox-acf35a42/build/Aws/Ec2/InstanceMetadata.o )
Aws/Ec2/InstanceMetadata.hs:20:41:
No instance for (exceptions-0.5:Control.Monad.Catch.MonadThrow
(ResourceT IO))
arising from a use of `HTTP.parseUrl'
Possible fix:
add an instance declaration for
(exceptions-0.5:Control.Monad.Catch.MonadThrow (ResourceT IO))
In a stmt of a 'do' block:
req <- HTTP.parseUrl ("http://169.254.169.254/" ++ p ++ '/' : x)
In the expression:
do { req <- HTTP.parseUrl
("http://169.254.169.254/" ++ p ++ '/' : x);
HTTP.responseBody <$> HTTP.httpLbs req mgr }
In an equation for `getInstanceMetadata':
getInstanceMetadata mgr p x
= do { req <- HTTP.parseUrl
("http://169.254.169.254/" ++ p ++ '/' : x);
HTTP.responseBody <$> HTTP.httpLbs req mgr }
Failed to install aws-0.8.5
OS X Lion, conduit 0.4.1.1, http-conduit 1.4.1.2
Will try on other systems, too, and with the stable version of aws.
@meteficha Can you reproduce this?
EC2 instances have access to a rich source of meta-data
It contains directories (names ending with "/") and files. No authentication is required, but access is local to the machine. Some EC2 images lock it down, so it isn't always accessible.
To do it properly, probably depends on #36.
This should be configurable, too.
Hello,
i get this error in response back on S3 Put Object command.
"The request signature we calculated does not match the signature you provided. Check your key and signing method".
I checked that credentials are valid.
I would like to have support for the PUT Object (Copy) command.
I can write the patch if you want but I don't know when I have time for it.
Is it possible to make it work with aeson-0.8?
Although it is tedious, I would recommend manually rendering the timestamps. formatTime is extremely inefficient. The function is responsible for about 10% of my CPU utilization when benchmarking an application that uploads to S3.
We also have a timestamp parsing library that may be faster than the parseTime function: http://hackage.haskell.org/package/timeparsers. The library is still pretty new and probably needs some work before being used in something like 'aws'.
It looks like the current DynamoDB code only supports the parts of the API dealing with the data definition (creating, updating, deleting tables) but not data retrieval and modification (get, put, update, query, and scan items).
This is something I'm willing to add, but wanted to check first whether anyone else is already working on it.
(pinging @wuzzeb)
It would be nice if aws would support Amazon Glacier. Possibly, this would make it possible for git-annex to support Glacier without external tools.
We need certificate checking for SSL requests.
Whenever I try to fetch an object from S3, I get the exception from parseObjectMetadata that the Last-Modified timestamp is wrong.
In the web interface, the AWS console reports that the last modified value is "Fri Aug 03 17:41:20 GMT-700 2012". I do not know for sure if this is actually the string it yields when you make the HTTP request. If it is, then it certainly does not conform to the rfc822Time string in the aws package (which is "%a, %_d %b %Y %H:%M:%S GMT").
On GetObject, is it possible to support giving the HTTP Range header? That way we can access partial data.
See http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html Range headers for more information
Allow making requests using STS (http://docs.amazonwebservices.com/IAM/latest/UsingSTS/Welcome.html / http://docs.amazonwebservices.com/STS/latest/APIReference/Welcome.html).
This issue is NOT about full support for STS, just making requests using these temporary credentials. There are two caveats:
This is especially interesting in conjunction with #29 and #37, so temporary credentials can be periodically obtained with the EC2 instance metadata.
Just started running into an issue after upgrading to latest with xml-conduit-1.1.0.3:
Name {nameLocalName = "SendRawEmailResponse", nameNamespace = Just "http://ses.amazonaws.com/doc/2010-12-01/", namePrefix = Nothing}, but received: EventEndDocument.
This functionality was working without any issues right before the update.
Some requests don't require authentification and no credentials. 'aws' currently has no real support for that.
This is an oft-cited trick for checking whether a specific object exists, rather than listing the objects in the bucket that match a prefix.
Every request prints out debug information into standard output - there is no way to disable this. Debug output has the following format:
AWS Debug: String to sign - ...
When I make an issue I can't seem to find it anymore. Are issues being filtered upon creation?
Doesn't show up unless I manually go to the URL of the issue. #91
Building of 0.4.* fails on OS X with HP 2011.4.0.0. 0.3.2 builds just fine. The build error is:
cabal: cannot configure void-0.5.5.1. It requires semigroups >=0.8.2 && <0.9
For the dependency on semigroups >=0.8.2 && <0.9 there are these packages:
semigroups-0.8.2, semigroups-0.8.3, semigroups-0.8.3.1 and semigroups-0.8.3.2.
However none of them are available.
semigroups-0.8.2 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3.1 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3.2 was excluded because semigroups-0.8 was selected instead
We need a test system.
CC @meteficha
hi,
i am writing a mws (marketplace web service) extension to aws and noticed, that there don't seem to be any convenience functions that, say, send one specific message, process the result in one way or another and then send another specific message.
one example on where i want to offer such a function is the mws' subscriptions api, so that one only has to call (name and type to change)
setupQueueForMwsSubscription cfg sqsCfg mwsCfg mgr
and it will add the right permissions to the queue and register the queue as a destination in MWS' subscription api.
is something like that not desirable or just not important for the other parts of aws?
if so, should it be in Aws.Mws
or in an extra module as Aws.Mws.Convenience
?
i'd really like to follow the spirit of aws.
You can break a 5 GB upload (the current limit on the size of an S3 object) into as many as 1024 separate parts and upload each one independently, as long as each part has a size of 5 megabytes (MB) or more. If an upload of a part fails it can be restarted without affecting any of the other parts. Once you have uploaded all of the parts you ask S3 to assemble the full object with another call to S3.
This would greatly improve upload times for large objects.
Is it possible to make this library work with transformers 0.4.1 ?
Some examples of how to use the package would be very useful.
Folks, today I realised that although I have been using my elastic-transcoder aws-extensions in a production context, I hadn't done anything with the code since mid-March, when it was pretty-much ready to go.
So today I re-spun the code as two extension packages:
AWS have since built out the elastic transcoder api but I thought it was important to just get it out. I was very pleasantly pleased by how easy it was to just make up these two packages which just hook into the current aws -- a huge validation of its design IMHO.
As this arrangement seems to be so natural and productive I am inclined to stick with it. I consider it a part of the aws.
Have you considered moving a more module package structure? The design seems clearly to support it.
I have also published cloudfront-signer for signing CloudFront URLs. This is a fairly simple (if dead useful) standalone component without any dependencies on the aws package. It does live in module that fits with the aws scheme -- Aws.CloudFront.Signer -- and could be consider part of the aws family.
The current packages have been put out there on version 0.0.0.1 to get things moving. I expect to be revising them fairly quickly in light of feedback. They are all hosted on my personal github account (cdornan) but I expect to be re-housing them on the iconnect account soon.
The API seems to be missing a way to delete a S3 bucket (once it's already been emptied of all objects).
Once that API is available, a good example program could be added, that did a mass-removal of all the contents of a bucket and then deleted it. Particularly handy when you've been learning the API, and so creating various junk in a bucket. ;)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.