weavejester / clj-aws-s3 Goto Github PK
View Code? Open in Web Editor NEWS3 client library for Clojure
S3 client library for Clojure
I'm using fake-s3 and openstack as my s3 providers & i'm currently writing patch to add function to change default domain of s3 server.
Changing host is simple just (.setEndpoint client "http://new-host") and works well.
But i'm little stuck. How to change PathStyleAccess?
I'm not very familiar with Java and it's documentation is very confusing. Changing PathStyleAccess is important because none of my s3 server dont support accessing buckets as virtualhost .s3.localhost.
Here is documentation: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/S3ClientOptions.html
I have a simple function to post a file in a bucket
(defn post-pic
"Uploads a picture to S3"
[file]
(let [{:keys [filename tempfile]} file]
(s3/put-object cred my-bucket filename tempfile)))
It works like a charm, however, it throws everytime an IllegalArgumentException saying that
No implementation of method: :render of protocol: #'compojure.response/Renderable found for class: com.amazonaws.services.s3.model.PutObjectResult
The jar in clojars has a dependency of aws sdk 1.4.2.1. Please update.
The current aws sdk version is 1.7.1, Is there any reason not to upgrade to it?
I need to get hold of an object for a particular version. I can provide a patch if necessary.
=> (s3/list-objects cred bucket)
NoClassDefFoundError Could not initialize class com.amazonaws.ClientConfiguration com.amazonaws.services.s3.AmazonS3Client.<init> (AmazonS3Client.java:365)
Happens on both versions 0.3.3 and 0.3.10 fwiw.
I'm getting com.amazonaws.services.s3.model.AmazonS3Exception
exception with this description:
Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 2A700E5BB5693E09, AWS Error Code: InvalidRequest, AWS Error Message: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
The only operation that I can execute successfully is:
(s3/list-buckets cred)
That works, but only if I don't specify :endpoint
in by cred's. If I add :endpoint "s3-eu-central-1.amazonaws.com"
to my cred's, then the list-buckets
gives that same exception.
Any ideas what could be wrong?
Related: aws/aws-sdk-java#421
We've been working to track down a strange issue for a week or two, and believe we have isolated it. My theory is that when S3 generates a 500-level server issue, the AmazonS3Client
instance is no longer usable and all requests generate "Stream Closed" exceptions.
I believe the memoization of s3-client
as referenced in #3 has made it such that once we get such a server exception, we cannot use clj-aws-s3 without a JVM restart. I don't have a great way to replicate and prove this just yet, however. I'm working on it though.
Presuming so, I see a few possible patches I could submit that fix the issue but remain usuable for the use case described in #3 but would like your input - make memoization optional, create a with-client
form of invoking the methods that lets the invoker decide on a client re-use strategy, or wrap calls to most methods in a handler that will regenerate the s3-client
in the event of an exception.
Thoughts?
(s3/put-object cred "my_bucket" thumb (io/file (str uploads-dir thumb)))
I'd like to make a fully public / web-accessible image
what goes next?
Just a heads-up that this dependency seems to be broken atm - am assuming this is a problem with the Maven repo. Not sure if/when they'll get around to fixing it.
Is any way to set metadata of an new object when put it into a bucket , or set metadata of an existing object?
It would be nice to have this.
I could do a PR for it if wanted.
Thanks!
I had a ClassNotFoundException using datomic-pro "0.9.5130" and clj-aws-s3 "0.3.10" when I connect the DynamoDB. The datomic uses aws-java-sdk-core 1.8.11 and clj-aws-s3 the 1.7.5.
When downloading large objects with get-object, I reliably get "socket is closed". This is caused because the (s3-client cred) is GC'd before my file is done downloading.
See https://forums.aws.amazon.com/thread.jspa?threadID=83326, the reply by Yosuke on Jan 23,2012 3:46 PM for coroboration.
I'm doing
(let [resp (s3/get-object bucket key)]
(io/copy (-> resp :content) (io/file dest-path))))
My download takes several minutes to finish, and I reliably get the "socket is closed" less than a minute into the download. Replacing (s3/get-object) with its source code, and keeping a reference on the s3-client object fixes the bug.
A small feature request. To be honest this isn't particularly hard to do directly:
(def s3-client
;; This looks for credentials in the standard places.
(AmazonS3Client. (DefaultAWSCredentialsProviderChain.)))
But it'd be nice if this could be achieved via your s3-client
function too so you can combine it the clojurey conventions for specifying the ClientConfiguration.
Looks like it's pretty easy to request byte ranges using the (.setRange ...)
method on the GetObjectRequest
object.
Not sure what the best way to expose it is. In the meantime I'll be doing the interop myself.
i need to create a hierarchy of date folders..
Edit: nevermind i am dumb i nad a leading / as soon as i removed it worked fine and created the folder
Looks like at least in aws-java-sdk "1.9.33" you can add call .setGeneralProgressListener
on PutObjectRequest
objects.
๐ช
Related clj-time/clj-time#190
I even copied the title :)
When using Java 1.8u60 this exception is thrown when calling aws.sdk.s3/get-object.
It seems to be that Java 1.8u60 requires a later version of joda-time (2.8.1). This problem was fixed in clj-time 0.11.0.
So I forked, changed the dependency to 0.11.0, installed and the error was no more.
I'm trying to retrieve a large list of files form an s3 bucket. To do so I need to call listNextBatchOfObjects
on a ListObject
, which is what is returned from listing files. Is this currently possible in this library?
Hi -- great lib!,
However, been struggling with put-object and a default permission to be :all-user :read but keep getting error.
I think this is a common use case and an extra example to put-object with permission will be helpful.
at s3.clj:121
In a deployed service, I don't want to hardcode AWS credentials in my code or even in environment variables. I would like to rely on the IAM role of the machine I'm on.
Is this possible with this library?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.