confetti-clj / confetti Goto Github PK
View Code? Open in Web Editor NEW🎉 A tool to help creating and deploying static sites with AWS
License: Mozilla Public License 2.0
confetti's Issues
sync-bucket task option to specify file-maps directly
- without intermediate file
- useful for very simple upload scenarios
Clarify Readme regarding access key
The Readme currently says
Creating a site
Let's say you want to deploy a site at my-app.com. To create an S3 bucket, a CloudFront distribution and restricted access keys you can run the following:
This sounds like its saying that the access key and secret key will be created, but the function signature of create-site sounds like its expecting values for these options. I'm assuming I need to create an AWS user and create the access key ahead of time, so what does the Confetti Readme mean when it says it will create restricted access keys?
Consider Lambda@Edge for immutable deploys
Allow custom invalidation path for cloudfront
Sometimes it may be better to invalidate /* than to invalidate many paths (costs maybe)
Support CloudFront Invalidations
Allow Record Set to be added to an existing HostedZone
A user might already have a hosted zone for abc.com but now wants to add a static CDN to cdn.abc.com. It would be possible to create a separate hosted zone but this would require adding this hosted zones nameservers to the domain introducing additional friction.
Alternatively confetti's Cloudformation tooling could check if a hosted zone for the root domain already exists. And optionally add the required record set to the existing hosted zone.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-recordset.html
Provide way to create Cloudformation template
I'm not going to let any program access AWS with keys that can do lots of things. Maybe there is a way to create proper IAM role with only necessary policies to allow stack creation, but I don't know how that would work.
But good way allow stack creation manually would be to add option (or modify --dry-run) to print the CloudFormation JSON or Yaml template. User can then check the template is fine and create the stack from AWS Console.
Instead of pprint here, https://github.com/confetti-clj/confetti/blob/master/src/confetti/boot_confetti.clj#L193 this could call cheshire/generate-string or similar (with key-fn that converts snake-case to CamelCase). Amazonica probably already has a function to translate clj data to JSON.
Allow specification of other regions than us-east
I just took a look at the possible args for create-site, and I don't see anything immediately referencing regions.
validate credentials?
Current assertion is useless
(assert creds "Credentials are required!")
maybe no validation is required since the utility libs validate things as well.
Incorrect Clojure version
I cloned project locally and run boot
confetti git:(master) boot
Downloading https://github.com/boot-clj/boot/releases/download/2.0.0/boot.sh...done.
clojure.lang.ExceptionInfo: Unable to resolve symbol: update in this context
data: {:file "confetti/serialize.clj", :line 8}
java.lang.RuntimeException: Unable to resolve symbol: update in this context
...
clojure.core/load/fn core.clj
Probably you want to include boot.properties file int the repo with minimum requirements?
Revisit instructions
- DNS configuration
- SSL setup
- Adding GZIP
- Future Improvements is outdated
- Who is this for?
Update objects when metadata changed
I'm using :cache-control metadata for caching purposes but confetti doesn't update objects when the file has not changed and the metadata has. Not sure if it should but it would be nice if it did ;-)
Fell free to close it if this is not a reasonable feature for the library.
Problem creating site for root domain
I tried to use confetti for my clojurecup entry but hit following problem:
Tried to create site without dns (I wanted to use cloudflare). I used root domain (lifelines.io).
Got an error that it's not possible to create site for root domain without setting up dns.
Is this by design?
make create-site return earlier
an improvement could be to check if only cloudfront is outstanding and exit at that point with a note that it may take up to 10 more minutes to have the distribution synced
Required for that is that is that outputs for stacks can be fetched before creation is fully done.
Not sure if outputs are available before the stack is fully created.
Also cloudfront ID is a useful output value that most likely is not available before the distribution is fully created.
Write stack outputs to edn file in current dir
Isolate dependencies in pods
Failed to contact the origin
After setting up a site automatically, when I visit the cloudfront url, I get the error message
ERROR
Failed to contact the origin.
Looking at the CloudFront settings, it looks like the S3 bucket is configured as a custom backend. To fix it I set it up to use the S3 bucket directly (you can select the S3 bucket you want from a list when creating an origin). The origin type is "S3 origin".
Improve sync-bucket docstring
should mention what options are helpful with what, which are required and so on
print Hosted Zone name servers at the end when `dns` option is used
file-map option needs some de-/serialization
Users are supposed to specify an edn file but edn can't easily hold java.io.File objects.
- document file-map.edn should provide canonical paths (strings)
- transform those paths into file objects before passing to
sync!
Exception while deploying a large number of files
The title is actually an attempt to an explanation, but it can be something else.
The exception is the following:
Deploying on bucket devel.clojurescript.io...
Pack source files...
Sifting output files...
clojure.lang.ExceptionInfo: java.lang.RuntimeException: Method code too large!, compiling:(NO_SOURCE_PATH:0:0)
data: {:file "/tmp/boot.user8095246900604741649.clj", :line 67}
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Method code too large!, compiling:(NO_SOURCE_PATH:0:0)
java.lang.RuntimeException: Method code too large!
clojure.asm.MethodWriter.getSize MethodWriter.java: 1872
clojure.asm.ClassWriter.toByteArray ClassWriter.java: 775
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 437
...
boot.pod/eval-in* pod.clj: 440
confetti.boot-confetti/eval2817/fn/fn/fn boot_confetti.clj: 155
replumb.boot-pack-source/eval2560/fn/fn/fn boot_pack_source.clj: 45
replumb.boot-pack-source/eval2606/fn/fn/fn boot_pack_source.clj: 57
boot.task.built-in/fn/fn/fn/fn built_in.clj: 416
boot.user/eval3014/fn/fn/fn boot.user8095246900604741649.clj: 45
boot.core/run-tasks core.clj: 927
boot.core/boot/fn core.clj: 937
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
I am basically including the following dependencies:
(def deps '[[org.clojure/clojurescript "1.7.228"]
[org.clojure/core.async "0.2.374"]
[reagent "0.5.1"]
[re-frame "0.5.0"]
[replumb/replumb "0.2.2-SNAPSHOT"]
[org.clojure/tools.reader "1.0.0-alpha3"]])
And the following task:
(core/deftask count-files
[]
(core/with-pre-wrap fileset
(util/warn "Fileset count %s" (count (tmpdir/ls fileset)))
fileset))
I get Fileset count 1704. Maybe I am pushing it to the limit 😄
"Cannot create enum from eu-west-2 value"
Retrieving jsr-275-0.9.3.jar from https://repo1.maven.org/maven2/
clojure.lang.ExceptionInfo: java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
data: {:file
"/var/folders/8j/27sz_2rj0kd1b3v9y4n1_1t00000gn/T/boot.user7464173975716694273.clj",
:line 35}
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
com.amazonaws.regions.Regions.fromName Regions.java: 71
com.amazonaws.services.s3.internal.S3RequestEndpointResolver.resolveRequestEndpoint S3RequestEndpointResolver.java: 68
com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy.redirectToRegionInHeader S3V4AuthErrorRetryStrategy.java: 88
com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy.shouldRetryWithAuthParam S3V4AuthErrorRetryStrategy.java: 73
com.amazonaws.http.AmazonHttpClient.executeOneRequest AmazonHttpClient.java: 781
com.amazonaws.http.AmazonHttpClient.executeHelper AmazonHttpClient.java: 489
com.amazonaws.http.AmazonHttpClient.execute AmazonHttpClient.java: 310
com.amazonaws.services.s3.AmazonS3Client.invoke AmazonS3Client.java: 3604
com.amazonaws.services.s3.AmazonS3Client.invoke AmazonS3Client.java: 3557
com.amazonaws.services.s3.AmazonS3Client.listObjects AmazonS3Client.java: 647
...
amazonica.core/fn-call/fn core.clj: 789
amazonica.core/intern-function/fn core.clj: 860
...
confetti.s3-deploy/get-bucket-objects/fn s3_deploy.clj: 29
confetti.s3-deploy/get-bucket-objects s3_deploy.clj: 29
confetti.s3-deploy/sync! s3_deploy.clj: 134
pod$eval3061.invoke NO_SOURCE_FILE
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 369
...
boot.pod/eval-in* pod.clj: 372
confetti.boot-confetti/eval3106/fn/fn/fn boot_confetti.clj: 276
boot.task.built-in/fn/fn/fn/fn built_in.clj: 402
boot.task.built-in/fn/fn/fn/fn built_in.clj: 402
jeluard.boot-notify/eval11725/fn/fn/fn boot_notify.clj: 60
io.perun/eval2147/fn/fn/fn perun.clj: 396
io.perun/eval1651/fn/fn/fn perun.clj: 116
boot.core/run-tasks core.clj: 794
boot.core/boot/fn core.clj: 804
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
>>> elapsed time 24m53s
Don't create confetti.edn; recommend direnv
create-site never finishes
$ boot -d confetti create-site --domain domain.name --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --dns
Reporting events generated while creating your stack.
Be aware that creation of CloudFront distributions may take up to 15min.
- [create-in-progress] AWS::CloudFormation::Stack - User Initiated
- [create-in-progress] AWS::S3::Bucket
- [create-in-progress] AWS::Route53::HostedZone
- [create-in-progress] AWS::S3::Bucket - Resource creation Initiated
- [create-in-progress] AWS::Route53::HostedZone - Resource creation Initiated
- [create-complete] AWS::S3::Bucket
- [create-in-progress] AWS::S3::BucketPolicy
- [create-in-progress] AWS::CloudFront::Distribution
- [create-in-progress] AWS::IAM::User
- [create-in-progress] AWS::S3::BucketPolicy - Resource creation Initiated
- [create-in-progress] AWS::IAM::User - Resource creation Initiated
- [create-in-progress] AWS::CloudFront::Distribution - Resource creation Initiated
- [create-complete] AWS::S3::BucketPolicy
- [create-complete] AWS::Route53::HostedZone
- [create-complete] AWS::IAM::User
- [create-in-progress] AWS::IAM::AccessKey
- [create-in-progress] AWS::IAM::AccessKey - Resource creation Initiated
- [create-complete] AWS::IAM::AccessKey
The command has been stuck at this point for more than 30 minutes. The cloudfront distribution seems to have finished in the meantime.
Auhentication warning during s3 sync
Mar 14, 2018 5:59:39 PM com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy buildRetryParams
WARNING: Attempting to re-send the request to testi-metosin-fi-sitebucket-1vvyctvha8ad5.s3.eu-central-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
- [update] blog/schema-spec-web-devs/index.html
Updating of stacks
It might be useful to save the template in the working dir and allow users to modify it and update the existing stack that way.
Another possibility is to just recommend re-creating stacks if a change is desired. This might not help as much with customizations that are not directly supported at creation time though (e.g. CORS policies currently)
The parameter previousObjectListing must be specified
When I re-deploy something, and I actually don't know what triggers the error at the moment, I get:
clojure.lang.ExceptionInfo:
data: {:file "/tmp/boot.user6757680306871265920.clj", :line 67}
java.lang.reflect.InvocationTargetException:
java.lang.IllegalArgumentException: The parameter previousObjectListing must be specified.
com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest.setPreviousObjectListing ListNextBatchOfObjectsRequest.java: 51
com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest.<init> ListNextBatchOfObjectsRequest.java: 33
Do you have some hint on how to debug it?
sync-bucket NullPointerException
When I attempt to run sync-bucket, with or without -y switch, I get the following null pointer exception:
❯ boot -d confetti sync-bucket --bucket my-bucket-redacted --access-key REDACTED --secret-key REDACTED --dir target/public
clojure.lang.ExceptionInfo: java.lang.NullPointerException
data: {:file
"/var/folders/8j/27sz_2rj0kd1b3v9y4n1_1t00000gn/T/boot.user8801486601348864520.clj",
:line 35}
java.util.concurrent.ExecutionException: java.lang.NullPointerException
java.lang.NullPointerException:
...
confetti.boot-confetti/find-confetti-edn boot_confetti.clj: 53
confetti.boot-confetti/read-confetti-edn boot_confetti.clj: 58
confetti.boot-confetti/eval282/fn/fn/fn boot_confetti.clj: 223
boot.core/run-tasks core.clj: 794
boot.core/boot/fn core.clj: 804
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
The exact same error occurs when I run sync-bucket as part of a comp pipeline.
Problem creating site
What I did:
- Cloned confetti locally and run
boot build-jar - Required confetti into my project boot file: https://github.com/hashobject/code.hashobject.com/blob/master/build.boot
- Run
boot create-site --domain "code.hashobject.com" - Got error:
clojure.lang.ExceptionInfo: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
data: {:file
"/var/folders/7d/9c32qg7x2_z9slnznbhq2dg80000gn/T/boot.user7083640999180883928.clj",
:line 27}
java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
clojure.lang.Compiler$CompilerException: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient
...
amazonica.core/eval523/loading--auto-- core.clj: 1
amazonica.core/eval523 core.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
amazonica.aws.cloudformation/eval517/loading--auto-- cloudformation.clj: 1
amazonica.aws.cloudformation/eval517 cloudformation.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
confetti.cloudformation/eval58/loading--auto-- cloudformation.clj: 1
confetti.cloudformation/eval58 cloudformation.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
pod$eval54.invoke NO_SOURCE_FILE
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 224
...
boot.pod/eval-in* pod.clj: 227
confetti.boot-confetti/prep-pod boot_confetti.clj: 21
confetti.boot-confetti/eval10209/fn/fn/fn boot_confetti.clj: 56
boot.core/run-tasks core.clj: 730
boot.core/boot/fn core.clj: 740
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
created user should be allowed to invalidate cloudfront cache items
Names
- bloy (via namewhale)
Cloudflare
I would suggest looking also into Cloudflare and potentially implement integration into confetti.
I want to share my current setup/process for static sites.
- I buy domain at namecheap.com
- I use stout which will create s3 bucket and cloudfront distribution
- I go to cloudflare, setup new domain.
- Change name servers in namecheap to point to cloudflare
- Setup CNAME in the cloudflare pointing to the cloudfront distribution host
- Setup page rules in cloudflare to automatically redirect http to https
I used to use Route53 before cloudflare. But now I think Cloudflare is better because:
- it's free and for Route53 you still need to pay
- it has free SSL, not setup required
- it has caching features, analytics etc
It seems that cloudflare has nice API and I suggest looking into it so instead of setting up Route53 and provisioning certificates user would be able to get same features using cloudflare (and for free).
I think if I have cloudflare support I would use confetti instead of stout:)
exclude unused aws java libs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.