dwyl / imgup Goto Github PK
View Code? Open in Web Editor NEWπ Effortless image uploads to AWS S3 with automatic resizing including REST API.
Home Page: https://imgup.fly.dev/
π Effortless image uploads to AWS S3 with automatic resizing including REST API.
Home Page: https://imgup.fly.dev/
When running the application, I get this error: ReferenceError: require is not defined
I tried to remove the npm package and reinstall it with mrt.
At present the automatic continuous deployment (CD) from GitHub actions to Fly.io for the imgup
app works partially.
https://imgup.fly.dev/liveview
However the REST API
fails when we attempt to upload in Hoppscotch
:
The app was recently deployed: https://fly.io/apps/imgup
In the logs following the failed API
request we see the following error:
https://fly.io/apps/imgup/monitoring
[error] ** (MatchError) no match of right hand side value: {:error, "Required key: :secret_access_key is nil in config!"}
Which relates to the following line of code:
Not super informative, but we know the upload is failing due to unavailability of the AWS :secret_access_key
... π
So we know what needs to be fixed ... π€
AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
from ci.yml
:imgup/.github/workflows/ci.yml
Lines 61 to 62 in 7eb43e7
FLY_API_TOKEN
to the dwyl
account.At present this repo is just a loose collection of notes and some sample code.
What it should be instead is an end-to-end working example of uploading, compressing, resizing, rotating and cropping images.
Our goal with image uploads is to be able to seamlessly upload any type of image
have it saved in a reliable place e.g. S3
(though we can research cheaper comparable options later...)
and display that image in our App
and other places.
We want to be able to upload in a Web
interface and in a Flutter
App.
_archive
directory in case we need to refer to any of it.
README.md
to _archive
too so you have a fresh start.Note: it will all be
DELETED
once we have fully working version.
LiveView
Uploads Demo:Phoenix LiveView Uploads Deep Dive: https://youtu.be/PffpT2eslH8
Create a new Phoenix
Project and replicate what Chris but document the steps
e.g: https://github.com/dwyl/phoenix-chat-example
Upload images to AWS S3
: https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073
Use a Content ID
cid
as the image name so that its unique.
Once images are uploaded to S3
, they should be "processed" by a Lambda
function.
Note: resizing and compressing images is a well-known/resolved challenge:
https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/
In-browser resizing. This will be it's own Epic
which will start with research for existing solutions. π
Deploy the app to Fly.io: https://fly.io/dashboard/dwyl-image-uploads/ e.g: https://image-upload.fly.dev/ π
If we can get to the point where we are uploading images in a Phoenix LiveView
App we will already be showing great progress.
Don't worry too much about the UI/UX at this stage. Use TailwindUI
:
https://tailwindui.com/components/application-ui/forms/form-layouts
@LuchoTurtle as discussed on standup this morning, this should be the next "big chunk" of work you should attempt to tackle.
If you have questions, please comment. π
I am getting this warning whilst following the SDK example:
I chose the following access permissions:
Based on the guidance from this step of the existing tutorial:
I wanted to get a second opinion on whether I had interpreted the updated version of the steps correctly and therefore whether I should react to this warning message?
The REST API
PR #61 was merged. β
But sadly when we attempt to upload an image via Hoppscotch
we get the following error:
https://imgup.fly.dev/api/images
Logs: https://fly.io/apps/imgup/monitoring
[error] ** (UndefinedFunctionError) function :hackney.request/5 is undefined (module :hackney is not available)
[info] :hackney.request(:post, "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhg1D3wGz3ps1wsUURdkSPLeFG5NTihkMtnpHHnXnFvSSo.png?uploads=1",
[{"Authorization", "AWS4-HMAC-SHA256
Credential=AKIAETC/20230614/eu-west-3/s3/aws4_request,SignedHeaders=content-length;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=65011ed0fc91075d3d429cda2ab70eda47ae4d3b214f187b34dbb0e8de8244a4"},
{"host", "s3.eu-west-3.amazonaws.com"}, {"x-amz-date", "20230614T102209Z"},
{"content-length", "0"}, {"x-amz-acl", "public-read"},
{"x-amz-content-sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}],
"", [:with_body, {:recv_timeout, 30000}])
Very curious why the {"content-length", "0"}
is it not reading the file? π
@LuchoTurtle do you mind investigating this? π
Feel free to:
a) revert back to your version of the API from before I made changes.
b) deploy directly to Fly to figure out why the uploads don't work.
There are two example setups in the 'examples' folder. One is an SDK and the other a 'direct' example.
I wanted to know why I might choose to use one version over the other?
eBook by Addy Osmani https://images.guide
Discussion:
https://news.ycombinator.com/item?id=19474292
To ensure that everything works all of the time we need to have a progressive enhancement demo.
I'm having issues with the config.json file I think (credentials or permissions).
This is the error I'm getting on the frontend side:
This one on Meteor console:
If I try loading the link that gave an error on the frontend I get this:
S3 account:
config.json file on the App:
I gave a bucket policy for testingboris since I was getting denied error but didn't do the trick:
It seems to be a permission issue with the bucket.
Another thing I had to do with the project was to change require() to Meteor.require if not I got an error on console. (require is not defined).
Really Appreciate some help on this!
I want to use this repo for an elixir project and saw the section about setting up a node server and using node packages crypto
and path
. I don't think it's good practice to have the two running simultaneously.
We don't have the time on this occasion to rewrite this repo entirely in elixir and so I wanted to know if there is another alternative?
How should we test the image uploading functionality on a project?
I can see this nightwatch example on this repo:
https://github.com/dwyl/image-uploads/blob/master/examples/sdk-upload/test/e2e/sdk-upload.test.js
However my concern is that every time you perform this test you are uploading a new file to S3. Unless someone in a team remembers to delete the test files on a regular basis you could end up with a lot of files. This comes as a cost on S3 if it was a large amount of data being stored.
Would it be possible/better to delete the files once they're uploaded? Or can we mock the upload in some way but still test it?
Image uploading is a common feature in many web/mobile applications.
We are going to be using the Image Upload feature extensively in
Time
to allow people to upload a photo showing evidence of their completed activity.
(if you cannot visualise the use-case of this, please comment in: dwyl/app#215)
Additionally there are several classes of applications where image uploading is relevant:
By building a clearly-explained Image Upload example/tutorial with re-useable code,
we can help others get creative and spawn new Apps/Projects in the community.
Questions we need to ask/answer:
/upload
endpoint to avoid people using it maliciously to upload undesirable images.
Origin
of the upload?
Origin
be spoofed by a command-line script?I'm not saying that we are going to create an "image upload" product,
But the ability to upload files/images is a "standalone" feature that people pay for!
e.g: https://apps.shopify.com/uploadery
This is a feature that has real value to end-users so if we build something great
not only will the people using the @dwyl app find it useful,
the wider community will use it and help us to improve it!
We are able to upload images to S3
via REST API
π
URL: https://imgup.fly.dev/api/images
Docs: https://github.com/dwyl/imgup/blob/main/api.md#4-testing-the-api-from-hoppscotch
{
"compressed_url": "https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhoFgi1PXSfoeFi9PGT3R33fpq4iAsQJRuU2Yoy7yMzhj3.jpg",
"url": "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhoFgi1PXSfoeFi9PGT3R33fpq4iAsQJRuU2Yoy7yMzhj3.jpg"
}
The problem is when we attempt to view one of these images directly in the browser e.g. by copy-pasting the URL
into a new browser tab: https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhoFgi1PXSfoeFi9PGT3R33fpq4iAsQJRuU2Yoy7yMzhj3.jpg
It automatically downloads the file instead of displaying it in the browser.
I suspect this is because the MIME Type is not correctly set on S3
...
This is why it does not display the image in a GitHub
issue comment or .md
file ...
![img](https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhoFgi1PXSfoeFi9PGT3R33fpq4iAsQJRuU2Yoy7yMzhj3.jpg)
... π€·ββοΈ
@LuchoTurtle do you have time to investigate this?
I've tried reading the docs: https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#t:initiate_multipart_upload_opt/0
but can't seem to find where to put the Content-Type for the uploaded file ... π€·ββοΈ
We can do this using Nightwatch.js:
see: http://stackoverflow.com/questions/25919673/file-upload-testing-in-nightwatch-js
and: https://github.com/dwyl/learn-nightwatch
_or_ Karma with PhantomJS
https://github.com/dwyl/learn-karma
What is the best way to allow for multiple image uploads?
For an e-commerce site where a product can has one primary photo plus multiple secondary photos, what is the best way to enable uploading multiple images?
Question 1 is of the highest priority for me right now but I will also want to consider the other questions in due course.
Instead of uploading the file to AWS S3 as a single blob we should be streaming the file to S3.
This avoids "blocking" while we wait for the file to be uploaded.
Once we have the basic Web-based image uploading working there are many enhancements we can make. β¨
Hopefully by making everything Open Source - as always - we invite contributions from the community. π€
However from our perspective @dwyl what we want is the ability to upload from our Flutter
(Native
Mobile) App.
JS
or Flutter
to upload an imageNote: this issue is not complete. It's a place-holder for the discussion around features/requirements.
Once we have #51 working in the Web interface, this is the next logical step.
looks like old package is no longer available, thus project does not work after cloning git repo
The "demo" version of the imgup
App will be open to allow people to test it.
We will not DELETE
the files unless the AWS S3
bill gets out of hand e.g: bots start uploading.
I don't expect nefarious people to discover/use the imgup
App ...
but if they do we will rapidly respond and put all uploading behind Auth
.
The REST API
will have Auth
ASAP because that is where abuse is most likely to originate.
See:
But we need to enable logging
: dwyl/logs#1
so that we can:
A. See who
is trying the demo and from where.
At present there is a policy to automatically DELETE
images after a period of time.
While this is useful for some projects, it's not a requirement for us.
Note: we can easily manually delete images if needed.
https://github.com/dwyl/imgup/actions/runs/5261384190/jobs/9509426771#step:4:26
/home/runner/work/_temp/.setup-beam/elixir/bin/mix local.rebar --force
** (System.EnvError) could not fetch environment variable "AWS_ACCESS_KEY_ID" because it is not set
(elixir 1.14.2) lib/system.ex:706: System.fetch_env!/1
/home/runner/work/imgup/imgup/config/config.exs:54: (file)
(stdlib 4.1.1) erl_eval.erl:748: :erl_eval.do_apply/7
(stdlib 4.1.1) erl_eval.erl:961: :erl_eval.expr_list/7
(stdlib 4.1.1) erl_eval.erl:290: :erl_eval.expr/6
(stdlib 4.1.1) erl_eval.erl:[28](https://github.com/dwyl/imgup/actions/runs/5261384190/jobs/9509426771#step:4:31)2: :erl_eval.expr/6
(stdlib 4.1.1) erl_eval.erl:961: :erl_eval.expr_list/7
Error: The process '/home/runner/work/_temp/.setup-beam/elixir/bin/mix' failed with exit code 1
Tried to add: https://github.com/dwyl/imgup/settings/secrets/actions but got the following error:
Failed to add secret, a secret with the same name (AWS_ACCESS_KEY_ID) already exists.
The time has come to update (revive) this project and bring it into 2016!
git mv README.md REF.md
(temporarily keep a reference to the old README.md so we can pluck sections from it e.g. Research on Orientation and other uploaders etc.)Next we need to Remove all references to Meteor (people writing meteor apps will still be able to use this project, but we need to break the tight-coupling)
git rm -r .meteor
git rm -r client
git rm -r packages
git rm -r private
git rm -r public
git mv server lib
git mv lib/server_save_file.js lib/index.js
(we will remove most of the code in this file shortly but it gives us some history #nostalgia ...)git rm npm-debug.log
git mv packages.json package.json
(convert from Meteor project to standard node.js project)git rm npm-debug.log
.gitignore
with the relevant definitions from https://github.com/github/gitignore/blob/master/Node.gitignore (_we only need to ignore node_modules
, npm-debug.log*
and coverage
for now...git rm smart.json
git rm smart.lock
git rm style
git rm style.css
Hi @jackcarlisle please create a pull request with the above changes and call it "reset" and assign to me. Then we can start working on the actual features... thanks! π
I am working on a mobile-first project that is using image uploads for product images. In this instance the products are drinks (bottles). The client has asked what specs they should follow for providing the images for these products. We want to ensure images are of a good enough quality but that they also load quickly. So I want to know what specs I should request?
Should we allow people to upload to Google Drive? (is this a useful feature) ?
A server side example might be easier to implement for someone who already has an existing code base. The direct form upload example might clash with existing forms because they can't be easily nested.
We are not using nightwatch in our project at the moment, but the tests in the sdk-upload example are only done using nightwatch. It would be really helpful to see another way to test this upload method.
We are using Hapi's server.inject
method and after a lot of playing around, eventually managed to upload a file; we only managed this with a .txt file. There are a couple of working examples linked at the bottom which may hold the solution.
Here is a link to the PR on my project that contains the image upload with a test that successfully uploads a file to our S3 bucket using server.inject.
I thought it would be useful to copy our method into an issue here, and we can develop it as and when we work out a better/more comprehensive way of testing
// our form has the inputs: name, file_name, mission_statement, logo
// logo represents the file, and file_name is the name of the file being uploaded
// name and mission statement are additional fields that our form needs to track
// at the moment, we are struggling to upload an image, so the logo is actually a .txt file
var payloadString =
'--AaB03x\r\n' +
'content-disposition: form-data; name="name"\r\n' + // input on our form called `name`
'\r\n' +
'Apple\r\n' + // The value of the input `name` is "Apple"
'--AaB03x\r\n' +
'content-disposition: form-data; name="file_name"\r\n' + // input on our form called `file_name`
'\r\n' +
'foxy.txt\r\n' + // The value of the input `file_name` is "foxy.txt"
'--AaB03x\r\n' +
'content-disposition: form-data; name="mission_statement"\r\n' + // input on our form called `mission_statement`
'\r\n' +
'Change the economy!\r\n' + // The value of the input `mission_statement` is "Change the economy!"
'--AaB03x\r\n' +
'content-disposition: form-data; name="logo"; filename="foxy.txt"\r\n' + // input on our form called `logo`
'Content-Type: text/plain\r\n' + // content type of the file being sent in the form
'\r\n' +
'foxxxxyy\r\r\n' + // contents of the file being sent in the form
'--AaB03x--\r\n'
var addLogoOptions = {
method: 'POST',
url: '/add-image',
headers: {
'Content-Type': 'multipart/form-data; boundary=AaB03x' // notice this boundary repeated before/after each input in the payload string
},
payload: payloadString
};
}
// the tape test
tape('/add-image route successfully adds an image to S3 --> ' + __filename, function (t) {
server.inject(addLogoOptions, function (res) {
t.equal(res.statusCode, 200, 'status code is 200 we can upload a file to S3');
t.end();
});
});
This test passes, the payload string was taken and adapted from this issue on the hapi repo
It looks like there might be a slightly simpler way of testing the file upload in this issue
Finally this is a working example of the potentially more simple way of solving this problem
Related to #51
AWS
access to @LuchoTurtle for uploading images to S3
and processing on Lambda
At present the imgup
app has no DB
π
So we have zero insight into what images are being uploaded. π π€·ββοΈ
This is "OK" for a basic demo...
But it's not great for a "production" app where we want to track metrics. π
We need to "productionize"! π
DB
schema for storing anon logging data.dwyl/auth/lib/auth/log.ex
user_agent
table should mimmic the one in auth
: lib/auth/user_agent.ex
so that we store the relevant detail. The other field we could add ismigrations
for the schema we define.Getting the following error when attempting to upload on: https://imgup.fly.dev/liveview
Couldn't upload files to S3. Open an issue on Github and contact the repo owner.
@LuchoTurtle can you please investigate why if/when you have time? Thanks. π
Note: this is almost certainly related to the work done for the "Edge Cases" PR: #86 / #69
The tests all pass on that branch so either theLiveView
upload is insufficiently tested or the tests are afalse
positive. π
The ideal (MVP) would be to provide people with a simple HTML form that POSTs an image to S3 and returns a URL to that image:
@jackcarlisle do you have time to look into this?
With most cloud storage services there is a cost associated with storing photos.
Granted, the cost is low e.g $0.03 per GB (per month) on AWS S3 ... but it can add up if you have many photos e.g. 100GB is $36 per year and that would only increase over time...
So, if instead of using AWS S3 we use Google Drive or Google Photos we get 15 GB "Free":
or "Unlimited Free" (Google Photos):
See:
- Drive Storage: https://www.google.com/settings/storage
- Photos: https://www.google.com/photos/about/
- http://www.makeuseof.com/tag/free-unlimited-photo-storage-more-google-photos/
- http://www.marketwatch.com/story/google-will-store-your-photos-for-free-but-is-there-a-privacy-cost-2015-06-03
- http://www.extremetech.com/mobile/206990-google-photos-unlimited-storage-for-free-with-a-few-gotchas
@iteles thoughts...?
Possibly use: https://github.com/hevnly/image-resizer ?
As discussed in #70 (review), the buckets should be env variables because they are used across the application to upload files, retrieve the URLs of the latter, and show them to the person using the app/API.
Because the bucket names are implicitly used in these URLs and when creating the needed metadata to upload files from the LiveView, we ought to have them config
'd and use them from there.
We are trying to implement an image upload from our server.
We need to replicate the form submit from the direct-upload example on the back end but we're having trouble structuring our http POST request.
The form data that we want to send are the params returned from our getS3Credentials
function.
The url of the form is the endpoint url from the same function.
Our form looks like this:
<form>
<input id="fileInput" type="file" name="file" onchange="uploadDemo.saveFile(this.files)"/>
</form>
<button id="submit" onclick="uploadDemo.submitFile()">Submit</button>
And this is what we're dynamically adding before we submit the form:
function buildAndSubmitForm (s3Data) {
var form = document.querySelector('form')
var keyInput = document.createElement('input')
keyInput.setAttribute('type', 'hidden')
keyInput.setAttribute('name', 'key')
keyInput.setAttribute('value', s3Data.filename)
form.setAttribute('method', 'post')
form.setAttribute('action', s3Data.endpoint_url)
form.setAttribute('enctype', 'multipart/form-data')
form.insertBefore(keyInput, form.firstChild)
form.url = s3Data.endpoint_url
form.formData = s3Data.params
form.submit()
}
Does anyone know how our POST request should look in order for our form data to be sent in the same fashion as our front end example?
The app is deployed to Fly: https://imgup.fly.dev/
But when we attempt to upload via the REST API
in Hoppscotch
we see the following error:
{
"errors": {
"detail": "Error uploading file #26"
}
}
Viewing the monitoring: https://fly.io/apps/imgup/monitoring we see the following error in the logs:
** (MatchError) no match of right hand side value: {:error, "Required key: :secret_access_key is nil in config!"}
Just updated the Secrets: https://fly.io/apps/imgup/secrets
Still getting the same issue. Works on localhost
... π€·ββοΈ
Going to investigate this now as the code is still fresh in my head. π
S3 works well. see: https://github.com/nelsonic/stream-to-s3 (which is in the process of being updated...)
could we use the code to upload any kind of file...?
At present Unique File Names are using DateTime.utc_now()
:
This is does not take advantage of cid
.
The idea of cid
is that when a file is uploaded the name of the file is based on the actual content of the file ...
Therefore we need to read the binary of the file and feed that into cid/1
such that when the same file is uploaded again, it will have the same filename on S3
.
The reason we want to do this is simple: we avoid duplicates.
key
variable to use a cid
of the actual contents of the file.W20150221-18:55:05.788(-8)? (STDERR) ReferenceError: require is not defined
W20150221-18:55:05.788(-8)? (STDERR) at app/node_modules/imagemagick/imagemagick.js:1:52
W20150221-18:55:05.789(-8)? (STDERR) at app/node_modules/imagemagick/imagemagick.js:417:3
I've spotted some possible errors with the following code:
Lines 23 to 28 in 81de078
{:ok, file_binary} = File.read(image.path)
This will break if File.read
returns an error, {:errror, reason}
as the match will fail, see https://hexdocs.pm/elixir/1.12/File.html#read/1
file_cid = Cid.cid(file_binary)
The Cid.cid
function returns a string, however the string can represent an error: invalid data type
, see https://hexdocs.pm/excid/Cid.html#cid/1
Not an error in itself but I think we can make this a bit more readable:
file_extension = image.content_type
|> MIME.extensions()
|> List.first()
file_name = "#{file_cid}.#{file_extension}"
Also MIME.extension
can return an empty list, so in this case the file extension would be nil
. Do you still want to upload the file if the extension is unknown? If yes maybe the file name should be just the file_cid
see https://hexdocs.pm/mime/MIME.html#extensions/1-examples
As a user attempting to upload an image,
I would like to see a progress bar
So that I know that my file is uploading ...
see: https://www.poeticoding.com/add-a-progress-bar-in-the-phoenix-file-upload-app/
At present the compressed images work fine until we have a wider image ...
https://jsfiddle.net/6t7p0eqc/7/
AWS
: https://github.com/dwyl/imgup#7-resizingcompressing-files.resize
code that currently creates a 200x200
square:imgup/an_aws_sam_imgup-compressor/src/index.js
Lines 20 to 24 in a6596c6
Should only constrain in one dimension e.g: only width
or only height
i.e. preserve the aspect ratio of images.
Redeploy the compressor
app on AWS
.
Confirm that compressed images are preserving aspect ratio (not adding black border)
Create a flow diagram to explain how this works!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.