dkataskin / erlazure Goto Github PK
View Code? Open in Web Editor NEWWindows Azure Erlang bindings
License: BSD 3-Clause "New" or "Revised" License
Windows Azure Erlang bindings
License: BSD 3-Clause "New" or "Revised" License
Hi,
thanks for creating erlazure.
I can list and create containers, but even on 1kb files i get time out when i try to upload something.
:erlazure.put_block_blob(pid, 'test', 'x.pdf', File.read!("x.pdf"))
Please help me. I use elixir 1.2.4 with erlang 18.3
From what I understood from the source code erlazure is a genserver using synchronous server calls. Calling new put_block_blob
functions on a single process will just queue up my requests and will take them one after another from the process mailbox. Also erlazure server process created with a start
method never exits (or is clean up).
What I'm trying to achieve are concurrent uploads to the Blob Storage, but starting new erlazure process in every upload process seems like an overkill, because started processes are never cleaned up and this can easily lead to system blowing up.
Am I right with my assumptions? What are the best ways to clean up erlazure process after every upload gracefully?
Thanks!
Hiho @dkataskin,
OTP24 removed some crypto functions which were deprecated in OTP23
http://erlang.org/doc/general_info/deprecations.html
I do not write erlang usually but e.g. https://github.com/epgsql/epgsql/pull/239/files seem to fix it in a backwards compatible way.
I tried to create a PR but the formatting is completely messed up 🤦
#31
Are there plans to support OTP24?
Best regards.
Get messages operation doesn't return the full list of messages.
It would be good to handle transient faults.
Any way to set the host for blob operations? Need to work with both an emulator and with the Azure Government services.
I appreciate your work on this library.
What would you say to having get_host
be a handle_call
and pull from the state of the actor instead, and default it to use the core.windows.net
domain if nothing else is configured? We're looking at running functional tests and it would be neat to use azurite in a container for running the more basic tests.
EDIT: If you are interested but swamped and would prefer I make a PR to review, I can take a swipe at it.
I have a super simple piece of code to upload file to my container called test
:
defmodule Azure do
def upload do
account = "<SECRET_ACCOUNT_NAME>"
key = "<SECRET_ACCOUNT_KEY>"
{:ok, pid} = :erlazure.start(account, key)
file = File.read!(Path.join([Path.expand("~/Downloads"), "srs.pdf"]))
:erlazure.put_block_blob(
pid,
"test",
"srs.pdf",
file
)
end
end
When I run this in iex
console it blows up and hangs the process with following stacktrace:
Interactive Elixir (1.7.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Azure.upload
** (EXIT from #PID<0.144.0>) shell process exited with reason: an exception was raised:
** (FunctionClauseError) no function clause matching in :lists.thing_to_list/1
(stdlib) lists.erl:603: :lists.thing_to_list("test")
(stdlib) lists.erl:1250: :lists.flatmap/2
(erlazure) /Users/lukaleli/Dev/azure/deps/erlazure/src/erlazure.erl:521: :erlazure.handle_call/3
(stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:665: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Interactive Elixir (1.7.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
18:54:32.344 [error] GenServer #PID<0.146.0> terminating
** (FunctionClauseError) no function clause matching in :lists.thing_to_list/1
(stdlib) lists.erl:603: :lists.thing_to_list("test")
(stdlib) lists.erl:1250: :lists.flatmap/2
(erlazure) /Users/lukaleli/Dev/azure/deps/erlazure/src/erlazure.erl:521: :erlazure.handle_call/3
(stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:665: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.144.0>): {:put_blob, "test", "srs.pdf", :block_blob, <<37, 80, 68, 70, 45, 49, 46, 51, 10, 37, 196, 229, 242, 229, 235, 167, 243, 160, 208, 196, 198, 10, 52, 32, 48, 32, 111, 98, 106, 10, 60, 60, 32, 47, 76, 101, 110, 103, 116, 104, 32, 53, 32, 48, 32, ...>>, []}
State: {:state, "<SECRET_ACCOUNT_NAME>", "<SECRET_ACCOUNT_KEY>", [], [blob_block_id: {:param_spec, :blob_block_id, :uri, 'blockid', #Function<10.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_content_length: {:param_spec, :blob_content_length, :header, 'x-ms-blob-content-length', #Function<14.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_copy_source: {:param_spec, :blob_copy_source, :header, 'x-ms-copy-source', #Function<13.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_range: {:param_spec, :blob_range, :header, 'Range', #Function<12.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_type: {:param_spec, :blob_type, :header, 'x-ms-blob-type', #Function<20.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, block_list_type: {:param_spec, :block_list_type, :uri, 'blocklisttype', #Function<9.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, comp: {:param_spec, :comp, :uri, 'comp', #Function<6.83802320/1 in :erlazure.get_req_common_param_specs/0>}, include: {:param_spec, :include, :uri, 'include', #Function<10.83802320/1 in :erlazure.get_req_common_param_specs/0>}, lease_action: {:param_spec, :lease_action, :header, 'x-ms-lease-action', #Function<19.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_break_period: {:param_spec, :lease_break_period, :header, 'x-ms-break-period', #Function<18.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_duration: {:param_spec, :lease_duration, :header, 'x-ms-lease-duration', #Function<17.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_id: {:param_spec, :lease_id, :header, 'x-ms-lease-id', #Function<16.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, marker: {:param_spec, :marker, :uri, 'marker', #Function<11.83802320/1 in :erlazure.get_req_common_param_specs/0>}, max_results: {:param_spec, :max_results, :uri, 'maxresults', #Function<8.83802320/1 in :erlazure.get_req_common_param_specs/0>}, message_ttl: {:param_spec, :message_ttl, :uri, 'messagettl', #Function<9.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, message_visibility_timeout: {:param_spec, :message_visibility_timeout, :uri, 'visibilitytimeout', #Function<10.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, num_of_messages: {:param_spec, :num_of_messages, :uri, 'numofmessages', #Function<6.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, peek_only: {:param_spec, :peek_only, :uri, 'peekonly', #Function<8.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, pop_receipt: {:param_spec, :pop_receipt, :uri, 'popreceipt', #Function<7.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, prefix: {:param_spec, :prefix, :uri, 'prefix', #Function<9.83802320/1 in :erlazure.get_req_common_param_specs/0>}, proposed_lease_id: {:param_spec, :proposed_lease_id, :header, 'x-ms-proposed-lease-id', #Function<15.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, res_type: {:param_spec, :res_type, :uri, 'restype', #Function<11.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, timeout: {:param_spec, :timeout, :uri, 'timeout', #Function<7.83802320/1 in :erlazure.get_req_common_param_specs/0>}]}
Client #PID<0.144.0> is alive
(stdlib) gen.erl:169: :gen.do_call/4
(stdlib) gen_server.erl:210: :gen_server.call/3
(stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
(elixir) src/elixir.erl:233: :elixir.eval_forms/4
(iex) lib/iex/evaluator.ex:245: IEx.Evaluator.handle_eval/5
(iex) lib/iex/evaluator.ex:225: IEx.Evaluator.do_eval/3
(iex) lib/iex/evaluator.ex:203: IEx.Evaluator.eval/3
(iex) lib/iex/evaluator.ex:89: IEx.Evaluator.loop/1
Looks like genserver has no corresponding handler that matches calling pattern.
NOTE: I've checked if erlazure
connects successfully to the Azure and it does. I was able to successfully list containers.
NOTE2: I'm using Elixir version 1.7.0-dev
, but I'm not sure if it really affects that...
When trying to create a block blob I ran into the following issue from my Elixir app.
** (exit) bad return value: "<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.\nRequestId:41797b79-0001-0002-0d7d-93ecbe000000\nTime:2015-05-21T04:18:52.3985127Z</Message><HeaderName>x-ms-blob-type</HeaderName><HeaderValue>block_blob</HeaderValue></Error>"
I trace the error down to the call at
erlazure/src/erlazure_blob.erl
Line 196 in d27e076
where erlang:atom_to_list
on block_blob
returns block_blob instead of BlockBlob. I was able to quick fix the issue for my use case jakubgarfield@edfb1e5 but my Erlang knowledge prevents me from fixing it properly.
Parse and present errors returned as result of a request.
Hello @dkataskin,
I have upgraded to OTP 24, and getting issues in erlazure, the errors comes while uploading files into azure storage.
Could you please help to resolve this issue as this is a urgent requirement.
Attached Error File
erlazure-Error-OTP24.txt
Implement Set Queue Acl operation.
Test functionality against cloud.
Hey!
Is it possible to set "Content-Disposition" header on the uploaded blob?
I am using erlazure via ex_azure and the call to put_blob
fails with:
[error] GenServer #PID<0.1678.0> terminating
** (stop) bad return value: "\uFEFF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.\nRequestId:b66001fa-d01e-005e-308e-3d06c6000000\nTime:2019-07-18T17:32:06.5364180Z</Message></Error>"
Last message (from #PID<0.93.0>): {:put_blob, 'container', 'new_file.pdf', :block_blob, <<37, 80, 68, 70, 45, 49, 46, 51, 10, 37, 255, 255, 255, 255, 10, 49, 32, 48, 32, 111, 98, 106, 10, 60, 60, 32, 47, 67, 114, 101, 97, 116, 111, 114, 32, 60, 102, 101, 102, 102, 48, 48, 53, 48, 48, ...>>,
When I was trying to find the culprit in ex_azure I realized it simply just calls handle_call({put_blob, Container, Name, Type = block_blob, Data, Options}, _From, State) ->
from erlazure.
The issue is that Azure returns the XML error, but there is no graceful failure of it. It simply makes the GenServer to fail... I would probably except that an error tuple with the error message is returned.
1> {ok,Pid} = erlazure:start("key", "sig").
{ok,<0.389.0>}
2> erlazure:create_container(Pid, "test").
09:41:52.015 [error] gen_server <0.389.0> terminated with reason: bad return value: <<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n">>
09:41:52.020 [error] CRASH REPORT Process <0.389.0> with 1 neighbours exited with reason: bad return value: <<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n">> in gen_server:terminate/7 line 826
** exception exit: {bad_return_value,<<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITL"...>>}
Peek messages operation fails
Update service versions to 2013.
Get queue acl operation should return signed_id record.
Add type and function specifications.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.