Git Product home page Git Product logo

erlazure's People

Contributors

brunotcouto avatar dkataskin avatar jacobfunch avatar jgmchan avatar jparadasb avatar paranojik avatar percygrunwald avatar yatagan avatar zhaox avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

erlazure's Issues

time out on small files

Hi,

thanks for creating erlazure.

I can list and create containers, but even on 1kb files i get time out when i try to upload something.
:erlazure.put_block_blob(pid, 'test', 'x.pdf', File.read!("x.pdf"))

Please help me. I use elixir 1.2.4 with erlang 18.3

erlazure concurrency

From what I understood from the source code erlazure is a genserver using synchronous server calls. Calling new put_block_blob functions on a single process will just queue up my requests and will take them one after another from the process mailbox. Also erlazure server process created with a start method never exits (or is clean up).
What I'm trying to achieve are concurrent uploads to the Blob Storage, but starting new erlazure process in every upload process seems like an overkill, because started processes are never cleaned up and this can easily lead to system blowing up.
Am I right with my assumptions? What are the best ways to clean up erlazure process after every upload gracefully?

Thanks!

Set host?

Any way to set the host for blob operations? Need to work with both an emulator and with the Azure Government services.

erlazure:get_host to pull from state

I appreciate your work on this library.

What would you say to having get_host be a handle_call and pull from the state of the actor instead, and default it to use the core.windows.net domain if nothing else is configured? We're looking at running functional tests and it would be neat to use azurite in a container for running the more basic tests.

EDIT: If you are interested but swamped and would prefer I make a PR to review, I can take a swipe at it.

Uploading blob results in exception

I have a super simple piece of code to upload file to my container called test:

defmodule Azure do
  def upload do
    account = "<SECRET_ACCOUNT_NAME>"

    key = "<SECRET_ACCOUNT_KEY>"

    {:ok, pid} = :erlazure.start(account, key)

    file = File.read!(Path.join([Path.expand("~/Downloads"), "srs.pdf"]))

    :erlazure.put_block_blob(
      pid,
      "test",
      "srs.pdf",
      file
    )
  end
end

When I run this in iex console it blows up and hangs the process with following stacktrace:

Interactive Elixir (1.7.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Azure.upload
** (EXIT from #PID<0.144.0>) shell process exited with reason: an exception was raised:
    ** (FunctionClauseError) no function clause matching in :lists.thing_to_list/1
        (stdlib) lists.erl:603: :lists.thing_to_list("test")
        (stdlib) lists.erl:1250: :lists.flatmap/2
        (erlazure) /Users/lukaleli/Dev/azure/deps/erlazure/src/erlazure.erl:521: :erlazure.handle_call/3
        (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
        (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.7.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
18:54:32.344 [error] GenServer #PID<0.146.0> terminating
** (FunctionClauseError) no function clause matching in :lists.thing_to_list/1
    (stdlib) lists.erl:603: :lists.thing_to_list("test")
    (stdlib) lists.erl:1250: :lists.flatmap/2
    (erlazure) /Users/lukaleli/Dev/azure/deps/erlazure/src/erlazure.erl:521: :erlazure.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.144.0>): {:put_blob, "test", "srs.pdf", :block_blob, <<37, 80, 68, 70, 45, 49, 46, 51, 10, 37, 196, 229, 242, 229, 235, 167, 243, 160, 208, 196, 198, 10, 52, 32, 48, 32, 111, 98, 106, 10, 60, 60, 32, 47, 76, 101, 110, 103, 116, 104, 32, 53, 32, 48, 32, ...>>, []}
State: {:state, "<SECRET_ACCOUNT_NAME>", "<SECRET_ACCOUNT_KEY>", [], [blob_block_id: {:param_spec, :blob_block_id, :uri, 'blockid', #Function<10.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_content_length: {:param_spec, :blob_content_length, :header, 'x-ms-blob-content-length', #Function<14.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_copy_source: {:param_spec, :blob_copy_source, :header, 'x-ms-copy-source', #Function<13.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_range: {:param_spec, :blob_range, :header, 'Range', #Function<12.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, blob_type: {:param_spec, :blob_type, :header, 'x-ms-blob-type', #Function<20.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, block_list_type: {:param_spec, :block_list_type, :uri, 'blocklisttype', #Function<9.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, comp: {:param_spec, :comp, :uri, 'comp', #Function<6.83802320/1 in :erlazure.get_req_common_param_specs/0>}, include: {:param_spec, :include, :uri, 'include', #Function<10.83802320/1 in :erlazure.get_req_common_param_specs/0>}, lease_action: {:param_spec, :lease_action, :header, 'x-ms-lease-action', #Function<19.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_break_period: {:param_spec, :lease_break_period, :header, 'x-ms-break-period', #Function<18.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_duration: {:param_spec, :lease_duration, :header, 'x-ms-lease-duration', #Function<17.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, lease_id: {:param_spec, :lease_id, :header, 'x-ms-lease-id', #Function<16.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, marker: {:param_spec, :marker, :uri, 'marker', #Function<11.83802320/1 in :erlazure.get_req_common_param_specs/0>}, max_results: {:param_spec, :max_results, :uri, 'maxresults', #Function<8.83802320/1 in :erlazure.get_req_common_param_specs/0>}, message_ttl: {:param_spec, :message_ttl, :uri, 'messagettl', #Function<9.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, message_visibility_timeout: {:param_spec, :message_visibility_timeout, :uri, 'visibilitytimeout', #Function<10.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, num_of_messages: {:param_spec, :num_of_messages, :uri, 'numofmessages', #Function<6.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, peek_only: {:param_spec, :peek_only, :uri, 'peekonly', #Function<8.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, pop_receipt: {:param_spec, :pop_receipt, :uri, 'popreceipt', #Function<7.5724061/1 in :erlazure_queue.get_request_param_specs/0>}, prefix: {:param_spec, :prefix, :uri, 'prefix', #Function<9.83802320/1 in :erlazure.get_req_common_param_specs/0>}, proposed_lease_id: {:param_spec, :proposed_lease_id, :header, 'x-ms-proposed-lease-id', #Function<15.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, res_type: {:param_spec, :res_type, :uri, 'restype', #Function<11.30665059/1 in :erlazure_blob.get_request_param_specs/0>}, timeout: {:param_spec, :timeout, :uri, 'timeout', #Function<7.83802320/1 in :erlazure.get_req_common_param_specs/0>}]}
Client #PID<0.144.0> is alive
    (stdlib) gen.erl:169: :gen.do_call/4
    (stdlib) gen_server.erl:210: :gen_server.call/3
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
    (elixir) src/elixir.erl:233: :elixir.eval_forms/4
    (iex) lib/iex/evaluator.ex:245: IEx.Evaluator.handle_eval/5
    (iex) lib/iex/evaluator.ex:225: IEx.Evaluator.do_eval/3
    (iex) lib/iex/evaluator.ex:203: IEx.Evaluator.eval/3
    (iex) lib/iex/evaluator.ex:89: IEx.Evaluator.loop/1

Looks like genserver has no corresponding handler that matches calling pattern.

NOTE: I've checked if erlazure connects successfully to the Azure and it does. I was able to successfully list containers.

NOTE2: I'm using Elixir version 1.7.0-dev, but I'm not sure if it really affects that...

Invalid HTTP header x-ms-blob-type

When trying to create a block blob I ran into the following issue from my Elixir app.

** (exit) bad return value: "<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.\nRequestId:41797b79-0001-0002-0d7d-93ecbe000000\nTime:2015-05-21T04:18:52.3985127Z</Message><HeaderName>x-ms-blob-type</HeaderName><HeaderValue>block_blob</HeaderValue></Error>"

I trace the error down to the call at

#param_spec{ id = blob_type, type = header, name = "x-ms-blob-type", parse_fun = fun erlang:atom_to_list/1 },

where erlang:atom_to_list on block_blob returns block_blob instead of BlockBlob. I was able to quick fix the issue for my use case jakubgarfield@edfb1e5 but my Erlang knowledge prevents me from fixing it properly.

Error handling

Parse and present errors returned as result of a request.

Gracefully handle no container error

I am using erlazure via ex_azure and the call to put_blob fails with:

[error] GenServer #PID<0.1678.0> terminating
** (stop) bad return value: "\uFEFF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.\nRequestId:b66001fa-d01e-005e-308e-3d06c6000000\nTime:2019-07-18T17:32:06.5364180Z</Message></Error>"
Last message (from #PID<0.93.0>): {:put_blob, 'container', 'new_file.pdf', :block_blob, <<37, 80, 68, 70, 45, 49, 46, 51, 10, 37, 255, 255, 255, 255, 10, 49, 32, 48, 32, 111, 98, 106, 10, 60, 60, 32, 47, 67, 114, 101, 97, 116, 111, 114, 32, 60, 102, 101, 102, 102, 48, 48, 53, 48, 48, ...>>, 

When I was trying to find the culprit in ex_azure I realized it simply just calls handle_call({put_blob, Container, Name, Type = block_blob, Data, Options}, _From, State) -> from erlazure.

The issue is that Azure returns the XML error, but there is no graceful failure of it. It simply makes the GenServer to fail... I would probably except that an error tuple with the error message is returned.

HTTP Error 411. The request must be chunked or have a content length.

1> {ok,Pid} = erlazure:start("key", "sig").
{ok,<0.389.0>}
2> erlazure:create_container(Pid, "test").
09:41:52.015 [error] gen_server <0.389.0> terminated with reason: bad return value: <<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n">>
09:41:52.020 [error] CRASH REPORT Process <0.389.0> with 1 neighbours exited with reason: bad return value: <<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n">> in gen_server:terminate/7 line 826
** exception exit: {bad_return_value,<<"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITL"...>>}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.