Git Product home page Git Product logo

shotgun's Introduction

shotgun

For the times you need more than just a gun.

Rationale

After using the gun library on a project where we needed to consume Server-sent Events (SSE) we found that it provided great flexibility, at the cost of having to handle each raw message and data, including the construction of the response body data. Although this is great for a lot of scenarios, it can get cumbersome and repetitive after implementing it a couple of times. This is why we ended up creating shotgun, an HTTP client that uses gun behind the curtains but provides a simple API that has out-of-the-box support for SSE.

Usage

shotgun uses maps and hence requires Erlang 17 to compile and run.

shotgun is an OTP application, so before being able to use it, it has to be started. Either add it as one of the applications in your .app file or run the following code:

application:ensure_all_started(shotgun).

Regular Requests

Once the application is started a connection needs to be created in order to start making requests:

{ok, Conn} = shotgun:open("google.com", 80),
{ok, Response} = shotgun:get(Conn, "/"),
io:format("~p~n", [Response]),
shotgun:close(Conn).

Which results in:

#{body => <<"<HTML><HEAD>"...>>,
  headers => [
     {<<"location">>,<<"http://www.google.com/adfs">>},
     {<<"content-type">>,<<"text/html; charset=UTF-8">>},
     {<<"x-content-type-options">>,<<"nosniff">>},
     {<<"date">>,<<"Fri, 17 Oct 2014 17:18:32 GMT">>},
     {<<"expires">>,<<"Sun, 16 Nov 2014 17:18:32 GMT">>},
     {<<"cache-control">>,<<"public, max-age=2592000">>},
     {<<"server">>,<<"sffe">>},
     {<<"content-length">>,<<"223">>},
     {<<"x-xss-protection">>,<<"1; mode=block">>},
     {<<"alternate-protocol">>,<<"80:quic,p=0.01">>}
   ],
   status_code => 302}
}

%= ok

Immediately after opening a connection we did a GET request, where we didn't specify any headers or options. Every HTTP method has its own shotgun function that takes a connection, a uri (which needs to include the slash), a headers map or a proplist containing the headers, and an options map. Some of the functions (post/5, put/5 and patch/5) also take a body argument.

Alternatively there's a generic request/6 function in which the user can specify the HTTP method as an argument in the form of an atom: get, head, options, delete, post, put or patch.

IMPORTANT: When you are done using the shotgun connection remember to close it with shotgun:close/1.

HTTP Secure Requests

It is possible to tell shotgun to use SSL by providing the atom https as the third argument when creating a connection with to the open function. Just like when performing HTTP requests it is also necessary to specify a port. HTTPS servers typically listen for connections on port 443 and this will be the most likely value you'll need to use.

Basic Authentication

If you need to provide basic authentication credentials in your requests, it is as easy as specifying a basic_auth entry in the headers map:

{ok, Conn} = shotgun:open("site.com", 80),
{ok, Response} = shotgun:get(Conn, "/user", #{basic_auth => {"user", "password"}}),
, or
{ok, Response} = shotgun:get(Conn, "/user", [{basic_auth, {"user", "password"}}]),
shotgun:close(Conn).

Specifying a Timeout

The timeout option can be used to specify a value for all types of requests:

{ok, Conn} = shotgun:open("google.com", 80).
{error, Error} = shotgun:get(Conn, "/", #{}, #{timeout => 10}).
io:format("~p~n", [Error]).
%%= {timeout,{gen_fsm,sync_send_event,[<0.368.0>,{get,{"/",[],[]}},10]}}
shotgun:close(Conn).

The default timeout value is 5000 if none is specified.

Consuming Server-sent Events

To use shotgun with endpoints that generate SSE the request must be configured using some values in the options map, which supports the following entries:

  • async ::boolean(): specifies if the request performed will return a chunked response. It currently only works for GET requests.. Default value is false.

  • async_mode :: binary | sse: when async is true the mode specifies how the data received will be processed. binary mode treats each chunk received as raw binary. sse mode buffers each chunk, splitting the data received into SSE. Default value is binary.

  • handle_event :: fun((fin | nofin, reference(), binary()) -> any()): this function will be called each time either a chunk is received (async_mode = binary) or an event is parsed (async_mode = sse). If no handle_event function is provided the data received is added to a queue, whose values can be obtained calling the shotgun:events/1. Default value is undefined.

The following is an example of the usage of shotgun when consuming SSE.

{ok, Conn} = shotgun:open("localhost", 8080).
%= {ok,<0.6003.0>}

Options = #{async => true, async_mode => sse},
{ok, Ref} = shotgun:get(Conn, "/events", #{}, Options).
%= {ok,#Ref<0.0.1.186238>}

% Some event are generated on the server...
Events = shotgun:events(Conn).
%= [{nofin, #Ref<0.0.1.186238>, <<"data: pong">>}, {nofin, #Ref<0.0.1.186238>, <<"data: pong">>}]

shotgun:events(Conn).
%= []

Notice how the second call to shotgun:events/1 returns an empty list. This is because events are stored in a queue and each call to events returns all events queued so far and then removes these from the queue. So it's important to understand that shotgun:events/1 is a function with side-effects when using it.

Additionally shotgun provides a parse_event/1 helper function that turns a server-sent event binary into a map:

shotgun:parse_event(<<"data: pong\ndata: ping\nid: 1\nevent: pinging">>).
%= #{data => [<<"pong">>,<<"ping">>],event => <<"pinging">>,id => <<"1">>}

Building & Test-Driving

To build shotgun just run the following on your command shell:

rebar3 compile

To start up a shell where you can try things out run the following (after building the project as described above):

rebar3 shell

Contact Us

If you find any bugs or have a problem while using this library, please open an issue in this repo (or a pull request :)).

And you can check all of our open-source projects at inaka.github.io

shotgun's People

Contributors

arcusfelis avatar cabol avatar elbrujohalcon avatar euen avatar guidorumi avatar harenson avatar hekaldama avatar hernanrivasacosta avatar igaray avatar jfacorro avatar kennethlakin avatar kianmeng avatar kinyoklion avatar loguntsov avatar michalwski avatar mira01 avatar mworrell avatar optikfluffel avatar phunehehe avatar readmecritic avatar rmoorman avatar sata avatar spike886 avatar tothlac avatar unix1 avatar zurab-darkly avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shotgun's Issues

Crash when fetching from slow HTTP server: shotgun:wait_response/3 not exported

This bug can be triggered by doing the following:

  • Find a HTTP server that takes many seconds to connect, or simply find one with a single IP address, and add an iptables DROP rule for that server
  • Compile shotgun e5ae062
  • Start an erlang shell, passing in the proper -pa args
  • Run the following, replacing very.slow.website.or.blocked.website with the FQDN or IP of the server in question:
application:ensure_all_started(shotgun).
{ok, Pid}=shotgun:open("very.slow.website.or.blocked.website", 80).
TestFun=fun(F) ->
            case shotgun:get(Pid, "/") of 
              {ok, Response} -> io:format("ok~n");
              {error, {timeout, _}} -> F(F)
            end
        end.
TestFun(TestFun).
f().

Lightly redacted, but otherwise complete conversation with an Erlang shell follows:

Erlang/OTP 18 [erts-7.0] [source] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V7.0  (abort with ^G)
(erl-node@host)1> application:ensure_all_started(shotgun).
{ok,[ranch,crypto,cowlib,asn1,public_key,ssl,gun,shotgun]}
(erl-node@host)2> {ok, Pid}=shotgun:open("very.slow.website.or.blocked.website", 80).
{ok,<0.68.0>}
(erl-node@host)3> TestFun=fun(F) ->
(erl-node@host)3>             case shotgun:get(Pid, "/") of 
(erl-node@host)3>               {ok, Response} -> io:format("ok~n");
(erl-node@host)3>               {error, {timeout, _}} -> F(F)
(erl-node@host)3>             end
(erl-node@host)3>         end.
#Fun<erl_eval.6.54118792>
(erl-node@host)4> TestFun(TestFun).

=ERROR REPORT==== 19-Sep-2015::04:13:11 ===
** State machine <0.68.0> terminating 
** Last message in was {'$gen_sync_event',
                           {<0.41.0>,#Ref<0.0.2.128>},
                           {get,{"/",[],[]}}}
** When State == wait_response
**      Data  == #{async => false,
                   async_mode => binary,
                   buffer => <<>>,
                   data => <<>>,
                   from => {<0.41.0>,#Ref<0.0.2.122>},
                   handle_event => undefined,
                   headers => undefined,
                   pid => <0.69.0>,
                   responses => {[],[]},
                   status_code => undefined,
                   stream => #Ref<0.0.2.123>}
** Reason for termination = 
** {'function not exported',
       [{shotgun,wait_response,
            [{get,{"/",[],[]}},
             {<0.41.0>,#Ref<0.0.2.128>},
             #{async => false,
               async_mode => binary,
               buffer => <<>>,
               data => <<>>,
               from => {<0.41.0>,#Ref<0.0.2.122>},
               handle_event => undefined,
               headers => undefined,
               pid => <0.69.0>,
               responses => {[],[]},
               status_code => undefined,
               stream => #Ref<0.0.2.123>}],
            []},
        {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
** exception error: no case clause matching {error,{{undef,[{shotgun,wait_response,
                                                                     [{get,{"/",[],[]}},
                                                                      {<0.41.0>,#Ref<0.0.2.128>},
                                                                      #{async => false,
                                                                        async_mode => binary,
                                                                        buffer => <<>>,
                                                                        data => <<>>,
                                                                        from => {<0.41.0>,#Ref<0.0.2.122>},
                                                                        handle_event => undefined,
                                                                        headers => undefined,
                                                                        pid => <0.69.0>,
                                                                        responses => {[],[]},
                                                                        status_code => undefined,
                                                                        stream => #Ref<0.0.2.123>}],
                                                                     []},
                                                            {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,518}]},
                                                            {proc_lib,init_p_do_apply,3,
                                                                      [{file,"proc_lib.erl"},{line,239}]}]},
                                                    {gen_fsm,sync_send_event,
                                                             [<0.68.0>,{get,{"/",[],[]}},5000]}}}
(erl-node@host)5> f().
ok
(erl-node@host)6> 

After making and exporting this zero-effort change in shotgun.erl

wait_response(Event, _, StateData) ->
  wait_response(Event, StateData).

I still get an error, which is as follows:

=ERROR REPORT==== 19-Sep-2015::04:26:54 ===
** State machine <0.68.0> terminating 
** Last message in was {'$gen_sync_event',
                           {<0.41.0>,#Ref<0.0.2.128>},
                           {get,{"/",[],[]}}}
** When State == wait_response
**      Data  == #{async => false,
                   async_mode => binary,
                   buffer => <<>>,
                   data => <<>>,
                   from => {<0.41.0>,#Ref<0.0.2.122>},
                   handle_event => undefined,
                   headers => undefined,
                   pid => <0.69.0>,
                   responses => {[],[]},
                   status_code => undefined,
                   stream => #Ref<0.0.2.123>}
** Reason for termination = 
** {unexpected,{get,{"/",[],[]}}}
** exception error: no case clause matching {error,{{unexpected,{get,{"/",[],[]}}},
                                                    {gen_fsm,sync_send_event,
                                                             [<0.68.0>,{get,{"/",[],[]}},5000]}}}

If I handle that case in the test program, then it looks like either shotgun or gun has crashed, as every subsequent call to shotgun:get returns a tuple that looks like:

{error,{noproc,{gen_fsm,sync_send_event,
                                   [<0.69.0>,{get,{"/",[],[]}},5000]}}}

SSE Comments are unrecognised

Lines starting with : are comments according to the SSE RFC, yet parse_event crashes on them:

1> shotgun:parse_event(<<": Comment 1\n: Comment 2\ndata: some data">>).
** exception error: no case clause matching <<": Comment 2">>
     in function  shotgun:'-parse_event/1-fun-0-'/2 (src/shotgun.erl, line 244)
     in call from lists:foldr/3 (lists.erl, line 1274)

SSE and Server Errors

When an SSE endpoint returns an error (e.g. status code 500), it gets listed as an event on shotgun:events.

As an example, consider this code from fiar:

    {ok, Pid1} = shotgun:open("localhost", 8080),

    {ok, Ref1} = shotgun:get( Pid1
                            , "/matches/" ++ Mid ++ "/events"
                            , Headers2
                            , #{ async => true
                               , async_mode => sse}),

    api_call(put, "/matches/" ++ Mid, Headers1, Body),

    [XX] = shotgun:events(Pid1),

    io:format("~p~n", [XX]).

That prints out…

#{headers => [{<<"connection">>,<<"keep-alive">>},
  {<<"server">>,<<"Cowboy">>},
  {<<"date">>,<<"Fri, 03 Oct 2014 13:32:19 GMT">>},
  {<<"content-length">>,<<"0">>}],
  status_code => 500}

It should either throw an exception instead, or shotgun:events should return a response description element (with the headers, status, etc.) besides the list of events.

Allow to specify a body for all HTTP methods or...

... expose a request/6.

Rationale

Even though it doesn't make sense to send a body in a GET, there are some APIs that are not so rigorous and require or expect a body for requests using such HTTP methods (i.e. ElasticSearch's HTTP REST API).

Queue operations into gun

All operations requested to a shotgun process should be queued while it is busy doing something else.

Replace pop by events

events() should return a list with all the events that arrived up to that moment. i think queue:to_list should be used.

Fulfill the open-source checklist

General Items

  • It has a github repo
  • It has a proper LICENSE file
  • It's hooked to a hipchat room
  • It's has a clear and useful README.md
  • It's documented (with examples)
  • It's tested

Exhibition

  • There is a blog post about it
  • It's shared on social networks
  • It's shared on reddit
  • It's shared on hacker news with a title like Show HN: description
  • It has a landing page built in github pages

For Libraries

  • Examples of use are documented in the README or linked from there

For Erlang Projects

  • It's checked with Elvis

Shotgun supervisor does not realize children die

Running tests which use shotgun usually leave the following messages in the logs:

16:33:23.060 [error] <0.272.0> [::] Supervisor shotgun_sup had child shotgun started with shotgun:start_link() at undefined exit with reason noproc in context shutdown_error

When closing the gun process, if you call is_alive on its pid it will report false, asking for the process info will return undefined, but calling supervisor:which_children(shotgun_sup) the pid will be included in the children's list.

Correctly split lines by SSE spec

http://www.w3.org/TR/2011/WD-eventsource-20110208/#processField

Lines must be separated by either a U+000D CARRIAGE RETURN U+000A LINE FEED (CRLF) character pair, a single U+000A LINE FEED (LF) character, or a single U+000D CARRIAGE RETURN (CR) character.

Since connections established to remote servers for such resources are expected to be long-lived, UAs should ensure that appropriate buffering is used. In particular, while line buffering with lines are defined to end with a single U+000A LINE FEED (LF) character is safe, block buffering or line buffering with different expected line endings can cause delays in event dispatch.

shotgun fails to compile on 0.1.6 and fbe44e2 using R16B03

Steps:
git clone https://github.com/inaka/shotgun
cd shotgun
make

Result using erlang R16B03:
...
./rebar compile
==> sync (compile)
Compiled src/sync_options.erl
Compiled src/sync_utils.erl
Compiled src/sync_scanner.erl
Compiled src/sync.erl
Compiled src/sync_notify.erl
make[1]: Leaving directory `/root/git/sipxecs/custom/reach-app/deps/shotgun/deps/sync'
ERLC shotgun_app.erl shotgun.erl shotgun_sup.erl
src/shotgun.erl:59: syntax error before: '{'
src/shotgun.erl:65: syntax error before: '{'
src/shotgun.erl:67: syntax error before: '{'
src/shotgun.erl:120: syntax error before: '{'
src/shotgun.erl:125: syntax error before: '{'
src/shotgun.erl:201: syntax error before: '{'
src/shotgun.erl:242: syntax error before: '{'
src/shotgun.erl:271: syntax error before: '{'
src/shotgun.erl:284: syntax error before: '{'
src/shotgun.erl:301: syntax error before: '{'
src/shotgun.erl:311: syntax error before: '{'
src/shotgun.erl:338: syntax error before: '{'
src/shotgun.erl:380: syntax error before: '{'
src/shotgun.erl:419: syntax error before: '{'
src/shotgun.erl:451: syntax error before: '{'
src/shotgun.erl:492: syntax error before: '{'
src/shotgun.erl:519: syntax error before: '{'
src/shotgun.erl:12: function get/2 undefined
src/shotgun.erl:12: function get/3 undefined
src/shotgun.erl:12: function request/6 undefined
src/shotgun.erl:41: function handle_sync_event/4 undefined
src/shotgun.erl:41: function init/1 undefined
src/shotgun.erl:41: function terminate/3 undefined
src/shotgun.erl:50: function at_rest/3 undefined
src/shotgun.erl:50: function parse_event/1 undefined
src/shotgun.erl:50: function receive_data/2 undefined
src/shotgun.erl:50: function wait_response/2 undefined
src/shotgun.erl:64: type response() undefined
src/shotgun.erl:118: spec for undefined function shotgun:get/2
src/shotgun.erl:123: spec for undefined function shotgun:get/3
src/shotgun.erl:155: function request/6 undefined
src/shotgun.erl:161: function request/6 undefined
src/shotgun.erl:167: function request/6 undefined
src/shotgun.erl:173: function request/6 undefined
src/shotgun.erl:179: function request/6 undefined
src/shotgun.erl:185: function request/6 undefined
src/shotgun.erl:191: function request/6 undefined
src/shotgun.erl:196: spec for undefined function shotgun:request/6
src/shotgun.erl:196: type headers() undefined
src/shotgun.erl:196: type options() undefined
src/shotgun.erl:238: spec for undefined function shotgun:parse_event/1
src/shotgun.erl:259: spec for undefined function shotgun:init/1
src/shotgun.erl:279: spec for undefined function shotgun:handle_sync_event/4
src/shotgun.erl:300: spec for undefined function shotgun:terminate/3
src/shotgun.erl:310: spec for undefined function shotgun:at_rest/3
src/shotgun.erl:334: spec for undefined function shotgun:wait_response/2
src/shotgun.erl:376: spec for undefined function shotgun:receive_data/2
src/shotgun.erl:403: function manage_chunk/4 undefined
src/shotgun.erl:434: function maps_get/3 is unused
src/shotgun.erl:444: function do_http_verb/3 is unused
src/shotgun.erl:500: function basic_auth_header/1 is unused
src/shotgun.erl:513: function encode_basic_auth/2 is unused
src/shotgun.erl:534: function check_uri/1 is unused
make: *** [ebin/shotgun.app] Error 1

Manage headers as map, not as proplist

It is not a good idea to have maps and proplists as arguments of the functions of shotung. We should receive a map for the headers since we are using maps for all the rest. Then we should transform the map into a proplist so that gun can understand it.

Add all http verbs

Add support for all http verbs apart from GET that is already implemented. This should be pretty straightforward.

basic auth

Add headers support for basic auth (in the future we can add other auth support) so that we do not need to do this ugliness:

shotgun:get(Pid, "/conversations", [{<<"authorization">>, <<"Basic dGVzdDp0ZXN0">>}]).
shotgun:get(Pid, "/conversations", [{<<"basic_auth">>, {<<"user">>, <<"passwd">>}}]).

Or even better

shotgun:get(Pid, "/conversations", #{basic_auth => {<<"user">>, <<"passwd">>}).

binary headers?

HTTP defines headers (both the name and the value) as an ASCII subset

   header-field = field-name ":" OWS field-value OWS
   field-name = token
   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   token = 1*tchar

field-value allows for VCHAR, which is still an ASCII subset.

So why the binary headers? to be "future proof"? :)

Show more descriptive errors when asking for events but there is no connection

When you run

Headers = #{ <<"content-type">> => <<"application/json">>
           , <<"Cookie">> => list_to_binary(CookieString)},

{ok, Pid} = shotgun:open("localhost", 8080),

{ok, #{status_code := 401}} =
      shotgun:get( Pid
                 , "/events"
                 , Headers
                 , #{ async => true
                    , async_mode => sse}),

shotgun:events(Pid),

And have no sse connection (perhaps because of a 401 error) you get this strange error

** exception exit: {{badarg,[{queue,to_list,
                                    [{#Ref<0.0.1.149676>,
                                      {ok,#{headers => [{<<"connection">>,<<"keep-alive">>},
                                             {<<"server">>,<<"Cowboy">>},
                                             {<<"date">>,<<"Fri, 24 Jul 2015 14:08:24 GMT">>},
                                             {<<"content-length">>,<<"0">>}],
                                            status_code => 401}}}],
                                    [{file,"queue.erl"},{line,87}]},
                             {shotgun,handle_sync_event,4,
                                      [{file,"/Users/euen/Documents/fiar/deps/shotgun/src/shotgun.erl"},
                                       {line,289}]},
                             {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,503}]},
                             {proc_lib,init_p_do_apply,3,
                                       [{file,"proc_lib.erl"},{line,237}]}]},
                    {gen_fsm,sync_send_all_state_event,[<0.4453.0>,get_events]}}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.