Git Product home page Git Product logo

erlastic_search's Introduction

ErlasticSearch

An Erlang client for Elasticsearch.

Build and Run

Start a rebar3 shell

rebar3 shell

Create an index :

erlastic_search:create_index(<<"index_name">>).
{ok, [{<<"ok">>,true},{<<"acknowledged">>,true}]}

Index a document :

erlastic_search:index_doc(<<"index_name">>, <<"type">>, [{<<"key1">>, <<"value1">>}]).
{ok,[{<<"ok">>,true},
     {<<"_index">>,<<"index_name">>},
     {<<"_type">>,<<"type">>},
     {<<"_id">>,<<"T-EzM_yeTkOEHPL9cN5B2g">>},
     {<<"_version">>,1}]}

Index a document (providing a document id) :

erlastic_search:index_doc_with_id(<<"index_name">>, <<"type">>, <<"id1">>, [{<<"key1">>, <<"value1">>}]).
{ok,[{<<"ok">>,true},
     {<<"_index">>,<<"index_name">>},
     {<<"_type">>,<<"type">>},
     {<<"_id">>,<<"id1">>},
     {<<"_version">>,2}]}

Search for a document :

erlastic_search:search(<<"index_name">>, <<"type">>, <<"key1:value1">>).
{ok,[{<<"took">>,6},
     {<<"timed_out">>,false},
     {<<"_shards">>,
      [{<<"total">>,5},{<<"successful">>,5},{<<"failed">>,0}]},
     {<<"hits">>,
      [{<<"total">>,3},
       {<<"max_score">>,0.30685282},
       {<<"hits">>,
        [[{<<"_index">>,<<"index_name">>},
          {<<"_type">>,<<"type">>},
          {<<"_id">>,<<"T-EzM_yeTkOEHPL9cN5B2g">>},
          {<<"_score">>,0.30685282},
          {<<"_source">>,[{<<"key1">>,<<"value1">>}]}],
         [{<<"_index">>,<<"index_name">>},
          {<<"_type">>,<<"type">>},
          {<<"_id">>,<<"id1">>},
          {<<"_score">>,0.30685282},
          {<<"_source">>,[{<<"key1">>,<<"value1">>}]}],
         [{<<"_index">>,<<"index_name">>},
          {<<"_type">>,<<"type">>},
          {<<"_id">>,<<"MMNcfNHUQyeizDkniZD2bg">>},
          {<<"_score">>,0.30685282},
          {<<"_source">>,[{<<"key1">>,<<"value1">>}]}]]}]}]}

Testing

In another terminal use docker-compose to start an Elasticsearch instance :

docker-compose up

For convenience, you can also start a Kibana instance for analysis/visualization :

docker-compose -f docker-compose.yml -f docker-compose-kibana.yml up

Run Common Test:

rebar3 ct

Using another JSON library than jsx

By default, we assume all the JSON erlang objects passed to us are in jsx's representation. And similarly, all of Elasticsearch's replies will be decoded with jsx.

However, you might already be using another JSON library in your project, which might encode and decode JSONs from and to a different erlang representation. For example, jiffy:

1> SimpleJson = <<"{\"key\":\"value\"}">>.
<<"{\"key\":\"value\"}">>
2> jiffy:decode(SimpleJson).
{[{<<"key">>,<<"value">>}]}
3> jsx:decode(SimpleJson).
[{<<"key">>,<<"value">>}]

In that case, you probably want erlastic_search to use your JSON representation of choice instead of jsx's.

You can do so by defining the ERLASTIC_SEARCH_JSON_MODULE environment variable when compiling erlastic_search, for example:

export ERLASTIC_SEARCH_JSON_MODULE=jiffy
rebar compile

The only constraint is that ERLASTIC_SEARCH_JSON_MODULE should be the name of a module, in your path, that defines the two following callbacks:

-callback encode(erlastic_json()) -> binary().
-callback decode(binary()) -> erlastic_json().

where erlastic_json() is a type mapping to your JSON representation of choice.

erlastic_search's People

Contributors

akovari avatar andreashasse avatar anushamorappanavar2024 avatar bluesalt avatar bryanhuntesl avatar chensongz avatar cstar avatar ddeboer avatar drobakowski avatar facundoolano avatar getong avatar jlecour avatar laxmanpottimuthi avatar linuss avatar marcelog avatar mdojwa avatar paulperegud avatar prasadbonthu avatar rj avatar sashaafm avatar stefanrusek avatar stevepm avatar t-mw avatar tsloughter avatar wk8 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

erlastic_search's Issues

Tests please

I mostly just merge peoples PRs with minimal review.

I don't use this library so would be great if someone who does would be interested in submitting a travis or circle config that can run the ct tests (and we need more ct tests, a lot more) against a running ElasticSearch that travis or circle provides.

Missing passing options to index_doc_with_id_opts

Hi,

Could you apply the following patch to master branch ?:

--- a/src/erlastic_search.erl
+++ b/src/erlastic_search.erl
@@ -143,7 +143,7 @@ index_doc_with_id(Params, Index, Type, Id, Doc) ->

 -spec index_doc_with_id_opts(record(erls_params), binary(), binary(), binary(), list() | binary(), list()) -> {ok, list()} | {error, any()}.
 index_doc_with_id_opts(Params, Index, Type, Id, Doc, Opts) when is_list(Doc), is_list(Opts) ->
-    index_doc_with_id_opts(Params, Index, Type, Id, jsx:encode(Doc), []);
+    index_doc_with_id_opts(Params, Index, Type, Id, jsx:encode(Doc), Opts);
 index_doc_with_id_opts(Params, Index, Type, Id, Doc, Opts) when is_binary(Doc), is_list(Opts) ->
     erls_resource:post(Params, filename:join([Index, Type, Id]), [], Opts, Doc, Params#erls_params.http_client_options).

The only change is in passing Opts to index_doc_with_id_opts.

Currenlty I am using this patch on my local copy of erlastic_search but it would be much simpler with this one available in public master branch :)

Thanks.

Generic way to access and pass http client options

This discussion started with #8. Summary:

...having some specific way to pass hackney options, right from the app, and in a transparent way through erlastic_search.

Maybe having an extra "http_client_options" field in the #erls_params record, that end up being passed verbatim to hackney, so we don't have to stay in sync with every existing (and possibly useful) option.

...one should be aware that using it requires knowledge of the http client in use behind the scenes, that can change without further notice (although a big increment in the version number would be helpful).

I will work on this and open a proper PR for your review, but wanted to open the issue first :)

exception

So I'm getting this when running against this vagrant box of elastic search:

2> erlastic_search:create_index(<<"resources">>).
** exception exit: {noproc,
                       {gen_server,call,
                           [hackney_manager,
                            {new_request,<0.437.0>,#Ref<0.0.1.7008>,
                                {client,undefined,hackney_dummy_metrics,
                                    hackney_tcp_transport,"127.0.0.1",9200,
                                    <<"127.0.0.1:9200">>,
                                    [{connect_timeout,infinity},
                                     {recv_timeout,infinity}],
                                    nil,nil,nil,true,hackney_pool,infinity,
                                    false,5,false,0,...}},
                            infinity]}}
     in function  gen_server:call/3 (gen_server.erl, line 212)
     in call from hackney_manager:init_request/1 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/hackney/src/hackney_client/hackney_manager.erl, line 65)
     in call from hackney_manager:new_request/1 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/hackney/src/hackney_client/hackney_manager.erl, line 55)
     in call from hackney_connect:socket_from_pool/4 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/hackney/src/hackney_connect/hackney_connect.erl, line 181)
     in call from hackney_connect:connect/5 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/hackney/src/hackney_connect/hackney_connect.erl, line 41)
     in call from hackney:request/5 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/hackney/src/hackney_client/hackney.erl, line 317)
     in call from erls_resource:do_request/6 (c:/Users/drozdyuka/Projects/NRC/drozdyuka/erlastic_search/_build/default/lib/erlastic_search/src/erls_resource.erl, line 67)
3>

Does it support update_doc API ?

I found it only support upsert_doc API. If the doc does not exist, new doc will be inserted. I need the API that update doc partly, and if the doc dose not exist, 404(Not Found) will be returned.

example

can be an example of the use of

Autocomplete works locally but not on my production application

Hello,

Locally I am able to type in queries in my local development environment while phoenix server is running, and everything works as expected. However in production, when I type any queries, I get this error message:

[info] Sent 500 in 30ms
[error] #PID<0.470.0> running Trophus.Endpoint terminated
Server: www.trophus.com:80 (http)
Request: GET /get_nearest?query=joe
** (exit) an exception was raised:
    ** (ArgumentError) argument error
        (erlastic_search) src/erls_resource.erl:67: :erls_resource.do_request/6
        (trophus) web/controllers/search_controller.ex:11: Trophus.SearchController.get_nearest/2
        (trophus) web/controllers/search_controller.ex:1: Trophus.SearchController.phoenix_controller_pipeline/2
        (trophus) lib/phoenix/router.ex:281: Trophus.Router.dispatch/2
        (trophus) web/router.ex:1: Trophus.Router.do_call/2
        (trophus) lib/trophus/endpoint.ex:1: Trophus.Endpoint.phoenix_endpoint_pipeline/1
        (trophus) lib/plug/debugger.ex:90: Trophus.Endpoint."call (overridable 3)"/2
        (trophus) lib/phoenix/endpoint/render_errors.ex:34: Trophus.Endpoint.call/2

Do you have any thoughts on what this could be due to? Thanks in advance!

Support for bulk update, please ?

Hi, can you make the app to bulk update the documents please ? instead of sending a lot of requests to update the document, it would be nice to have bulk update.

Thanks

Mappings

I could not find any way to put a mapping when creating and index, or PUTing a mapping to a index/type after its creation.

Will you add support for this?

Cannot compile after adding into deps list

Beginner question, After adding repo into deps list and compiling it throws this error

src/erls_json.erl:19: undefined macro 'ERLASTIC_SEARCH_JSON_MODULE'
src/erls_json.erl:30: undefined macro 'ERLASTIC_SEARCH_JSON_MODULE'
src/erls_json.erl:3: function decode/1 undefined
src/erls_json.erl:3: function encode/1 undefined
src/erls_json.erl:17: spec for undefined function encode/1
src/erls_json.erl:28: spec for undefined function decode/1

Blocking request issue

Hi,
I'm currently using this app in my production code, I had an issue with blocking requests. I have a timeout returned from init() to be handled in to handle_info(), and from there I'm casting a request with gen_server:cast() to be handled into handle_cast().
Inside I'm calling a function that has this code snippet:
try erlastic_search:search_limit(<<"message">>, <<"inbox">>, <<<<"userId:">>/binary, A/binary, "&_source=''">>, 9999) of
{ok, Resp} ->
{<<"hits">>, Hits} = lists:keyfind(<<"hits">>, 1, Resp),
{<<"hits">>, Hits1} = lists:keyfind(<<"hits">>, 1, Hits),
parse_es_response_data_to_id_list(Hits1);
_ -> []
catch
: -> []
end.

First time this code will work, but then it will not work, I think there is something related to Hackney, I'm not sure.
Currently I solved the issue by adding error() before {noreply, State} inside handle_cast(), and now the code is working on production:

handle_cast({disableMessages, UserId}, State) ->
IdList = fetch_msg_id_list_for(UserId),
set_msg_availability(IdList, <<"false">>),
error(break_blocking), <------------------------ this solved my issue
{noreply, State};

Thanks for your time and effort for providing libraries and applications for the Erlang community.

crash dump?

I cannot make it work. Any clue ?

$ ./start-dev.sh
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]

{"init terminating in do_boot",{undef,[{erlastic_search,start,[],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

is it possible to build a release from erlastic_search repo?

hello,

I am having issues with including erlastic_search into my app and building a release.

reltool keeps throwing exceptions

bash-3.2$ rebar generate
==> rel (generate)
ERROR: generate failed while processing /Users/romanshestakov/development/erlastic_search/rel: {'EXIT',{{badmatch,{error,"erlastic_search: Missing application directory."}},
[{rebar_reltool,generate,2,
[{file,"src/rebar_reltool.erl"},{line,53}]},
{rebar_core,run_modules,4,[{file,"src/rebar_core.erl"},{line,446}]},
{rebar_core,execute,5,[{file,"src/rebar_core.erl"},{line,371}]},
{rebar_core,process_dir1,6,[{file,"src/rebar_core.erl"},{line,235}]},
{rebar_core,process_commands,2,
[{file,"src/rebar_core.erl"},{line,90}]},
{rebar,main,1,[{file,"src/rebar.erl"},{line,58}]},
{escript,run,2,[{file,"escript.erl"},{line,752}]},
{escript,start,1,[{file,"escript.erl"},{line,276}]}]}}
bash-3.2$ pwd

I suspect one of the apps in elastic_search dependency tree might be missing something.

in any case, I am wandering if anybody have ever tried to build a successful release out of erlastic_search project and might be able to share a config for reltool?

Regards, Roman

Support for bulk index docs with default params

This function would be useful if we are using bulk_index_docs with default params i.e #erls_params{}. Since record erls_params will not be able to available if this repo is used as library for any OTP applciation.

PS: Created pull request.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.