Git Product home page Git Product logo

bolt_sips's People

Contributors

cheerfulstoic avatar dnesteryuk avatar dominique-vassard avatar florinpatrascu avatar hissssst avatar kalamarski-marcin avatar kristofka avatar ndac-todoroki avatar sylph01 avatar tcrossland avatar vic avatar wli0503 avatar zediogoviana avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bolt_sips's Issues

Unsupported authentication token, missing key `scheme`: {user_agent='BoltSips/1.5.1'}

Following the directions here:

https://hexdocs.pm/bolt_sips/1.5.1/readme.html

I get the following when attempting connection:

14:23:59.458 [error] Bolt.Sips.Protocol (#PID<0.276.0>) failed to connect: ** (Bolt.Sips.Internals.Error) Unsupported authentication token, missing key scheme: { user_agent='BoltSips/1.5.1' }

I've double-check my config, which is as follows:

` def application do
[
applications: [:bolt_sips], mod: {Bolt.Sips.Application, []},
extra_applications: [:logger]
]
end

defp deps do
[
{:bolt_sips, "~> 1.5"}
]
end
config :bolt_sips, Bolt,
hostname: 'localhost',
basic_auth: [username: 'neo4j', password: 'somepasswordhere'],
port: 7687
`
using:
Erlang/OTP 21 [erts-10.3.4]
Elixir/Mix 1.8.1 (from Brew on Mac OS High Sierra)

Connection works from the Neo4j browser. I'm not sure what to make of the error message. Any suggestions?

Publish latest version (0.5) to Hex?

Hey Florin,

Thanks for creating bolt_sips, it's really useful!

In the README, you recommend using version 0.5:

def deps do
  [{:bolt_sips, "~> 0.5"}]
end

But on Hex, it appears that the latest version is 0.4.12.

I see the disclaimer in the README that you're working on implementing the db_connection behavior. Is the reason you haven't pushed version 0.5 to Hex because that's experimental?

Thanks for your help!

Starting Manually

Hi,

You specify:

2. Ensure bolt_sips is started before your application:

def application do
  [applications: [:bolt_sips]]
end

This makes it impossible to provide configuration at runtime, using :weave.

I've tried adding runtime: false to stop BoltSips automatically starting up and instead manually starting it in my application module callback, with

supervisor(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)])

but this causes everything to "hang"

Nil error message when sending invalid parameter type on query.

I was doing a Cypher query and unintentionally sent an elixir date-time as parameter for it (instead of actually converting it to an integer timestamp). The issue is, even when the query correctly fails with an error, the message for it is just nil.

{:error, [code: :failure, message: nil]}

It would be a lot nicer if the message was something useful like "we cant serialize that data type to neo4j" or something.

I've added a test for this case here:

https://github.com/florinpatrascu/bolt_sips/compare/master...vic:invalid_param?expand=1

Is the error not being placed by boltex ? or bolt_sips ?

Query failed

Hi Florin,
I would like to thank you for a great library!
But, I'm having troubles running through the example in the readme.

I get this match error:
"no function clause matching in :gen_tcp.send/2"

I'm using Erlang 20 and Elixir 1.5.2 and Bolt 0.4.11 (latest from Hex)

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, pid} = Bolt.Sips.start_link(url: "localhost")
{:ok, #PID<0.5408.0>}
iex(2)> conn = Bolt.Sips.conn
{:ok, %{"server" => "Neo4j/3.1.0"}}
iex(3)> Bolt.Sips.query!(conn, "CREATE (a:Person {name:'Bob'})")
** (exit) exited in: :gen_server.call(#PID<0.5415.0>, {{:ok, %{"server" => "Neo4j/3.1.0"}}, "CREATE (a:Person {name:'Bob'})", %{}}, 15000)
    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in :gen_tcp.send/2
            (kernel) gen_tcp.erl:265: :gen_tcp.send({:ok, %{"server" => "Neo4j/3.1.0"}}, <<0, 2, 176, 14, 0, 0>>)
            (elixir) lib/enum.ex:675: Enum."-each/2-lists^foreach/1-0-"/2
            (elixir) lib/enum.ex:675: Enum.each/2
            (boltex) lib/boltex/bolt.ex:209: Boltex.Bolt.ack_failure/3
            (bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
            (elixir) lib/enum.ex:3176: Enumerable.List.reduce/3
            (elixir) lib/stream.ex:849: Stream.do_list_transform/10
            (elixir) lib/enum.ex:1887: Enum.reduce_while/3
            (bolt_sips) lib/bolt_sips/connection.ex:57: Bolt.Sips.Connection.handle_call/3
            (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
            (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
    (stdlib) gen_server.erl:214: :gen_server.call/3
    (poolboy) /Users/semtne/Projects/xxx/deps/poolboy/src/poolboy.erl:76: :poolboy.transaction/3
    (bolt_sips) lib/bolt_sips/query.ex:80: Bolt.Sips.Query.tx/3
    (bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3
iex(3)> [error] GenServer #PID<0.5415.0> terminating
** (FunctionClauseError) no function clause matching in :gen_tcp.send/2
    (kernel) gen_tcp.erl:265: :gen_tcp.send({:ok, %{"server" => "Neo4j/3.1.0"}}, <<0, 2, 176, 14, 0, 0>>)
    (elixir) lib/enum.ex:675: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:675: Enum.each/2
    (boltex) lib/boltex/bolt.ex:209: Boltex.Bolt.ack_failure/3
    (bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
    (elixir) lib/enum.ex:3176: Enumerable.List.reduce/3
    (elixir) lib/stream.ex:849: Stream.do_list_transform/10
    (elixir) lib/enum.ex:1887: Enum.reduce_while/3
    (bolt_sips) lib/bolt_sips/connection.ex:57: Bolt.Sips.Connection.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.5406.0>): {{:ok, %{"server" => "Neo4j/3.1.0"}}, "CREATE (a:Person {name:'Bob'})", %{}}
State: [auth: {}, socket: :gen_tcp, hostname: 'localhost', retry_linear_backoff: [delay: 150, factor: 2, tries: 3], with_etls: false, ssl: false, timeout: 15000, max_overflow: 2, pool_size: 5, port: 7687, url: "localhost"]
Client #PID<0.5406.0> is alive
    (stdlib) gen.erl:169: :gen.do_call/4
    (stdlib) gen_server.erl:210: :gen_server.call/3
    (poolboy) /Users/semtne/Projects/xxx/deps/poolboy/src/poolboy.erl:76: :poolboy.transaction/3
    (bolt_sips) lib/bolt_sips/query.ex:80: Bolt.Sips.Query.tx/3
    (bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
    (elixir) src/elixir.erl:239: :elixir.eval_forms/4
    (iex) lib/iex/evaluator.ex:231: IEx.Evaluator.handle_eval/5
    (iex) lib/iex/evaluator.ex:212: IEx.Evaluator.do_eval/3
    (iex) lib/iex/evaluator.ex:190: IEx.Evaluator.eval/3
    (iex) lib/iex/evaluator.ex:89: IEx.Evaluator.loop/1
    (iex) lib/iex/evaluator.ex:24: IEx.Evaluator.init/4
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Phoenix 1.3 Unable to start

I've followed the install guide, but get the following error.

exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.120961608.535560200.138616>, true}, :infinity) ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
nofile

Runtime configuration

The docs only explain configuration via classic elixir config mechanism. We are using releases and try to configure all services during runtime via environment variables. What would be the suggested way to achieve it with bolt_sips? Creating my own worker / supervisor which loads config with init/1? Or is there a way to hook into bolt_sips' init/1 callback?

Simple Query taking too much time to process

Precheck

  • For bugs, do a quick search and make sure the bug has not yet been reported.
  • Finally, be nice and have fun!

Environment

  • Elixir version (elixir -v): 1.9.1
  • Neo4j and version (Neo4j 3.5.3, etc.): 3.5.8
  • Connection scheme (bolt://, bolt+routing:// or neo4j://): bolt://
  • Bolt.Sips version (mix deps): 2.0.0-rc.2
  • Operating system: Phoenix running on MacOS 10.14, Neo4j running on a docker inside linux VM

Current behavior

Hello!
I'm developing a simple server using phoenix and neo4j and started using Bolt_Sips for that. But strangely, a simple query, returning a list of 120 nodes, takes too much time.

Query:

    cypher = """
      MATCH (p: Place)
        RETURN p as place;
    """

(note: p has only a "name" variable)

The following code takes too much time to execute (around 2 seconds in the first try and 1.4 seconds in the second try):

Bolt.Sips.query!(Bolt.Sips.conn, cypher)

I have checked the execution time using

    :timer.tc(fn -> Bolt.Sips.query!(Bolt.Sips.conn, cypher) end)
      |> elem(0)
      |> Kernel./(1_000)
      |> IO.inspect

When querying directly from neo4j using the web dashboard, I got 281ms in the first try and 56ms in the second try, much less than using bolt_sips.

Expected behavior

Much faster query.


Am I doing something wrong? :(

Or is this the expected behaviour?

Time out for some queries

Hello, I started to play with bolt sips 3 weeks ago, and today I deployed my service to my staging enviroment but I have found some issues:

I'm receiving a time out error when I made query like this:

"MATCH (s) WHERE id(s) = {start_node_id}\nMATCH (e) WHERE id(e) = {end_node_id}\nCREATE (s)-[relationship:WRITES{created_at: {created_at}, source: {source}}]->(e)\nRETURN relationship", %{created_at: 1521829096, end_node_id: 1888, source: "messenger", start_node_id: 2467}

My logs:

3/23/2018 12:18:31 PM ** (exit) exited in: :gen_server.call(#PID<0.1494.0>, {#Port<0.3962>, "MATCH (s) WHERE id(s) = {start_node_id}\nMATCH (e) WHERE id(e) = {end_node_id}\nCREATE (s)-[relationship:WRITES{created_at: {created_at}, source: {source}}]->(e)\nRETURN relationship", %{created_at: 1521829096, end_node_id: 1888, source: "messenger", start_node_id: 2467}}, 15000)
3/23/2018 12:18:31 PM    ** (EXIT) time out

But when I made queries like this:

"MATCH (node:USER)-[:WRITES]->(:BOT{bot_id:{bot_id}})\nRETURN node\nORDER BY node.created_at", %{bot_id: "my-bot"}

I got the results that I expect.

Environment:

  • Phoenix 1.3
  • Elixir 1.6
  • Bolt sips ~> 0.4
  • Neo4j 3.3

Annd my bolt sips configuration:

config :bolt_sips, Bolt,
  hostname: "localhost",
  port: 7687,
  pool_size: 5,
  max_overflow: 1

It would be great to get your feedback about this issue.

Thanks for your help

date & datetime support

Is there a way to get date and datetime properties working with 0.4.12?

I am currently getting Neo.ClientError.Request.Invalid with DateTime is not yet supported as a return type in Bolt.

Application exited: shutdown

Environment

  • Elixir version (elixir -v): 1.8.0
  • Neo4j and version (Neo4j 3.5.3, etc.): Neo4j 3.3
  • Connection scheme (bolt://, bolt+routing:// or neo4j://): bolt://
  • Bolt.Sips version (mix deps): ~> 2.0.0-rc
  • Operating system: Ubuntu 18.04

Current behavior

I just followed your Getting started guide, but when I try to do: DEBUG=1 iex -S mix I just got:

[info]  Application n4d exited: shutdown

My files are:

mix.exs

defp deps do
  [
    {:bolt_sips, "~> 2.0.0-rc"},
    {:jason, "~> 1.1"}
  ]
end

application.ex

def start(_type, _args) do
  # List all child processes to be supervised
  children = [
    # Starts a worker by calling: N4D.Worker.start_link(arg)
    # {N4D.Worker, arg}
    {Bolt.Sips, Application.get_env(:bolt_sips, Bolt)}
  ]

  # See https://hexdocs.pm/elixir/Supervisor.html
  # for other strategies and supported options
  opts = [strategy: :one_for_one, name: N4D.Supervisor]
  Supervisor.start_link(children, opts)
end

config.exs

config :bolt_sips, Bolt,
  url: "bolt://localhost:7687",
  pool_size: 10

I don't know, what I'm doing wrong, can someone help me, thanks!

Pool of connections

Hello Florin,

Turns out I misunderstand something regarding the pool. When I call:

Bolt.Sips.conn

I expect to get an established connection to a Neo4j instance, but actually, it starts establishing a new connection from scratch, it doesn't get it from the pool. The Bolt.Sips.Connection uses the pool to be sure there is a limited number of process which establish connections, but those processes don't preserve established connections for further use. The connection operation is quite expensive (40-50 ms in my case).

Did I get it wrong? I am looking forward for your response. Thanks. ๐Ÿ˜„

Problem with negative integers

Hi Florin, thank you for the efforts you are putting in to this project.

Today I`ve seen a strange behaviour when using bolt_sips driver. It somewhat gives wrong result when there are negative integers on neo4j. for example: an integer which value is -1 become 256 and -256 becomes 65280 in bolt_sips query result.

It seems bolt_sips does not interpret neo4j`s negative integers.

using bolt_sips before db_connection v2

as ecto v2 requries db_connection: ~> 1.1, what commit should I use to avoid upgrading ecto to v3?
I am using bolt_sips 0.4.12 and experience an error in production. Neo4j worked for a few hours and threw

ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: unable to create new native thread unable to create new native thread java.lang.OutOfMemoryError: unable to create new native thread

I am wondering if the latest commits would fix this problem.

chores: port some of the official Neo4j driver tests to Bolt.Sips, using boltkit

The official Neo4j drivers are using more and more the support offered by the
boltkit toolkit, and we too would like to use it.

boltkit is extremely useful for development, offering a great test-bed for simulating server responses in various scenarios. You can see how the above mentioned drivers are implementing their tests with it, for example these "mocks".

To integrate boltkit in our tests, we'd have to start the boltkit (stub)server at the beginning of our tests, and kill/stop it when they end; we could most probably use System.cmd/3, for bootstrapping it. Then, with the boltkit server running as a background process, we can send and receive scripted scenarios, making our tests depending much-much less on having various Neo4j server instances, of different versions, clustered servers or not clustered, simulate nodes going away or corrupted routing tables and so on. The later being a very tedious and time consuming process, especially given the hardware resources eaten up from our dev machines, when in conjuncture with docker/docker-compose & Co. ๐Ÿ˜ข

We'd appreciate any support for donating from your skills and dev time to improving our development environment and for also making our core-driver development easier and more robust nevertheless.

Thank you!
โค๏ธ

hex package contains an erl_crash.dump file

More of a stylistic thing but most likely not intended.

When I install the current (0.2.5) package and look at its content, I find an erl_crash.dump file in lib.

Big Kudos and Many Thanks for providing this package by the way ๐Ÿ‘

Getting unpack error

I'm getting:

    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in Boltex.Bolt.unpack/1
            (bolt_sips) lib/boltex/bolt.ex:234: Boltex.Bolt.unpack({:error, :timeout})
            (bolt_sips) lib/boltex/bolt.ex:198: Boltex.Bolt.receive_data/3
            (bolt_sips) lib/boltex/bolt.ex:154: Boltex.Bolt.run_statement/4
            (bolt_sips) lib/bolt_sips/connection.ex:52: Bolt.Sips.Connection.handle_call/3
            (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
            (stdlib) gen_server.erl:647: :gen_server.handle_msg/5
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

For this query: USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM \"file:/import_1482495188_5.csv\" AS row MERGE (:A {aid: toInteger(row.aid), name: row.name})
which can take up to 20 seconds to process.

module Bolt.Sips is not available

I followed the minimalist configuration from the README and receive the following error:

iex(1)> {:ok, _p} = Bolt.Sips.start_link(host: "localhost")
** (UndefinedFunctionError) function Bolt.Sips.start_link/1 is undefined (module Bolt.Sips is not available) Bolt.Sips.start_link([host: "localhost"])

Also, iex -S mix fails:
=INFO REPORT==== 4-Nov-2016::20:39:19 ===
application: logger
exited: stopped
type: temporary
** (Mix) Could not start application boltex: could not find application file: boltex.app

"Unsupported authentication token, missing key `scheme`"

Hi,

I am trying to get the driver working in iex but unfortunately am running into the titular error. It works just fine from the terminal, but I am getting these unexpected return values elsewhere.

screen shot 2017-06-02 at 12 37 02 am

I am having the iex issue on both Mac and Windows. However, none of it works on Windows, whereas the terminal command on OS X does work.

The link to my project is here if you want to look at it.

I apologize if this is not the proper forum, but I cannot find any reference to this issue anywhere else. I also have checked my configuration against multiple examples, and everything seems fine to me. I'm fairly stumped. Any help would be appreciated.

Thanks!

README improvements.

I'm thinking of doing the following improvements to the README.md

Installation

  • Just indicate the standard way of declaring a mix dependency.
    the example for the github and local dependency should be removed just to keep documentation concise.

  • Adapt the Add the bolt_sips dependency the applications list section.
    remove the mod: {Bolt.Sips.Application, []} part as Bolt.Sips.Application actually
    is an empty supervisor and does nothing.

So I'm guessing the original intention of Bolt.Sips.Application (correct me if I'm wrong)
was to start Bolt.Sips with some configuration options (see snippet bellow). And in that case, remove the
part that shows how to use Bolt.Sips.start_link manually.

I'm currently doing something like this in my own app's supervision tree:

children = [
       worker(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)])
]

Usage

  • Remove logger and mix_test_watch configurations.
  • The snippet for adding bolt_sips to mix :applications should be only on Installation section.

Testing

  • dockerized neo4j
    Add example on how to start a local neo4j for testing (like the one travis uses from #2):
    docker run -p 7687:7687 -e 'NEO4J_AUTH=none' neo4j:3.0.6

Consistent bad connection state after malformed query: "... You need to\n`ACK_FAILURE` or `RESET` ..."

I have the following fulltext index on my database:

CALL db.index.fulltext.createNodeIndex(
  "topic_label",
  ["topic"],
  ["label"],
  { analyzer: "simple" });

And the following Bolt.Sips configuration (note the pool_size of just 1):

[
  url: "bolt://localhost:7687",
  ssl: true,
  basic_auth: [
    username: "neo4j",
    password: "[REDACTED]"
  ],
  pool_size: 1
]

If I issue the following malformed query (because badparen) is not quoted and is therefore treated like a Lucene query containing syntax errors):

Bolt.Sips.query(
  Bolt.Sips.conn(),
  "CALL db.index.fulltext.queryNodes(\"topic_label\", \"badparen)\") YIELD node RETURN node")

Then that query correctly fails with an error from Apache Lucene:

{:error,
 [
   code: nil,
   message: "run_statement: Unknown failure: {:failure, %Bolt.Sips.Internals.Error{code: \"Neo.ClientError.Procedure.ProcedureCallFailed\", connection_id: 48, function: :pull_all, message: \"Failed to invoke procedure `db.index.fulltext.queryNodes`: Caused by: org.apache.lucene.queryparser.classic.ParseException: Encountered \\\" \\\")\\\" \\\") \\\"\\\" at line 1, column 8.\\nWas expecting one of:\\n    <EOF> \\n    <AND> ...\\n    <OR> ...\\n    <NOT> ...\\n    \\\"+\\\" ...\\n    \\\"-\\\" ...\\n    <BAREOPER> ...\\n    \\\"(\\\" ...\\n    \\\"*\\\" ...\\n    \\\"^\\\" ...\\n    <QUOTED> ...\\n    <TERM> ...\\n    <FUZZY_SLOP> ...\\n    <PREFIXTERM> ...\\n    <WILDTERM> ...\\n    <REGEXPTERM> ...\\n    \\\"[\\\" ...\\n    \\\"{\\\" ...\\n    <NUMBER> ...\\n    \", type: :cypher_error}}\n"
 ]}

But! If I then try to issue any further queries:

Bolt.Sips.query(Bolt.Sips.conn(), "Match (n) return n") 

Then they fail with the following exception:

{:error,
 [
   code: nil,
   message: "The session is in a failed state and ignores further messages. You need to\n`ACK_FAILURE` or `RESET` in order to send new messages.\n"
 ]}

I wasn't sure if this was a bug or if I was missing a configuration, but it looked like the failed state error was leaking out of internals; was something supposed to ACK or RESET on my behalf? The connection seems to stay in this bad state indefinitely, which could poison a pool over time if a running system has the right bugs in it.

Thanks!

Environment

  • Elixir version (elixir -v):
    Erlang/OTP 21 [erts-10.3.5.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
    Elixir 1.9.0 (compiled with Erlang/OTP 21)
  • Neo4j and version (Neo4j 3.5.3, etc.):
    Neo4j 3.5.12
  • Connection scheme (bolt://, bolt+routing:// or neo4j://):
    bolt://
  • Bolt.Sips version (mix deps):
    1.5.1
  • Operating system:
    Fedora 30 (Workstation Edition)

Support for bolt+routing

Not reporting an issue here but asking if there are plans to build in support for the routing extension.
I am currently at graphconnect and learning about how this is implemented in other drivers.
I am happy to share what I learn and to support implementation.
Curious if there is already work in progress on this?

Documentation: public / private / guides?

Hi,
As project grows, document does too.
I found current Hex documentation a little bit messy and it doesn't help user to leverage the true power of bolt_sips.

What can we do?

  • Expose public API in a clear way. As an example today, Bolt.Sips.ConfigAgent is at the same level as Bolt.Sips.Types, and we know that the first is for internal use only.
    We could benefit of the grouping capability of ExDoc to have a clear separation between public API and internal doc (because it's nice to not have to look into the code for some docs...)
  • Keep the README.md for github and have some nice "Guide" section on Hex with visible TOC and all

What do you think,

Boltex.Bolt.generate_chunks fails with too much data

Hi, first of all thank you very much for this project!

When running a query with parameters I encountered the following error message:

(FunctionClauseError) no function clause matching in Boltex.Bolt.generate_chunks/3
    (bolt_sips) lib/boltex/bolt.ex:115: Boltex.Bolt.generate_chunks([<<178, 16, 209, 7, 244, 77, 69, 82, 71, 69, 32, 40, 103, 105, 116, 104, 117, 98, 58, 71, 105, 116, 104, 117, 98, 80, 114, 111, 102, 105, 108, 101, 32, 123, 32, 103, 105, 116, 104, 117, 98, 73, 100, 58, 32, 36, 117, 115, 101, ...>>, <<177, 63, 192>>], [], "")
    (bolt_sips) lib/boltex/bolt.ex:102: Boltex.Bolt.send_messages/3
    (bolt_sips) lib/boltex/bolt.ex:157: Boltex.Bolt.run_statement/4

With a larger amount of parameters it seems like generating chunks isn't working correctly. Unfortunately I wasn't able to find the exact size.

What I saw is that on this line the chunks variable is mentioned twice in the function signature. Could that be an issue?

errors on semicolon

Hello!
a little example whith semicolon in text, which raise exeption

Bolt.Sips.query!(Bolt.Sips.conn, "MERGE (:Test {text: "test with semicolon;"})")

21:58:18.753 [debug] [{:sock_ref, "", #PID<0.624.0>, #PID<0.625.0>, #PID<0.626.0>}] cypher: "MERGE (:Test {text: "test with semicolon" - params: %{} - bolt: {:error, %Boltex.Error{code: "Neo.ClientError.Statement.SyntaxError", connection_id: nil, function: :run_statement, message: "Unexpected end of input: expected '\', ANY or '"' (line 1, column 41 (offset: 40))\n"MERGE (:Test {text: "test with semicolon"\n ^", type: :cypher_error}}
(elixir) lib/exception.ex:494: RuntimeError.exception(%Boltex.Error{code: "Neo.ClientError.Statement.SyntaxError", connection_id: nil, function: :run_statement, message: "Unexpected end of input: expected '\', ANY or '"' (line 1, column 41 (offset: 40))\n"MERGE (:Test {text: "test with semicolon"\n ^", type: :cypher_error})
(bolt_sips) lib/bolt_sips/query.ex:100: Bolt.Sips.Query.send!/4
(elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
(bolt_sips) lib/bolt_sips/query.ex:86: Bolt.Sips.Query.tx/3
(bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3

In web editor this request process correctly.

[bolt v3] Bolt.Sips.transaction crash with invalid query

Considering the awaited return, which happened in bolt version 1 and 2 (Neo4j < 3.4)

iex(5)> Bolt.Sips.transaction(Bolt.Sips.conn(), fn conn ->
...(5)> Bolt.Sips.query(conn, "RETRN 1 AS num")           
...(5)> end)
{:ok,
 {:error,
  [
    code: "Neo.ClientError.Statement.SyntaxError",
    message: "Invalid input 'R': expected 'u/U' (line 1, column 4 (offset: 3))\n\"RETRN 1 AS num\"\n    ^"
  ]}}

The same code in a bolt V3 Neo4j server:

iex(7)> Bolt.Sips.transaction(Bolt.Sips.conn(), fn conn ->  
...(7)> Bolt.Sips.query(conn, "RETRN 1 AS num")             
...(7)> end)
** (MatchError) no match of right hand side value: {:error, %Bolt.Sips.Internals.Error{code: "Neo.ClientError.Request.Invalid", connection_id: 10928, function: :commit, message: "Message 'COMMIT' cannot be handled by a session in the READY state.", type: :cypher_error}}
    (bolt_sips) lib/bolt_sips/protocol.ex:117: Bolt.Sips.Protocol.handle_commit/2
    (db_connection) lib/db_connection/holder.ex:270: DBConnection.Holder.holder_apply/4
    (db_connection) lib/db_connection.ex:1468: DBConnection.run_commit/3
    (db_connection) lib/db_connection.ex:1006: DBConnection.checkin/4
    (db_connection) lib/db_connection.ex:1450: DBConnection.commit/3
    (db_connection) lib/db_connection.ex:1377: DBConnection.run_transaction/4

This is due to how DBConnection handles the transaction:

  • execute the query
  • rollback if there is an error

But in Bolt v3, a rollback is automatically performed at server-level if query is invalid.
All is required is a RESET message...

Parameters not working?

I'm trying to use parameters to create nodes and not having much luck. I started out with:

Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report = {props}", %{props: my_map})

That gives me:

** (Bolt.Sips.Exception) socket is not connected
    (bolt_sips) lib/bolt_sips/query.ex:57: Bolt.Sips.Query.query!/3
    lib/mix/tasks/test.load_db.ex:32: anonymous fn/2 in Mix.Tasks.Test.LoadDb.run/1
    (elixir) lib/enum.ex:1841: Enum.reduce_range_inc/4
    lib/mix/tasks/test.load_db.ex:18: Mix.Tasks.Test.LoadDb.run/1
    (mix) lib/mix/task.ex:301: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:75: Mix.CLI.run_task/2

To simplify I tried this:

Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report.test = {props}", %{props: 'foo'})

That successfully queries, but then the test property has the value "102,111,111" in the database.

This works, however:

Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report.test = 'foo'")

So it seems like maybe there's just something wrong with params (?)

Project's own config for bolt.cypher task

This was mentioned on line 32 of the task definition.

I personally would like to use this as a starting point for writing other tasks for Bolt, like setting up constraints, so I'm interested in this too, but I have no Elixir skill yet.

Error Connecting to Remote DB

Thank you for all of your hard work! I'm a huge fan of Neo4J, and it's because of this project that I can use Elixir.

However, being new to Elixir, I'm not sure what I'm doing wrong or how to read this stack trace. I'm trying to connect to GrapheneDB and get the following error:

iex(1)> Bolt.Sips.conn
** (exit) exited in: :gen_server.call(#PID<0.427.0>, :connect)
    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in Boltex.Utils."-hex_encode/1-lc$^0/1-0-"/1
            (bolt_sips) lib/boltex/utils.ex:9: Boltex.Utils."-hex_encode/1-lc$^0/1-0-"({:error, :closed})
            (bolt_sips) lib/boltex/bolt.ex:44: Boltex.Bolt.handshake/2
            (bolt_sips) lib/bolt_sips/connection.ex:41: Bolt.Sips.Connection.handle_call/3
            (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
            (stdlib) gen_server.erl:647: :gen_server.handle_msg/5
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
     (stdlib) gen_server.erl:204: :gen_server.call/2
    (poolboy) src/poolboy.erl:76: :poolboy.transaction/3
iex(1)> [error] GenServer #PID<0.427.0> terminating
** (FunctionClauseError) no function clause matching in Boltex.Utils."-hex_encode/1-lc$^0/1-0-"/1
    (bolt_sips) lib/boltex/utils.ex:9: Boltex.Utils."-hex_encode/1-lc$^0/1-0-"({:error, :closed})
    (bolt_sips) lib/boltex/bolt.ex:44: Boltex.Bolt.handshake/2
    (bolt_sips) lib/bolt_sips/connection.ex:41: Bolt.Sips.Connection.handle_call/3
    (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:647: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :connect
State: [timeout: 15000, hostname: "bolt://hobby.dbs.graphenedb.com", basic_auth: [username: "fakeusername", password: "fakepassword"], port: 24786, pool_size: 5, max_overflow: 1]

My config:

config :bolt_sips, Bolt,
 hostname: "bolt://hobby.dbs.graphenedb.com",
 basic_auth: [username: "fakeusername", password: "fakepassword"],
 port: 24786,
 pool_size: 5,
 max_overflow: 1

Any ideas?

UndefinedFunctionError during Bolt.Sips.start_link after migrating from 1.3.0 to 1.4.0 or 1.5.1

Environment

  • Elixir 1.8.2 (compiled with Erlang/OTP 20), Interactive Elixir (1.8.2)
  • Neo4J 3.5.6
  • Connecting via bolt:// (see below)
  • Bolt.Sips 1.5.1
  • Ubuntu 18.04.2 LTS (Bionic Beaver)

Current behavior

I call the following code in IEX to start my bolt.sips connection, unchanged from how I did with version 1.3.0 of Bolt Sips:

Bolt.Sips.start_link([       
  url: "bolt://localhost:7687",
  ssl: true,
  basic_auth: [username: "neo4j", password: "secret"]
])

and the following is logged in response:

18:29:55.186 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [3, 2, 1, 0]"
 
18:29:55.186 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [3, 2, 1, 0]"
 
18:29:55.187 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x3>>
 
18:29:55.187 [debug] S: HANDSHAKE ~ 3
 
18:29:55.187 [debug] C: HELLO ~ [%{credentials: "secret", principal: "neo4j", scheme: "basic", user_agent: "BoltSips/1.5.1"}]
 
18:29:55.187 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x3>>
 
18:29:55.187 [debug] S: HANDSHAKE ~ 3

18:29:55.187 [debug] C: HELLO ~ [%{credentials: "secret", principal: "neo4j", scheme: "basic", user_agent: "BoltSips/1.5.1"}]

18:29:55.187 [error] GenServer #PID<0.574.0> terminating
** (RuntimeError) connect raised UndefinedFunctionError exception. The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true when starting your connection if you wish to see all of the details
    (bolt_sips) Bolt.Sips.Internals.PackStream.Encoder.encode/2
    (bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:145: Bolt.Sips.Internals.PackStream.Message.Encoder.encode_message/4
    (bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:105: Bolt.Sips.Internals.PackStream.Message.Encoder.call_encode/3
    (bolt_sips) lib/bolt_sips/internals/bolt_protocol_helper.ex:20: Bolt.Sips.Internals.BoltProtocolHelper.send_message/4 
    (bolt_sips) lib/bolt_sips/internals/bolt_protocol_v3.ex:28: Bolt.Sips.Internals.BoltProtocolV3.hello/5
    (bolt_sips) lib/bolt_sips/protocol.ex:39: Bolt.Sips.Protocol.connect/1
    (db_connection) lib/db_connection/connection.ex:69: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
Last message: nil
State: Bolt.Sips.Protocol

Several such errors are logged for various PIDs.

[EDIT]
If I downgrade to 1.4.0 I receive a similar repeated error:

23:48:54.658 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [2, 1, 0, 0]"
 
23:48:54.659 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x2>>
 
23:48:54.659 [debug] S: HANDSHAKE ~ 2

23:48:54.659 [debug] C: INIT ~ ["BoltSips/", %{credentials: "secret", principal: "neo4j", scheme: "basic"}]
 
23:48:54.660 [error] GenServer #PID<0.338.0> terminating
** (RuntimeError) connect raised UndefinedFunctionError exception. The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true when starting your connection if you wish to see all of the details
    (bolt_sips) Bolt.Sips.Internals.PackStream.Encoder.encode/2
    (bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:209: Bolt.Sips.Internals.PackStream.Message.Encoder.do_encode/3
    (bolt_sips) lib/bolt_sips/internals/bolt_protocol.ex:177: Bolt.Sips.Internals.BoltProtocol.send_message/4
    (bolt_sips) lib/bolt_sips/internals/bolt_protocol.ex:155: Bolt.Sips.Internals.BoltProtocol.init/5
    (bolt_sips) lib/bolt_sips/protocol.ex:39: Bolt.Sips.Protocol.connect/1
    (db_connection) lib/db_connection/connection.ex:69: DBConnection.Connection.connect/2
    (connection) lib/connection.ex:622: Connection.enter_connect/5
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: nil
State: Bolt.Sips.Protocol

Downgrading further to 1.3.0 resolves the issue as expected, so this appears to be an issue for me only since 1.4.0.

Expected behavior

I expected this to establish a database connection with Neo4j, as it did in version 1.3.0.

I tried passing the :show_sensitive_... flag along to Bolt.Sips.start_link, not knowing if it would ultimately be passed down to DBConnection, but the same message was logged without additional information.

Response is not transformed

Hi, it seems that the current verison of Bolt.Sips.Response.transform is not working correctly. When i do the next neo4j query...

    cypher = """
      MATCH (a:Person)
      WHERE a.uid = {uid}
      RETURN a
    """
    data = Bolt.query!(Bolt.conn(), cypher, %{uid: uid})
    (hd data)["a"]

I get the next response:

%Bolt.Sips.Types.Node{
  id: 55,
  labels: ["Person"],
  properties: %{
    "last_payment" => [sig: 70, fields: [1550835424, 283000000, 0]],
    "uid" => "test1112"
  }
}

So the DateTime doesn't get parsed correctly.
Currently I solve this using the next code in my view:

    data = Enum.reduce(
      Map.to_list(person.properties), 
      %{}, 
      fn obj, acc -> 
        Map.put(acc, 
          elem(obj, 0),
          if is_binary(elem(obj, 1)) do 
            elem(obj, 1) 
          else # We expect :sig to be 70
            [seconds, nanoseconds, offset] = elem(obj, 1)[:fields]
            naive_dt = NaiveDateTime.add(
              ~N[1970-01-01 00:00:00.000],
              seconds * 1_000_000_000 + nanoseconds,
              :nanosecond
            )
            NaiveDateTime.to_iso8601(naive_dt) <> 
              Bolt.Sips.TypesHelper.formated_time_offset(offset)
          end) 
      end)

This is just a workaround using the code I stripped from your Bolt.Sips.Response class.

Timeout on all queries after 15 minutes of inactivity

Thanks for your hard work on this driver.

I never ran into this issue whilst working in a development environment, since there was little downtime between subsequent queries. However, after testing out the current build on a production server, I noticed that all the query execution attempts after ~15 minutes of inactivity produce a timeout exception.

My configuration for the driver is as follows:

config :bolt_sips, Bolt,
  url: "bolt://<host>:8124",
  basic_auth: [username: "neo4j", password: "***"]

I use the driver API as it is recommended in the documentation, i.e.

Bolt.Sips.conn()
|> Bolt.Sips.query!( ... )

The Neo4j server is a Docker image with default configuration.

Any ideas on what causes the trouble?
Thank you.

Boltex

Hey @florinpatrascu,

I noticed that you are no longer using Boltex as a dependency but instead have the files in this project.

Could you elaborate what lead to that decision? I see multiple disadvantages from diverging bolt implementations and would love for us to find a way to keep the protocol and a client library separate.

Please let me know if there is anything in particular that you were looking for or that I did that led you to copy the code over to here. If it was the implementation of the db_connection library in boltex, I was under the impression that I kept it isolated enough for custom connection handling to still work (let me know if that was not the case - I am open to breaking out the DB Connection part into a separate library).

I think it'd be ideal if we could go back to separate projects so users have the choice between using just low-level protocol implementations (as I am doing in one project) or using a fully-featured library such as yours. This would have the added benefit that we would bundle our resources in ensuring the best possible bolt implementation in Elixir.

Thank you,
Michael

etls doesn't compile under Windows

Just for your information. :)
Even with all deps installed it doesn't work.
The owner has said that he doesn't have experience with Windows, so probably no fix is coming soon.

Dependency on boltex fork

Is there still any reason to depend on florinpatrascu/boltex instead of mschae/boltex ?

mschae/boltex#10 has been merged and would like to remove the TODO I had once we depend on the orignal boltex code.

Etls cannot be used or I got it wrong

Context:

  1. I added the bolt_sips to my project.
  2. I added export BOLT_WITH_ETLS=true to my .profile in Ubuntu.
  3. I installed dependencies.

Expected result: The etls package is installed along with the bolt_sips.
Actual result: The etls is missing (the project crashes at launch).

** (Mix) Could not start application etls: could not find application file: etls.app

I successfully launched the project after adding the etls into dependencies of the project. I think the mix installs only dependencies listed in the mix.lock of the bolt_sips. Actually, I am ok to keep the etls among dependencies of my project. Probably, the approach with the BOLT_WITH_ETLS env variable should be changed. My advice is to get rid of the BOLT_WITH_ETLS and mention in the readme that the etls should be added to dependencies of a project which uses the bolt_sips. Meantime, the bolt_sips can provide a configuration option to select the ssl lib, example:

config :bolt_sips, Bolt,
  ssl_lib: :etls

What do you think?

Bolt.Sips.conn returns bare :error

Ahoy!

I'm able to use Bolt.Sips.conn in my test cases and interactively in iex, but an escript I've compiled is misbehaving. I'm sure this is something wrong with my configuration, but Bolt.Sips.conn returns a bare :error symbol, which causes Bolt.Sips.query to fail rather loudly. It looks like the :error is coming from :poolboy's transaction logic (In the v0.4.12 branch at Connection.ex pool_server/3, line 94).

I thought I'd raise this up in case this isn't an intentional hand-off. Normally I would expect to receive { :ok, conn } from Sips.conn if getting an error was possible, since that would force me to match and coincidentally root out :errors early in the stack; that might be a worthwhile change to the Bolt.Sips.conn api.

Thanks for Bolt.Sips! It's been fantastic to work with so far. I'll go see if I can get some logging out of poolboy. :D

Stacktrace if you want it:

7:27:01.222 [error] GenServer #PID<0.113.0> terminating
** (FunctionClauseError) no function clause matching in :ssl.send/2
    ssl.erl:322: :ssl.send(:error, <<0, 2, 176, 14, 0, 0>>) # :error is from Bolt.Sips.conn via :poolboy
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (boltex) lib/boltex/bolt.ex:211: Boltex.Bolt.ack_failure/3
    (bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
    (elixir) lib/enum.ex:3317: Enumerable.List.reduce/3
    (elixir) lib/stream.ex:911: Stream.do_list_transform/7
    (elixir) lib/enum.ex:1990: Enum.reduce_while/3
Last message (from TextClient.Agent): {:error, "MATCH (a:Card { id: $id })\nRETURN a", %{id: "5"}}

Timeout sometimes after a lot of queries

Hello, in the latest weeks I was working in a project using Bolt Sips to connect to Neo4j, and today I deployed my service to production but I'm experience some issues, I'm receiving a lot of requests and doing almost 5 queries to Neo4j per request, but sometimes that queries give me a timeout and then my requests answer with an 500 status code.

This is the error:

7/10/2018 11:52:29 AM16:52:29.337 [error] #PID<0.2632.2> running UsersAPIWeb.Endpoint terminated
7/10/2018 11:52:29 AMServer: people-service:4000 (http)
7/10/2018 11:52:29 AMRequest: POST /api/bots/MTCenter/users
7/10/2018 11:52:29 AM** (exit) exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.2148655786.3957063682.261483>, true}, 5000)
7/10/2018 11:52:29 AM    ** (EXIT) time out
7/10/2018 11:52:29 AM16:52:29.366 request_id=6qd7pdbvcenep2evgp0dic4uic8vnb0j [info] Updating whatsapp status of user: 7228977818 with: %{"input" => "7228977818", "status" => "valid", "wa_id" => "5217228977818"}
7/10/2018 11:52:29 AM16:52:29.429 [error] #PID<0.2705.2> running UsersAPIWeb.Endpoint terminated
7/10/2018 11:52:29 AMServer: people:4000 (http)
7/10/2018 11:52:29 AMRequest: POST /api/bots/coppel/users/update-last-interaction
7/10/2018 11:52:29 AM** (exit) exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.2148655786.3957063682.261495>, true}, 5000)
7/10/2018 11:52:29 AM    ** (EXIT) time out

Environment:

  • Phoenix 1.3
  • Elixir 1.6
  • Bolt sips ~> master branch
  • Neo4j 3.3

And my bolt sips configuration is:

# Set configuration for Neo4j with Bolt
config :bolt_sips, Bolt,
  hostname: "${NEO4J_HOST}",
  port: 7687,
  basic_auth: [username: "${NEO4J_USER}", password: "${NEO4J_PASSWORD}"],
  pool_size: 10,
  max_overflow: 5,
  retry_linear_backoff: [delay: 150, factor: 2, tries: 3]

My application supervisor has this childrens

children = [
  # Start Neo4j connection
  worker(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)]),
  # Start the endpoint when the application starts
  supervisor(UsersAPIWeb.Endpoint, [])
]

It would be great to get your feedback about this issue.

Thanks for your help

timeout is happening while querying.

stacktrace

** (Bolt.Sips.Exception) timeout
    (bolt_sips) lib/bolt_sips/query.ex:59: Bolt.Sips.Query.query!/3

Environment

Erlang/OTP 22 
IEx 1.9.1 (compiled with Erlang/OTP 22)
* Neo4j version (Neo4j 3.5.1):
* Bolt.Sips version: "1.5.1",
* Operating system: macOS 10.14.4
* Connection config 
config :bolt_sips, Bolt,
  hostname: 'hostname.com',
  basic_auth: [username: "user", password: "****"],
  port: 7687,
  pool_size: 50,
  max_overflow: 5,
  timeout: 5_000,
  retry_linear_backoff: [delay: 150, factor: 2, tries: 3]

while executing query randomly it gives timeout then again after retrying it works fine. I was using previously :neo4j_sips, "0.2.18", it never gave such timeouts. I decided to move it to this library then this is happening let me know if you want any other info which can help you to debug this issue or in case I'm doing something wrong.

Refactoring: Encode / Decode Sips.types at low level

How things are

Today, all Bolt.Sips.Types are encoded / decoded at "high level" while this operations should be done a "low level", i.e in the bolt encoding layer.
This means that they are encoded twice:

  • first encoded to a internals-compliant structure / decoded from binary to list
  • then encoded into Bolt-compliant binary format

Or decoded twice:

  • first decoded from binary to list
  • then converted to a Bolt.Sips.Type

How they will be

Each Bolt.Sips.Type should encoded / decoded at the bolt encoding / decoding layer.

Why?

  • Performance: I imagine that encoding / decoding only once will be faster that twice, even if there is little gain
  • Further development: Code things once
  • Readability: When searching for encoding / decoding, check only one place

Implications

End user won't be impacted, as everything will be internally handled.

Some big changes are expected: Bolt version will be split in their own files.
AS I figure out when testing, a request with unsupported data hangs, then it has to be handled before being sent to server.
Architecture will reflect the one used in: https://github.com/dominique-vassard/bolt-neo4j
with encoder_v1, encoder_v2, etc.
This will be clearer and future version could be easily added, tested and not lost inside one big file.
bolt_protocol.ex should follow the same pattern as bolt v3 introduces ne messages and having everything in the same file is just a mess.

This change also means that bolt version should be passed all the way down to the encoding / decoding layer.
In order to hide this from a user perspective (and to not break current API), it will be stored in conn.

Let's do it! :)

DBConnection from Hex

We'd like to stop overriding DBConnection everywhere in our app. In the process of upgrading our outdated dependencies, something broke and another library stopped working due to it expecting a different method in DBConnection. db_connection 1.1.3 was released on 13/01/2018 on Hex - is this a recent enough version for bolt_sips, and if so, can bolt_sips's mixfile be updated to use the Hex package?

Connection pool creates much more connections than specified.

My team and I found bolt_sips has two apparently wrong behaviors regarding to connection pooling.

  1. It opens much more tcp connections than specified with the pool size configuration. Although the number of processes in the pool is correct, the number of sockets at the OS level grows almost indefinitely, limited only by the limit on the number of file descriptors the process can afford.
  2. When reaching such limit, the behavior of query execution is erratic and unpredictable. You may get a response if you wait long enough, or after a while you may get an error which makes no sense to the client application. It seems like the timeout parameter only applies to the connection between the driver and the neo4j server, but it doesn't apply to the client application requests.

You can try by yourself starting this minimal application. Just change the query/queries you want to execute in BoltSipsLoad.

After cloning, and deps compilation, try:

$ iex -S mix
ex(1)> BoltSipsLoad.load_test(5, 1, 500)

This would repeat 5 times launching 1 process to execute a query and then will wait 500ms for the next iteration. In another terminal you can check for the number of opened sockets to the neo4j server (substitute pid for watherver the process id of your beam is):

$ lsof -nP -i4TCP  | awk -v pid=$pid '$2 == pid {print $0}' | grep 7687 | wc -l
5

5 sockets to the bolt socket, as expected.

Now something more interesting. Let's make 10 iterations, launching 100 processes in each one.

ex(2)> BoltSipsLoad.load_test(10, 100, 500)
$ lsof -nP -i4TCP  | awk -v pid=$pid '$2 == pid {print $0}' | grep 7687 | wc -l
470

What?!

This is confirmed in the observer application. Look at all those tcp_inet ports

image

And each one of them is a port to the neo4j bolt port.

image

Unfortunately this is causing trouble in a production system that was just handed to me. Increasing the limit of file descriptors the process can open just moves the problem somewhere else, because the neo4j server can't handle thousands of connections without getting into performance problems. In any case, there shouldn't be there as many open connections, that's what the pool is intended for, isn't it?

It would be great to get your confirmation/feedback about this issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.