florinpatrascu / bolt_sips Goto Github PK
View Code? Open in Web Editor NEWNeo4j driver for Elixir
License: Apache License 2.0
Neo4j driver for Elixir
License: Apache License 2.0
Following the directions here:
https://hexdocs.pm/bolt_sips/1.5.1/readme.html
I get the following when attempting connection:
14:23:59.458 [error] Bolt.Sips.Protocol (#PID<0.276.0>) failed to connect: ** (Bolt.Sips.Internals.Error) Unsupported authentication token, missing key scheme
: { user_agent='BoltSips/1.5.1' }
I've double-check my config, which is as follows:
` def application do
[
applications: [:bolt_sips], mod: {Bolt.Sips.Application, []},
extra_applications: [:logger]
]
end
defp deps do
[
{:bolt_sips, "~> 1.5"}
]
end
config :bolt_sips, Bolt,
hostname: 'localhost',
basic_auth: [username: 'neo4j', password: 'somepasswordhere'],
port: 7687
`
using:
Erlang/OTP 21 [erts-10.3.4]
Elixir/Mix 1.8.1 (from Brew on Mac OS High Sierra)
Connection works from the Neo4j browser. I'm not sure what to make of the error message. Any suggestions?
this would ease tracking diagnostics against previous versions.
Hey Florin,
Thanks for creating bolt_sips
, it's really useful!
In the README, you recommend using version 0.5:
def deps do
[{:bolt_sips, "~> 0.5"}]
end
But on Hex, it appears that the latest version is 0.4.12.
I see the disclaimer in the README that you're working on implementing the db_connection
behavior. Is the reason you haven't pushed version 0.5 to Hex because that's experimental?
Thanks for your help!
how do you feel about multi database connections?
i.e. polyglot persistence
its possible with mysql and postgres apparently
http://geoffreylessel.com/2016/connecting-to-multiple-databases-with-ecto/
Hi,
You specify:
2. Ensure bolt_sips is started before your application:
def application do
[applications: [:bolt_sips]]
end
This makes it impossible to provide configuration at runtime, using :weave
.
I've tried adding runtime: false
to stop BoltSips automatically starting up and instead manually starting it in my application module callback, with
supervisor(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)])
but this causes everything to "hang"
I was doing a Cypher query and unintentionally sent an elixir date-time as parameter for it (instead of actually converting it to an integer timestamp). The issue is, even when the query correctly fails with an error, the message for it is just nil.
{:error, [code: :failure, message: nil]}
It would be a lot nicer if the message was something useful like "we cant serialize that data type to neo4j" or something.
I've added a test for this case here:
https://github.com/florinpatrascu/bolt_sips/compare/master...vic:invalid_param?expand=1
Is the error not being placed by boltex ? or bolt_sips ?
Hi Florin,
I would like to thank you for a great library!
But, I'm having troubles running through the example in the readme.
I get this match error:
"no function clause matching in :gen_tcp.send/2"
I'm using Erlang 20 and Elixir 1.5.2 and Bolt 0.4.11 (latest from Hex)
Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, pid} = Bolt.Sips.start_link(url: "localhost")
{:ok, #PID<0.5408.0>}
iex(2)> conn = Bolt.Sips.conn
{:ok, %{"server" => "Neo4j/3.1.0"}}
iex(3)> Bolt.Sips.query!(conn, "CREATE (a:Person {name:'Bob'})")
** (exit) exited in: :gen_server.call(#PID<0.5415.0>, {{:ok, %{"server" => "Neo4j/3.1.0"}}, "CREATE (a:Person {name:'Bob'})", %{}}, 15000)
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in :gen_tcp.send/2
(kernel) gen_tcp.erl:265: :gen_tcp.send({:ok, %{"server" => "Neo4j/3.1.0"}}, <<0, 2, 176, 14, 0, 0>>)
(elixir) lib/enum.ex:675: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:675: Enum.each/2
(boltex) lib/boltex/bolt.ex:209: Boltex.Bolt.ack_failure/3
(bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
(elixir) lib/enum.ex:3176: Enumerable.List.reduce/3
(elixir) lib/stream.ex:849: Stream.do_list_transform/10
(elixir) lib/enum.ex:1887: Enum.reduce_while/3
(bolt_sips) lib/bolt_sips/connection.ex:57: Bolt.Sips.Connection.handle_call/3
(stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:665: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
(stdlib) gen_server.erl:214: :gen_server.call/3
(poolboy) /Users/semtne/Projects/xxx/deps/poolboy/src/poolboy.erl:76: :poolboy.transaction/3
(bolt_sips) lib/bolt_sips/query.ex:80: Bolt.Sips.Query.tx/3
(bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3
iex(3)> [error] GenServer #PID<0.5415.0> terminating
** (FunctionClauseError) no function clause matching in :gen_tcp.send/2
(kernel) gen_tcp.erl:265: :gen_tcp.send({:ok, %{"server" => "Neo4j/3.1.0"}}, <<0, 2, 176, 14, 0, 0>>)
(elixir) lib/enum.ex:675: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:675: Enum.each/2
(boltex) lib/boltex/bolt.ex:209: Boltex.Bolt.ack_failure/3
(bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
(elixir) lib/enum.ex:3176: Enumerable.List.reduce/3
(elixir) lib/stream.ex:849: Stream.do_list_transform/10
(elixir) lib/enum.ex:1887: Enum.reduce_while/3
(bolt_sips) lib/bolt_sips/connection.ex:57: Bolt.Sips.Connection.handle_call/3
(stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:665: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.5406.0>): {{:ok, %{"server" => "Neo4j/3.1.0"}}, "CREATE (a:Person {name:'Bob'})", %{}}
State: [auth: {}, socket: :gen_tcp, hostname: 'localhost', retry_linear_backoff: [delay: 150, factor: 2, tries: 3], with_etls: false, ssl: false, timeout: 15000, max_overflow: 2, pool_size: 5, port: 7687, url: "localhost"]
Client #PID<0.5406.0> is alive
(stdlib) gen.erl:169: :gen.do_call/4
(stdlib) gen_server.erl:210: :gen_server.call/3
(poolboy) /Users/semtne/Projects/xxx/deps/poolboy/src/poolboy.erl:76: :poolboy.transaction/3
(bolt_sips) lib/bolt_sips/query.ex:80: Bolt.Sips.Query.tx/3
(bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3
(stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
(elixir) src/elixir.erl:239: :elixir.eval_forms/4
(iex) lib/iex/evaluator.ex:231: IEx.Evaluator.handle_eval/5
(iex) lib/iex/evaluator.ex:212: IEx.Evaluator.do_eval/3
(iex) lib/iex/evaluator.ex:190: IEx.Evaluator.eval/3
(iex) lib/iex/evaluator.ex:89: IEx.Evaluator.loop/1
(iex) lib/iex/evaluator.ex:24: IEx.Evaluator.init/4
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
I've followed the install guide, but get the following error.
exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.120961608.535560200.138616>, true}, :infinity) ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
nofile
please see: issue #32, for background info.
The docs only explain configuration via classic elixir config mechanism. We are using releases and try to configure all services during runtime via environment variables. What would be the suggested way to achieve it with bolt_sips? Creating my own worker / supervisor which loads config with init/1? Or is there a way to hook into bolt_sips' init/1 callback?
bolt://
, bolt+routing://
or neo4j://
): bolt://Hello!
I'm developing a simple server using phoenix and neo4j and started using Bolt_Sips for that. But strangely, a simple query, returning a list of 120 nodes, takes too much time.
Query:
cypher = """
MATCH (p: Place)
RETURN p as place;
"""
(note: p has only a "name" variable)
The following code takes too much time to execute (around 2 seconds in the first try and 1.4 seconds in the second try):
Bolt.Sips.query!(Bolt.Sips.conn, cypher)
I have checked the execution time using
:timer.tc(fn -> Bolt.Sips.query!(Bolt.Sips.conn, cypher) end)
|> elem(0)
|> Kernel./(1_000)
|> IO.inspect
When querying directly from neo4j using the web dashboard, I got 281ms in the first try and 56ms in the second try, much less than using bolt_sips.
Much faster query.
Am I doing something wrong? :(
Or is this the expected behaviour?
Hello, I started to play with bolt sips 3 weeks ago, and today I deployed my service to my staging enviroment but I have found some issues:
I'm receiving a time out error when I made query like this:
"MATCH (s) WHERE id(s) = {start_node_id}\nMATCH (e) WHERE id(e) = {end_node_id}\nCREATE (s)-[relationship:WRITES{created_at: {created_at}, source: {source}}]->(e)\nRETURN relationship", %{created_at: 1521829096, end_node_id: 1888, source: "messenger", start_node_id: 2467}
My logs:
3/23/2018 12:18:31 PM ** (exit) exited in: :gen_server.call(#PID<0.1494.0>, {#Port<0.3962>, "MATCH (s) WHERE id(s) = {start_node_id}\nMATCH (e) WHERE id(e) = {end_node_id}\nCREATE (s)-[relationship:WRITES{created_at: {created_at}, source: {source}}]->(e)\nRETURN relationship", %{created_at: 1521829096, end_node_id: 1888, source: "messenger", start_node_id: 2467}}, 15000)
3/23/2018 12:18:31 PM ** (EXIT) time out
But when I made queries like this:
"MATCH (node:USER)-[:WRITES]->(:BOT{bot_id:{bot_id}})\nRETURN node\nORDER BY node.created_at", %{bot_id: "my-bot"}
I got the results that I expect.
Environment:
Annd my bolt sips configuration:
config :bolt_sips, Bolt,
hostname: "localhost",
port: 7687,
pool_size: 5,
max_overflow: 1
It would be great to get your feedback about this issue.
Thanks for your help
Is there a way to get date and datetime properties working with 0.4.12
?
I am currently getting Neo.ClientError.Request.Invalid
with DateTime is not yet supported as a return type in Bolt
.
@dominique-vassard - we use the -rc postfix mostly to stay close to the db_connection's own status. Now that db_connection is stable, should we too drop the -rc? What do you think?
bolt://
, bolt+routing://
or neo4j://
): bolt://I just followed your Getting started guide, but when I try to do: DEBUG=1 iex -S mix
I just got:
[info] Application n4d exited: shutdown
My files are:
mix.exs
defp deps do
[
{:bolt_sips, "~> 2.0.0-rc"},
{:jason, "~> 1.1"}
]
end
application.ex
def start(_type, _args) do
# List all child processes to be supervised
children = [
# Starts a worker by calling: N4D.Worker.start_link(arg)
# {N4D.Worker, arg}
{Bolt.Sips, Application.get_env(:bolt_sips, Bolt)}
]
# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: N4D.Supervisor]
Supervisor.start_link(children, opts)
end
config.exs
config :bolt_sips, Bolt,
url: "bolt://localhost:7687",
pool_size: 10
I don't know, what I'm doing wrong, can someone help me, thanks!
Hello Florin,
Turns out I misunderstand something regarding the pool. When I call:
Bolt.Sips.conn
I expect to get an established connection to a Neo4j instance, but actually, it starts establishing a new connection from scratch, it doesn't get it from the pool. The Bolt.Sips.Connection uses the pool to be sure there is a limited number of process which establish connections, but those processes don't preserve established connections for further use. The connection operation is quite expensive (40-50 ms in my case).
Did I get it wrong? I am looking forward for your response. Thanks. ๐
Hi Florin, thank you for the efforts you are putting in to this project.
Today I`ve seen a strange behaviour when using bolt_sips driver. It somewhat gives wrong result when there are negative integers on neo4j. for example: an integer which value is -1 become 256 and -256 becomes 65280 in bolt_sips query result.
It seems bolt_sips does not interpret neo4j`s negative integers.
Heads-up; please don't use the Bolt.Sips.Success module, for custom implementations - it will go away in the new version, together with some other components, as a result of a large driver refactoring process.
as ecto v2 requries db_connection: ~> 1.1
, what commit should I use to avoid upgrading ecto to v3?
I am using bolt_sips 0.4.12 and experience an error in production. Neo4j worked for a few hours and threw
ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: unable to create new native thread unable to create new native thread java.lang.OutOfMemoryError: unable to create new native thread
I am wondering if the latest commits would fix this problem.
I don't see any sample to connect to a cluster, is that option supported by the driver?
cc @yoelfme
The official Neo4j drivers are using more and more the support offered by the
boltkit toolkit, and we too would like to use it.
boltkit
is extremely useful for development, offering a great test-bed for simulating server responses in various scenarios. You can see how the above mentioned drivers are implementing their tests with it, for example these "mocks".
To integrate boltkit
in our tests, we'd have to start the boltkit
(stub)server at the beginning of our tests, and kill/stop it when they end; we could most probably use System.cmd/3
, for bootstrapping it. Then, with the boltkit
server running as a background process, we can send and receive scripted scenarios, making our tests depending much-much less on having various Neo4j server instances, of different versions, clustered servers or not clustered, simulate nodes going away or corrupted routing tables and so on. The later being a very tedious and time consuming process, especially given the hardware resources eaten up from our dev machines, when in conjuncture with docker/docker-compose & Co. ๐ข
We'd appreciate any support for donating from your skills and dev time to improving our development environment and for also making our core-driver development easier and more robust nevertheless.
Thank you!
โค๏ธ
More of a stylistic thing but most likely not intended.
When I install the current (0.2.5) package and look at its content, I find an erl_crash.dump file in lib.
Big Kudos and Many Thanks for providing this package by the way ๐
I'm getting:
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in Boltex.Bolt.unpack/1
(bolt_sips) lib/boltex/bolt.ex:234: Boltex.Bolt.unpack({:error, :timeout})
(bolt_sips) lib/boltex/bolt.ex:198: Boltex.Bolt.receive_data/3
(bolt_sips) lib/boltex/bolt.ex:154: Boltex.Bolt.run_statement/4
(bolt_sips) lib/bolt_sips/connection.ex:52: Bolt.Sips.Connection.handle_call/3
(stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:647: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
For this query: USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM \"file:/import_1482495188_5.csv\" AS row MERGE (:A {aid: toInteger(row.aid), name: row.name})
which can take up to 20 seconds to process.
I followed the minimalist configuration from the README and receive the following error:
iex(1)> {:ok, _p} = Bolt.Sips.start_link(host: "localhost")
** (UndefinedFunctionError) function Bolt.Sips.start_link/1 is undefined (module Bolt.Sips is not available) Bolt.Sips.start_link([host: "localhost"])
Also, iex -S mix
fails:
=INFO REPORT==== 4-Nov-2016::20:39:19 ===
application: logger
exited: stopped
type: temporary
** (Mix) Could not start application boltex: could not find application file: boltex.app
Hi,
I am trying to get the driver working in iex
but unfortunately am running into the titular error. It works just fine from the terminal, but I am getting these unexpected return values elsewhere.
I am having the iex
issue on both Mac and Windows. However, none of it works on Windows, whereas the terminal command on OS X does work.
The link to my project is here if you want to look at it.
I apologize if this is not the proper forum, but I cannot find any reference to this issue anywhere else. I also have checked my configuration against multiple examples, and everything seems fine to me. I'm fairly stumped. Any help would be appreciated.
Thanks!
Bolt protocol has moved to its separate repository, can you update the links to the actual protocol documentation to serialization https://github.com/boltprotocol/boltprotocol/blob/master/v1/_serialization.asciidoc#path-structure
I'm thinking of doing the following improvements to the README.md
Just indicate the standard way of declaring a mix dependency.
the example for the github and local dependency should be removed just to keep documentation concise.
Adapt the Add the bolt_sips dependency the applications list
section.
remove the mod: {Bolt.Sips.Application, []}
part as Bolt.Sips.Application
actually
is an empty supervisor and does nothing.
So I'm guessing the original intention of Bolt.Sips.Application
(correct me if I'm wrong)
was to start Bolt.Sips with some configuration options (see snippet bellow). And in that case, remove the
part that shows how to use Bolt.Sips.start_link
manually.
I'm currently doing something like this in my own app's supervision tree:
children = [
worker(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)])
]
bolt_sips
to mix :applications
should be only on Installation section.docker run -p 7687:7687 -e 'NEO4J_AUTH=none' neo4j:3.0.6
I have the following fulltext index on my database:
CALL db.index.fulltext.createNodeIndex(
"topic_label",
["topic"],
["label"],
{ analyzer: "simple" });
And the following Bolt.Sips configuration (note the pool_size of just 1):
[
url: "bolt://localhost:7687",
ssl: true,
basic_auth: [
username: "neo4j",
password: "[REDACTED]"
],
pool_size: 1
]
If I issue the following malformed query (because badparen)
is not quoted and is therefore treated like a Lucene query containing syntax errors):
Bolt.Sips.query(
Bolt.Sips.conn(),
"CALL db.index.fulltext.queryNodes(\"topic_label\", \"badparen)\") YIELD node RETURN node")
Then that query correctly fails with an error from Apache Lucene:
{:error,
[
code: nil,
message: "run_statement: Unknown failure: {:failure, %Bolt.Sips.Internals.Error{code: \"Neo.ClientError.Procedure.ProcedureCallFailed\", connection_id: 48, function: :pull_all, message: \"Failed to invoke procedure `db.index.fulltext.queryNodes`: Caused by: org.apache.lucene.queryparser.classic.ParseException: Encountered \\\" \\\")\\\" \\\") \\\"\\\" at line 1, column 8.\\nWas expecting one of:\\n <EOF> \\n <AND> ...\\n <OR> ...\\n <NOT> ...\\n \\\"+\\\" ...\\n \\\"-\\\" ...\\n <BAREOPER> ...\\n \\\"(\\\" ...\\n \\\"*\\\" ...\\n \\\"^\\\" ...\\n <QUOTED> ...\\n <TERM> ...\\n <FUZZY_SLOP> ...\\n <PREFIXTERM> ...\\n <WILDTERM> ...\\n <REGEXPTERM> ...\\n \\\"[\\\" ...\\n \\\"{\\\" ...\\n <NUMBER> ...\\n \", type: :cypher_error}}\n"
]}
But! If I then try to issue any further queries:
Bolt.Sips.query(Bolt.Sips.conn(), "Match (n) return n")
Then they fail with the following exception:
{:error,
[
code: nil,
message: "The session is in a failed state and ignores further messages. You need to\n`ACK_FAILURE` or `RESET` in order to send new messages.\n"
]}
I wasn't sure if this was a bug or if I was missing a configuration, but it looked like the failed state error was leaking out of internals; was something supposed to ACK or RESET on my behalf? The connection seems to stay in this bad state indefinitely, which could poison a pool over time if a running system has the right bugs in it.
Thanks!
bolt://
, bolt+routing://
or neo4j://
):Not reporting an issue here but asking if there are plans to build in support for the routing extension.
I am currently at graphconnect and learning about how this is implemented in other drivers.
I am happy to share what I learn and to support implementation.
Curious if there is already work in progress on this?
Hi,
As project grows, document does too.
I found current Hex documentation a little bit messy and it doesn't help user to leverage the true power of bolt_sips.
What can we do?
Bolt.Sips.ConfigAgent
is at the same level as Bolt.Sips.Types
, and we know that the first is for internal use only.What do you think,
Hi, first of all thank you very much for this project!
When running a query with parameters I encountered the following error message:
(FunctionClauseError) no function clause matching in Boltex.Bolt.generate_chunks/3
(bolt_sips) lib/boltex/bolt.ex:115: Boltex.Bolt.generate_chunks([<<178, 16, 209, 7, 244, 77, 69, 82, 71, 69, 32, 40, 103, 105, 116, 104, 117, 98, 58, 71, 105, 116, 104, 117, 98, 80, 114, 111, 102, 105, 108, 101, 32, 123, 32, 103, 105, 116, 104, 117, 98, 73, 100, 58, 32, 36, 117, 115, 101, ...>>, <<177, 63, 192>>], [], "")
(bolt_sips) lib/boltex/bolt.ex:102: Boltex.Bolt.send_messages/3
(bolt_sips) lib/boltex/bolt.ex:157: Boltex.Bolt.run_statement/4
With a larger amount of parameters it seems like generating chunks isn't working correctly. Unfortunately I wasn't able to find the exact size.
What I saw is that on this line the chunks
variable is mentioned twice in the function signature. Could that be an issue?
Hello!
a little example whith semicolon in text, which raise exeption
Bolt.Sips.query!(Bolt.Sips.conn, "MERGE (:Test {text: "test with semicolon;"})")
21:58:18.753 [debug] [{:sock_ref, "", #PID<0.624.0>, #PID<0.625.0>, #PID<0.626.0>}] cypher: "MERGE (:Test {text: "test with semicolon" - params: %{} - bolt: {:error, %Boltex.Error{code: "Neo.ClientError.Statement.SyntaxError", connection_id: nil, function: :run_statement, message: "Unexpected end of input: expected '\', ANY or '"' (line 1, column 41 (offset: 40))\n"MERGE (:Test {text: "test with semicolon"\n ^", type: :cypher_error}}
(elixir) lib/exception.ex:494: RuntimeError.exception(%Boltex.Error{code: "Neo.ClientError.Statement.SyntaxError", connection_id: nil, function: :run_statement, message: "Unexpected end of input: expected '\', ANY or '"' (line 1, column 41 (offset: 40))\n"MERGE (:Test {text: "test with semicolon"\n ^", type: :cypher_error})
(bolt_sips) lib/bolt_sips/query.ex:100: Bolt.Sips.Query.send!/4
(elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
(bolt_sips) lib/bolt_sips/query.ex:86: Bolt.Sips.Query.tx/3
(bolt_sips) lib/bolt_sips/query.ex:55: Bolt.Sips.Query.query!/3
In web editor this request process correctly.
Considering the awaited return, which happened in bolt version 1 and 2 (Neo4j < 3.4)
iex(5)> Bolt.Sips.transaction(Bolt.Sips.conn(), fn conn ->
...(5)> Bolt.Sips.query(conn, "RETRN 1 AS num")
...(5)> end)
{:ok,
{:error,
[
code: "Neo.ClientError.Statement.SyntaxError",
message: "Invalid input 'R': expected 'u/U' (line 1, column 4 (offset: 3))\n\"RETRN 1 AS num\"\n ^"
]}}
The same code in a bolt V3 Neo4j server:
iex(7)> Bolt.Sips.transaction(Bolt.Sips.conn(), fn conn ->
...(7)> Bolt.Sips.query(conn, "RETRN 1 AS num")
...(7)> end)
** (MatchError) no match of right hand side value: {:error, %Bolt.Sips.Internals.Error{code: "Neo.ClientError.Request.Invalid", connection_id: 10928, function: :commit, message: "Message 'COMMIT' cannot be handled by a session in the READY state.", type: :cypher_error}}
(bolt_sips) lib/bolt_sips/protocol.ex:117: Bolt.Sips.Protocol.handle_commit/2
(db_connection) lib/db_connection/holder.ex:270: DBConnection.Holder.holder_apply/4
(db_connection) lib/db_connection.ex:1468: DBConnection.run_commit/3
(db_connection) lib/db_connection.ex:1006: DBConnection.checkin/4
(db_connection) lib/db_connection.ex:1450: DBConnection.commit/3
(db_connection) lib/db_connection.ex:1377: DBConnection.run_transaction/4
This is due to how DBConnection handles the transaction:
But in Bolt v3, a rollback
is automatically performed at server-level if query is invalid.
All is required is a RESET
message...
I'm trying to use parameters to create nodes and not having much luck. I started out with:
Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report = {props}", %{props: my_map})
That gives me:
** (Bolt.Sips.Exception) socket is not connected
(bolt_sips) lib/bolt_sips/query.ex:57: Bolt.Sips.Query.query!/3
lib/mix/tasks/test.load_db.ex:32: anonymous fn/2 in Mix.Tasks.Test.LoadDb.run/1
(elixir) lib/enum.ex:1841: Enum.reduce_range_inc/4
lib/mix/tasks/test.load_db.ex:18: Mix.Tasks.Test.LoadDb.run/1
(mix) lib/mix/task.ex:301: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:75: Mix.CLI.run_task/2
To simplify I tried this:
Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report.test = {props}", %{props: 'foo'})
That successfully queries, but then the test
property has the value "102,111,111"
in the database.
This works, however:
Neo4j.query!(Neo4j.conn, "CREATE (report:Report) SET report.test = 'foo'")
So it seems like maybe there's just something wrong with params (?)
This was mentioned on line 32 of the task definition.
I personally would like to use this as a starting point for writing other tasks for Bolt, like setting up constraints, so I'm interested in this too, but I have no Elixir skill yet.
Thank you for all of your hard work! I'm a huge fan of Neo4J, and it's because of this project that I can use Elixir.
However, being new to Elixir, I'm not sure what I'm doing wrong or how to read this stack trace. I'm trying to connect to GrapheneDB and get the following error:
iex(1)> Bolt.Sips.conn
** (exit) exited in: :gen_server.call(#PID<0.427.0>, :connect)
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in Boltex.Utils."-hex_encode/1-lc$^0/1-0-"/1
(bolt_sips) lib/boltex/utils.ex:9: Boltex.Utils."-hex_encode/1-lc$^0/1-0-"({:error, :closed})
(bolt_sips) lib/boltex/bolt.ex:44: Boltex.Bolt.handshake/2
(bolt_sips) lib/bolt_sips/connection.ex:41: Bolt.Sips.Connection.handle_call/3
(stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:647: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
(stdlib) gen_server.erl:204: :gen_server.call/2
(poolboy) src/poolboy.erl:76: :poolboy.transaction/3
iex(1)> [error] GenServer #PID<0.427.0> terminating
** (FunctionClauseError) no function clause matching in Boltex.Utils."-hex_encode/1-lc$^0/1-0-"/1
(bolt_sips) lib/boltex/utils.ex:9: Boltex.Utils."-hex_encode/1-lc$^0/1-0-"({:error, :closed})
(bolt_sips) lib/boltex/bolt.ex:44: Boltex.Bolt.handshake/2
(bolt_sips) lib/bolt_sips/connection.ex:41: Bolt.Sips.Connection.handle_call/3
(stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:647: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :connect
State: [timeout: 15000, hostname: "bolt://hobby.dbs.graphenedb.com", basic_auth: [username: "fakeusername", password: "fakepassword"], port: 24786, pool_size: 5, max_overflow: 1]
My config:
config :bolt_sips, Bolt,
hostname: "bolt://hobby.dbs.graphenedb.com",
basic_auth: [username: "fakeusername", password: "fakepassword"],
port: 24786,
pool_size: 5,
max_overflow: 1
Any ideas?
I call the following code in IEX to start my bolt.sips connection, unchanged from how I did with version 1.3.0 of Bolt Sips:
Bolt.Sips.start_link([
url: "bolt://localhost:7687",
ssl: true,
basic_auth: [username: "neo4j", password: "secret"]
])
and the following is logged in response:
18:29:55.186 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [3, 2, 1, 0]"
18:29:55.186 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [3, 2, 1, 0]"
18:29:55.187 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x3>>
18:29:55.187 [debug] S: HANDSHAKE ~ 3
18:29:55.187 [debug] C: HELLO ~ [%{credentials: "secret", principal: "neo4j", scheme: "basic", user_agent: "BoltSips/1.5.1"}]
18:29:55.187 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x3>>
18:29:55.187 [debug] S: HANDSHAKE ~ 3
18:29:55.187 [debug] C: HELLO ~ [%{credentials: "secret", principal: "neo4j", scheme: "basic", user_agent: "BoltSips/1.5.1"}]
18:29:55.187 [error] GenServer #PID<0.574.0> terminating
** (RuntimeError) connect raised UndefinedFunctionError exception. The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true when starting your connection if you wish to see all of the details
(bolt_sips) Bolt.Sips.Internals.PackStream.Encoder.encode/2
(bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:145: Bolt.Sips.Internals.PackStream.Message.Encoder.encode_message/4
(bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:105: Bolt.Sips.Internals.PackStream.Message.Encoder.call_encode/3
(bolt_sips) lib/bolt_sips/internals/bolt_protocol_helper.ex:20: Bolt.Sips.Internals.BoltProtocolHelper.send_message/4
(bolt_sips) lib/bolt_sips/internals/bolt_protocol_v3.ex:28: Bolt.Sips.Internals.BoltProtocolV3.hello/5
(bolt_sips) lib/bolt_sips/protocol.ex:39: Bolt.Sips.Protocol.connect/1
(db_connection) lib/db_connection/connection.ex:69: DBConnection.Connection.connect/2
(connection) lib/connection.ex:622: Connection.enter_connect/5
Last message: nil
State: Bolt.Sips.Protocol
Several such errors are logged for various PIDs.
[EDIT]
If I downgrade to 1.4.0 I receive a similar repeated error:
23:48:54.658 [debug] C: HANDSHAKE ~ "<<0x60, 0x60, 0xB0, 0x17>> [2, 1, 0, 0]"
23:48:54.659 [debug] S: HANDSHAKE ~ <<0x0, 0x0, 0x0, 0x2>>
23:48:54.659 [debug] S: HANDSHAKE ~ 2
23:48:54.659 [debug] C: INIT ~ ["BoltSips/", %{credentials: "secret", principal: "neo4j", scheme: "basic"}]
23:48:54.660 [error] GenServer #PID<0.338.0> terminating
** (RuntimeError) connect raised UndefinedFunctionError exception. The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true when starting your connection if you wish to see all of the details
(bolt_sips) Bolt.Sips.Internals.PackStream.Encoder.encode/2
(bolt_sips) lib/bolt_sips/internals/pack_stream/message/encoder.ex:209: Bolt.Sips.Internals.PackStream.Message.Encoder.do_encode/3
(bolt_sips) lib/bolt_sips/internals/bolt_protocol.ex:177: Bolt.Sips.Internals.BoltProtocol.send_message/4
(bolt_sips) lib/bolt_sips/internals/bolt_protocol.ex:155: Bolt.Sips.Internals.BoltProtocol.init/5
(bolt_sips) lib/bolt_sips/protocol.ex:39: Bolt.Sips.Protocol.connect/1
(db_connection) lib/db_connection/connection.ex:69: DBConnection.Connection.connect/2
(connection) lib/connection.ex:622: Connection.enter_connect/5
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: nil
State: Bolt.Sips.Protocol
Downgrading further to 1.3.0 resolves the issue as expected, so this appears to be an issue for me only since 1.4.0.
I expected this to establish a database connection with Neo4j, as it did in version 1.3.0.
I tried passing the :show_sensitive_...
flag along to Bolt.Sips.start_link, not knowing if it would ultimately be passed down to DBConnection, but the same message was logged without additional information.
Hi, it seems that the current verison of Bolt.Sips.Response.transform is not working correctly. When i do the next neo4j query...
cypher = """
MATCH (a:Person)
WHERE a.uid = {uid}
RETURN a
"""
data = Bolt.query!(Bolt.conn(), cypher, %{uid: uid})
(hd data)["a"]
I get the next response:
%Bolt.Sips.Types.Node{
id: 55,
labels: ["Person"],
properties: %{
"last_payment" => [sig: 70, fields: [1550835424, 283000000, 0]],
"uid" => "test1112"
}
}
So the DateTime doesn't get parsed correctly.
Currently I solve this using the next code in my view:
data = Enum.reduce(
Map.to_list(person.properties),
%{},
fn obj, acc ->
Map.put(acc,
elem(obj, 0),
if is_binary(elem(obj, 1)) do
elem(obj, 1)
else # We expect :sig to be 70
[seconds, nanoseconds, offset] = elem(obj, 1)[:fields]
naive_dt = NaiveDateTime.add(
~N[1970-01-01 00:00:00.000],
seconds * 1_000_000_000 + nanoseconds,
:nanosecond
)
NaiveDateTime.to_iso8601(naive_dt) <>
Bolt.Sips.TypesHelper.formated_time_offset(offset)
end)
end)
This is just a workaround using the code I stripped from your Bolt.Sips.Response class.
Thanks for your hard work on this driver.
I never ran into this issue whilst working in a development environment, since there was little downtime between subsequent queries. However, after testing out the current build on a production server, I noticed that all the query execution attempts after ~15 minutes of inactivity produce a timeout exception.
My configuration for the driver is as follows:
config :bolt_sips, Bolt,
url: "bolt://<host>:8124",
basic_auth: [username: "neo4j", password: "***"]
I use the driver API as it is recommended in the documentation, i.e.
Bolt.Sips.conn()
|> Bolt.Sips.query!( ... )
The Neo4j server is a Docker image with default configuration.
Any ideas on what causes the trouble?
Thank you.
Hey @florinpatrascu,
I noticed that you are no longer using Boltex as a dependency but instead have the files in this project.
Could you elaborate what lead to that decision? I see multiple disadvantages from diverging bolt implementations and would love for us to find a way to keep the protocol and a client library separate.
Please let me know if there is anything in particular that you were looking for or that I did that led you to copy the code over to here. If it was the implementation of the db_connection library in boltex, I was under the impression that I kept it isolated enough for custom connection handling to still work (let me know if that was not the case - I am open to breaking out the DB Connection part into a separate library).
I think it'd be ideal if we could go back to separate projects so users have the choice between using just low-level protocol implementations (as I am doing in one project) or using a fully-featured library such as yours. This would have the added benefit that we would bundle our resources in ensuring the best possible bolt implementation in Elixir.
Thank you,
Michael
Just for your information. :)
Even with all deps installed it doesn't work.
The owner has said that he doesn't have experience with Windows, so probably no fix is coming soon.
Is there still any reason to depend on florinpatrascu/boltex
instead of mschae/boltex
?
mschae/boltex#10 has been merged and would like to remove the TODO I had once we depend on the orignal boltex code.
Context:
export BOLT_WITH_ETLS=true
to my .profile
in Ubuntu.Expected result: The etls package is installed along with the bolt_sips.
Actual result: The etls is missing (the project crashes at launch).
** (Mix) Could not start application etls: could not find application file: etls.app
I successfully launched the project after adding the etls into dependencies of the project. I think the mix installs only dependencies listed in the mix.lock of the bolt_sips. Actually, I am ok to keep the etls among dependencies of my project. Probably, the approach with the BOLT_WITH_ETLS env variable should be changed. My advice is to get rid of the BOLT_WITH_ETLS and mention in the readme that the etls should be added to dependencies of a project which uses the bolt_sips. Meantime, the bolt_sips can provide a configuration option to select the ssl lib, example:
config :bolt_sips, Bolt,
ssl_lib: :etls
What do you think?
Ahoy!
I'm able to use Bolt.Sips.conn in my test cases and interactively in iex, but an escript I've compiled is misbehaving. I'm sure this is something wrong with my configuration, but Bolt.Sips.conn returns a bare :error
symbol, which causes Bolt.Sips.query to fail rather loudly. It looks like the :error
is coming from :poolboy's
transaction logic (In the v0.4.12 branch at Connection.ex pool_server/3, line 94).
I thought I'd raise this up in case this isn't an intentional hand-off. Normally I would expect to receive { :ok, conn }
from Sips.conn if getting an error was possible, since that would force me to match and coincidentally root out :error
s early in the stack; that might be a worthwhile change to the Bolt.Sips.conn api.
Thanks for Bolt.Sips! It's been fantastic to work with so far. I'll go see if I can get some logging out of poolboy. :D
Stacktrace if you want it:
7:27:01.222 [error] GenServer #PID<0.113.0> terminating
** (FunctionClauseError) no function clause matching in :ssl.send/2
ssl.erl:322: :ssl.send(:error, <<0, 2, 176, 14, 0, 0>>) # :error is from Bolt.Sips.conn via :poolboy
(elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:769: Enum.each/2
(boltex) lib/boltex/bolt.ex:211: Boltex.Bolt.ack_failure/3
(bolt_sips) lib/bolt_sips/connection.ex:65: anonymous fn/3 in Bolt.Sips.Connection.handle_call/3
(elixir) lib/enum.ex:3317: Enumerable.List.reduce/3
(elixir) lib/stream.ex:911: Stream.do_list_transform/7
(elixir) lib/enum.ex:1990: Enum.reduce_while/3
Last message (from TextClient.Agent): {:error, "MATCH (a:Card { id: $id })\nRETURN a", %{id: "5"}}
Hello, in the latest weeks I was working in a project using Bolt Sips to connect to Neo4j, and today I deployed my service to production but I'm experience some issues, I'm receiving a lot of requests and doing almost 5 queries to Neo4j per request, but sometimes that queries give me a timeout and then my requests answer with an 500 status code.
This is the error:
7/10/2018 11:52:29 AM16:52:29.337 [error] #PID<0.2632.2> running UsersAPIWeb.Endpoint terminated
7/10/2018 11:52:29 AMServer: people-service:4000 (http)
7/10/2018 11:52:29 AMRequest: POST /api/bots/MTCenter/users
7/10/2018 11:52:29 AM** (exit) exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.2148655786.3957063682.261483>, true}, 5000)
7/10/2018 11:52:29 AM ** (EXIT) time out
7/10/2018 11:52:29 AM16:52:29.366 request_id=6qd7pdbvcenep2evgp0dic4uic8vnb0j [info] Updating whatsapp status of user: 7228977818 with: %{"input" => "7228977818", "status" => "valid", "wa_id" => "5217228977818"}
7/10/2018 11:52:29 AM16:52:29.429 [error] #PID<0.2705.2> running UsersAPIWeb.Endpoint terminated
7/10/2018 11:52:29 AMServer: people:4000 (http)
7/10/2018 11:52:29 AMRequest: POST /api/bots/coppel/users/update-last-interaction
7/10/2018 11:52:29 AM** (exit) exited in: :gen_server.call(:bolt_sips_pool, {:checkout, #Reference<0.2148655786.3957063682.261495>, true}, 5000)
7/10/2018 11:52:29 AM ** (EXIT) time out
Environment:
And my bolt sips configuration is:
# Set configuration for Neo4j with Bolt
config :bolt_sips, Bolt,
hostname: "${NEO4J_HOST}",
port: 7687,
basic_auth: [username: "${NEO4J_USER}", password: "${NEO4J_PASSWORD}"],
pool_size: 10,
max_overflow: 5,
retry_linear_backoff: [delay: 150, factor: 2, tries: 3]
My application supervisor has this childrens
children = [
# Start Neo4j connection
worker(Bolt.Sips, [Application.get_env(:bolt_sips, Bolt)]),
# Start the endpoint when the application starts
supervisor(UsersAPIWeb.Endpoint, [])
]
It would be great to get your feedback about this issue.
Thanks for your help
stacktrace
** (Bolt.Sips.Exception) timeout
(bolt_sips) lib/bolt_sips/query.ex:59: Bolt.Sips.Query.query!/3
Erlang/OTP 22
IEx 1.9.1 (compiled with Erlang/OTP 22)
* Neo4j version (Neo4j 3.5.1):
* Bolt.Sips version: "1.5.1",
* Operating system: macOS 10.14.4
* Connection config
config :bolt_sips, Bolt,
hostname: 'hostname.com',
basic_auth: [username: "user", password: "****"],
port: 7687,
pool_size: 50,
max_overflow: 5,
timeout: 5_000,
retry_linear_backoff: [delay: 150, factor: 2, tries: 3]
while executing query randomly it gives timeout then again after retrying it works fine. I was using previously :neo4j_sips, "0.2.18", it never gave such timeouts. I decided to move it to this library then this is happening let me know if you want any other info which can help you to debug this issue or in case I'm doing something wrong.
Today, all Bolt.Sips.Types are encoded / decoded at "high level" while this operations should be done a "low level", i.e in the bolt encoding layer.
This means that they are encoded twice:
Or decoded twice:
Each Bolt.Sips.Type should encoded / decoded at the bolt encoding / decoding layer.
Why?
End user won't be impacted, as everything will be internally handled.
Some big changes are expected: Bolt version will be split in their own files.
AS I figure out when testing, a request with unsupported data hangs, then it has to be handled before being sent to server.
Architecture will reflect the one used in: https://github.com/dominique-vassard/bolt-neo4j
with encoder_v1, encoder_v2, etc.
This will be clearer and future version could be easily added, tested and not lost inside one big file.
bolt_protocol.ex
should follow the same pattern as bolt v3 introduces ne messages and having everything in the same file is just a mess.
This change also means that bolt version should be passed all the way down to the encoding / decoding layer.
In order to hide this from a user perspective (and to not break current API), it will be stored in conn
.
Let's do it! :)
We'd like to stop overriding DBConnection everywhere in our app. In the process of upgrading our outdated dependencies, something broke and another library stopped working due to it expecting a different method in DBConnection. db_connection
1.1.3
was released on 13/01/2018 on Hex - is this a recent enough version for bolt_sips
, and if so, can bolt_sips
's mixfile be updated to use the Hex package?
My team and I found bolt_sips
has two apparently wrong behaviors regarding to connection pooling.
timeout
parameter only applies to the connection between the driver and the neo4j server, but it doesn't apply to the client application requests.You can try by yourself starting this minimal application. Just change the query/queries you want to execute in BoltSipsLoad
.
After cloning, and deps compilation, try:
$ iex -S mix
ex(1)> BoltSipsLoad.load_test(5, 1, 500)
This would repeat 5 times launching 1 process to execute a query and then will wait 500ms for the next iteration. In another terminal you can check for the number of opened sockets to the neo4j server (substitute pid for watherver the process id of your beam is):
$ lsof -nP -i4TCP | awk -v pid=$pid '$2 == pid {print $0}' | grep 7687 | wc -l
5
5 sockets to the bolt socket, as expected.
Now something more interesting. Let's make 10 iterations, launching 100 processes in each one.
ex(2)> BoltSipsLoad.load_test(10, 100, 500)
$ lsof -nP -i4TCP | awk -v pid=$pid '$2 == pid {print $0}' | grep 7687 | wc -l
470
What?!
This is confirmed in the observer application. Look at all those tcp_inet
ports
And each one of them is a port to the neo4j bolt port.
Unfortunately this is causing trouble in a production system that was just handed to me. Increasing the limit of file descriptors the process can open just moves the problem somewhere else, because the neo4j server can't handle thousands of connections without getting into performance problems. In any case, there shouldn't be there as many open connections, that's what the pool is intended for, isn't it?
It would be great to get your confirmation/feedback about this issue.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.