I am trying to export data from mysql to neo4j using neo4j-etl tool.
Neo4j Version : 3.3.4
Neo4j-etl Version : 1.2.0
Operating System : Mac OS X
I am running below command
./neo4j-etl export --rdbms:url jdbc:mysql://localhost:3306/openmrs --rdbms:user root --rdbms:password asdf --rdbms:schema openmrs --neo4j:user neo4j --neo4j:password qwas --import-tool /usr/local/Cellar/neo4j/3.3.4/libexec/bin --csv-directory /usr/local/Cellar/neo4j/3.3.4/libexec/import --mapping-file mappingNew.json --options-file options.json --using cypher:direct --quote '",' --debug --force
I am getting following error while importing. This is due to escape characters in a CSV file
org.neo4j.driver.v1.exceptions.DatabaseException: At /usr/local/Cellar/neo4j/3.3.4/libexec/import/csv-008/openmrs/NODE_openmrs.raxacoremessage_8e1dbe64-8a43-4c39-9513-1cd0aa175ab2.csv:44188 - there's a field starting with a quote and whereas it ends that quote there seems to be characters in that field after that ending quote. That isn't supported. This is what I read: 'ghjg
","dd'
at org.neo4j.driver.internal.net.SocketResponseHandler.handleFailureMessage(SocketResponseHandler.java:83)
at org.neo4j.driver.internal.messaging.PackStreamMessageFormatV1$Reader.unpackFailureMessage(PackStreamMessageFormatV1.java:470)
at org.neo4j.driver.internal.messaging.PackStreamMessageFormatV1$Reader.read(PackStreamMessageFormatV1.java:431)
at org.neo4j.driver.internal.net.SocketClient.receiveOne(SocketClient.java:196)
at org.neo4j.driver.internal.net.SocketConnection.receiveOne(SocketConnection.java:217)
at org.neo4j.driver.internal.net.ConcurrencyGuardingConnection.receiveOne(ConcurrencyGuardingConnection.java:165)
at org.neo4j.driver.internal.net.pooling.PooledSocketConnection.receiveOne(PooledSocketConnection.java:183)
at org.neo4j.driver.internal.InternalStatementResult.receiveOne(InternalStatementResult.java:335)
at org.neo4j.driver.internal.InternalStatementResult.consume(InternalStatementResult.java:291)
at org.neo4j.etl.util.CypherBoltRunner.execute(CypherBoltRunner.java:64)
at org.neo4j.etl.commands.rdbms.importer.BoltDriverImportFromRdbms.doLoadCsv(BoltDriverImportFromRdbms.java:29)
at org.neo4j.etl.commands.rdbms.importer.AbstractLoadCsvImportFromRdbms.extractAndLoad(AbstractLoadCsvImportFromRdbms.java:58)
at org.neo4j.etl.commands.rdbms.importer.BoltDriverImportFromRdbms.extractAndLoad(BoltDriverImportFromRdbms.java:16)
at org.neo4j.etl.cli.rdbms.ImportFromRdbmsCli.run(ImportFromRdbmsCli.java:150)
at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)
I have created a dummy database neocheck for replicating this issue
Here is the csv file
"1","1","p/a soft\","Ob"
"2","2","p/a soft\","Ob"
"3","3","dsfsad
dfsafsd
dsafds
dsafasd
dsfads
","Ob"
"4","4","dsaf
dfsadf
/dffas
dfsa","Ob"
When I am running this command
./neo4j-etl export --rdbms:url jdbc:mysql://localhost:3306/neocheck --rdbms:user root --rdbms:password qwas --rdbms:schema neocheck --using bulk:neo4j-import --neo4j:user neo4j --neo4j:password qwas --import-tool $NEO4J_HOME/bin --destination $NEO4J_HOME/data/databases/graph.db/ --csv-directory $NEO4J_HOME/import --options-file options.json --debug --force
It is giving me the following error
Error in input data
Caused by:ERROR in input
data source: BufferedCharSeeker[source:/Users/zakirsaifi/Desktop/Office/Task/ETL/neo4j-community-3.3.5/import/csv-001/neocheck/NODE_neocheck.OBS_6427d0d8-cdc5-47b9-9d39-c271c2646ed2.csv, position:4194376, line:0]
in field: valueText:string:3
for header: [:ID(neocheck.obs), id:long, valueText:string, :LABEL]
raw field value: 1
original error: At /Users/zakirsaifi/Desktop/Office/Task/ETL/neo4j-community-3.3.5/import/csv-001/neocheck/NODE_neocheck.OBS_6427d0d8-cdc5-47b9-9d39-c271c2646ed2.csv:0 - there's a field starting with a quote and whereas it ends that quote there seems to be characters in that field after that ending quote. That isn't supported. This is what I read: 'p/a soft","OO'
WARNING Import failed. The store files in /Users/zakirsaifi/Desktop/Office/Task/ETL/neo4j-community-3.3.5/data/databases/graph.db are left as they are, although they are likely in an unusable state. Starting a database on these store files will likely fail or observe inconsistent records so start at your own risk or delete the store manually', DurationMillis: 1731 }]
at org.neo4j.etl.process.ProcessHandle.await(ProcessHandle.java:84)
at org.neo4j.etl.neo4j.importcsv.ImportFromCsvCommand.execute(ImportFromCsvCommand.java:29)
at org.neo4j.etl.commands.rdbms.importer.Neo4jImportImportFromRdbms.doImport(Neo4jImportImportFromRdbms.java:63)
at org.neo4j.etl.commands.rdbms.importer.Neo4jImportImportFromRdbms.extractAndLoad(Neo4jImportImportFromRdbms.java:41)
at org.neo4j.etl.cli.rdbms.ImportFromRdbmsCli.run(ImportFromRdbmsCli.java:150)
at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)
It seems 'java' command is not available.
Please check your JAVA_HOME environment variable.
This is sql file for the my dummy database.
https://drive.google.com/file/d/1y_KrA0QpZOZ0WMKxTWcFyy_Z2B1Mqz-p/view?usp=sharing(url)
When I am using neo4j-etl-cli-1.2.0-RC1. I was able to import the dummy database in offline bulk mode (using bulk:neo4j-import of etl) but not in any other mode. If I could import that in earlier version why in the new version it is not escaping the newline and other chracters.
I have discussed the same in below issue [#20]