aarreedd / csv-to-elasticsearch Goto Github PK
View Code? Open in Web Editor NEWSimple tool to import CSV into ElasticSearch
License: MIT License
Simple tool to import CSV into ElasticSearch
License: MIT License
I feel so lucky to find this amazing tool for quick importing csv file to elastic and thanks a lot!
While I am new to python and came into a problem: it ran properly but returned 401 in the console.
I would appreciate so much if you can check and tell me what's wrong.
1 console
$ python3 csv_to_elastic.py --json-struct '{"answer" : "%answer%","question" : "%question%"}' --csv-file data.csv --elastic-index qa --elastic-address 'localhost:9200'
----- CSV to ElasticSearch -----
Importing all rows into qa
from 'data.csv'
Reached end of CSV - sending to Elastic
Returned status code: 401
2 I put data.csv in the same direction of py script and it looks like this:
answer | question
A1 | Q1
3 my index name is qa
Thanks a lot :)
Fields are automatically inferred to strings or numbers. This causes problems in some cases; in particular some long digitnumbers were turned into E-notation, for example 1231231231234 -> '1.231231E12'. This format was then refused by logstash since it didn't match the type.
In the json-format, it's possible to double-quote ""%key%"" to make it a string in the output; however the type conversion that happens before can still cause problems.
So there should be a way to disable this automatic conversion for specific fields, or alltogether.
I'm having issues with the first column of the file not being imported into Elasticsearch. First discovered this issue when I tried to set the first column as the "id" and it couldn't be found.
After importing my CSV, I tried a search by items in first column and it returned empty.
Is this by design? Are we supposed to be inserting a blank column in position 1?
I am getting a JSON decode error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The top of my CSV file looks like following:
"date","temperature"
"1981-01-01",20.7
"1981-01-02",17.9
"1981-01-03",18.8
I am trying to run your script using the following command:
python csv_to_elastic.py --elastic-address 'localhost:9200' --csv-file daily-minimum-temperatures-in-me.csv --elastic-index 'temperatures' --datetime-field=date --json-struct '{ "date" : "%date%", "temperature" : "%temperature%" }' --delimiter ','
What could be a reason for such issue and how can I solve it?
I tried to use the script for large file content >500000 the script gives this error
----- CSV to ElasticSearch -----
Importing all rows into `bus` from 'mrget.csv'
Traceback (most recent call last):
File "csv_to_elastic.py", line 241, in <module>
elastic_address=parsed_args.elastic_address, ssl=parsed_args.ssl, username=parsed_args.username, password=parsed_args.password, id_column=parsed_args.id_column)
File "csv_to_elastic.py", line 101, in main
elif len(row[0]) == 0: # Empty rows on the end of document
IndexError: list index out of range
Hello,
I imported my CSV file using this command :
python csv_to_elastic.py --elastic-address 'localhost:9200' --csv-file 'full_test.csv' --elastic-index 'stops_index' --datetime-field=dateField --json-struct '{"stop_id" : "%stop_id%", "stop_name" : "%stop_name%", "stop_lat" : "%stop_lat%", "stop_lon" : "%stop_lon%"}'
and I got this output :
----- CSV to ElasticSearch -----
Importing all rows into `stops_index` from 'full_test.csv'
Traceback (most recent call last):
File "csv_to_elastic.py", line 130, in <module>
elastic_address=parsed_args.elastic_address, id_column=parsed_args.id_column)
File "csv_to_elastic.py", line 54, in main
int(row[pos])
IndexError: list index out of range
Microsoft Windows [version 10.0.16299.371]
(c) 2017 Microsoft Corporation. Tous droits r▒serv▒s.
I'm running Python 3.6.5 on Windows 10.
Thank you in advance!
Hi, first off this is an incredibly helpful tool, thanks so much for putting it together!
I might suggeset that in your README, for noobs like me, you could specify that this is written for python3, and that you will need to pip3 install python-dateutil and http in order for this to work. It could save some folks a little time figuring it out if they're not fully versed in python.
You may also want to add documentation that the --delimiter and --max-rows field exists as well- I got stuck for a few minutes trying to figure out why it said it was "Importing None rows into...." to elasticsearch.
Once again- thanks!
i have about 10000 csv files, 200million rows total. i want to import them to es once. so can anybody tell me how to solve this problems;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.