phutchins / logstash-input-mongodb Goto Github PK
View Code? Open in Web Editor NEWMongoDB input plugin for Logstash
License: Other
MongoDB input plugin for Logstash
License: Other
ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html
It would be cool if the flatten and dig parse methods can format the event names using the bracket format v.s. underscores. logstash uses this format to translate to elastic's Object datatype. If done correctly, embedded mongodb docs can be mapped to elastic objects.
E.g.
change: new_hash[k1.to_s+"_"+k2.to_s] = v2
to: new_hash["[#{k1.to_s}][#{k2.to_s}]"] = v2
thus =>
Mongo doc:
{
some_field: "some value",
foo: { bar: "yay"}
}
mongo logstash:
"some_field" = "some value"
"[foo][bar]" = "yay"
elastic:
{
"some_field": "some value",
"foo": { bar: "yay"}
}
and of course, the property needs to be mapped as an object type before applying the import:
{
"some_mapping": {
"properties": {
"some_field": {"type": "text"},
"foo": {"type": "object"}
}
}
}
When the input MongoDB document has a non UTF-8 (or something like that), the input plugin gets into infinite loop, as execption is raised and process is restarted, just to hit the same case again.
logstash.stdout:
D, [2016-09-17T21:14:03.445000 #22826] DEBUG -- : MONGODB | X.X.X.X:27017 | logs.find | STARTED | {"find"=>"logs", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('57d948b1995d5258198b457c')}}, "limit"=>500}
D, [2016-09-17T21:14:03.458000 #22826] DEBUG -- : MONGODB | X.X.X.X:27017 | logs.find | SUCCEEDED | 0.013s
logstash.
{:timestamp=>"2016-09-17T21:14:03.423000+0000", :message=>"MongoDB Input threw an exception, restarting", :exception=>#<ArgumentError: invalid byte sequence in UTF-8>, :level=>:warn}
Proposed solution:
Skip error record, by adding a fake record to ES, just to make sure that when Max is requestied, the failed will not be counted
I am trying to get mongodb data into Elasticssearch using logstash, with logstash-input-mongodb plugin. I tried to update the plugin to support version 5 of ES.
In terminal it shows version 0.4.1 but in error logs it throws error for version 0.1.X.
Also it's not connecting with mongodb to get data in to ES.
Can someone help with this ?
Error log
[2017-07-10T12:50:31,447][INFO ][logstash.inputs.mongodb ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2017-07-10T12:50:32,061][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2017-07-10T12:50:32,062][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-10T12:50:32,225][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x4e62230d>}
[2017-07-10T12:50:32,227][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-07-10T12:50:32,289][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-07-10T12:50:32,299][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x38dfd049>]}
[2017-07-10T12:50:32,302][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-07-10T12:50:32,644][INFO ][logstash.inputs.mongodb ] Registering MongoDB input
[2017-07-10T12:50:32,867][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://127.0.0.1:27017/jtrade\", placeholder_db_dir=>\"/home/jtrade/logstash\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"activities\", batch_size=>5000, id=>\"961874444def3510514cefaccdbeb8ecd95670de-1\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_03e61419-953b-4267-803b-f8550e160cf7\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/home/jtrade/logstash/logstash_sqlite.db': '/home/jtrade' does not exist"}
[2017-07-10T12:50:33,239][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/home/jtrade/logstash/logstash_sqlite.db': '/home/jtrade' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>(org/sqlite/SQLiteConnection.java:45)", "org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)", "org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:226)", "RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool.rb:116)", "RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:228)", "RUBY.available(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:201)", "RUBY._acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:137)", "RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:151)", "RUBY.sync(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:282)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)", "RUBY.sync(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:282)", "RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:150)", "RUBY.hold(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/connection_pool/threaded.rb:106)", "RUBY.synchronize(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/database/connecting.rb:306)", "RUBY.execute(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:251)", "RUBY.execute(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:1081)", "RUBY.fetch_rows(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/adapters/jdbc.rb:763)", "RUBY.with_sql_each(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:942)", "RUBY.with_sql_first(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:950)", "RUBY.single_record!(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:748)", "RUBY.first(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:248)", "RUBY.[](/usr/share/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.48.0/lib/sequel/dataset/actions.rb:39)", "RUBY.get_placeholder(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:112)", "RUBY.update_watched_collections(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:160)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.update_watched_collections(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:158)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:182)", "RUBY.register_plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:281)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292)", "RUBY.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:442)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:336)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:226)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2017-07-10T12:50:33,304][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-07-10T12:50:36,267][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
Thanks you so much.
my logstash config:
input {
mongodb {
uri => 'mongodb://127.0.0.1:27017/wiki'
collection => 'wiki'
placeholder_db_dir => "E:/mongo2es/data"
placeholder_db_name => "datalogstash_sqlite_wiki.db"
batch_size => 1000
}
}
filter {
mutate {
remove_field => [ "host", "@version", "@timestamp", "logdate", "log_entry" ]
}
}
output {
stdout { codec => rubydebug }
file {
path => "E:/mongo2es/logs/mongo2es-wiki.log"
}
elasticsearch {
index => "wiki"
document_type => "wiki"
document_id => "%{mongo_id}"
hosts => ["127.0.0.1:9200"]
}
}
2767278 in mongodb, but 2767277 in elasticsearch.
Any thought?
Thanks in Advance
Hi,
When exe command
sudo bin/plugin install –no-verify /home/local/logstash-input-mongodb/logstash-input-mongodb-1.0.0-java.gem
get below error
ERROR: Mixed source of plugins, you can't mix local
.gem` and remote gems
See: 'bin/plugin install --help'
`
The lack of this information forces us to create separate logstash pipelines for every collection in order to add this information to the output.
I'm using Elasticsearch 5.4, Logstash 5.4.2, MongoDB 3.4.5.
When I import from MongoDB to Elasticsearch through Logstash, I found the following error:
[2017-07-11T16:57:06,180][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mongodb-2017.07.11", :_type=>"logs", :_routing=>nil}, 2017-07-11T08:57:03.744Z %{message}], :response=>{"index"=>{"_index"=>"mongodb-2017.07.11", "_type"=>"logs", "_id"=>"AV0w3bobj2yUz3xTJdU4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."}}}}
It mentioned that there was a conflict between the name of [_id] meta field in Kibana and [_id] meta column of MongoDB. How should I solve this conflict in my logstash pipeline config? I think this is a critical problem between ES and Mongo, how did this plugin work before?
Thank you!
Hi
Still trying to figure out how to actually install your plugin and I simply cant.
Logstash 2.1.1 install by repo
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-mongodb (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.
Install by file
Validating ./logstash-input-mongodb-0.3.3.gem
Error reading plugin file ./logstash-input-mongodb-0.3.3.gem, caused by Gem::Package::FormatError
ERROR: Installation aborted, verification failed for ./logstash-input-mongodb-0.3.3.gem
That's for 0.3.1 and 0.3.2 as well
CentOS Linux release 7.1.1503 (Core)
Hi,
Is there any plans to implement tailing the Mongo oplog in order to pickup changes to existing docs?
We use this logstash configuration:
input {
mongodb {
uri => 'mongodb://127.0.0.1/test'
placeholder_db_dir => '/users/sth00/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'Users'
batch_size => 5000
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
host => "127.0.0.1"
protocol => "http"
index => "mongodb-%{+YYYY.MM.dd}"
}
}
in mongoDB the collection "Users" only have 4 fields: _id, name, email, age
we have only 7 records and in elasticsearch these 7 records have been duplicate every second and it doesn't stop. why?
Among other things, how I can delete the fileds: logdate and logentry, we only need dump the mongodb data (Fields) into ES and keep synced these data between mongoDB and ES thanks to logstash and the plugin.
cordial regards
Hello
How to fix this exception:
D, [2016-04-07T22:37:20.095000 #1] DEBUG -- : MONGODB | my-db:27017 | my.find | SUCCEEDED | 0.006s
MongoDB Input threw an exception, restarting {:exception=>#<TypeError: The field '@timestamp' must be a (LogStash::Timestamp, not a String ("2016-04-03T15:40:39.794Z")>, :level=>:warn}
D, [2016-04-07T22:37:20.099000 #1] DEBUG -- : MONGODB | my-db:27017 | my.find | STARTED | {"find"=>"coll", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5701397834ba8f0b61000001')}}, "limit"=>5000}
How to select or exclude fields in collection!
Thx
Hi.
I've got a problem regarding the authentication with the MongoDB.
The tools I use:
This is the output of my log file containing the problem:
D, [2015-09-15T14:13:25.455000 #21908] DEBUG -- : MONGODB | Adding team-db1:27017 to the cluster.
D, [2015-09-15T14:13:25.480000 #21908] DEBUG -- : MONGODB | Adding team-db2:27017 to the cluster.
D, [2015-09-15T14:13:25.523000 #21908] DEBUG -- : MONGODB | Server team-db1:27017 discovered as primary in replica set: 'team_replSet'. Changing topology to replica set.
D, [2015-09-15T14:13:25.558000 #21908] DEBUG -- : MONGODB | team-db1:27017 | team.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
D, [2015-09-15T14:13:25.561000 #21908] DEBUG -- : MONGODB | team-db1:27017 | team.listCollections | FAILED | not authorized on team to execute command { listCollections: 1, cursor: {}, filter: { name: { $not: /system\.|\$/ } } } (13) | 0.002s
The error reported is:
not authorized on team to execute command { listCollections: 1, cursor: {}, filter: { name: { $not: /system\.|\$/ } } } (13)
My logstash Config:
input {
mongodb {
uri => "mongodb://admin:****@team-db1,team-db2/team"
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'auftraggeber.db'
collection => 'auftraggeber'
batch_size => 5000
}
}
output {
elasticsearch {
host => ["team-es2:9300"]
cluster => 'team-es-cluster'
index => 'auftraggebers'
}
}
The password contains a comma. To escape the comma I have used url-encode (%2C).
The authenticationDatabase I use is admin.
If I remove the username and password in the Logstash config, I get the same error.
@phutchins Currently, the plugin processes all the documents in the collection and infinitely waits/listens for new documents. I am trying to make the process exit once all the documents are read, so I tried this fix within 'get_cursor_for_collection' method -
public
def get_cursor_for_collection(mongodb, mongo_collection_name, last_id_object, batch_size)
collection = mongodb.collection(mongo_collection_name)
# Need to make this sort by date in object id then get the first of the series
# db.events_20150320.find().limit(1).sort({ts:1})
cursors = collection.find({:_id => {:$gt => last_id_object}}).limit(batch_size).to_a #store the array
if cursors.empty? then exit end #If cursor array empty, then exit
return collection.find({:_id => {:$gt => last_id_object}}).limit(batch_size)
end
Although i was able to test this out, when I run it against my actual collection it doesn't work. It doesn't error out but simply exits after a few seconds.
Is there a better way to do this ?
Thanks!
hi there:
i have one separate job updating a single document in mongodb multiple times a minute, but i'd like to have the logstash mongodb input only pull documents from mongodb, say, once every 2 minutes or so. this would essentially 'consolidate' all of those updates to the same document into one. i looked through the code briefly for something that might support that, but didn't find such an option. is this something you've considered before? thank you for your time and thanks for your excellent work on this plugin -- works very nicely for me.
uri => 'mongodb://uu:[email protected]:7397,10.14.5.130:7397,10.14.5.131:7397/shinken_dev2?ssl=true
Error:
D, [2016-11-22T02:16:44.288000 #28422] DEBUG -- : MONGODB | SSL handshake failed. MongoDB may not be configured with SSL support.
Hi,
Is there any way to config a specific collection instead of the regex?
So if I have these collections: hits, hits-old, hits-1, I want to be able to define "hits" and get docs only from this collection without the other collections.
Thanks!
My collection configuration looks like this: collection => '.*'
Is there a way i can find out the collection of which the document is coming from?
By looking into the source code i could not find anything like that.
I am trying to build logstash-input-mongodb. Are there any build instructions?
I have been using Elastic's guide. Below are the steps I have followed so far
git clone https://github.com/phutchins/logstash-input-mongodb.git
cd logstash-input-mongodb
rm Gemfile.lock
bundle install
bundle exec rspec
gem build logstash-input-mongodb.gemspec
I am unable to successfully run the tests. When I run bundle exec rspec
I get the following error
Using Accessor#strict_set for specs
NameError: uninitialized constant LogStash::Inputs::Mongodb
const_missing at org/jruby/RubyModule.java:3212
<top> at /Users/nreese/projects/logstash-input-mongodb/spec/inputs/mongodb_spec.rb:10
load at org/jruby/RubyKernel.java:955
block in (root) at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1
each at org/jruby/RubyArray.java:1560
load_spec_files at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
load_spec_files at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
setup at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96
run at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84
run at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69
<top> at /Users/nreese/projects/jruby-9.0.5.0/lib/ruby/gems/shared/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37
load at org/jruby/RubyKernel.java:955
<top> at /Users/nreese/projects/jruby-9.0.5.0/bin/rspec:23
Looks like the test file can not find the class LogStash::Inputs::Mongodb. Am I missing a build step?
Thank you for the plugin, so far it works well and was quick to set up.
I have many fields that contain other fields and are nested to many levels. For example a field may contain the following:
"[{\"Field_1\"=>\"Field_1_Value\", \"Field_2\"=>\"Field_2_Value\", \"Field_3\"=>\"Field_3_Value\", \"Field_4\"=>\"Field_4_Value\", \"Field_5\"=>\"Field_5_Value\", \"Parent_Field\"=>[{\"Child_Field_1\"=>\"Child_Field_1_Value\", \"Child_Field_2\"=>nil, \"Child_Field_3\"=>\"Child_Field_3_Value\", \"Child_Field_4\"=>nil}]}]"
Is there a way to either have the plugin parse the contents as children of the parent field, or if there are existing filters that can parse the data properly?
I'm currently looking into using the ruby filter to iterate through the contents, but thought maybe I missed something and maybe there is something out there that can already do this.
Thanks
Command:
bin/logstash-plugin install --no-verify plugins/logstash-input-mongodb/logstash-input-mongodb-0.3.3.gem
Installing logstash-input-mongodb
WARNING: SSLSocket#session= is not supported
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "jdbc-sqlite3":
In snapshot (Gemfile.lock):
jdbc-sqlite3 (= 3.8.11.2)
In Gemfile:
jdbc-sqlite3 (>= 0) java
logstash-input-mongodb (= 0.3.3) java depends on
jdbc-sqlite3 (= 3.8.10.1) java
Running bundle update
will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Hi guys,I have a Invalid Objectid problem in version 0.4.1.
The log looks like:
[2017-06-02T23:34:24,310][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<BSON::ObjectId::Invalid: '0000abf12901b36a15fc8354e3328ddc' is an invalid ObjectId.>}
And my data in Mongodb looks like:
{
"_id" : "0000abf12901b36a15fc8354e3328ddc",
"data" : {
"uid" : "C6TJVe2kpTfjosl1hc",
.........
},
"connection_id" : "0000abf12901b36a15fc8354e3328ddc"
}
Logstash config like this:
input {
mongodb {
uri => 'mongodb://127.0.0.1:27017/test'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'mycollection_'
batch_size => 1
unpack_mongo_id => true
}
}
output {
stdout {codec => rubydebug}
elasticsearch {
hosts => "127.0.0.1"
index => "%{collection}"
template_name => "logstash_1"
codec => json
}
}
I want to know is there anyone else has this problem? and how to solve this?
I'm working on using this input gem to pull data from a MongoDB instance (that I don't own or manage) and sync it up with Elasticsearch. This input seems to do just that, but I'm having an issue actually fetching the data.
I get the following error on all objects:
MongoDB Input threw an exception, restarting {:exception=>#<BSON::ObjectId::Invalid: '[domain name or MD5]' is an invalid ObjectId.>, :level=>:warn, :file=>"logstash/inputs/mongodb.rb", :line=>"346", :method=>"run"}
I believe the error is because the IDs in this mongo collection are domains, and another collection uses md5 hashes as the ID. Is there any way to get around this or fix it?
I wanna say I think it's [https://github.com/phutchins/logstash-input-mongodb/blob/master/lib/logstash/inputs/mongodb.rb#L227](Line 227) that's erroring, but I could be wrong. I can post my logstash config if needed. Any help would be greatly appreciated!
Hi
I have several databases and collections in them.
How do you specify this in config?
Ie. want to read from myExampleDb.collectionABC
I have a problem when i configure the plugin. I've created a config file :
input {
mongodb {
uri => 'mongodb://mongourl/mng?ssl=false'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'pages_feed'
batch_size => 5000
}
}
filter {
}
output {
elasticsearch {
hosts => ["elasticurl:9200"]
index => "pagesfeed"
document_type => "feed"
document_id => "%{id}"
}
stdout { codec => rubydebug }
}
When logstash executes the file i get the error :
{
: timestamp=>"2016-09-06T09:10:32.424000+0000",
: message=>"An unexpected error occurred!",
: error=>#<Sequel: : DatabaseConnectionError: Java: : JavaSql: : SQLException: pathto'/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb'doesnotexist>,
: class=>"Sequel::DatabaseConnectionError",
: backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)",
"org.sqlite.core.CoreConnection.(org/sqlite/core/CoreConnection.java:74)",
"org.sqlite.jdbc3.JDBC3Connection.(org/sqlite/jdbc3/JDBC3Connection.java:24)",
"org.sqlite.jdbc4.JDBC4Connection.(org/sqlite/jdbc4/JDBC4Connection.java:23)",
"org.sqlite.SQLiteConnection.(org/sqlite/SQLiteConnection.java:45)",
"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)",
"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)",
"java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)",
"RUBY.connect(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:222)",
"RUBY.make_new(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool.rb:103)",
"RUBY.make_new(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:224)",
"RUBY.available(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:197)",
"RUBY._acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:133)",
"RUBY.acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:147)",
"RUBY.sync(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)",
"org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)",
"RUBY.sync(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)",
"RUBY.acquire(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:146)",
"RUBY.hold(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:104)",
"RUBY.synchronize(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/database/connecting.rb:256)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:247)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:952)",
"RUBY.fetch_rows(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:738)",
"RUBY.with_sql_each(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:835)",
"RUBY.with_sql_first(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:843)",
"RUBY.single_record!(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:661)",
"RUBY.single_record(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:649)",
"RUBY.first(/opt/logstash/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/dataset/actions.rb:200)",
"RUBY.",
"RUBY.get_placeholder(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:98)",
"RUBY.update_watched_collections(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:146)",
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)",
"RUBY.update_watched_collections(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:144)",
"RUBY.register(/opt/logstash/vendor/local_gems/9a1942b2/logstash-input-mongodb-0.3.3/lib/logstash/inputs/mongodb.rb:168)",
"RUBY.start_inputs(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:322)",
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)",
"RUBY.start_inputs(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:321)",
"RUBY.start_workers(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126)",
"RUBY.execute(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90)",
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
"RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95)",
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
"RUBY.initialize(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24)",
"java.lang.Thread.run(java/lang/Thread.java:745)"],
: level=>: warn
}
My problem is with logstash_sqlite.db. The database should exist?
Hello,
I'm trying to install this plugin on a logstash 5.0.0 installation, but Gemfile.lock
limits logstash-core (>= 1.4.0, < 2.0.0)
.
~ # /opt/logstash/current/bin/logstash-plugin install --no-verify logstash-output-mongodb
Installing logstash-output-mongodb
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core-plugin-api":
In snapshot (Gemfile.lock):
logstash-core-plugin-api (= 2.1.16)
In Gemfile:
logstash-devutils (~> 1.1) java depends on
logstash-core-plugin-api (~> 2.0) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-output-file (>= 0) java depends on
logstash-core-plugin-api (< 2.99, >= 2.0.0) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-output-stdout (>= 0) java depends on
logstash-core-plugin-api (< 2.99, >= 1.60.1) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-input-s3 (>= 0) java depends on
logstash-mixin-aws (>= 0) java depends on
logstash-core-plugin-api (<= 2.99, >= 1.60) java
logstash-output-mongodb (>= 0) java depends on
logstash-core-plugin-api (~> 1.0) java
logstash-core-plugin-api (>= 0) java
Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash-core":
In snapshot (Gemfile.lock):
logstash-core (= 5.0.0)
In Gemfile:
logstash-core-plugin-api (>= 0) java depends on
logstash-core (= 5.0.0) java
logstash-output-mongodb (>= 0) java depends on
logstash-core (< 2.0.0, >= 1.4.0) java
logstash-core (>= 0) java
Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-output-mongodb (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-output-mongodb (>= 0) java', in any of the sources.
Hey, I've got an issue with connection between LogStash and MongoDB.
MongoDB Version 3.3.9
Logstash Version 2.3.4-1
2016-07-21T14:03:54.930913520Z Settings: Default pipeline workers: 2
2016-07-21T14:03:55.346991872Z D, [2016-07-21T14:03:55.346000 #1] DEBUG -- : MONGODB | Adding IP:27018 to the cluster.
2016-07-21T14:03:55.397462102Z D, [2016-07-21T14:03:55.397000 #1] DEBUG -- : MONGODB | Adding 59dab37a6e17:27017 to the cluster.
2016-07-21T14:03:55.582269918Z D, [2016-07-21T14:03:55.581000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:03:55.586333543Z D, [2016-07-21T14:03:55.585000 #1] DEBUG -- : MONGODB | Server IP:27018 discovered as primary in replica set: 'rs0'. Changing topology to replica set.
2016-07-21T14:03:55.681482935Z D, [2016-07-21T14:03:55.680000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.690459992Z D, [2016-07-21T14:03:55.687000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.004s
2016-07-21T14:03:55.823540480Z D, [2016-07-21T14:03:55.821000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:55.838810649Z D, [2016-07-21T14:03:55.838000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.013s
2016-07-21T14:03:55.878286624Z D, [2016-07-21T14:03:55.877000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.881207073Z D, [2016-07-21T14:03:55.880000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:55.934179617Z D, [2016-07-21T14:03:55.933000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:55.936905464Z D, [2016-07-21T14:03:55.936000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:55.948433473Z D, [2016-07-21T14:03:55.947000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:55.951456885Z D, [2016-07-21T14:03:55.950000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:56.026110027Z D, [2016-07-21T14:03:56.024000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.030478215Z D, [2016-07-21T14:03:56.029000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.047168308Z D, [2016-07-21T14:03:56.046000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.051397686Z D, [2016-07-21T14:03:56.049000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.003s
2016-07-21T14:03:56.166734805Z D, [2016-07-21T14:03:56.166000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.168845318Z D, [2016-07-21T14:03:56.168000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.179821637Z D, [2016-07-21T14:03:56.179000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.182601771Z D, [2016-07-21T14:03:56.182000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.003s
2016-07-21T14:03:56.360019595Z D, [2016-07-21T14:03:56.357000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.363478803Z D, [2016-07-21T14:03:56.362000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.003s
2016-07-21T14:03:56.371330707Z D, [2016-07-21T14:03:56.371000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.373536476Z D, [2016-07-21T14:03:56.373000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:03:56.534728827Z Pipeline main started
2016-07-21T14:03:56.734602817Z D, [2016-07-21T14:03:56.734000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:56.737201519Z D, [2016-07-21T14:03:56.737000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.002s
2016-07-21T14:03:56.748718347Z D, [2016-07-21T14:03:56.748000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:56.751771200Z D, [2016-07-21T14:03:56.750000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.001s
2016-07-21T14:03:57.418523856Z D, [2016-07-21T14:03:57.417000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:57.426493735Z D, [2016-07-21T14:03:57.426000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.007s
2016-07-21T14:03:57.435871242Z D, [2016-07-21T14:03:57.435000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:57.445850020Z D, [2016-07-21T14:03:57.441000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.006s
2016-07-21T14:03:58.811576284Z D, [2016-07-21T14:03:58.811000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:03:58.816773559Z D, [2016-07-21T14:03:58.816000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.004s
2016-07-21T14:03:58.828617310Z D, [2016-07-21T14:03:58.828000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:03:58.838503594Z D, [2016-07-21T14:03:58.838000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.009999999999999998s
2016-07-21T14:04:01.451048043Z D, [2016-07-21T14:04:01.450000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | STARTED | {"find"=>"reports", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5790d01b6b469fb9e13f7014')}}, "limit"=>5000}
2016-07-21T14:04:01.457235672Z D, [2016-07-21T14:04:01.456000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.find | SUCCEEDED | 0.005s
2016-07-21T14:04:01.476911916Z D, [2016-07-21T14:04:01.476000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system\.|\$/}}}
2016-07-21T14:04:01.479310542Z D, [2016-07-21T14:04:01.478000 #1] DEBUG -- : MONGODB | IP:27018 | routine_reports.listCollections | SUCCEEDED | 0.002s
2016-07-21T14:04:05.592725429Z D, [2016-07-21T14:04:05.592000 #1] DEBUG -- : MONGODB | IP:27018 being removed from the cluster.
2016-07-21T14:04:10.848395264Z D, [2016-07-21T14:04:10.847000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:10.850026777Z D, [2016-07-21T14:04:10.849000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:36.680396897Z D, [2016-07-21T14:04:36.679000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:04:36.685525505Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:04:37.180376679Z D, [2016-07-21T14:04:37.179000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:07.126564411Z D, [2016-07-21T14:05:07.126000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:07.130668424Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:05:07.133406692Z D, [2016-07-21T14:05:07.132000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:36.141040717Z D, [2016-07-21T14:05:36.140000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:36.640674677Z D, [2016-07-21T14:05:36.640000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:37.140718973Z D, [2016-07-21T14:05:37.140000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
2016-07-21T14:05:37.144758436Z �[33mMongoDB Input threw an exception, restarting {:exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x206510 tag_sets=[] server_selection_timeout=30 local_threshold=0.015>>, :level=>:warn}�[0m
2016-07-21T14:05:37.640441332Z D, [2016-07-21T14:05:37.640000 #1] DEBUG -- : MONGODB | getaddrinfo: name or service not known
input {
mongodb {
uri => 'mongodb://IP:27018/routine_reports?replicaSet=rs0'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => '^reports$'
batch_size => 5000
}
}
output {
stdout { codec => rubydebug }
file {
path => "/opt/logstash-mongodb/mongo2es.log"
}
elasticsearch {
index => "routine_reports"
document_type => "test_report"
document_id => "%{mongo_id}"
hosts => ["IP:9201"]
}
}
I'm not sure what's missing (or it's a bug?). Any help? Thanks in advance!
Regards,
drptbl.
First of all thanks for the implementation of a MongoDB-input for Logstash. It seems to be very reasonable as there might be many use-cases out there demanding a connection of Elasticsearch to MongoDB.
The same demand is present in the use-cases I am currently working on.
I am quite new to the ELK-stack.
Thus, in case you believe the error is related to issues on my side, please let me know.
Problem:
Milliseconds of date-times are cut off when ingesting data from MongoDB to Elasticsearch.
Thus, aggregations over large amounts of data-points might lead to skewed results.
Description of the problem:
Example data in MongoDB:
{ "_id" : ObjectId("5721d60a1c4a9e0fc06dc0ba"), "State" : <State>, "thingId" : <another id>, "Name" : <Name>, "StartTime" : ISODate("2016-03-16T16:30:22.000Z"), "EndTime" : ISODate("2016-03-16T16:37:17.453Z"), "Duration" : 0.115403681777778 }
Logstash configuration:
`input {
mongodb {
uri =>
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'AggregatedData'
generateId => true
batch_size => 5000
}
}
filter {
date {
match => [ "StartTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "aggregated_data"
}
}`
Error Message:
The error message occurs for all data-points that are to be imported into ES.
Failed parsing date from field {:field=>"StartTime", :value=>"2016-03-21 03:03:15 UTC", :exception=>"Invalid format: \"2016-03-21 03:03:15 UTC\" is malformed at \" UTC\"", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}
In my eyes, the following part of the error-message indicates, that the information is cut off by the input-plugin before reaching the filter in the logstash config-file.
Also I have no clue, where the "UTC"-string is coming from.
:field=>"StartTime", :value=>"2016-03-21 03:03:15 UTC",
Do you have an idea about what is going wrong here?
Do you have a schedule for a possible fix, in case this is directly related to your plugin and you want to fix it.
Cheers and thanks in advance,
StetoGraph
I have field in mongodb that contains json object.
How can I save this document structure into elastic?
Example:
in mongodb
"location" : {
"lon" : -71.715313,
"lat" : 42.286992
}
parse_method => "flatten" returns
"location": "{"lon"=>-71.715313, "lat"=>42.286992}"
parse_method => "dig" returns
"location": "{"lon"=>-71.715313, "lat"=>42.286992}"
parse_method => "simple" returns
"location_lon": "-71.715313",
"location_lat": "42.286992",
How can I leave same json structure?
"location" : {
"lon" : -71.715313,
"lat" : 42.286992
}
I am using ELK stack on Windows, with logstash-2.2.0. When I try installing logstash-input-mongodb plugin using the following command in the terminal.
logstash\bin>plugin install <path to logstash-input-mongodb-0.3.3.gem>
I get the following error
Validating D:\........\logstash-input-mongodb-0.3.3.gem Installing logstash-input-mongodb Error Bundler::HTTPError, retrying 1/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 2/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 3/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 4/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 5/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 6/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 7/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 8/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 9/10 Could not fetch specs from https://rubygems.org/ Error Bundler::HTTPError, retrying 10/10 Could not fetch specs from https://rubygems.org/ Too many retries, aborting, caused by Bundler::HTTPError ERROR: Installation Aborted, message: Could not fetch specs from https://rubygems.org/
I am new to ELK. and haven't installed any plugin in the logstash before. Please help me install the mongo db input plugin.
when i enter "./plugin install logstash-input-mongodb" , error below raised:
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-mongodb (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.
I want to copy only data for last month to elasticsearch.
How can I specify some query to mongodb?
(EDIT) Error has changed after mucking with it - first issue was my fault.
LoadError: no such file to load -- file-dependencies/gem
require at org/jruby/RubyKernel.java:1072
require at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
(root) at /local/logstash/lib/pluginmanager/install.rb:4
require at org/jruby/RubyKernel.java:1072
(root) at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
require at /local/logstash-1.5.3/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
Hi @phutchins ,
I tried using logstash-input-mongodb-0.2.0 for syncing mongodb data with ES.
However when we remove documents in mongoldb corresponding documents are not getting removed from ES.
Is this known issue or expected behaviour.
Please let us know how to resolve this issue if already fixed.
regards,
chakrapani
Hello,
I've been trying out this plugin and it's great!
I'd like to get data from a system.profile collection in order to create an operation analysis dashboard in Kibana.
When I try to do that, logstash gives me this message:
D, [2015-08-16T12:17:14.914000 #30642] DEBUG -- : MONGODB | COMMAND | namespace=collection1.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system.|$/}}} flags=[] limit=-1 skip=0 project=nil | runtime: 32.0001ms
So system collections are filtered out. The code responsible for that is in the ruby mongo driver:
https://github.com/mongodb/mongo-ruby-driver/blob/15f5707794e852a8124d8d9d89019223e0183ebe/lib/mongo/operation/commands/list_collections.rb#L49
Is there any way to get around that in your plugin?
Thanks.
Hi,
I want used logstash with the plugin, but I understand that is not avalaible for logstash>2.0.0. Are you planning to upgrade?
plugin install logstash-input-mongodb
---> Running in 062b87f7ae18
Validating logstash-input-mongodb
Installing logstash-input-mongodb
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-mongodb (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-mongodb (>= 0) java', in any of the sources.
The command '/bin/sh -c ${LOGSTASH_SRC_DIR}/bin/plugin install logstash-input-mongodb' returned a non-zero code: 1
@phutchins My mongo contains complex nested JSON documents that I am trying to index into Elastic search (with a predefined mapping). By using the plugin in either 'flatten' or 'dig' mode, the complex JSON structure is lost. For example, array of nested json documents are represented as a string so Elastic does apply the mapping against it.
Any thoughts on how to retain the structure from Mongo without any flattening ?
Thanks!
Am trying to configure this plugin to connect to my mongo replica set and am getting the following error: "not authorized for query on uilogs.system.namespace (13)"
From the mongo logs, reports something similar: "assertion 13 not authorized for query on uilogs.system.namespaces ns:uilogs.system.namespaces query: { name:
My conf file looks like this:
...
mongodb {
uri => 'mongodb://uilogsAdmin:@host-abc123:27017/uilogs'
placeholder_db_dir => '/home/somewhere/logstash-1.5.3/logstash-mongodb'
placeholder_db_name => 'logstatsh_sqlite.db'
collection => 'daily.apps'
}
am using logstash 1.5.3, the most recent version of this plugin 0.2.0, and mongodb 2.6.4
the login id 'uilogsAdmin' currently has the following roles: dbOwner, clusterAdmin, dbAdminAnyDatabase.
If I log into the primary with the above id and run uilogs.system.namespaces.find() I do not get the auth error.
Please advise what could be wrong here.
Hi Team ,
I want to pull the data from Collection with in a specific interval time . Once the data is available it should pull the data automatically from where it left off initially .
Can we do this with this plugin
I think it's self explanatory. The "simple" parse_method converts them to strings, which is what the flatten method should do as well.
Hi,
I am trying out logstash and I was very happy to find this plugin.
I installed the plugin but when I use to pull the database i get the following error:
D, [2016-01-07T03:59:54.337000 #9574] DEBUG -- : MONGODB | Adding localhost:27017 to the cluster.
D, [2016-01-07T03:59:54.501000 #9574] DEBUG -- : MONGODB | localhost:27017 | umbrella.find | STARTED | {"find"=>"system.namespaces", "filter"=>{:name=>{"$not"=>/system.|$/}}}
D, [2016-01-07T03:59:54.509000 #9574] DEBUG -- : MONGODB | localhost:27017 | umbrella.find | FAILED | comparison of Fixnum with nil failed | 0.006s
The error reported is:
comparison of Fixnum with nil failed
This is my config:
input {
mongodb {
uri => 'mongodb://localhost:27017/umbrella'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'users'
}
}
I have the umrella db and users collection.
I am using logstash version 2.1.1
Any idea what is causing this issue?
Thank you in advance
I'm trying to run the plugin on logstash 5.1.2, and getting the error:
Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.
19:20:09.456 [LogStash::Runner] INFO logstash.inputs.mongodb - Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
D, [2017-02-09T19:20:09.766000 #7068] DEBUG -- : MONGODB | Adding localhost:27017 to the cluster. | runtime: 0.0000ms
D, [2017-02-09T19:20:12.839000 #7068] DEBUG -- : MONGODB | COMMAND | namespace=admin.$cmd selector={:ismaster=>1} flags=[] limit=-1 skip=0 project=nil | runtime: 3066.0000ms
19:20:12.845 [[main]-pipeline-manager] INFO logstash.inputs.mongodb - Registering MongoDB input
D, [2017-02-09T19:20:12.936000 #7068] DEBUG -- : MONGODB | COMMAND | namespace=test.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system.|$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 22.0001ms
19:20:13.140 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
19:20:13.151 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
D, [2017-02-09T19:20:13.356000 #7068] DEBUG -- : MONGODB | QUERY | namespace=test.auditcommunications selector={:_id=>{:$gt=>BSON::ObjectId('0761b057fccc5c277c23dfbe')}} flags=[:slave_ok] limit=50 skip=0 project=nil | runtime: 224.9999ms
19:20:13.420 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
19:20:13.647 [[main]<mongodb] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::MongoDB uri=>"mongodb://localhost:27017/test", placeholder_db_dir=>"/elk/logstash-mongodb/", placeholder_db_name=>"logstash_sqlite.db", collection=>"auditcommunications", batch_size=>50, id=>"797b2fd78162084f6d57991a196d656a6c95f190-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_cd856514-8fcc-43ad-becd-9497e98577b9", enable_metric=>true, charset=>"UTF-8">, since_table=>"logstash_since", parse_method=>"flatten", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>"Default message...", interval=>1>
Error: Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.
D, [2017-02-09T19:20:14.782000 #7068] DEBUG -- : MONGODB | QUERY | namespace=test.auditcommunications selector={:_id=>{:$gt=>BSON::ObjectId('0761b057fccc5c277c23dfbe')}} flags=[:slave_ok] limit=50 skip=0 project=nil | runtime: 128.9999ms
My configuration is pretty much copy pasted from the instructions:
input {
mongodb {
uri => 'mongodb://localhost:27017/test'
placeholder_db_dir => '/elk/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'auditcommunications'
batch_size => 50
}
}
filter {
date {
match => [ "timestamp", "ISO8601" ]
}
}
output {
stdout { codec => rubydebug }
}
Thanks in advance
I get this exception in loop (I'm running v0.3.3):
{:timestamp=>"2016-04-21T01:15:09.431000+0000", :message=>"MongoDB Input threw an exception, restarting", :exception=>#<BSON::ObjectId::Invalid: 'Infinity' is an invalid ObjectId.>, :level=>:warn}
and no document are indexed after that.
Tried to search for any invalid ID but can't find one. The IDs are given by Mongo, I'm not setting any custom IDs. Also run query in Mongo: ({"_id":{$not:{$type:7}}}) which return empty.
Anyone else get this exception??
Please help.
i am a new user to elk i am getting the error please suggest me
Hi
I sucessfully installed logstash-input-mongodb plugin in my logstash instance but probably there's something missing somewhere in my configuration. Just for test, I added a single entry in my mongodb collection and started up logstash with the following configuration:
input{
mongodb {
uri => 'mongodb://127.0.0.1:27017/syslog?ssl=no'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'messages'
batch_size => 5000
}}
output {
file{
path => "/var/log/mongoteste"
}}
When checking "/var/log/mongoteste" I realize that, in fact, it's writing in in the file, but at each 5 seconds (checked in timestamp) , the entry is replicated and keeps in this loop everytime, just like this:
{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:31.632Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}
{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:36.675Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}
{"host":"localhost.localdomain","@Version":"1","@timestamp":"2015-09-05T00:58:41.714Z","logdate":"2015-09-04T17:31:13+00:00","log_entry":"{"_id"=>BSON::ObjectId('55e9d5611adb62fa4d000001'), "TAGS"=>".source.s_tcp", "SOURCEIP"=>"172.16.102.148", "SEQNUM"=>"1", "PROGRAM"=>"ola", "PRIORITY"=>"notice", "LEGACY_MSGHDR"=>"ola", "HOST_FROM"=>"172.16.102.148", "HOST"=>"172.16.102.148", "FACILITY"=>"user", "DATE"=>"Sep 4 18:31:13"}","mongo_id":"55e9d5611adb62fa4d000001","TAGS":".source.s_tcp","SOURCEIP":"172.16.102.148","SEQNUM":1,"PROGRAM":"ola","PRIORITY":"notice","LEGACY_MSGHDR":"ola","HOST_FROM":"172.16.102.148","HOST":"172.16.102.148","FACILITY":"user","DATE":"Sep 4 18:31:13"}
If I add a new entry in my collection, the plugin will consume new data but just like before, will write it in file constantly each 5 seconds.
Has anyone run into the same problem? My conf is logstash 1.5.4 + logstash-input-mongodb-0.2.0 .
Best Regards
Hi,
Is anyone tested this input with the new Logstash 2.0 version?
I want to upgrade soon...
Thanks.
Currently this plugin sleeps and continues to look for new data once collection is finished. This behavior should be configurable.
Please add flag in input config to decide the behavior.
We can also provide pull request if needed.
How to pass SSL mongo connector URI and truststore
Hey, I would like to input my mongodb data into elasticsearch via logstash. So I've searched for a logstash input plugin and this repo popped up.
@phutchins What is your vision for this plugin?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.