Hi,

The ELK_docker_setup example I contributed previously targets vesion 1.x of both Logstash and Elasticsearch.

With 2.x released for both LS and ES now, I can contribute another example that targets these releases.

The ELKv2_docker_setup is very similar to v1.x but does have slight configuration changes in ES.

What would also be different in this new example is demonstrated use of tagged docker images, which makes it easy to set the exact versions of the 3 pieces (ELK) you wish to deploy.

If I contribute this new example I suggest perhaps keeping them both and renaming to something like:

ELKv1_docker_setup
ELKv2_docker_setup

Let me know if you'd like me to proceed.

Thanks.

Hello, I'm a newbie and got kibana and elasticsearch running.
I managed to ingest the data & do get a count response of approximately 473039

But I cannot complete the next step to load the dashboard. Folllowing this in Kibana:
Click the Settings tab >> Objects tab >> Import, and select restaurants_kibana.json

I do get:
Saved Objects: Cannot read property 'listeners' of undefined

Can someone help walk me through? What am I missing?

https://github.com/salyh/elk2-out-of-the-box-demo

This is in response to Issue #1 -- we should add a .gitattributes file for this repo to tell git clients to preserve the original line ending.

https://help.github.com/articles/dealing-with-line-endings

#54

#54

See elastic/Example-Watches#27 for original issue
Will add:

• cluster state - alert if yellow or red.
• split brain detection
• master changing frequently
• free disk space
• High file descriptor usage
• Minimum masters incorrect
• Queue Rejections (Index, Search or Bulk)
• Avg. Query Response Time Exceeds Bound

Dear, I am totally new to ELK and experimeting with your example.

Although, I´m facing this issue when i run nyc_collision_logstash.conf. I noticed that many fields are "nil" in the logs.

Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}
Ruby exception occurred: comparison of NilClass with String failed {:level=>:error}

I tried downloading both v1 and v2 demo's, but when I install the NYC traffic demo i get the error:

Error: Could not parse application options: invalid option: --manifestdir

at the point where puppet is doing the provisioning. Looking around it seems this was deprecated in the 4.x puppet versions. hashicorp/vagrant#3740

Not really clear where to take it from here though. Any help to get the demo working would be appreciated.

Error: Could not locate that index-pattern-field (id: geoip.location)
at FieldAggParamFactory.FieldAggParam.deserialize (http://localhost:5601/bundles/kibana.bundle.js?v=9889:99917:16)
at http://localhost:5601/bundles/kibana.bundle.js?v=9889:102896:29
at Array.forEach (native)
at AggConfigFactory.AggConfig.fillDefaults (http://localhost:5601/bundles/kibana.bundle.js?v=9889:102874:28)
at new AggConfig (http://localhost:5601/bundles/kibana.bundle.js?v=9889:102788:13)
at http://localhost:5601/bundles/kibana.bundle.js?v=9889:103492:19
at Array.map (native)
at new AggConfigs (http://localhost:5601/bundles/kibana.bundle.js?v=9889:103490:35)
at VisFactory.Vis.setState (http://localhost:5601/bundles/kibana.bundle.js?v=9889:99388:20)
at SavedVis._updateVis (http://localhost:5601/bundles/kibana.bundle.js?v=9889:104372:17)

Hi,

I've tried a couple different ways to output data to an S3 bucket on Amazon. I followed the documentation for output plug-ins to s3 but the function is not work. Suggestions please?

filter { }

output {
stdout {
codec => dots
}
elasticsearch {
hosts => "http://localhost:9200"
index => "twitter"
document_type => "tweets"
template => "D:/Downloads/datasets/iot/twitter_template.json"
template_name => "twitter_example"
template_overwrite => true
}
file {
path => "D:/Downloads/datasets/twitter_iot/twitter.csv"
}
s3 {
credentials => ["accessKeyID", "SecretKeyID"]

access_key_id => "accessKeyID"

secret_access_key => "SecretKeyID"

 region => "us-west-2"
 bucket => "s3://mybucket/path"

}
}

editcap, tcpdump and wireshark don't seem to like dns-tunnel-iodine.pcap is it corrupt, or some windows/mac binary file issue?

@asawariS can we move away from this "ElasticStack" prefix and do a restructuring. I'd rather do this now rather whilst the repo is still moderate in size.

Please update to warn that there is a risk that invalid JSON strings are generated by nginx.
See elastic/logstash#4339 (comment)

Hi,

I'm using the ELK stack with Docker and I think adding some Install and setup steps would be helpful for those also interested to use Docker.

I've actually gone ahead and done this already and posted it to my blog.

If you like, we could use that content (modified anywhere you deem appropriate) and add it here.

I'd be happy to put the content together and create a pull request - or not, it's up to you really.

The content I've used in my blog is just a single markdown file and 3 images.

Here's a link to the blog post HTML you can review:

http://rudijs.github.io/2015-10/docker-elk-quickstart/

Here's a link to the markdown source

https://raw.githubusercontent.com/rudijs/rudijs.github.io/master/_posts/2015-10-05-docker-elk-quickstart.markdown

There's some jekyll tags that'd be stripped out, perhaps just the content from Overview to Summary could be used here.

Anyways, let me know if you're interested and I'll help to prepare it.

Cheers.

Hi,

The Apache example works like a charm! I also want to know if you can provide an example for Tomcat logs (localhost_access_log). By analysing the log, I can get the download statistics based on IP or country over a period os time.

Much appreciated.

Cheers,
Kayne.

Requires porting of tag cloud to v5. Nice to have.

Any chance that Elasticsearch could post the code from the OSCON drone demo? Would love to see the source even if not fully functional. Cheers, Carol

First of all thanks for great examples as kick-start for newbies like me :-)

Few updates in examples :

apache_template.json => [ "path": "full",] option is deprecated logstash-plugins/logstash-output-elasticsearch#94

apache_logstash.conf =>

1. protocol => default elasticsearch output protocol set to http [https://www.elastic.co/guide/en/logstash/2.0/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch"
2. host => host configuration parameter renamed to hosts https://github.com/elastic/logstash/blob/master/CHANGELOG.md
3. cluster => cluster configuration parameter disappeared completely without any trace

Hi,
I'm a newbie, I'm running the nyc collision data example, and have elasic and kibana up and running, data and examples files are downloaded.

I'm getting an error with step 1. ingest data into elasticsearch using logstash:
cat nyc_collision_data.csv | d:/downloads/logstash-2.3.0/bin/logstash -f nyc_collision_logstash.conf

I'm using windows command prompt, and the message I receive is: 'cat' is not recognized as an internal or external command, operable program or batch file.

how do i get past this error?
thx,
rains

#54

es version 1.4.4

When I execute

curl -XPOST "localhost: 9200 / _snapshot / usfec / 1 / _restore

es the process will kill

[2015-03-04 11:02:02,500][INFO ][cluster.metadata         ] [node@elk-node01-6000] closing indices [[usfec_indiv_contrib]]
[2015-03-04 11:02:23,143][WARN ][cluster.metadata         ] [node@elk-node01-6000] [usfec_indiv_contrib] re-syncing mappings with cluster state for types [[indiv_contrib]]
Killed

Currently all demos use the installation instructions. A recent pull #104 uses a docker setup to ease deployment. Given the prevalence of docker, and our official images, maybe we should provide a set of docker instructions in addition to the current approach

@asawariS @tylerhannan

Hello!

Did this example function because it doesn't work for me I couldn't use it to detect Tunnel
Could you please help me to fix the problem
thanks!

Hi I'm a EKL newbie and am trying to work through an issue.
I'm trying to load json file with below format. I've created a mapping for it and followed the instructions at https://www.elastic.co/guide/en/kibana/current/getting-started.html#tutorial-load-dataset. I get an error trying to load the JSON data into Elastic. Note: I was able to load the account data and shakespeare JSON data into Elastic.

sample data
{
"ID": "alpha1",
"timestamp": "2016-01-29T22:00:17.334Z",
"deviceID": "1",
"deviceType": "watch1",
"codec": "Z1",
"notes": "blue watch"
},
{
"ID": "alpha2",
"timestamp": "2016-01-30T22:00:17.334Z",
"deviceID": "2",
"deviceType": "watch2",
"codec": "Z2",
"notes": "yellow watch"
},
{
"ID": "alpha3",
"timestamp": "2016-01-31T22:00:17.334Z",
"deviceID": "3",
"deviceType": "watch3",
"codec": "Z3",
"notes": "orage watch"
},
{
"ID": "alpha4",
"timestamp": "2016-02-02T22:00:17.334Z",
"deviceID": "4",
"deviceType": "watch4",
"codec": "Z4",
"notes": "red watch"
}

mapping
curl -XPUT "localhost:9200/devices" -d"{""mappings"":{""default"":{""properties"":{""_id"":{""type"":""string"",""index"":""not_analyzed""},""created"":{""type"":""string"",""index"":""not_analyzed""},""serialNumber"":

{""type"":""integer""},""mtype"":{""type"":""string"",""index"":""not_analyzed""},""code"":{""type"":""string"",""index"":""not_analyzed""},""desc"":{""type"":""string"",""index"":""not_analyzed""}}}}}"

*data load command *
curl -XPOST "localhost:9200/devices/_bulk?pretty" --data-binary ""@d:\downloads\datasets\elastic\deviceList.json""

error

The Sample "CPU - Change in IOWait" is not suitable for the metricbeat, which is the replacement for topbeat.
Error informaiton:
SearchPhaseExecutionException[all shards failed]; nested: RemoteTransportException[[elk5-es-poc-node-3][10.193.105.128:9301][indices:data/read/search[phase/query]]]; nested: IllegalArgumentException[Fielddata is disabled on text fields by default. Set fielddata=true on [beat.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.];

hi,

Using Twitter Example, i've tried to import the twitter_kibana.json , but it failed :(
kibana says: Saved Objects: The file could not be processed.

using kibana Version: 4.1.1

any clues maybe ?

thanks for your help

Hi Guys,

I'm trying out the vagrant file on a windows 7 64 bit install and getting the error output below:

I'm using Vagrant 1.6.3 and Virtualbox 4.3.14

Stderr from the command:

stdin: is not a tty
Warning: Setting manifestdir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1095:in block in issue_deprecations') Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1095:inblock in issue_deprecations')
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Error: /var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_nginx_sites-available_kibana_vhost.conf/fragments.concat.out -d /var/lib/puppet/concat
/etc_nginx_sites-available_kibana_vhost.conf returned 1 instead of one of [0]
Error: /Stage[main]/Elasticsearch-demo_vagrant_env/Nginx::Resource::Vhost[kibana_vhost]/Concat[/etc/nginx/sites-available/kibana_vhost.conf]/Exec[concat/etc/nginx/sites-
available/kibana_vhost.conf]/returns: change from notrun to 0 failed: /var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_nginx_sites-available_k
ibana_vhost.conf/fragments.concat.out -d /var/lib/puppet/concat/etc_nginx_sites-available_kibana_vhost.conf returned 1 instead of one of [0]
Error: /Stage[main]/Elasticsearch-demo_vagrant_env/Nginx::Resource::Vhost[kibana_vhost]/Concat[/etc/nginx/sites-available/kibana_vhost.conf]/Exec[concat/etc/nginx/sites-
available/kibana_vhost.conf]: Failed to call refresh: /var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_nginx_sites-available_kibana_vhost.conf
/fragments.concat.out -d /var/lib/puppet/concat/etc_nginx_sites-available_kibana_vhost.conf returned 1 instead of one of [0]
Error: /Stage[main]/Elasticsearch-demo_vagrant_env/Nginx::Resource::Vhost[kibana_vhost]/Concat[/etc/nginx/sites-available/kibana_vhost.conf]/Exec[concat/etc/nginx/sites-
available/kibana_vhost.conf]: /var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_nginx_sites-available_kibana_vhost.conf/fragments.concat.out -d
/var/lib/puppet/concat/_etc_nginx_sites-available_kibana_vhost.conf returned 1 instead of one of [0]
Warning: /Stage[main]/Elasticsearch-demo_vagrant_env/Nginx::Resource::Vhost[kibana_vhost]/Concat[/etc/nginx/sites-available/kibana_vhost.conf]/File[/etc/nginx/sites-avail
able/kibana_vhost.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_vagrant_env/Nginx::Resource::Vhost[kibana_vhost]/File[kibana_vhost.conf symlink]: Skipping because of failed dependencies
Warning: /Stage[main]/Nginx::Service/Service[nginx]: Skipping because of failed dependencies
Warning: /Stage[main]/Nginx/Anchor[nginx::end]: Skipping because of failed dependencies
Error: Could not start Service[es-01]: Execution of '/etc/init.d/elasticsearch-es-01 start' returned 1:
Wrapped exception:
Execution of '/etc/init.d/elasticsearch-es-01 start' returned 1:
Error: /Stage[main]/Elasticsearch-demo_vagrant_env/Elasticsearch::Instance[es-01]/Elasticsearch::Service[es-01]/Elasticsearch::Service::Init[es-01]/Service[es-01]/ensure:
change from stopped to running failed: Could not start Service[es-01]: Execution of '/etc/init.d/elasticsearch-es-01 start' returned 1:
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[sanitize_from_old_remainings]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[get_data_from_download.elasticsearch.org]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[create_es_dir]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[unzip_snapshot]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[register_snapshot]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch-demo_nyc/Exec[restore_snapshot]: Skipping because of failed dependencies

the DNS Tunnel pcap file does not open in wireshark ! Can you please provide a correct one ?

Guys,
Please let me know the user name and password for the demo of virtual Box.

Hi,

Thanks for the effort in sharing the ELK examples. I have just tried NYC accidents Kibana visualization. I was able to import the data to ES and when tried to visualize the dashboard, getting error. Attached the screenshot for reference.

Thanks,
Selva

Hi,

Thank you very much for the example with DonorsChoose data, it's awesome!

I wanted to run the example on my local pc, unfortunately I have problem with downloaded snapshot - I can't unpack it.
I tried with winrar and 7-zip, but both failed. Error is always the same:

What is quite interesting to me, is fact that when I'm downloading file, the size is 7.5GB:

File downloads fine, but its size is different than expected:

Only 4.95GB.

OS: windows 10
File System: NTFS
Free space before unpacking: 10 GB

Any help much appreciated.
Thank you.

The data file for this is 7.5GB, which is pretty big for an example. Can we parse this down?

Otherwise we need a pretty prominent warning of the size for users.

Hi,
did not hear back on the nyc_collision example so tried the twitter_elk_example. logstash pipeline was started by 4 workers but then received this error in teh screen shot. any idea on how to resolve this error and get this example working?

For https://github.com/elastic/examples/tree/master/ELK_nginx-json, the highlighted text below in README should be nginx_json_logstash.conf instead of apache_logstash.conf

Note: Included nginx_json_logstash.conf configuration file assumes that you are running Elasticsearch on the same host as Logstash and have not changed the defaults. Modify the host and cluster settings in the output { elasticsearch { ... } } section of apache_logstash.conf, if needed.

Issue to track moving all examples to 5.0.