Write the code, Change the world.
🇨🇳 Xi'an China ・ 🕹Java developer
搭建ELK日志分析平台。
Write the code, Change the world.
🇨🇳 Xi'an China ・ 🕹Java developer
https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations
http://yaml-online-parser.appspot.com/
https://www.digitalocean.com/community/tutorials
https://aggarwalarpit.wordpress.com/2015/12/03/configuring-elk-stack-to-analyse-apache-tomcat-logs/
https://www.systemcodegeeks.com/web-servers/apache/configuring-elk-stack-analyse-apache-tomcat-logs/
https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html
https://github.com/logstash/log4j-jsonevent-layout
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
https://blog.lanyonm.org/articles/2015/12/29/log-aggregation-log4j-spring-logstash.html
http://www.tianmaying.com/tutorial/elastic-logstash-kibana
http://grokdebug.herokuapp.com/
http://grokconstructor.appspot.com/do/match
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
I'm using elasticsearch 1.5
and it is working perfectly the most part of the time, but everyday at the same time it becomes crazy, CPU % goes to ~70% when the average is around 3-5% there are SUPER servers with 32GB reserved for lucene, swap it is lock and clearing the cache doesn't solve the problem (it doesn't take down the heap mem)
Settings:
3 servers (nodes) 32 cores and 128GB RAM each
2 buckets (indices) one with ~18 million documents (this one doesn't receive updates pretty often just indexing new docs) the other one have around 7-8 million documents but we are constantly bombarding it with updates search delete and indexing as well
The best distribution for our structure, was to have only 1 shard per node with not replicas, we can afford to have a % of the data off for few seconds, that will be back as soon as the server get online again, and this process is fast enough since it doesn't need to relocate anything. previously we used to have 3 shards with 1 replica, but the issue mentioned above occurs as well, so is easy to figure it out that the problem is not related with the distribution.
Things that I already tried,
Merging, i try to use the Optimize API trying to give less load to the schedule merge, but actually the merging process takes a lot of R/W of the disk but it doesn't affect substantially the mem or the CPU load.
Flushing, I tried to flush with long and shot intervals, and the results were the same nothing
changed, since flushing affects directly the merging process and as mentioned above, merging process doesn't takes that much of the CPU or mem usage.
managing the cache, clearing it manually but it doesn't seems to take the cpu load to normal state not even for a moment.
Here is the most of the elasticsearch.yml configs
bootstrap.mlockall: true
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 200
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 3000
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 1000
indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb
indices.fielddata.cache.size: 30%
#indices.fielddata.cache.expire: 6h #will be depreciated & Dev recomend not to use it
indices.cache.filter.size: 30%
#indices.cache.filter.expire: 6h #will be depreciated & Dev recomend not to use it
index.refresh_interval: 30s
#index.translog.flush_threshold_ops: 50000
#index.translog.flush_threshold_size: 1024mb
index.translog.flush_threshold_period: 5m
index.merge.scheduler.max_thread_count: 1
here is the stats when the server is in a normal state:
node_stats_normal.txt
Node stats during the problem.
node_stats.txt
I will appreciate any help or discussion that can point me in the right direction to get rid of this behavior
thanks in advance..
Regards,
Daniel
Originally posted by @ACV2 in elastic/elasticsearch#4288 (comment)
docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -e ES_MIN_MEM=128m -e ES_MAX_MEM=1024m -it --name elk sebp/elk:es220_l222_k441
service logstash stop
/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'
service logstash start
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
/etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
yum install elasticsearch
chkconfig --add elasticsearch
centos 7
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
./bin/plugin install license
./bin/plugin install shield
./bin/elasticsearch
创建用户es_admin/es_admin 角色为admin
./bin/shield/esusers useradd es_admin -p es_admin -r admin
esusers useradd <username> [-p <password>] [-r <roles>]
-h,--help Shows this message
-p,--password <password> The user password
-r,--roles <roles> Comma-separated list of the roles of the user
curl -u es_admin -XGET 'http://115.28.163.63:9200/'
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
/etc/yum.repos.d/logstash.repo
[logstash-2.3]
name=Logstash repository for 2.3.x packages
baseurl=https://packages.elastic.co/logstash/2.3/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
/etc/yum.repos.d/kibana.repo
[kibana-4.5]
name=Kibana repository for 4.5.x packages
baseurl=http://packages.elastic.co/kibana/4.5/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
yum install kibana
chkconfig --add kibana
https://download.elastic.co/beats/winlogbeat/winlogbeat-1.2.3-windows.zip
PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1
winlogbeat.yml
winlogbeat:
registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml
event_logs:
- name: Application
- name: Security
- name: System
output:
elasticsearch:
hosts:
- 192.168.0.228:9200
logging:
to_files: true
files:
path: C:/ProgramData/winlogbeat/Logs
level: info
Test
PS C:\Program Files\Winlogbeat> .\winlogbeat.exe -c .\winlogbeat.yml -configtest -e
curl -XPUT 'http://192.168.0.228:9200/_template/winlogbeat' -d@/etc/winlogbeat/winlogbeat.template.json
curl -XDELETE 'http://192.168.0.228:9200/winlogbeat-*'
sudo yum install libpcap
curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-x86_64.rpm
sudo rpm -vi packetbeat-1.2.3-x86_64.rpm
/etc/packetbeat/packetbeat.yml
curl -XPUT 'http://115.28.163.63:9200/_template/packetbeat' -d@/etc/packetbeat/packetbeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/packetbeat-*'
sudo /etc/init.d/packetbeat start
curl -XGET 'http://115.28.163.63:9200/packetbeat-*/_search?pretty'
curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm
sudo rpm -vi filebeat-1.2.3-x86_64.rpm
/etc/filebeat/filebeat.yml
curl -XPUT 'http://115.28.163.63:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/filebeat-*'
sudo /etc/init.d/filebeat start
curl -L -O https://download.elastic.co/beats/topbeat/topbeat-1.2.3-x86_64.rpm
sudo rpm -vi topbeat-1.2.3-x86_64.rpm
/etc/topbeat/topbeat.yml
curl -XPUT 'http://115.28.163.63:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json
curl -XDELETE 'http://115.28.163.63:9200/topbeat-*'
sudo /etc/init.d/topbeat start
curl -XGET 'http://115.28.163.63:9200/topbeat-*/_search?pretty'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.