Yarnbeat
Yarnbeat is an elastic Beat which polls the YARN API and forwards YARN application and resource metric information to the ELK stack for monitoring and alerting.
Getting Started
Prerequisites
Yarnbeat was developed against Go 1.12. Other versions of Go may work, but I make no promises.
Configuration
Example yarnbeat configuration:
yarnbeat:
# Defines how often job metrics are polled in seconds
period: 60
# List of resource manager URLs to try when polling the YARN API
# For best performance, list our primary RM first, then failovers
resource_manager_urls:
- "http://localhost:8088"
# Set to true to enable MapReduce History Server integration
enable_mr: true
history_server_url: "http://localhost:9888"
# Configure which MapReduce counters should be included in the
# output, and what their field names should be
mr_counters:
- group_name: org.apache.hadoop.mapreduce.TaskCounter
counters:
# The format here is COUNTER_NAME: field_name
# counter fields will appear as "mapreduce.job.counters.field_name"
MAP_INPUT_RECORDS: map_input_records
MAP_OUTPUT_RECORDS: map_output_records
- group_name: My Custom Group
counters:
# Note that for MapReduce counter names containing the `.` character, the beats library will attempt
# to parse the key value into a subpath. To work around this, replace the `.` with `$$`. In the
# counter below, the MapReduce counter name would be `counter.with.periods`
counter$$with$$periods: counter_with_periodsNote that for MapReduce counter names containing the . character, the beats library will attempt to parse the key value into a subpath. To work around this, replace the . with $$.
For information concerning valid Beats outputs, see the Filebeat documentation
Build
To build Yarnbeat from source, clone the repository and execute the following command:
make yarnbeatRun
To execute Yarnbeat after building from source, run
./bin/yarnbeat -c /path/to/yarnbeat.ymlTo execute Yarnbeat directly with Go, run
go run github.com/IBM/yarnbeat -c /path/to/yarnbeat.ymlFor full command options run
./bin/yarnbeat --helpTest
To test the Yarnbeat module run
make testBuild and run the Docker image
To run Yarnbeat in a Docker image, mount the directory containing yarnbeat.yml to /etc/yarnbeat
make docker
docker run -v /path/containing/yarnbeatyml:/etc/yarnbeat yarnbeatKibana Dashboards
To load the prebuilt objects (index pattern, visualizations, dashboards) into Kibana, execute the following commands. Note that your config file must contain valid connection settings in setup.kibana.
make yarnbeat
make dashboards
./bin/yarnbeat setup --dashboards -c path/to/yarnbeat.yml
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent