中文文档

Compass is a big data task diagnosis platform, which aims to improve the efficiency of user troubleshooting and reduce the cost of abnormal tasks for users.

The key features:

• Non-invasive, instant diagnosis, you can experience the diagnostic effect without modifying the existing scheduling platform.

• Supports multiple scheduling platforms(DolphinScheduler, Airflow, or self-developed etc.)

• Supports Spark 2.x or 3.x, Hadoop 2.x or 3.x troubleshooting.

• Supports workflow layer exception diagnosis, identifies various failures and baseline time-consuming abnormal problems.

• Supports Spark engine layer exception diagnosis, including 14 types of exceptions such as data skew, large table scanning, and memory waste.

• Supports various log matching rule writing and abnormal threshold adjustment, and can be optimized according to actual scenarios.

Compass has supported the concept of diagnostic types:

git clone https://github.com/cubefs/compass.git
cd compass
mvn package -DskipTests

cd dist/compass

vi bin/compass_env.sh
# Scheduler MySQL
export SCHEDULER_MYSQL_ADDRESS="ip:port"
export SCHEDULER_MYSQL_DB="scheduler"
export SCHEDULER_DATASOURCE_USERNAME="user"
export SCHEDULER_DATASOURCE_PASSWORD="pwd"
# Compass MySQL
export COMPASS_MYSQL_ADDRESS="ip:port"
export COMPASS_MYSQL_DB="compass"
export SPRING_DATASOURCE_USERNAME="user"
export SPRING_DATASOURCE_PASSWORD="pwd"
# Kafka
export SPRING_KAFKA_BOOTSTRAPSERVERS="ip1:port,ip2:port"
# Redis
export SPRING_REDIS_CLUSTER_NODES="ip1:port,ip2:port"
# Zookeeper
export SPRING_ZOOKEEPER_NODES="ip1:port,ip2:port"
# Elasticsearch
export SPRING_ELASTICSEARCH_NODES="ip1:port,ip2:port"

vi conf/application-hadoop.yml
hadoop:
  namenodes:
    - nameservices: logs-hdfs # the value of dfs.nameservices
      namenodesAddr: [ "machine1.example.com", "machine2.example.com" ] # the value of dfs.namenode.rpc-address.[nameservice ID].[name node ID]
      namenodes: [ "nn1", "nn2" ] # the value of dfs.ha.namenodes.[nameservice ID]
      user: hdfs
      password:
      port: 8020
      # scheduler platform hdfs log path keyword identification, used by task-application
      matchPathKeys: [ "flume" ]

  yarn:
    - clusterName: "bigdata"
      resourceManager: [ "machine1:8088", "machine2:8088" ] # the value of yarn.resourcemanager.webapp.address
      jobHistoryServer: "machine3:19888" # the value of mapreduce.jobhistory.webapp.address

  spark:
    sparkHistoryServer: [ "machine4:18080" ] # the value of spark.history.ui

The Compass table structure consists of two parts, one is compass.sql, and the other is a table that depends on the scheduling platform (dolphinscheduler.sql or airflow.sql, etc.)

1. Please execute document/sql/compass.sql first

2. If you are using the DolphinScheduler scheduling platform, please execute document/sql/dolphinscheduler.sql; if you are using the Airflow scheduling platform, please execute document/sql/airflow.sql

3. If you are using a self-developed scheduling platform, please refer to the task-syncer module to determine the tables that need to be synchronized

./bin/start_all.sh

architecture document

deployment document

Welcome to join the community for the usage or development of Compass. Here is the way to get help:

• Submit an issue.
• Join the wechat group, search and add WeChat ID daiwei_cn or zebozhuang. Please indicate your intention in the verification information. After verification, we will invite you to the community group.

Compass is licensed under the Apache License, Version 2.0 For detail see LICENSE and NOTICE.