stormcloudtech / mammoth Goto Github PK

Container based hadoop distributed system log analyzer

Mammoth is a container based hadoop distributed system log analyzer.
It monitors the Hadoop system as a whole and plays a role in easily delivering meaningful data to users.

Manage your Hadoop system with ease!

Do not worry and use it in any OS!

Easy installation!

Just copy & Paste!

1. Hadoop Overview

   • Status monitoring of all nodes
   • Disk Usage/Quota monitoring with visualiztion
   • vCores/Memory monitoring
   • Status monitoring of all applications

2. Resources(Yarn monitoring)

   • vCores monitoring with graph visualization
   • Memory monitoring with graph visualization

3. HDFS(Hadoop File System monitoring)

   • Disk quota monitoring
   • Disk usage monitoring
   • Disk status monitoring with graph visualization
   • File browsing service

4. Applications(MapReduce applications monitoring)

   • Status monitoring of all applications
   • Detailed application information monitoring

1. Mammoth monitor

Python is installed on most systems these days.

The monitor gets the Hadoop log directly from where the Hadoop system is installed and sends data to the collector server.
Since most systems these days have Python installed, we could implement a monitor program in Python easily and conveniently.

Here is README.md of mammoth monitor!

2. Mammoth collector

Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic).

Collector server is responsible for processing data received from numerous monitors appropriately and passing the data to influxdb.
Since it is a server that has to handle a lot of traffic, we configured a collector server by balancing the load in our service.
The framework used for Collector Server is a framework called FastAPI.
This framework has been proven to be as fast as Node.js or Go , and we used it because it is a Python-based framework.

Here is README.md of mammoth collector!

3. Mammoth GUI

Flutter is Google's UI toolkit for building beautiful, natively compiled applications for mobile, web, desktop, and embedded devices from a single codebase.

There is only one reason I used flutter in the GUI.
Because it is a cross platform framework.
All apps can be compiled from one source to Windows , Mac , Linux , iPadOS , and Android tablet.
We do not plan to support the web environment and mobile phone environment.
Just because I want users to use it on the big screen.

Here is README.md of mammoth GUI!

1. Download the Mammoth and register as a member.
   • Windows, OSX, Linux, or iPad, you can use this app anywhere you can think of.
2. Install mammoth monitor in your hadoop system.
   • Mammoth monitor installation guide
3. Run Mammoth monitor

1. Download the Mammoth and register as a member.

   • Windows, OSX, Linux, or iPad, you can use this app anywhere you can think of.

2. Install mammoth monitor in your hadoop system.

   • Mammoth monitor installation guide

3. Install mammoth collector in your own server.

4. Install influxdb and mongodb in your own server.

   • Databases installation guide

5. Change conf.py in your collector

6. Change conf.py in your monitor

1. Collector(Load balanced)

   • mammoth-collector1 :
     • CPU : 4 cores
     • Memory : 8GB
   • mammoth-collector2 :
     • CPU : 4 cores
     • Memory : 8GB
   • mammoth-collector3 :
     • CPU : 4 cores
     • Memory : 8GB
   • mammoth-collector4 :
     • CPU : 4 cores
     • Memory : 8GB
   • mammoth-collector5 :
     • CPU : 4 cores
     • Memory : 8GB

2. InfluxDB

   • CPU : 8 cores
   • Memory : 64GB

3. InfluxDB

   • CPU : 4 cores
   • Memory : 32GB