glaracuente / insightproject2018 Goto Github PK

The free online encyclopedia Wikipedia says: "The term Network Monitoring describes the use of a system that constantly monitors a computer network for slow or failing systems and that notifies the network administrator in case of outages via email, pager or other alarms."

For any company to grow and be successful, it is essential the it establish a method of monitoring its network activity is essential to. This should include solutions that can automatically detect and respond to threats and performance issues in real time, as well as predict possible issues in the future.

The object of this project is to add the monitoring infrastructure to the data engineering pipeline. This provides the reporting capability on the capacity and performance of the hardware and applications so you can know when you are running into issues as well use it for forecasting future needs.

1. Learning curve

2. Identifying what needs to be monitored

   • Creating a “visual” map of the most critical parts of the network
   • avoiding information overload

3. Deciding how to measure what is being monitored

   • Setting base lines

Ingestion: AWS S3
Data Processing: Spark
Storage: MySQL
User Interface (UI): Flask

Display Grafana

Monitoring System Prometheus

• Containerize the monitoring system
• Add caching to the system

New York City Parking Violations

[Prometheus] (http://prometheus.io/)
[How to use prometheus ] (https://www.digitalocean.com/community/tutorials/how-to-use-prometheus-to-monitor-your-ubuntu-14-04-server)
[ How to Install and Secure Grafana] (https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-16-04)
[mysql_exporter] (https://github.com/prometheus/mysqld_exporter)
[Monitoring MySQL] (https://dzone.com/articles/monitoring-mysql)
Monitoring Read/Write Latency