This event logger was built for a lightweight event log and reporting purposes. The philosophy behind this API is that a mobile app will send events to /api/v1/event/heartbeat
API with the following parameters:
msisdn=8801XXXXXXXXX
// user identifierversion=1.0.0
// app versionplatform=android
// app platformeg: (android|ios)
event=sample_event_name
// a unique event name which the we are interested to trackidentifier=ABC123
// any kind of content ideg: C3P0
source=sample_event_source
// event source
This "event" is then logged and aggregated using the lightest and quickest possible system (nginx, fluentd and elasticsearch) and, from there we could then easily dump raw data, build dashboards and do other stuff that our wild imagination can perceive.
To run this project you will need the following
git
docker
anddocker-compose
installed- any code editor/IDE of your choice (optional)
Clone the git repo from [github]
cd /path/to/folder/
git clone https://github.com/muttalebm/nginx-event-collector.git
cd nginx-event-collector
Rename the .env.example file to .env in deploy folder
cd deploy
mv .env.example .env
Review the .env file. Check the Service Settings
section for any changes in the below configurations
#############################################################
# Services Settings
#############################################################
# Web Service
WEB_HTTP_PUBLISH_PORT=8121
WEB_HTTPS_PUBLISH_PORT=8122
## Where the log dumper application will be stored inside the container
WORK_DIR_PATH=/usr/src/app
## Any debug tool you might need
DEBUG_TOOLS=nano telnet
## Fluentd image version
FLUENTD_IMAGE_VERSION=v1.16.2-debian-1.0
## Nginx image version
NGINX_IMAGE_VERSION=1.25.2
## password for ES and Kibana
KIBANA_PASSWORD=default@123
ELASTIC_PASSWORD=default@123
## Fluentd Host
FLUEND_HOST=host.docker.internal
Build the docker image and run the services in background mode
docker compose build && docker compose up -d
docker compose logs -f
At first you will see a lot of logs and even some errors showing fluentd is unable to reach elasticsearch. I would suggest give it some time so that elasticsearch and kibana is intiated. If problem persists check if kibana can be accessed from your web browser http://localhost:5601
Now it's time to generate some logs. The nginx web application will expose an API endpoint where clients can send their logs
curl --location 'https://host.docker.internal:8122/api/v1/event/heartbeat?msisdn=FR3780700740043126777448625&version=225&platform=maroon&event=toffee_time&identifier=24462728&source=toffee' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <secret>'
Nginx is configured to write an access log to stdout which is then read by fluentd through docker log driver. Fluentd then parses the log and ships to ES
!! Work in progress !!
Run the log dumper which takes below arguments to and generates your desired csv dump. The dump will be available in your host machine's application folder.
Thank you for investing your time in contributing to our project! โจ.
To contribute to this project, please
- fork the repo
- make changes locally
- review your changes
- create PR