A simple service that queries the logs from a remote log aggregation service query endpoint (currently just New Relic's GraphQL API).

Written in Rust. Deployable as a Docker container.

You can run the log-scraper with Docker and start up a redis container to use for caching.

# optional: start up redis
docker run --name some-redis -p 6379:6379 -d redis

# run the log-scraper service
docker run -i \
  -e NRLS_ACCOUNT_ID='1234567' \
  -e NRLS_API_KEY='<my-new-relic-api-key>' \
  -e REDIS_URL='redis:6379' \
  -e LOG_DIRECTORY='/usr/src/app/logs' \
  -e LS_SVC_PORT=7777 \
  -p 8080:7777 \
  --link some-redis:redis \
  willko/log-scraper:0.3.0-with-docs

Note: provide -d instead of -i for detached (background) mode instead of interactive mode.

Test that the service is up:

curl -i localhost:8080

# HTTP/1.1 200 OK
# content-length: 92
# content-type: application/json
# date: Sun, 12 Mar 2023 18:01:36 GMT
#
# {"ok":true,"message":"Healthy and kicking! Docs: /docs/log_scraper/api/logs_api/index.html"}

There is a helm chart for deploying the service to a Kubernetes environment. See the ./helm directory for more information.

Deploy to Kubernetes:

helm install log-scraper ./helm --namespace <my-namespace> \
  --set service.newRelicAccountId=$NRLS_ACCOUNT_ID \
  --set service.newRelicApiKey=$NRLS_API_KEY \
  --set service.logPrefix='my-awesome-app-logs'

There are environment variable and helm variable names for each configuration setting the log-scraper accepts. The table below shows the corresponding names for each setting and their default value for quick reference followed by the same list with their descriptions for futher details.

LOG_DIRECTORY (service.logDirectory)

The location of where logs are stored on the system. Note this defaults to a volume path when deploying with Helm (mounted at /usr/src/app/logs).

LOG_FILE_PREFIX (service.logPrefix)

The filename prefix for saving log files. The resulting saved files wll have the pattern similar to the following:

{logPrefix}_{date}.{logExtension}

LOG_FILE_EXTENSION (service.logExtension)

The extension to use when saving log files. Does not include the "dot".

LS_POLL_SCHEDULE (service.pollSchedule)

The cron schedule to set for polling the remote server to search for new logs.

LS_SVC_PORT (service.port)

The port the service will be served at.

NRLS_ACCOUNT_ID (service.newRelicAccountId)

The New Relic Account ID to authenticate as. This is passed in the query that is sent to their API.

NRLS_API_KEY (service.newRelicApiKey)

This is an API Key from New Relic that works with their NRQL GraphQL API.

REDIS_URL (service.redisURL)

Redis URL with port.

REDIS_KEY_NAME (service.redisKeyName)

The key name to store the timestamp of the last seen log entry fetched from the remote server.

Building and running the service locally is also quite simple and straightforward.

• Rust / Cargo
• Docker

The following commands should get you up and running. There scripts provided as well to help build and run via Docker locally.

# Local Build
make

# Generate Docs
cargo doc --document-private-items --open

# Docker Build
make build-container

# Docker Run
make start-container

# Docker Cleanup
make clean-containers

# Docker Compose Run
make start-compose

# Docker Compose Cleanup
make compose-cleanup

# Tag a new Docker build
TAG=<tag> make tag