Git Product home page Git Product logo

logdna-cloudwatch's Introduction

LogDNA CloudWatch Lambda Function

The LogDNA AWS CloudWatch integration relies on AWS Lambda to route your CloudWatch Logs to LogDNA.

How to Use

Deploy the Code

  1. Create a new Lambda function and select Author from scratch
  2. Click on the Lambda function to edit the details:
  • Code entry type: Upload a .ZIP file
  • Upload our LogDNA Lambda function .ZIP File.
  • Handler: index.handler
  • Runtime: Node.js 20.x

Configuration

General Configuration

If the S3 Lambda is being used to stream from gzipped files:

  1. Set Timeout to, at least, 10 seconds.
  2. Set Memory limit to, at least, 128 MB.

Notes:

  • The recommended number of retries is 0 because retrying lambda execution can result in duplicate logs. It can be modified in Configuration > Asynchronous invocation.

Triggers

Add CloudWatch Logs as a trigger with the following configuration:

  • Select the CloudWatch Log Group to be sent to LogDNA.
  • Choose your own custom Filter Name.
  • Optional Filter Pattern option can be used to filter the logs before shipping to LogDNA.

Notes:

  • You can specify only one CloudWatch Log Group in one trigger.

Permissions

For Execution role, assign a role that has the following policies:

Environment Variables

Set LOGDNA_KEY variable to your LogDNA ingestion key. Optionally, you can use the following environment variables:

  • LOGDNA_HOSTNAME: Alternative Host Name
  • LOGDNA_TAGS: Comma-separated Tags
  • LOGDNA_URL: Custom Ingestion URL
  • LOG_RAW_EVENT: Setting line to Raw event.message (Default: false):
    • It can be enabled by setting LOG_RAW_EVENT to YES or TRUE
    • Enabling it moves the following event-related meta data from the line field to the meta field:
      • event.type: messageType of CloudWatch Log encoded inside awslogs.data in base64
      • event.id: id of each CloudWatch Log encoded inside awslogs.data in base64
      • log.group: LogGroup where the log is coming from
      • log.stream: LogStream where the log is coming from

Notes: The following optional environment variables can also be used to tune this Lambda function for specific use cases.

  • LOGDNA_MAX_REQUEST_TIMEOUT: Time limit (in milliseconds) for requests made by this HTTP Client (Default: 30000)
  • LOGDNA_FREE_SOCKET_TIMEOUT: How long (in milliseconds) to wait for inactivity before timing out on the free socket (Default: 300000)
  • LOGDNA_MAX_REQUEST_RETRIES: The maximum number of retries for sending a line when there are network failures (Default: 5)
  • LOGDNA_REQUEST_RETRY_INTERVAL: How frequently (in milliseconds) to retry for sending a line when there are network failures (Default: 100)

Monitoring

Enabling monitoring means forwarding the metrics and logs about the execution of the CloudWatch Lambda function to CloudWatch. You can also create and use a separate CloudWatch Lambda function to monitor the performance of this CloudWatch Lambda function.

Test

You can test the configuration and code package using the following test input containing the sample event data:

{
    "awslogs": {
        "data": "H4sIAAAAAAAAEzWQQW+DMAyF/wrKmaEkJCbhhjbWCzuBtMNUVSmkNBIQRMKqqep/X6Cb5Ivfs58++45G7ZzqdfMza5Sjt6IpTh9lXReHEsXI3ia9BJnQlHHIhMSEBnmw/WGx6xwcp8Z50M9uN2q/aDUGx2vn/5oYufXs2sXM3tjp3QxeLw7lX6hS47lTz6lTO9i1uynfXkOMe5lsp9Fxzyy/9eS3hTsyXYhOGVCaEsBSgsyEYBkGzrDMAIMQlAq+gQIQSjFhBFgqJOUMAog34WAfoFFOOM8kA0Y5SSH+f0SIb67GRaHq/baosn1UmUlHF7tErxvk5wa56b2Z+iRJ0OP4+AWj9ITzSgEAAA=="
    }
}

License

Copyright ยฉ LogDNA, released under an MIT license. See the LICENSE file and https://opensource.org/licenses/MIT

Contributing

Contributions are always welcome. See the contributing guide to learn how you can help.

Happy Logging!

logdna-cloudwatch's People

Contributors

dspv avatar gjanco avatar johnpadilla avatar leeliu avatar lextoumbourou avatar lyhuang avatar smusali avatar vilyapilya avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

logdna-cloudwatch's Issues

Improve README

The README is not really clear and should be updated (https://docs.logdna.com/docs/cloudwatch too)

What wasn't clear is:

  • 1 - Create a new Lambda function and skip to the Configure function section I guess this was about an old interface? It's impossible to find the Configure Function section see screenshot below
    screenshot 2018-07-20 at 17 02 20
  • Before I could click on create, I had to create a role (this wasn't mentioned)
  • Only after that I was able to click on Upload .zip from the Code entry type select input, and edit the handler see screenshot
    screenshot 2018-07-20 at 17 04 56
  • And the environment vars below

Host the package in S3 Bucket

Can you host the file on S3? No need to download the file that way - just copy and paste the URL in Actions->Upload a File from Amazon S3

Add test coverage

After finishing #27, the unit and integration tests must be added to have, at least, 90%-95% coverage.

Minimum RAM? Timeout value?

This wasn't mentioned in the README but what do you recommend for these values?
This is a screenshot of the default values:
screenshot 2018-07-20 at 17 08 33

Add this to AWS Serverless Application Repository

Hi there!

Thanks for the Lambda code, I think you should publish it on AWS Serverless Application Repository, it would be a lot easier to deploy and maintain ;-)

screenshot 2018-07-20 at 16 48 31

Is that something planned maybe?

Issues with JSON formatted logs

Migrating from LogDNA-agent to the Lambda integration, our JSON logs are not properly formatted in the LogDNA web app. The JSON body gets converted to a string, and log lines displayed in the web app as a string.

In the interim, we made our own fork with a hacky fix that would break other use-cases. But we would like the upstream package to have better support for JSON formatted logs so that we do not have to maintain a patch.

Setup:
Follow these instructions (NOTE: The link to the zip file is an old version sitting in an S3 bucket, we downloaded the latest from github). We did port the instructions to Terraform, it is possible that something is different than what the manual instructions would produce, but I could not find a meaningful difference.

Steps to reproduce:

  • Run an application with JSON formatted logs, ingest logs using LogDNA agent
  • Run the same application, use the Cloudwatch->Lambda->LogDNA integration to ingest

Expected result:
The log lines look the same in the LogDNA web app

Actual result:
The logs lines look different, are not properly parsed as JSON data

Node 14 end of support

Hi, are you please planning to upgrade this package to node 16 or later given that Node 14 will be at the end of their security support in about a month? Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.