Git Product home page Git Product logo

Comments (6)

PettitWesley avatar PettitWesley commented on August 15, 2024 1

@anton-demydov-zoral Sorry, with the holidays I was on vacation.

Please follow this example for adding a custom config file: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file

There are two problems with yours:

  1. You use the /fluent-bit/etc path for your config, which is the path used by FireLens for its generated config, you need to use a different file path. This should be noted in our documentation.
  2. You should not add the input for logs, FireLens will take care of that with the managed config. Your extra config is imported into the managed config.

Here is what you should have:

Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf

Config in extra.conf:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Log_Level debug

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True

[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index react-sdk
    Type  my_type

Use the link I provided to see what the Task Definition should look like, there are two key sections. On your Fluent Bit container you need to have:

"firelensConfiguration": {
				"type": "fluentbit",
				"options": {
					"config-file-type": "file",
					"config-file-value": "/extra.conf"
				}
			},

And then in your app container that produces logs you should have:

"logConfiguration": {
				 "logDriver":"awsfirelens",
}

from amazon-ecs-firelens-examples.

anton-demydov-zoral avatar anton-demydov-zoral commented on August 15, 2024

Also, there is a problem with config-file-type:

 {
        "essential": true,
        "image": "829824778702.dkr.ecr.us-east-1.amazonaws.com/custom-fluent-bit:latest",
        "name": "log_router",
        "firelensConfiguration": {
                "type": "fluentbit",
                "options":{
                  "enable-ecs-log-metadata":"false",
                  "config-file-type": "file",
                  "config-file-value": "/extra.conf"
            }
        },

extra.conf:

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log

I've tried to use latest, 2.2.0, 2.3.0 containers from amazon/aws-for-fluent-bit but no luck.
If I use latest - fluentbit doesn't even start:

[2020/11/24 11:21:38] [error] [lib] backend failed 

If I use 2.2.0:

[2020/11/24 10:58:02] [error] [filter_parser] requested parser 'nginx' not found

I've tried to add parsers.conf but it didn't help as well.

from amazon-ecs-firelens-examples.

PettitWesley avatar PettitWesley commented on August 15, 2024

Sorry I missed this, this example should work: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/parse-common-log-formats

Your extra.conf is missing the Service section to import parsers: https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/parse-common-log-formats/parse-apache.conf

That's all you should need. You should not add your own input in the extra config file.

from amazon-ecs-firelens-examples.

anton-demydov-zoral avatar anton-demydov-zoral commented on August 15, 2024

Thanks @PettitWesley . I'll check and let you know.

from amazon-ecs-firelens-examples.

anton-demydov-zoral avatar anton-demydov-zoral commented on August 15, 2024

Hey @PettitWesley
It seems I still see the issue with parsing. Please check my configuration.
Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD fluent-bit.conf /fluent-bit/etc/

fluent-bit.conf:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Log_Level debug

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True

[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index react-sdk
    Type  my_type

I can confirm that I see unparsed logs in elasticsearch:
Screenshot 2020-12-21 at 13 43 50

but I assume fluentbit should parse them to something like:

{
  "remote": "172.16.5.61",
  "host": "-",
  "user": "-",
  "method": "GET",
  "path": "/",
  "code": "200",
  "size": "5901",
  "referer": "-",
  "agent": "ELB-Healthchecker"
}

from amazon-ecs-firelens-examples.

anton-demydov-zoral avatar anton-demydov-zoral commented on August 15, 2024

Finally, it works, thank you @PettitWesley !
Screenshot 2021-01-05 at 17 59 28

just posting here my configs for future needs.
Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf

extra.conf:

[SERVICE]
    Log_Level info
    Parsers_File /fluent-bit/parsers/parsers.conf
[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True
[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index MY_INDEX
    Type  my_type
    tls     On
    tls.verify Off
    Aws_Region us-east-1

and Task Definition:

[
    {
	"essential": true,
	"image": "REPO",
	"name": "log_router",
	"firelensConfiguration": {
		"type": "fluentbit",
                "options":{
                  "enable-ecs-log-metadata": "false",
                  "config-file-type": "file",
                  "config-file-value": "/extra.conf"
            }
	},
	"logConfiguration": {
		"logDriver": "awslogs",
		"options": {
			"awslogs-group": "log_group",
			"awslogs-region": "us-east-1",
			"awslogs-create-group": "true",
			"awslogs-stream-prefix": "firelens"
		}
	},
	"memoryReservation": 50
	},
    {
      "name": "name",
      "image": "image_name}",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80
                }
      ],
      "logConfiguration": {
                "logDriver": "awsfirelens"
            },
      "memory": 512,
      "cpu": 256
    }
  ]

from amazon-ecs-firelens-examples.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.