Comments (6)
@anton-demydov-zoral Sorry, with the holidays I was on vacation.
Please follow this example for adding a custom config file: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file
There are two problems with yours:
- You use the /fluent-bit/etc path for your config, which is the path used by FireLens for its generated config, you need to use a different file path. This should be noted in our documentation.
- You should not add the input for logs, FireLens will take care of that with the managed config. Your extra config is imported into the managed config.
Here is what you should have:
Dockerfile:
FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf
Config in extra.conf:
[SERVICE]
Parsers_File /fluent-bit/parsers/parsers.conf
Log_Level debug
[FILTER]
Name parser
Match **
Parser nginx
Key_Name log
Reserve_Data True
[OUTPUT]
Name es
Match *
Host AWS_ES_HOST
Port 443
Index react-sdk
Type my_type
Use the link I provided to see what the Task Definition should look like, there are two key sections. On your Fluent Bit container you need to have:
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
},
And then in your app container that produces logs you should have:
"logConfiguration": {
"logDriver":"awsfirelens",
}
from amazon-ecs-firelens-examples.
Also, there is a problem with config-file-type
:
{
"essential": true,
"image": "829824778702.dkr.ecr.us-east-1.amazonaws.com/custom-fluent-bit:latest",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
"options":{
"enable-ecs-log-metadata":"false",
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
},
extra.conf:
[FILTER]
Name parser
Match **
Parser nginx
Key_Name log
I've tried to use latest
, 2.2.0
, 2.3.0
containers from amazon/aws-for-fluent-bit
but no luck.
If I use latest - fluentbit doesn't even start:
[2020/11/24 11:21:38] [error] [lib] backend failed
If I use 2.2.0
:
[2020/11/24 10:58:02] [error] [filter_parser] requested parser 'nginx' not found
I've tried to add parsers.conf but it didn't help as well.
from amazon-ecs-firelens-examples.
Sorry I missed this, this example should work: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/parse-common-log-formats
Your extra.conf is missing the Service section to import parsers: https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/parse-common-log-formats/parse-apache.conf
That's all you should need. You should not add your own input in the extra config file.
from amazon-ecs-firelens-examples.
Thanks @PettitWesley . I'll check and let you know.
from amazon-ecs-firelens-examples.
Hey @PettitWesley
It seems I still see the issue with parsing. Please check my configuration.
Dockerfile:
FROM amazon/aws-for-fluent-bit:latest
ADD fluent-bit.conf /fluent-bit/etc/
fluent-bit.conf:
[SERVICE]
Parsers_File /fluent-bit/parsers/parsers.conf
Log_Level debug
[INPUT]
Name forward
unix_path /var/run/fluent.sock
[FILTER]
Name parser
Match **
Parser nginx
Key_Name log
Reserve_Data True
[OUTPUT]
Name es
Match *
Host AWS_ES_HOST
Port 443
Index react-sdk
Type my_type
I can confirm that I see unparsed logs in elasticsearch:
but I assume fluentbit should parse them to something like:
{
"remote": "172.16.5.61",
"host": "-",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "5901",
"referer": "-",
"agent": "ELB-Healthchecker"
}
from amazon-ecs-firelens-examples.
Finally, it works, thank you @PettitWesley !
just posting here my configs for future needs.
Dockerfile:
FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf
extra.conf:
[SERVICE]
Log_Level info
Parsers_File /fluent-bit/parsers/parsers.conf
[FILTER]
Name parser
Match **
Parser nginx
Key_Name log
Reserve_Data True
[OUTPUT]
Name es
Match *
Host AWS_ES_HOST
Port 443
Index MY_INDEX
Type my_type
tls On
tls.verify Off
Aws_Region us-east-1
and Task Definition:
[
{
"essential": true,
"image": "REPO",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
"options":{
"enable-ecs-log-metadata": "false",
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "log_group",
"awslogs-region": "us-east-1",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"memoryReservation": 50
},
{
"name": "name",
"image": "image_name}",
"essential": true,
"portMappings": [
{
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awsfirelens"
},
"memory": 512,
"cpu": 256
}
]
from amazon-ecs-firelens-examples.
Related Issues (20)
- Aws Ecs Firelens to send the log to opensearch with custom filed
- Is there a way i can send the logs of my service to otel collector which is also a seperate service using the fluent forward protocol?(doubt)
- Metrics examples: remove need for metric filter by making FLB invoke a script that outputs EMF
- custom-fluent-bit-with-config Dockerfile contains unneeded file
- Is it possible to define custom input tag for fluentbit configs? HOT 4
- Feature Request: add Fluentbit Splunk Example HOT 3
- [Question] [Help] asctime not recognized as time field in Kibana HOT 2
- Feature Request: add Kinesis Data Streams Example HOT 4
- Feature Request: syslog example
- Issue with FARGATE 1.4.0 HOT 3
- OpenSearch Domain not found
- space after - in health check script fails the health check HOT 1
- Provide examples of using log key values in confs and task definitions HOT 1
- Log driver awslogs disallows options: log_key
- FireLens health check recommendation
- Firelens Multiple Output based on regex
- Any of containers log not shipping to destination
- Support of multiple headers in FireLens task definition
- Adding Dynatrace FluentBit example HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-ecs-firelens-examples.