Comments (5)
mirii1994
commented on September 14, 2024
Hi @rloporp , thanks for reporting this issue.
Can you please share details about your k8s environment? Where it runs on (cloud provider), which k8s version, what type are the nodes?
from logzio-k8s.
rloporp
commented on September 14, 2024
Sure, it's a kOps-managed cluster running on AWS EC2 instances:
- kOps version 1.23.2
- K8s version 1.23.9
- m5.xlarge instances (4 vCPU, 16GB RAM)
We've investigated on our own, and disk IOPS seems to be a bottleneck. We're testing memory-based buffers instead and the results look promising.
We've also noticed that the problem tends to begin when fluentd tries to forward a big log entry (even within limits) to LogZ. In that scenario, LogZ rejects the connection, fluentd keeps retrying the failed log entry and the buffer gets full with all subsequent logs.
from logzio-k8s.
mirii1994
commented on September 14, 2024
@rloporp thank you.
What was the size of the log entry that was rejected by logz.io's listener?
What was the response from logz.io?
from logzio-k8s.
rloporp
commented on September 14, 2024
About the log size, they are currently truncated to 5000 characters (in terms of byte size, I think they're UTF-8 characters, but I'm not 100% sure) + minimal Sping Boot headers.
About the error and Logz.io's response:
This is an old error (after one month, I could only retrieve a couple of lines):
2023-05-11 12:11:37 +0000 [error]: #0 [out_logzio] Error while sending POST to https://listener.logz.io:8071?token=****: {"malformedLines":0,"oversizedLines":1,"successfulLines":23} │
2023-05-11 12:11:37 +0000 [warn]: #0 [out_logzio] failed to flush the buffer. retry_times=9 next_retry_time=2023-05-11 12:12:09 +0000 chunk="5fb43eb75e431a2312837c16b91a1bdc" error_class=RuntimeError error="Logzio listener returned (400) for https://listener.logz. │
Current errors:
2023-06-16 15:32:08 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":6}
...
2023-06-16 15:33:25 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":54}
...
2023-06-16 16:31:59 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":56}
...
2023-06-16 16:33:12 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":4}
...
2023-06-16 16:51:52 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":17}
Zoom in on the last (16:51:52) error:
2023-06-16 16:51:38 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:40 +0000 [info]: #0 [filter_kubernetes_metadata] stats - namespace_cache_size: 5, pod_cache_size: 16, pod_cache_host_updates: 1545, pod_watch_gone_errors: 98, pod_watch_gone_notices: 98, namespace_cache_api_updates: 1063, pod_cache_api_updates: 1063, id_cache_miss: 1063
2023-06-16 16:51:41 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:44 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:44 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:45 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:46 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:46 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:48 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:50 +0000 [info]: #0 [spring_multi_lines] Timeout flush: kubernetes.var.log.containers.****.log:default
2023-06-16 16:51:52 +0000 [info]: #0 [out_logzio] Got 400 code from Logz.io. This means that some of your logs are too big, or badly formatted. Response: {"malformedLines":0,"oversizedLines":1,"successfulLines":17}
from logzio-k8s.
mirii1994
commented on September 14, 2024
Hi @rloporp ,
Please try the latest docker image (1.5.0) and let us know if this solves your issue. Thank!
from logzio-k8s.
Related Issues (20)
- AKS issue on namespace watch HOT 5
- Fluentbit support HOT 2
- Provide a convenient way to install (and use Kubernetes secrets) HOT 3
- Container crash (CrashLoopBackOff) HOT 3
- Logzio Daemonset on GKE is deleted on kube-system namespace after a few seconds HOT 1
- daemon.extensions wont create
- daemon.extensions wont create HOT 1
- Issue running on EKS HOT 3
- Dockerhub environment description seems to be outdated HOT 4
- deprecated apis in k8s config HOT 3
- json logs are not parsed HOT 8
- FR: exclude/only include logs from certain k8s namespaces HOT 12
- The DaemonSet "fluentd-logzio" is invalid HOT 1
- ARM support HOT 3
- token is still visible in the output HOT 3
- rewrite_tag_filter sets message key to null with this installation HOT 5
- include dedot plugin HOT 1
- Separate logging for each namespace HOT 4
- [EKS 1.21/1.22][BoundServiceAccountTokenVolume] Refresh AWS service account tokens automatically HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logzio-k8s.