Comments (5)
The bottleneck could be on the output side.
Do you see all the metrics being written to influxDB?
You can tweak batch-size: 1000
(set to a lower value) and flush-timer: 10s
(set to lower value) to speed up the flushing of metrics to the DB and avoid storing them in memory.
from gnmic.
Hi @karimra,
I've just tested: batch-size: 200, flush-timer: 2s; batch-size: 100, flush-timer: 1s. Didn't seem to make a difference. RAM increase jumps happen every minute, in sync with all the IXRs sending metrics to gNMIc.
When the VM RAM gets full, some of the metrics aren't sent to InfluxDB anymore. Influx data for a specific node might be missing "/state/port/statistics/out-octets", while ".../in-octets" are present.
EC
from gnmic.
Would you be able to test without processors, just to see if the behavior is different?
from gnmic.
Yes! I've disabled the event-processors for the influxdb output and RAM usage stays low.
This is the template I've been using, output1 was already disabled:
https://github.com/ebinachan/telemetry/blob/master/sros/gnmic.yaml
Edit: RAM usage is low even with source_trim processor. The moment I enable either convert_int, convert_float, or both, I'm seeing the gradual RAM increase and loss of metrics.
It seems to be related to these messages:
Mar 31 13:35:02 linux bash[24818]: 2023/03/31 13:35:02 influxdb2client E! Write error: 400 Bad Request: partial write: field type conflict: input field "/state/port/statistics/in-octets" on measurement "PS_SROS_PORTSTATS" is type string, already exists as type integer dropped=1
Mar 31 13:35:57 linux bash[24818]: 2023/03/31 13:35:57 influxdb2client E! Write error: 400 Bad Request: partial write: field type conflict: input field "/state/port/statistics/in-octets" on measurement "PS_SROS_PORTSTATS" is type string, already exists as type integer dropped=1
Mar 31 13:36:57 linux bash[24818]: 2023/03/31 13:36:57 influxdb2client E! Write error: 400 Bad Request: partial write: field type conflict: input field "/state/port/statistics/in-octets" on measurement "PS_SROS_PORTSTATS" is type string, already exists as type integer dropped=1
Mar 31 13:37:59 linux bash[24818]: 2023/03/31 13:37:59 influxdb2client E! Write error: 400 Bad Request: partial write: field type conflict: input field "/state/port/statistics/in-octets" on measurement "PS_SROS_PORTSTATS" is type string, already exists as type integer dropped=1
Then, I changed the config to match on exact field names as in:
https://github.com/ebinachan/telemetry/blob/master/sros/20230331_gnmic.yaml
Dropped all SROS telemetry measurements from influx, and started gNMIc again. I'm still seeing the logs above, and a gradual loss in metrics along with the RAM increase.
I should have pasted the "in-octets" write error initially, but I honestly thought it was a one-off unrelated error. Now I'm thinking this might cause some sort of soft-lockup and then RAM usage starts inflating.
Second Edit:
It could be that one of the SROS nodes was reporting a high enough "in-octets" value that was too large for the "int" converter.
Moved the in-octets/out-octets fields to the float converter, dropped InfluxDB measurements before process start, and the write errors are gone, RAM usage is low for now.
EC
from gnmic.
Thanks for the update! Glad the ram issues are gone.
from gnmic.
Related Issues (20)
- split-events: false not suppressing empty messages HOT 3
- stringArray in the main branch is not working as expected HOT 4
- Poll updates are not written to output provided in the config HOT 4
- Kafka input TLS authentication support HOT 3
- Installation script fails with ubuntu 24.04 HOT 1
- Prometheus metrics issue for string values HOT 14
- gnmic 0.38.2 regression test failure HOT 3
- Starlark LLDP script breaks prometheus output HOT 8
- Performance optimisation for prometheus output HOT 11
- Log Fail Attempt to Influxdb HOT 1
- [Question] Deploying behind NGINX ingress controller HOT 1
- gnmi-server not sending sync_response correctly in Subscribe mode = once HOT 2
- Support for the gNMI extension field HOT 3
- gNMIc targets redistribution doesn`t work after pod fails HOT 2
- can get response when using a disabled cipher HOT 1
- Question about setting stream window size HOT 5
- Feature request: Improve observability in api-server metrics implementation
- Question: What format to be used for targets added in consul HOT 1
- gnmic retries stream subscriptions after INVALID_ARGUMENT error HOT 1
- How to parse an SNE router ID to the gnmi request when the router ID is not exposed to outside network HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gnmic.