Comments (2)
GiedriusS
commented on June 15, 2024
Maybe you could try the version from main in your tests and check whether daa34a5 helps? 🤔
from thanos.
yliu138repo
commented on June 15, 2024
@GiedriusS I have tried with main-2024-04-03-f80fd94, I am still seeing following error context deadline exceeded. I've also tried to configure otel collector timeout. Does it mean I will need to increase aysnc workers --receive.forward.async-workers=x with number of async workers higher than 5?
2024-04-05T04:19:49.099Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n", "errorCauses": [{"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n"}], "dropped_items": 461230}
from thanos.
Related Issues (20)
- Rate query failing from Grafana HOT 2
- receive: memory spikes during tenant WAL truncation HOT 5
- Thanos Store fills up the entire hard disk at startup HOT 3
- Support for encryption in transit for Memcached HOT 2
- Why does thanos query-frontend not support multiple downstream addresses?
- Thanos query fanout has high latency when a strict-store is unavailable HOT 3
- Ruler UI: `alert.query-template` is not honored inside the Rules UI HOT 3
- [Thanos Storegateway ]"failed to read index-header from disk; recreating" path=/data/01H1K45K0FRT36S1RCAWHW7R9A/index-heade HOT 1
- Adding User Agent to HTTP Logs
- Compact: Display TODO plan HOT 7
- compactor: does not compact 4 consecutive 2-hour blocks HOT 6
- compactor: series not 16-byte aligned error HOT 2
- Improved file access logging
- Sidecar: reporting as ready on startup when no Prometheus process is running
- tools bucket: Add ability to discover external labels from prometheus address for `upload-blocks` HOT 1
- Thanos Sidecar - Flush Endpoint HOT 9
- Grafana only shows raw data from Thanos HOT 3
- Instance Principal Provider - Region issue
- Consider X-Forwarded-For on HTTP/GRPC Logging
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.