Comments (4)
udoprog
commented on May 24, 2024
We maintain a cache in the consumer process of which documents have previously been written, and each time it is started or entries time out we attempt to write the same document (with the same ID) again. This has so far worked and we get a code indicating that it was a duplicate, it should not be an issue as far as I know. I have not seen this in our logs before, but we use a different ingestion method (kafka-based).
Which version of Elasticsearch do you use? It should be possible to reproduce this by disabling the write cache.
EDIT: also, I'm assuming you are writing through the HTTP endpoint, is that correct?
from heroic.
mykolasmith
commented on May 24, 2024
Yes, correct, we are using the HTTP endpoint. Some things to note
- Occurs whether the service has just been restarted or has been active for weeks.
- Occurs even when no new metrics / tag combinations have are being written (i.e. new points with existing metrics / tags)
In the Heroic access logs, we'll see a few messages at a time return 500 Internal Server Error:
10.4.0.167 - - [08/Nov/2016:20:26:45 +0000] "POST //heroic.[REDACTED]/write HTTP/1.1" 500 179
10.4.0.167 - - [08/Nov/2016:20:26:45 +0000] "POST //heroic.[REDACTED]/write HTTP/1.1" 500 177
In the Heroic main logs, we'll see a corresponding org.elasticsearch.index.engine.DocumentAlreadyExistsException:
2016-11-08 20:26:45,215 ERROR c.s.h.c.CoreJavaxRestFramework [elasticsearch[Jebediah Guthrie][listener][T#1]] Request failed org.elasticsearch.index.engine.DocumentAlreadyExistsException: [heroic-1478131200000][3] [metadata][fcaca5641390bf8084fed957c60bee87]: document already exists
at org.elasticsearch.index.engine.InternalEngine.innerCreateNoLock(InternalEngine.java:329)
at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:287)
at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:259)
at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:482)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:206)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-08 20:26:45,815 ERROR c.s.h.c.CoreJavaxRestFramework [qtp1576277927-19944] Request failed org.elasticsearch.index.engine.DocumentAlreadyExistsException: [heroic-1478131200000][1] [series][a25734a5b0c3c1d418582bc64eb7e8eb]: document already exists
at org.elasticsearch.index.engine.InternalEngine.innerCreateNoLock(InternalEngine.java:329)
at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:287)
at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:259)
at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:482)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:206)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
from heroic.
mykolasmith
commented on May 24, 2024
Any progress on this? We're still seeing this error quite regularly (>1% of all requests to /write)
from heroic.
udoprog
commented on May 24, 2024
No, this is not actively being worked on at the moment.
from heroic.
Related Issues (20)
- Configurable Option to Index Resource Identifiers in Elasticsearch
- Heroic Bigtable Consumer does not handle failures as expected HOT 10
- ## Codestyle: enforce consistent statement-spacing and indentation (See issue #720)
- Implement new Bigtable timeout settings
- Rework all timer-based Tests
- upgrade bigtable-client-core library from 1.12.1 to 1.18.1 (latest) to eliminate retries bug HOT 1
- Replace bigtable-client-core lib with com.google.cloud:google-cloud-bigtable
- Bigtable server-side and client-side & Heroic-side behaviour analysis HOT 1
- RotatingIndexMapping interval is not read from config
- Dynamically determine metadata read indices based upon query range
- Analyse Heroic and user's perspective when hitting a timeout. Then implement necessary changes. HOT 2
- Distribution Documentation
- Mandatory client ID filter does not return informational message upon query rejection
- Investigate & resolve nondeterministic build errors HOT 1
- add x-client-id to markdown documentation examples for /query/[metrics|batch]
- Instrument Bigtable timeouts (add metrics and log messages)
- Fix "...Span <span name> is GC'ed without being ended." issue (caused by a BT timeout) HOT 1
- Investigate potentially serious performance implications of seemingly unnecessary thread-per-log message logging implementation
- integrationTests requires "hidden" quay.io/testcontainers/ryuk docker image HOT 1
- Downloads often cancel (stall) and no auto-resuming happens. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from heroic.