Comments (5)
Pinging @elastic/ml-core (Team:ML)
from elasticsearch.
Similar to #107266 (comment)
A stopped transform has a running thread that eventually fails to do whatever it is that it is doing (in this case, save to index)
from elasticsearch.
I cannot think of a good way to approach this, the current option is to explicitly check for this here:
else if (irrecoverableException instanceof IndexNotFoundException && IndexerState.ABORTING == getState()) {
logger.debug(
"[{}] Bulk index experienced IndexNotFoundException failure while Transform is aborting. This is likely "
+ "due to the Transform delete API called with delete_dest_index=true. Aborting indexer. ",
getJobId()
);
onAbort();
// do not call listener
}
This is similar to what happens before and after this code is invoked, where the Transform exits gracefully if it was moved into the ABORTING
state (via the DELETE
API).
The alternative is to have fail check for the ABORTING
state and exit gracefully there, but that is a more invasive change with broader impact
from elasticsearch.
The above isn't quite true, it is correlated but I cannot repro it.
This is actually the line that is throwing the exception: https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java#L832
I haven't fully digested what this code is doing enough to repro it, it feels vaguely like the node running the transform has a earlier cluster state version than the node processing the bulk request, so we fail out instead of retrying?
from elasticsearch.
This can kinda be consistently reproduced by trying to get the shards to relocate. When this happens, we want the Transform to retry the checkpoint and check if the Index exists.
It might be possible that a Delete API can still trigger this issue, in which case we will want to have the Indexer gracefully shut down.
It also might be possible that the user can delete the Index using the Index API without stopping the Transform, in that case the Transform seems to continue running and the Bulk API will recreate the Index as per its spec. Ideally we'd either stop the Transform or have the Transform create/update the Index as per the Transport API spec, but I don't think that is within the scope of this issue.
from elasticsearch.
Related Issues (20)
- [CI] ClientYamlTestSuiteIT classMethod failing HOT 8
- ES|QL when executed over CCS requires "read" local privilege for any (or none) local indices HOT 4
- [documentation] ES|QL over CCS with API key based security HOT 4
- Logical CCS Names: Dynamically reference additional cluster information HOT 1
- Limit shard snapshot work-in-progress HOT 1
- [ESQL] SimplifyComparisonArithmetics not being correctly applied HOT 1
- [CI] ReactiveStorageDeciderDecisionTests testStoragePreventsAllocation failing HOT 2
- Automatically add created, last_changed and changed_by metadata for index templates, component templates and ingest pipelines HOT 6
- Failures in elasticsearch / periodic / platform-support / 7.17 / windows-2016 / checkPart1 / platform-support-windows HOT 3
- ES|QL version of spatial intersects search slow on some benchmarks HOT 2
- [CI] MlDistributedFailureIT testFullClusterRestart failing HOT 4
- [CI] DataStreamsSnapshotsIT testRestoreSnapshotFully failing with assertion error HOT 1
- [CI] CoordinatorVotingConfigurationTests testClusterUUIDLogging failing HOT 2
- LogsDB cluster level setting in `logs@settings` HOT 1
- [CI] SSLConfigurationReloaderTests testReloadingKeyStore failing HOT 5
- [CI] Multiple test failures JDK23 on main HOT 2
- [CI] WildcardLikeTests testFold {TestCase= with text} failing HOT 2
- [CI] InferenceIT testInferClassificationModel failing HOT 2
- [CI] WildcardLikeTests testCrankyEvaluateBlockWithNulls {TestCase= with keyword} failing HOT 2
- Large synonyms sets inconsistently return synonym results HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.