Comments (6)
albertzaharovits
commented on August 18, 2024
1
ping @williamrandolph since you edited the test yesterday.
from elasticsearch.
elasticsearchmachine
commented on August 18, 2024
Pinging @elastic/es-core-infra (Team:Core/Infra)
from elasticsearch.
williamrandolph
commented on August 18, 2024
I don't know that I'll have time to get to this today, but I'll get a look as soon as I can. In the meantime I'm going to go ahead and mute. #107630
from elasticsearch.
pgomulka
commented on August 18, 2024
judging by the thread dump it is blocked for 19minutes on:
client().prepareSearch(USER_INDEX)
.setQuery(QueryBuilders.matchAllQuery())
// Request times out if max concurrent shard requests is set to 1
.setMaxConcurrentShardRequests(usually() ? SearchRequest.DEFAULT_MAX_CONCURRENT_SHARD_REQUESTS : randomIntBetween(2, 10))
.get()
"TEST-KibanaThreadPoolIT.testUserThreadPoolsAreBlocked-seed#[AB39E5A32B6FC95C]" ID=34 WAITING on org.elasticsearch.action.support.PlainActionFuture$Sync@7c36647d
at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on org.elasticsearch.action.support.PlainActionFuture$Sync@7c36647d
at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
at app//org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:273)
at app//org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:93)
at app//org.elasticsearch.client.internal.support.AbstractClient$RefCountedFuture.get(AbstractClient.java:1535)
at app//org.elasticsearch.client.internal.support.AbstractClient$RefCountedFuture.get(AbstractClient.java:1515)
at app//org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:45)
at app//org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:152)
at app//org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:43)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase.lambda$assertThreadPoolsBlocked$3(SystemIndexThreadPoolTestCase.java:96)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase$$Lambda$3815/0x000000080161f060.run(Unknown Source)
at app//org.apache.lucene.tests.util.LuceneTestCase._expectThrows(LuceneTestCase.java:3040)
at app//org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2877)
at app//org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2868)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase.assertThreadPoolsBlocked(SystemIndexThreadPoolTestCase.java:90)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase$$Lambda$3783/0x000000080160ef40.run(Unknown Source)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase.runWithBlockedThreadPools(SystemIndexThreadPoolTestCase.java:66)
at app//org.elasticsearch.indices.SystemIndexThreadPoolTestCase.testUserThreadPoolsAreBlocked(SystemIndexThreadPoolTestCase.java:75)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
there is also this in logs:
1> [2024-04-24T15:39:25,570][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][3] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,570][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][0] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,570][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][2] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,570][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][1] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,671][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][4] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,673][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][5] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,675][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][6] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,677][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][7] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,726][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][8] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2024-04-24T15:39:25,727][WARN ][o.e.i.s.IndexShard ] [node_s0] [user_index][9] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
I wonder if it is relevant
from elasticsearch.
pgomulka
commented on August 18, 2024
I don't think there were changes around this functionality recently. I would lower the status of this medium-risk
from elasticsearch.
pgomulka
commented on August 18, 2024
I am not sure if I got into the wrong direction here. but one way to reproduce this is to allow for the task to end up on a queue for search threadpool. so we would have to modify the fillThreadPoolQueues and run with smaller timeout ./gradlew ':modules:kibana:internalClusterTest' --tests "org.elasticsearch.kibana.KibanaThreadPoolIT" -Dtests.timeoutSuite=30000!
long queueSize = info.getQueueSize().singles();
queueSize = threadPoolName.equals("search") ?queueSize - 1 : queueSize;
for (int i = 0; i < queueSize; i++) {
however I cannot confirm how this would be even possible without this change. The search on user's index should fail - be rejected by the executor, but for some reason it got stuck.
Perhaps we could add some trace logging around submitting a task to check the queue size ? SearchService#runAsync
from elasticsearch.
Related Issues (20)
- Error 302 after Configuration alert msteam in watcher elasticsearch HOT 2
- Fix java.library.path in VectorScorerBenchmark HOT 3
- [CI] RestEsqlIT testDoLogWithDebug {p0=SYNC} failing on serverless HOT 4
- Netty Buffer Leak via FetchSearchResult HOT 1
- Prohibit creation of rollup jobs in clusters that have never used rollup. HOT 3
- `DATE_DIFF()` with just two args fails with NPE HOT 1
- [ES|QL] Support first(), last() functions HOT 3
- [ESQL] Class cast exception mixing division and comparison HOT 1
- [ES|QL][Kibana request] Return _id by default HOT 5
- [ES|QL][Kibana request] Support _ignored metadata HOT 2
- Add and require `?acknowledge=true` when performing destructive actions HOT 5
- [Learning to rank] Add support to feature variables which is not interact with field data HOT 1
- Apt repo for Elasticsearch 5.x is no longer signed HOT 1
- [Transforms] IndexNotFoundException in GetCheckpoint HOT 1
- [ML] IllegalStateException from duplicate keys when reporting machine learning usage HOT 2
- [ML] Cannot create inference endpoint with id `.elser_model_2` due to validation error HOT 1
- ClassCastException in SignificantTermsAggregation HOT 1
- [ES|QL] implicit datetime casting for binary operator args HOT 1
- [CI] IpFilterRemoteAddressFilterTests testThatFilteringWorksByIp failing HOT 4
- Searchable snapshot should not be able to be deleted if it's in use by a frozen index HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.