Comments (5)
elasticsearchmachine
commented on September 28, 2024
Pinging @elastic/es-analytical-engine (Team:Analytics)
from elasticsearch.
nik9000
commented on September 28, 2024
Doesn't reproduce. Checking the saved build bits.
from elasticsearch.
nik9000
commented on September 28, 2024
The logs don't show me anything interesting. Here's where the failure happened:
[2024-06-07T23:44:36,311][INFO ][o.e.c.r.a.AllocationService] [test-cluster-1] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][4]]])." previous.health="YELLOW" reason="shards started [[test][4]]"
[2024-06-07T23:45:36,244] ...failure happens here...
[2024-06-07T23:45:36,308][INFO ][o.e.x.m.MachineLearning ] [test-cluster-1] Starting machine learning feature reset
[2024-06-07T23:44:34,148][INFO ][o.e.x.e.EnrichPolicyRunner] [test-cluster-0] Policy [networks-policy]: Policy execution complete
[2024-06-07T23:45:36,244] ...failure happens here...
[2024-06-07T23:45:38,113][INFO ][o.e.n.Node ] [test-cluster-0] stopping ...
This isn't even the last of the tests. Three more run after this and pass.
from elasticsearch.
nik9000
commented on September 28, 2024
The failure is just a timeout, so I checked for GC pauses and don't see a long one:
[2024-06-07T13:44:37.224+0000][799157][info ][safepoint ] Safepoint "Cleanup", Time since last: 1361576542 ns, Reaching safepoint: 68883 ns, Cleanup: 22265 ns, At safepoint: 16977 ns, Total: 108125 ns
[2024-06-07T13:44:43.309+0000][799157][info ][safepoint ] Safepoint "Cleanup", Time since last: 6084797009 ns, Reaching safepoint: 71024 ns, Cleanup: 15188 ns, At safepoint: 15885 ns, Total: 102097 ns
[2024-06-07T13:45:06.311+0000][799157][info ][safepoint ] Safepoint "Cleanup", Time since last: 23002298383 ns, Reaching safepoint: 96985 ns, Cleanup: 19006 ns, At safepoint: 22753 ns, Total: 138744 ns
[2024-06-07T13:45:36.315+0000][799157][info ][safepoint ] Safepoint "Cleanup", Time since last: 30003396248 ns, Reaching safepoint: 114032 ns, Cleanup: 56463 ns, At safepoint: 23203 ns, Total: 193698 ns
and
[2024-06-07T13:44:36.165+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 1035249252 ns, Reaching safepoint: 28409 ns, Cleanup: 21751 ns, At safepoint: 16339 ns, Total: 66499 ns
[2024-06-07T13:44:36.190+0000][799165][info ][gc,start ] GC(114) Pause Young (Normal) (G1 Evacuation Pause)
[2024-06-07T13:44:36.190+0000][799165][info ][gc,task ] GC(114) Using 12 workers of 23 for evacuation
[2024-06-07T13:44:36.190+0000][799165][debug][gc,age ] GC(114) Desired survivor size 20971520 bytes, new threshold 15 (max threshold 15)
[2024-06-07T13:44:36.193+0000][799165][info ][gc,phases ] GC(114) Pre Evacuate Collection Set: 0.28ms
[2024-06-07T13:44:36.193+0000][799165][info ][gc,phases ] GC(114) Merge Heap Roots: 0.12ms
[2024-06-07T13:44:36.193+0000][799165][info ][gc,phases ] GC(114) Evacuate Collection Set: 1.97ms
[2024-06-07T13:44:36.193+0000][799165][info ][gc,phases ] GC(114) Post Evacuate Collection Set: 0.68ms
[2024-06-07T13:44:36.193+0000][799165][info ][gc,phases ] GC(114) Other: 0.16ms
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) Age table with threshold 15 (max threshold 15)
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 1: 398624 bytes, 398624 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 2: 103456 bytes, 502080 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 3: 13264 bytes, 515344 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 4: 6552 bytes, 521896 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 5: 162336 bytes, 684232 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 6: 12648 bytes, 696880 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 7: 79416 bytes, 776296 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 8: 527336 bytes, 1303632 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 9: 25512 bytes, 1329144 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 10: 88216 bytes, 1417360 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 11: 9448 bytes, 1426808 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 12: 2680 bytes, 1429488 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 13: 17592 bytes, 1447080 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 14: 4528 bytes, 1451608 total
[2024-06-07T13:44:36.193+0000][799165][trace][gc,age ] GC(114) - age 15: 3664 bytes, 1455272 total
[2024-06-07T13:44:36.193+0000][799165][info ][gc,heap ] GC(114) Eden regions: 75->0(75)
[2024-06-07T13:44:36.193+0000][799165][info ][gc,heap ] GC(114) Survivor regions: 1->1(10)
[2024-06-07T13:44:36.193+0000][799165][info ][gc,heap ] GC(114) Old regions: 26->26
[2024-06-07T13:44:36.193+0000][799165][info ][gc,heap ] GC(114) Humongous regions: 0->0
[2024-06-07T13:44:36.193+0000][799165][info ][gc,metaspace] GC(114) Metaspace: 165034K(169280K)->165034K(169280K) NonClass: 142440K(144768K)->142440K(144768K) Class: 22594K(24512K)->22594K(24512K)
[2024-06-07T13:44:36.193+0000][799165][info ][gc ] GC(114) Pause Young (Normal) (G1 Evacuation Pause) 402M->102M(512M) 3.481ms
[2024-06-07T13:44:36.194+0000][799165][info ][gc,cpu ] GC(114) User=0.02s Sys=0.00s Real=0.01s
[2024-06-07T13:44:36.194+0000][799165][info ][safepoint ] Safepoint "G1CollectForAllocation", Time since last: 25269916 ns, Reaching safepoint: 80636 ns, Cleanup: 4306 ns, At safepoint: 3783648 ns, Total: 3868590 ns
[2024-06-07T13:44:40.337+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 4143710283 ns, Reaching safepoint: 70969 ns, Cleanup: 12494 ns, At safepoint: 15723 ns, Total: 99186 ns
[2024-06-07T13:44:50.339+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 10001529701 ns, Reaching safepoint: 128063 ns, Cleanup: 14258 ns, At safepoint: 23328 ns, Total: 165649 ns
[2024-06-07T13:45:06.341+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 16001740619 ns, Reaching safepoint: 107554 ns, Cleanup: 16758 ns, At safepoint: 16185 ns, Total: 140497 ns
[2024-06-07T13:45:28.344+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 22002390915 ns, Reaching safepoint: 113024 ns, Cleanup: 12752 ns, At safepoint: 15838 ns, Total: 141614 ns
[2024-06-07T13:45:36.457+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 8113326463 ns, Reaching safepoint: 42546 ns, Cleanup: 34079 ns, At safepoint: 38836 ns, Total: 115461 ns
[2024-06-07T13:45:37.518+0000][799165][info ][gc,start ] GC(115) Pause Young (Normal) (G1 Evacuation Pause)
[2024-06-07T13:45:37.518+0000][799165][info ][gc,task ] GC(115) Using 12 workers of 23 for evacuation
[2024-06-07T13:45:37.518+0000][799165][debug][gc,age ] GC(115) Desired survivor size 20971520 bytes, new threshold 15 (max threshold 15)
[2024-06-07T13:45:37.522+0000][799165][info ][gc,phases ] GC(115) Pre Evacuate Collection Set: 0.34ms
[2024-06-07T13:45:37.522+0000][799165][info ][gc,phases ] GC(115) Merge Heap Roots: 0.14ms
[2024-06-07T13:45:37.522+0000][799165][info ][gc,phases ] GC(115) Evacuate Collection Set: 2.07ms
[2024-06-07T13:45:37.522+0000][799165][info ][gc,phases ] GC(115) Post Evacuate Collection Set: 0.90ms
[2024-06-07T13:45:37.522+0000][799165][info ][gc,phases ] GC(115) Other: 0.15ms
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) Age table with threshold 15 (max threshold 15)
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 1: 255600 bytes, 255600 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 2: 16160 bytes, 271760 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 3: 11208 bytes, 282968 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 4: 11824 bytes, 294792 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 5: 5440 bytes, 300232 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 6: 161680 bytes, 461912 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 7: 12520 bytes, 474432 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 8: 79416 bytes, 553848 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 9: 527264 bytes, 1081112 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 10: 25320 bytes, 1106432 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 11: 88216 bytes, 1194648 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 12: 9448 bytes, 1204096 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 13: 2680 bytes, 1206776 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 14: 17592 bytes, 1224368 total
[2024-06-07T13:45:37.522+0000][799165][trace][gc,age ] GC(115) - age 15: 4344 bytes, 1228712 total
[2024-06-07T13:45:37.522+0000][799165][info ][gc,heap ] GC(115) Eden regions: 75->0(75)
[2024-06-07T13:45:37.522+0000][799165][info ][gc,heap ] GC(115) Survivor regions: 1->1(10)
[2024-06-07T13:45:37.522+0000][799165][info ][gc,heap ] GC(115) Old regions: 26->26
[2024-06-07T13:45:37.522+0000][799165][info ][gc,heap ] GC(115) Humongous regions: 0->0
[2024-06-07T13:45:37.522+0000][799165][info ][gc,metaspace] GC(115) Metaspace: 165131K(169344K)->165131K(169344K) NonClass: 142528K(144832K)->142528K(144832K) Class: 22603K(24512K)->22603K(24512K)
[2024-06-07T13:45:37.522+0000][799165][info ][gc ] GC(115) Pause Young (Normal) (G1 Evacuation Pause) 402M->101M(512M) 3.900ms
[2024-06-07T13:45:37.522+0000][799165][info ][gc,cpu ] GC(115) User=0.02s Sys=0.00s Real=0.00s
[2024-06-07T13:45:37.522+0000][799165][info ][safepoint ] Safepoint "G1CollectForAllocation", Time since last: 1060950034 ns, Reaching safepoint: 85313 ns, Cleanup: 62213 ns, At safepoint: 4173388 ns, Total: 4320914 ns
[2024-06-07T13:45:39.004+0000][799165][info ][safepoint ] Safepoint "Cleanup", Time since last: 1481809813 ns, Reaching safepoint: 8752 ns, Cleanup: 83151 ns, At safepoint: 7741 ns, Total: 99644 ns
[2024-06-07T13:45:40.155+0000][799165][info ][gc,heap,exit] Heap
[2024-06-07T13:45:40.155+0000][799165][info ][gc,heap,exit] garbage-first heap total reserved 524288K, committed 524288K, used 280285K [0x00000000e0000000, 0x0000000100000000)
[2024-06-07T13:45:40.155+0000][799165][info ][gc,heap,exit] region size 4096K, 44 young (180224K), 1 survivors (4096K)
[2024-06-07T13:45:40.155+0000][799165][info ][gc,heap,exit] Metaspace used 165454K, committed 169664K, reserved 1245184K
[2024-06-07T13:45:40.155+0000][799165][info ][gc,heap,exit] class space used 22669K, committed 24576K, reserved 1048576K
from elasticsearch.
nik9000
commented on September 28, 2024
Since this has only failed once I'm going to reenable it and see if we can get another failure with more fun information.
from elasticsearch.
Related Issues (20)
- Add docs for additional fields when level parameter is set to shards in nodes stats HOT 2
- [CI] HaHdfsFailoverTestSuiteIT testHAFailoverWithRepository failing HOT 1
- [CI] SpatialPushDownGeoPointIT testPushedDownQueriesSingleValue failing HOT 2
- We ignore `similarity` when gathering inner_hits for vectors HOT 1
- [CI] MultiClusterSpecIT test {stats.MaxOfIpGrouping} failing HOT 3
- Segment merging for deleted documents on upgrade causing 100% CPU usage. HOT 10
- [CI] GlobalCheckpointSyncActionIT class failing HOT 4
- [CI] MultiClusterSpecIT test {stats.MinOfIpGrouping} failing HOT 3
- [CI] SearchProgressActionListenerIT testSearchProgressWithAggs failing HOT 2
- [CI] HeapAttackIT testSortByManyLongsTooMuchMemoryAsync failing HOT 1
- [CI] SearchProgressActionListenerIT testSearchProgressSimple failing HOT 5
- [CI] DockerTests test600Interrupt failing HOT 8
- [CI] TextSimilarityRankTests testRerankInferenceResultMismatch failing HOT 3
- [CI] PrevalidateShardPathIT testCheckShards failing HOT 2
- [CI] MlDistributedFailureIT testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign failing HOT 2
- [CI] EsqlSpecIT test {stats.MaxOfIpGrouping SYNC} failing HOT 2
- [CI] EsqlSpecIT test {stats.MinOfIpGrouping SYNC} failing HOT 3
- [CI] EsqlSpecIT test {stats.MinOfIpGrouping ASYNC} failing HOT 3
- [CI] InternalDistributionBwcSetupPluginFuncTest class failing HOT 2
- [CI] SecureHdfsSnapshotRepoTestKitIT class failing HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.