Comments (4)
I forked this repository and made a commit that replaces the old Jackson with the newer FasterXML/Jackson:
brunnsbe@ca36d5e
I however have problems running all the tests as quite many of them throws an OutOfMemoryException:
...
[ERROR] TestZkHelixPropertyStore.testSet » OutOfMemory unable to create new native thr...
[ERROR] TestZkHelixPropertyStore.testSetInvalidPath » OutOfMemory unable to create new...
[ERROR] TestZkHelixPropertyStore.testZkClientMonitor » OutOfMemory unable to create ne...
[ERROR] TestZkHelixPropertyStore.testZkTriggeredCallback » OutOfMemory unable to creat...
[ERROR] TestZkManagerWithAutoFallbackStore.testBasic » OutOfMemory unable to create ne...
[INFO]
[ERROR] Tests run: 866, Failures: 177, Errors: 0, Skipped: 103
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Helix 0.9.1-SNAPSHOT:
[INFO]
[INFO] Apache Helix ....................................... SUCCESS [ 2.061 s]
[INFO] Apache Helix :: Core ............................... FAILURE [ 01:00 h]
[INFO] Apache Helix :: Admin Webapp ....................... SKIPPED
[INFO] Apache Helix :: Restful Interface .................. SKIPPED
[INFO] Apache Helix :: HelixAgent ......................... SKIPPED
[INFO] Apache Helix :: Recipes ............................ SKIPPED
[INFO] Apache Helix :: Recipes :: Rabbitmq Consumer Group . SKIPPED
[INFO] Apache Helix :: Recipes :: Rsync Replicated File Store SKIPPED
[INFO] Apache Helix :: Recipes :: distributed lock manager SKIPPED
[INFO] Apache Helix :: Recipes :: distributed task execution SKIPPED
[INFO] Apache Helix :: Recipes :: service discovery ....... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:00 h
[INFO] Finished at: 2019-08-13T09:23:19+03:00
[INFO] ------------------------------------------------------------------------
...
from helix.
Will take a look.
from helix.
Hi @brunnsbe,
Thanks so much for working on Helix!
We did a performance experiment profile and find for a specific use case -- Helix controller leadership switch, using FasterXML/jackson along with some other performance improv changes, we get round around ~3x faster performance result.
Baseline profile:
The org.apache.helix.zookeeper.datamodel.serializer.ZNRecordSerializer.deserialize took ~60% of the time for org.apache.helix.manager.zk.DistributedLeaderElection.onControllerChange, where onControllerChange overall takes ~60s. This function call is CPU heavy.
After changing to FasterXML/jackson and made the ObjectMapper static, ZNRecordSerializer.deserialize only take ~10% of the total time, where onControllerChange overall takes ~20s. It becomes IO bounded.
It is another evidence that we should replace org.codehaus.jackson with FasterXML/jackson.
Do you want to continue the work and create a PR :D?
Xiaoyuan
from helix.
BTW, we improved our unit test and you won't see that much OutOfMemoryException now.
from helix.
Related Issues (20)
- [Failed CI Test] testGetAllMessages(org.apache.helix.rest.server.TestPerInstanceAccessor) HOT 4
- Accessing HelixManager ExecutorService HOT 6
- Apache Pinot facing issues with Helix version 1.4.0 HOT 1
- [Failed CI Test] testLostZkConnection(org.apache.helix.integration.TestZkConnectionLost) HOT 1
- [Failed CI Test] testComplexParentListenerPuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testCreatePuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testCrudPuppies(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testDeletePuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testGetPuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testSetPuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testUpdatePuppy(org.apache.helix.metaclient.impl.zk.TestMultiThreadStressTest.TestMultiThreadStressZKClient) HOT 1
- [Failed CI Test] testEvacuationWithOfflineInstancesInCluster(org.apache.helix.integration.rebalancer.TestInstanceOperation) HOT 3
- ZkBucketDataAccessor GC never occurs in high frequency write scenarios
- [Failed CI Test] testNodeSwapWithSwapOutInstanceOffline(org.apache.helix.integration.rebalancer.TestInstanceOperation) HOT 1
- AutoRebalanceStrategy generate empty IdealState
- [Failed CI Test] testDistributedController(org.apache.helix.integration.manager.TestConsecutiveZkSessionExpiry) HOT 1
- [Failed CI Test] testBatchMessageThreadPoolSize(org.apache.helix.messaging.handling.TestResourceThreadpoolSize) HOT 2
- Controller can violate fault zone placement constraints during intermediate state
- WagedInstanceCapacity Null Pointer Exception due to stale _instanceCapacityMap
- Request - Make batch instances stoppable request work for clusters without topology setup
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from helix.