Comments (2)
I need to sync configuration for about 50-100 servers around the globe.
Olric is specifically designed to handle that cluster size. But it's also designed to work on a low latency network. If there are too latency between hosts, you may experience unexpected behaviors.
Another very important topic on this is your consistency requirements: if you need strong consistency, you probably need a different kind of solution or maybe you need to revise your system design. Olric is a PA/EC product in the context of the PACELC principle. This means that Olric can be considered as a strongly consistent data store under certain conditions: such as stable network and ReplicationFactor=1
. But the network is generally unreliable.
There is one writer and the rest are readers. Can this project handle this?
Yes it does. Look at this configuration: PartitionCount: 271, ClusterSize: 50. In this scenario Olric distributes the partitions among the cluster members. So most of the cluster members host 4 or 5 partitions and the writer thread will be routed to the partition owner transparently for every key.
So far, everything that uses RAFT turned out to be unusable
A plain RAFT or Paxos based design cannot handle this kind of load but Amazon published a paper called "Millions of tiny databases". I don't know what the problem you want to solve but the paper may give some inspiration.
Also, is there a way to write snapshot into storage so when a machine goes down and reboots it does not need to fetch the entire data set from scratch but only changes?
Unfortunately Olric doesn't provide this feature.
from olric.
ok, thanks for the info. i kinda new i will have to write my own solution, just wanted to be sure i didn't miss some existing project.
from olric.
Related Issues (20)
- panic: runtime error: integer divide by zero in setLRUEvictionStats HOT 1
- panic on args HOT 1
- Panic on start when BindAddr cannot be resolved
- Data race when getting stats in embedded mode
- remove dtopic listener didn't work well in cluster mode
- DNS based service discovery HOT 8
- Fast restart will cause data lost HOT 1
- Flaky integration test
- Using olric from PHP programs HOT 3
- storage overhead per object HOT 3
- Allow developer to pass his/her logging framework of choice HOT 9
- Use of DM Lock with Embedded Member HOT 2
- Failed to fetch the latest version of the routing table
- Can't reliably get PubSub - "no available client found"
- Data race in cluster setup using Olric v0.5.5 HOT 1
- Performance issues for production usage HOT 3
- Entire cluster fail to serve requests when one pod fails to restart with DMap creation operation timeout HOT 4
- Potential lock contention: Coordinator is not able to publish routing table to new node which causes it fails to be bootstrapped HOT 1
- Olric Clustered over different Kubernetes Clusters HOT 1
- panic: DMapPipeline.execOnPartition
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from olric.