Comments (8)
@roshkhatri / @zuiderkwast WDYT ?
from valkey.
CLUSTERMSG_TYPE_UPDATE
I kind of think this one should stay. We're indicating state changes, it seems prudent to send our state of the world as well.
from valkey.
I believe the messages listed here are rare and don't make up a large percentage of the cluster bus traffic, so I don't think we need to prioritize these. The bulk of the cluster bus traffic is PING and PONG. If we can figure out a way to use light ping pong 🏓, then we're saving a lot of cluster bus overhead for large clusters.
A basic idea is that each node remembers what it sent to any other node (for example a hash of the content) and if it hasn't changed since the last ping/pong between the nodes, then the node can send a light PING or PONG, just as a keepalive message and to say that nothing has changed. Do you want to explore this idea?
from valkey.
The bulk of the cluster bus traffic is PING and PONG.
This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.
from valkey.
At a high level, it does seem like a nice idea!
However for,
CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST
we do requirecurrentEpoch
,configEpoch
andmyslots
.- 'CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST' would require
mflags
This means we will be adding new field to the clusterMsgLight
which will make it a bit large again or add these in some other way in the data field.
@zuiderkwast's idea is what we had discussed some time ago and I do agree with that too and would like to explore.
This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.
Yeah this would he the best gain stating that I am alive and no changes since last time.
from valkey.
The bulk of the cluster bus traffic is PING and PONG.
This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.
@madolson Yes, the slot bitmap is 2KB (16K bits). A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?
Anyway, the light packet doesn't solve our scaling problem. The problem is still the all-to-all communication. In a raft cluster, nodes only communicate with the leader, except when the leader isn't reachable anymore. To focus on that might be a better idea.
from valkey.
A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?
This will be a constant for every msg, right? The thing we can focus on is how we can reduce our overhead for every msg.
from valkey.
A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?
This will be a constant for every msg, right? The thing we can focus on is how we can reduce our overhead for every msg.
Yes, the overhead per message is constant, but it's important to discuss it. It makes little sense to reduce a small message to even smaller (say from 20 to 10 bytes) because we would only save very little compare to all the overhead. If we can reduce a message from 2K to 400 including all protocol overhead, we save 80% so this is good.
We still have very many messages though and this will still be a bottleneck, because the number of ping-pong messages in a cluster grows exponentially with the size of the cluster (right?). If we save 80% of the size of each message, we can maybe achieve a 600 nodes cluster instead of 500 nodes, but if we can reduce the number of message to just grow linearly or quadratically with the size of the cluster, then we can scale to huge clusters for real. We could introduce features like non-voting nodes, and less active nodes that don't ping anyone but they just answer pings from other nodes. Or... we take the step to use a raft protocol.
from valkey.
Related Issues (20)
- [BUG] The result of the PING command does not match the documentation when using RESP3 HOT 1
- [BUG] A large negative number(e.g. -LONG_MAX) in HRANDFIELD/ZRANDMEMBER can easily cause Valkey to crash. HOT 2
- [NEW] Migrate from Redis 7.4.0 to Valkey 7.2.5 HOT 9
- [NEW] Introduce slot-level memory metrics
- [Test failure] Test issues with empty shard migrations
- [BUG] "CLIENT CAPA redirect" has no effect in scripts HOT 1
- [BUG] Cluster: Inconsistent MOVED/READONLY errors for read-only client
- [NEW] Introduce a new client capability for throttling
- [NEW] Add support for setting the group on a unix domain socket HOT 1
- Feature Request: Add Error Status Field for Diskless Syncs HOT 5
- [NEW] HTTP support for Valkey HOT 4
- [NEW] Support proxy protocol HOT 2
- Migrate TLS.md into README.md HOT 2
- [Test failure] test_reclaimFilePageCache fails on Amazon Linux 2023 AARCH64
- Remove direct reference to conn->fd and use the connection abstraction
- [BUG] cluster rebalance --cluster-weight <node>=0 fails with clusterManagerMoveSlot: NOREPLICAS error HOT 8
- [NEW] Benchmark CLI tool rewrite HOT 9
- [NEW] New Commands for Module: Module Set-Argument <name> [ARGS1, ARGS2...] and Module Reset-Argument <name> HOT 2
- [NEW] Replication stream access via Module API
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from valkey.