Comments (8)
TubeMQ is suitable for high concurrency, massive data and tolerate a small amount of data loss scenarios under abnormal conditions.
From the actual application situation, the Broker encounters an abnormality that the disk fails under RAID10 and cannot be recovered, or the broker suddenly loses power and the memory has not been flushed, and the unconsumed data will be lost; if the disk failure can be recovered and the data is recovered The message has been swiped into the file, and the unconsumed message can be consumed after the Broker restarts.
We have encountered difficulties when considering multi-copy solutions: in the same scenario, multi-node asynchronous replication also has data loss. Only when multi-node synchronous replication is used can data be guaranteed, but this will directly reduce system throughput. At the same time, the cost of the solution is increased, so the TubeMQ implementation focuses on providing a high-performance and low-cost solution to tolerate a small amount of data loss.
--- --- --- --- --- --- --- --- ---
TubeMQ适合高并发、海量数据且异常情况下容忍少量数据丢失场景。
从实际应用情况看,Broker遇到磁盘在RAID10下故障且无法恢复的异常,或者Broker突然断电处在内存还未刷盘宕机,未消费到的数据会丢失;如果磁盘故障能恢复且数据已刷盘到文件中,未消费的消息在Broker重启后能被继续消费。
我们在考虑多副本方案时遇到难题:同样场景下,多节点异步复制同样存在丢数据情况,只有在采用多节点同步复制时才能保证数据一条不丢,但这样做会直接降低了系统吞吐量同时增加了方案成本,所以TubeMQ实现侧重以容忍少量数据丢失为前提供高性能又低成本的方案。
from tubemq.
TubeMQ plans to provide different data service guarantees in the service classification mode. At present, the use of zk for offset persistence is also based on this consideration.
--- --- --- --- --- --- --- --- ---
TubeMQ计划后续以服务分级方式提供不同的数据服务保证,目前offset持久化使用zk也是基于这点考虑暂未改造
from tubemq.
@gosonzhang 多副本同步复制或者异步复制,可以做成可配置,让具体业务去选择。
from tubemq.
Yes, we plan to provide multiple copies to allow the business to choose different data quality guarantees.
--- --- --- --- --- --- --- --- ---
是的,我们计划提供多副本功能,让业务能选择不同的数据质量保证
from tubemq.
@gosonzhang Consider using hdfs as the storage layer? if one broker is down ,we can start the same broker at any other node immediately.
from tubemq.
If the hdfs nodes are outside the cluster, they are equivalent to adding additional machine costs. If they are in the cluster, the same machine resource will be shared by multiple copies.
--------------------------
hdfs节点如果在集群之外,相当于是增加了额外机器成本,如果是集群内则相同机器情况下会为多副本分摊同份资源
from tubemq.
可以参考其他消息系统,新增一个生产者配置,设定级别,由使用者应用开发选择,比如针对准确性要求不高的可以异步或同步一个副本,针对金融对账等场景同步所有副本,在吞吐量和可靠性上做一些权衡处理
from tubemq.
from tubemq.
Related Issues (20)
- Error when I use mvn compile HOT 10
- Comb the use points of isBlank() function to reduce unnecessary checks
- After the default flow control, add consumer group level flow control,consumer group level flow control does not take effect. HOT 1
- Broker is slower than the consumer's VisitAuthorizedToken update,the consumer will continue to warn the failure of certificate. HOT 1
- Can tubemq support the compatibility of Kafka interface? HOT 2
- com.tencent.tubemq.corebase.protobuf.generated.ClientMaster HOT 3
- Shielding abnormal broker should not be limited to IOException, any exception in the writing process should be managed. HOT 1
- IOException occurs in the broker and the number of abnormal brokers exceeded blocking threshold which will affect data writing. Unfortunately, consumers will timeout constantly. HOT 2
- I strongly suggest some command-line tools like kafka command-line tools HOT 1
- Should we refactor the front-end console page because it is too old, both in terms of technology and architecture HOT 6
- If the broker was blocked, it can't be restored to the disabled state after restarting the master. HOT 2
- Inconsistency in the default conf and docs HOT 1
- TubeMQ web show Chinese garbled HOT 1
- Can you support TubeMQ, provide SDKs in various languages, or maintenance tools? HOT 7
- Why copy file to SSD HOT 1
- TubeMQ connector for spark HOT 1
- There is a vulnerability in fastjson 1.2.60,upgrade recommended
- There is a vulnerability in netty 3.8.0.Final,upgrade recommended HOT 1
- 请问怎么实现的服务端过滤,可以大概说一下不
- 不维护了吗
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tubemq.