broker的HA方案是什么，有规划吗？ about tubemq HOT 8 OPEN

TubeMQ is suitable for high concurrency, massive data and tolerate a small amount of data loss scenarios under abnormal conditions.

From the actual application situation, the Broker encounters an abnormality that the disk fails under RAID10 and cannot be recovered, or the broker suddenly loses power and the memory has not been flushed, and the unconsumed data will be lost; if the disk failure can be recovered and the data is recovered The message has been swiped into the file, and the unconsumed message can be consumed after the Broker restarts.

We have encountered difficulties when considering multi-copy solutions: in the same scenario, multi-node asynchronous replication also has data loss. Only when multi-node synchronous replication is used can data be guaranteed, but this will directly reduce system throughput. At the same time, the cost of the solution is increased, so the TubeMQ implementation focuses on providing a high-performance and low-cost solution to tolerate a small amount of data loss.

---　---　---　---　---　---　---　---　---

TubeMQ适合高并发、海量数据且异常情况下容忍少量数据丢失场景。

从实际应用情况看，Broker遇到磁盘在RAID10下故障且无法恢复的异常，或者Broker突然断电处在内存还未刷盘宕机，未消费到的数据会丢失；如果磁盘故障能恢复且数据已刷盘到文件中，未消费的消息在Broker重启后能被继续消费。

我们在考虑多副本方案时遇到难题：同样场景下，多节点异步复制同样存在丢数据情况，只有在采用多节点同步复制时才能保证数据一条不丢，但这样做会直接降低了系统吞吐量同时增加了方案成本，所以TubeMQ实现侧重以容忍少量数据丢失为前提供高性能又低成本的方案。

from tubemq.

TubeMQ plans to provide different data service guarantees in the service classification mode. At present, the use of zk for offset persistence is also based on this consideration.

---　---　---　---　---　---　---　---　---

TubeMQ计划后续以服务分级方式提供不同的数据服务保证，目前offset持久化使用zk也是基于这点考虑暂未改造

from tubemq.

@gosonzhang 多副本同步复制或者异步复制，可以做成可配置，让具体业务去选择。

from tubemq.

Yes, we plan to provide multiple copies to allow the business to choose different data quality guarantees.

---　---　---　---　---　---　---　---　---

是的，我们计划提供多副本功能，让业务能选择不同的数据质量保证

from tubemq.

@gosonzhang Consider using hdfs as the storage layer? if one broker is down ,we can start the same broker at any other node immediately.

from tubemq.

If the hdfs nodes are outside the cluster, they are equivalent to adding additional machine costs. If they are in the cluster, the same machine resource will be shared by multiple copies.

－－－－－－－－－－－－－－－－－－－－－－－－－－

hdfs节点如果在集群之外，相当于是增加了额外机器成本，如果是集群内则相同机器情况下会为多副本分摊同份资源

from tubemq.

可以参考其他消息系统，新增一个生产者配置，设定级别，由使用者应用开发选择，比如针对准确性要求不高的可以异步或同步一个副本，针对金融对账等场景同步所有副本，在吞吐量和可靠性上做一些权衡处理

from tubemq.

from tubemq.