Git Product home page Git Product logo

ccr-syncer's Introduction

CCR Syncer

CCR(Cross Cluster Replication)也就是跨集群数据复制,能够在库/表级别将源集群的数据变更同步到目标集群,可用于提升在线服务的数据可用性、隔离在离线负载、建设两地三中心等。

原理

名词解释

源集群 (srcCluster):业务写入数据的集群
目标集群 (destCluster):跨集群复制的目标集群
binlog:源集群变更日志,记录了源集群的数据修改和操作,是目标集群数据重放和恢复的凭据
Syncer:一个轻量的CCR任务控制节点

架构说明

framework
Syncer从源集群批量获取库/表的binlog,并根据binlog中的信息在目标集群重放,从而实现数据的全量/增量复制。
如果binlog是数据变更,则通知目标集群从源集群拉取数据。
如果binlog是元数据变更,则在目标集群发起对应的操作。

使用说明

  1. 在fe.conf、be.conf中打开binlog feature配置项

    enable_feature_binlog = true
  2. 部署源、目标doris集群

  3. 部署Syncer

    git clone https://github.com/selectdb/ccr-syncer
    cd ccr-syncer
    
    # -j 开启多线程编译
    # --output指定输出的路径名称,默认名称为output
    bash build.sh <-j NUM_OF_THREAD> <--output SYNCER_OUTPUT_DIR>
    
    cd SYNCER_OUTPUT_DIR
    
    # 启动syncer,加上--daemon使syncer在后台运行
    bash bin/start_syncer.sh --daemon
    
    # 停止syncer
    bash bin/stop_syncer.sh

    更多选项相见启动配置

  4. 打开源集群中被同步库/表的binlog

    -- enable database binlog
    ALTER DATABASE db_name SET properties ("binlog.enable" = "true");
    
    -- enable table binlog
    ALTER TABLE table_name SET ("binlog.enable" = "true");

    如果是库同步,则需要打开库中所有表的binlog.enable,这个过程可以通过脚本快速完成,脚本的使用方法见脚本说明文档

  5. 向Syncer发起同步任务

    curl -X POST -H "Content-Type: application/json" -d '{
        "name": "ccr_test",
        "src": {
        "host": "localhost",
        "port": "9030",
        "thrift_port": "9020",
        "user": "root",
        "password": "",
        "database": "demo",
        "table": "example_tbl"
        },
        "dest": {
        "host": "localhost",
        "port": "9030",
        "thrift_port": "9020",
        "user": "root",
        "password": "",
        "database": "ccrt",
        "table": "copy"
        }
    }' http://127.0.0.1:9190/create_ccr
    • name: CCR同步任务的名称,唯一即可
    • host、port:对应集群master的host和mysql(jdbc) 的端口
    • thrift_port:对应FE的rpc_port
    • user、password:syncer以何种身份去开启事务、拉取数据等
    • database、table:
      • 如果是db级别的同步,则填入dbName,tableName为空
      • 如果是表级别同步,则需要填入dbName、tableName

    其他操作详见操作列表

ccr-syncer's People

Contributors

jackdrogon avatar deadlinefen avatar dataroaring avatar rfcube avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.