Comments (5)
Would you please post more details of your configuration?
from bireme.
# target database where the data will sync into.
target.url = jdbc:postgresql://gpmaster:1921/gpcluster
target.user = gpadmin
target.passwd = gpadmin
# data source name list, separated by comma.
data_source = maxwell1
# data source "mysql1" type
maxwell1.type = maxwell
# kafka server which maxwell write binlog into.
maxwell1.kafka.server = middle:9092
# kafka topic which maxwell write binlog into.
maxwell1.kafka.topic = testpay
# application performance monitor report type: "none", "console", "jmx"
metrics.reporter=console
# interval of console APM reporter.
metrics.reporter.console.interval = 10
# JDBC connection pool size of target database.
loader.conn_pool.size = 10
# queue size of task for each table which is waiting for load.
loader.task_queue.size = 2
# number of threads used to transform data source record into target format.
transform.thread_pool.size = 10
# number of threads used to generate load tasks.
merge.thread_pool.size = 10
# interval of generating a load task in milliseconds.
merge.interval = 10000
# max tuple size for a load task
merge.batch.size = 50000
from bireme.
Would please post some examples of Kafka consume record?
from bireme.
HI, kafka record like this :
{"database":"testpayment","table":"tbpaymentorder","type":"insert","ts":1505879683,"xid":9753352,"commit":true,"position":"mysql-bin-m.000014:626525214","data":{"PM_SEQ_NO":"20170920FT018880133","BIZ_PM_SEQ_NO":"2017092004018920133","RELATE_PM_SEQ_NO":null,"PM_TYPE":"FT","GMT_PM":"2017-09-20 03:55:44","GMT_MODIFIED":"2017-09-20 03:54:43","GMT_CREATE":"2017-09-20 03:54:43","MEMO":"2017092011552570735005004","PAY_CHANNEL":"01","BIZ_PRODUCT_CODE":"20201"}}
from bireme.
@aStonegod Currently, the column name of table must be the same in MySQL and GPDB.
from bireme.
Related Issues (20)
- 2.0.0版本启动一段时间就挂了 HOT 2
- 目标数据库支持mysql吗?应该怎么配置。
- 库表字段添加,库表新建,库表字段修改这类DML语言能自动同步吗? HOT 1
- kafka+Maxwell+bireme同步问题
- 开发环境配置 HOT 1
- 数据同步丢失问题
- jsvc error: Invalid option -cwd HOT 1
- Greenplum table and MySQL table size are inconsistent! HOT 1
- bireme可以多实例吗
- is kafka 0.9.0.1 support? HOT 2
- com.google.gson.stream.MalformedJsonException
- Greenplum table and MySQL table size are inconsistent!
- 当缓存数据达到一定量,将这些数据合并为一个任务 (Task)
- bire2.0支持mongo配置吗 HOT 5
- kafka数据中有转义符如何处理
- cn.hashdata.bireme.BiremeException: Transform failed.
- 怎样设置同步ddl,比如 alter table tbName add/drop columnNm... HOT 1
- postgres执行DELETE操作时brieme报错:cn.hashdata.bireme.BiremeException: Transform failed.
- Delete event cause failure
- Kafka重复消费问题
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bireme.