databendcloud / flink-connector-databend Goto Github PK

Because MongoDB using _id as primary key, it it String type and will get error in databend.

Now: DELETE from target_table where _id=xxxxx.
Expected: DELETE from target_table where id='xxxxx'

feature description :

such as we have these tables that need to be synchronized.
mysql tables
a_0
a_1
b_0
b_1
c

databend tables
a
b
c

We can add two parameters to achieve this effect.
--including-tables="a_.|b_.|c"
--multi-to-one-origin="a_.|b_."
--multi-to-one-target="a|b"

Implementation principle :
1.Create only one table
2.Multiple tables to one side output stream

Test sync data from MongoDB using MongoDB flink connector and databend flink connector.

There are many users who want this function, so we need to support delete in CDC, otherwise the connector is not a complete CDC plan.

Flink CDC has released 3.0.0 and this version has many enhancements such as schema evolution, Flink CDC Cli, database sync etc. More details in https://mp.weixin.qq.com/s/rkjIK2UH_IeC5QIwHw_4fw.

So we plan to support flink-connector -databend to integrate with 3.0.

• Support full-database synchronization.

• The table-name also need to support regular expressions to monitor multiple tables that satisfy the regular expressions.

There are many tables with string primary key, now delete statement of this kind of table is not supported by flink databend sink.

After the entire database synchronization defaults to one table and one sink, and tables cannot be automatically created. So we need to add that when use single sink, the table is automatically created without restarting the flink task.