Comments (17)
I knew that we have no proper privilege to run mysqldump with --master-data flag in AWS RDS, so it is not allowed in AWS RDS.
from go-mysql-elasticsearch.
I had same issue when use go-mysql-elasticsearch with Amazon RDS MySQL server. managed to workaround it by following:
- use mysqldump to dump the RDS MySQL server and restore it to local MySQL db.
- shutdown RDS MySQL server
- start go-mysql-elasticsearch with
my_addr
set to local db and wait until it's finished - stop go-mysql-elasticsearch and delete var/master.info
- restart RDS MySQL server
- restart go-mysql-elasticsearch with
my_addr
to set to RDS MySQL server andmysqldump
to empty
it seems working properly, @siddontang any thoughts on this? thanks
from go-mysql-elasticsearch.
go-mysql-elasticsearch will try to use mysqldump to dump all data then sync with the position saved in mysqldump.
We can support starting from a binlog position to avoid using mysqldump.
Btw if the binlog is truncated, we may have not enough data to sync.
from go-mysql-elasticsearch.
@siddontang thanks for your reply.
the problem is as you posted above: "no proper privilege to run mysqldump with --master-data flag in AWS RDS, so it is not allowed in AWS RDS."
from what i tried yesterday, it seems still the case. is go-mysql-elasticsearch has way to avoid using --master-data flag to dump all the data?
starting from binlog position sounds good, but if i run show binary logs;
it only show 2 rows. i would expect there should be more rows to keep all the binlog files, does RDS delete obsolete binlog files?
from go-mysql-elasticsearch.
Hi @tpeng
go-mysql-elasticsearch doesn't support starting from a position now, but you can update the master.info directly to let it work.
I have another way if possible:
- Get the current binlog position with
show master status
- Dump the MySQl without
--master-data
- Sync from the got postion in step 1.
This can work well even you insert new data when do step 2 because binlog is row format, so any change synced to elasticsearch is idempotent.
from go-mysql-elasticsearch.
Hey @siddontang
just to clarity, the 3 steps above still requires dump the initial data from MySQL server (e.g. run mysqldump from command line), right?
from go-mysql-elasticsearch.
No, we can guarantee it.
from go-mysql-elasticsearch.
I will support this later if mysqldump with --master-data
failed. Maybe this week.
from go-mysql-elasticsearch.
thanks @siddontang, that would be great. but i wonder how is it going to work when RDS MySQL deleted expired binlog files?
from go-mysql-elasticsearch.
If the binlog files are deleted after we dumping the data, we may meet an error, sadly.
from go-mysql-elasticsearch.
let me know when you done with the change, i'm happy to test it!
thanks,
from go-mysql-elasticsearch.
Hi @tpeng
Can you help me to make sure we can run show master status
in RDS?
from go-mysql-elasticsearch.
from go-mysql-elasticsearch.
Hi @tpeng
I post a new branch siddontang/skip-master-data to fix this problem, see #116
Can you help me to check in RDS, you should set skip_master_data = true
in the config file then run.
from go-mysql-elasticsearch.
it seems working quite well. see the following logs:
...
2017/07/16 23:37:31 status.go:52: [info] run status http server 127.0.0.1:12800
2017/07/16 23:37:31 dump.go:114: [info] skip master data, get current binlog position (mysql-bin-changelog.023727, 422)
2017/07/16 23:37:31 dump.go:120: [info] try dump MySQL and parse
mysqldump: [Warning] Using a password on the command line interface can be insecure.
2017/07/16 23:39:52 dump.go:126: [info] dump MySQL and parse OK, use 140.72 seconds, start binlog replication at (mysql-bin-changelog.023727, 422)
2017/07/16 23:39:52 sync.go:21: [info] start sync binlog at (mysql-bin-changelog.023727, 422)
2017/07/16 23:39:52 binlogsyncer.go:268: [info] begin to sync binlog from position (mysql-bin-changelog.023727, 422)
...
thanks @siddontang
however seems to me using show master status
to get binlog position seems superior than run mysqldump with --master-data
, would it be better to always use it, so skip_master_data
can be avoid?
from go-mysql-elasticsearch.
Hi @tpeng
Skipping master data will sync duplicated data if we insert new data before dumping although this may be not a big problem for ES.
The old mechanism has been running for a long time and it works well, I will remove the config later. Btw, you can send me a PR to do this :-)
from go-mysql-elasticsearch.
I will close the issue, it is fixed now.
from go-mysql-elasticsearch.
Related Issues (20)
- es 7 使用 Parent-Child Relationship
- how can I sync mysql which in docker container to elasticsearch?
- binlog error on restart (invalid character 'R' after top-level value)
- 执行exe同步报错 invalid sequence 8!=1 HOT 2
- 多个子表聚合到一个index
- 能支持dockerfile
- 能否同步one-many嵌套表作为对象关联 到es中?需要怎么配置
- 无法实现同步
- 你好,同步报错如下:sync.go:156 do ES bulk err invalid character '<' looking for beginning of value, close sync HOT 1
- mysql8 mysqldump出错 HOT 1
- 阿里云/腾讯云 mysql to es WITH MAX_USER_CONNECTIONS 10000 HOT 1
- mysql,datetime字段同步到ES的date字段,会变成加8时区的格式 HOT 1
- How do I connect to an ES cluster HOT 1
- Startup error
- start canal err line 1 column 29 near "ON *.* FROM 'root'@'%'"
- go mod tidy HOT 1
- MySql int filed to ES text filed.
- panic: runtime error: index out of range [16] with length 16
- 同步es8需要删除client.go 文件的type字段检测 HOT 1
- sync.go:145 parse query(SAVEPOINT `trans2`) err line 1 column 9 near "SAVEPOINT `trans2`" , will skip this event 这个报错对数据有没有影响 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-mysql-elasticsearch.