No data synced about go-mysql-elasticsearch HOT 24 CLOSED

can you insert some data and then flush logs, the mysql binlog will be rotated and go-mysql-elasticsearch will log this.

from go-mysql-elasticsearch.

Hi @itarafath

Do you have a users table in test database?

from go-mysql-elasticsearch.

Yes..available.

from go-mysql-elasticsearch.

Thanks . It works . But sync only newly inserted data, existing data are missing.

from go-mysql-elasticsearch.

can you use select * from users to see the existing data which are not synced?

from go-mysql-elasticsearch.

it won't work. Where can I find MySQL dump? Below I attached screenshot

from go-mysql-elasticsearch.

seem that you don't install mysqldump, so go-mysql-elesticsearch can't sync the data which does not exist in binlog.

mysqldump is contained in MySQL, maybe you can install it? Sorry that I have no Windows computer so I don't know how to do it.

from go-mysql-elasticsearch.

MySQL dump is working fine. Is there any specific path for MySQL dump file.
Which folder should contain MySQL dump file? It should be SQL or CSV?

from go-mysql-elasticsearch.

We don't save dump file, just dump the data from MySQL and sync to ES with a Pipe.

from go-mysql-elasticsearch.

If I run mysqldump dbName it's working. Then what should be wrong?

from go-mysql-elasticsearch.

Delete the data directory set in data_dir configuration and restart the again. Hope this can solve your problem.

from go-mysql-elasticsearch.

No :( . Not worked.

from go-mysql-elasticsearch.

Can you show me the console output?

from go-mysql-elasticsearch.

from go-mysql-elasticsearch.

seem that you find the user with id 11?

from go-mysql-elasticsearch.

Yeah. That's newly inserted one. But existing data are missing.

from go-mysql-elasticsearch.

what is your existing data? can you show me an example, like selecting the result in MySQL and then in ES?

from go-mysql-elasticsearch.

from go-mysql-elasticsearch.

how about curl -XGET "http://localhost:9200/users/user/1" or curl -XGET "http://localhost:9200/users/user/11"

from go-mysql-elasticsearch.

from go-mysql-elasticsearch.

very strange. I will dive this issue later.
Can you show me the user table schema in MySQL.
What's your MySQL version, your ES version?

from go-mysql-elasticsearch.

MySQL : 5.6.21
ES : 5.0.0

Table Schema
CREATE TABLEa( idint(11) NOT NULL AUTO_INCREMENT, namevarchar(30) NOT NULL DEFAULT '', PRIMARY KEY (id) ) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=latin1;

from go-mysql-elasticsearch.

Hi @itarafath

I use your config and data to test in my Mac OSX. But to my surprised, it works ok.

1. Create the table a and insert (1, name1) and (2, name2).

2. Use reset master to force MySQL clear its binlog.

3. Start go-mysql-elasticsearch with your configuration.

   ➜  go-mysql-elasticsearch git:(master) ✗ ./bin/go-mysql-elasticsearch -config etc/river.toml
   2016/11/10 20:21:44 binlogsyncer.go:71: [info] create BinlogSyncer with config &{1001 mysql 127.0.0.1 3306 root   false false}
   2016/11/10 20:21:44 dump.go:108: [info] try dump MySQL and parse
   2016/11/10 20:21:44 status.go:52: [info] run status http server 127.0.0.1:12800
   mysqldump: [Warning] Using a password on the command line interface can be insecure.
   2016/11/10 20:21:45 dump.go:114: [info] dump MySQL and parse OK, use 1.29 seconds, start binlog replication at (mysql.000001, 154)
   2016/11/10 20:21:45 sync.go:17: [info] start sync binlog at (mysql.000001, 154)
   2016/11/10 20:21:45 binlogsyncer.go:226: [info] begin to sync binlog from position (mysql.000001, 154)
   2016/11/10 20:21:45 binlogsyncer.go:130: [info] register slave for master server 127.0.0.1:3306
   2016/11/10 20:21:45 binlogsyncer.go:552: [info] rotate to (mysql.000001, 154)
   2016/11/10 20:21:45 sync.go:53: [info] rotate binlog to (mysql.000001, 154)

4. Select in ES

   curl -XGET "http://localhost:9200/users/user/1"
   {"_index":"users","_type":"user","_id":"1","_version":1,"found":true,"_source":{"id":1,"name":"name1"}}

5. Insert (3, name3) and select in ES

   curl -XGET "http://localhost:9200/users/user/3"
   {"_index":"users","_type":"user","_id":"3","_version":1,"found":true,"_source":{"id":3,"name":"name3"}}

I use MySQL 5.7.16 and ES 5.0.0.
I don't know what is wrong in your test now......

from go-mysql-elasticsearch.

@siddontang Ok. I will try . Thanks for your support . 👍

from go-mysql-elasticsearch.