GME & the future? about go-mysql-elasticsearch HOT 5 CLOSED

Hi @RVN-BR

I know that ES will discard river feature, but this doesn't affect GME, because I don't use river feature at all.

GME is a service acting as a MySQL slave to sync MySQL binlog and then update the data changes into ES with ES http bulk API.

Btw, if you want to sync MySQL binlog and write the changes into logstash, you can use my another project github.com/siddontang/go-mysql, you can use the canal package to sync binlog and write the json format changes in a place which logstash can get.

from go-mysql-elasticsearch.

@siddontang Makes perfect sense! I didnt realize GME was using bulk api and not river api... I guess thats future-proof then :)

Thanks

from go-mysql-elasticsearch.

Hi @RVN-BR

Thank you reminding me, maybe we can build another repository named go-mysql-logstash, it will sync the mysql data and then write the json format in a place.

But I haven't used logstash, so I don't know the proper format the logstash needs.

from go-mysql-elasticsearch.

@siddontang It appears it shouldnt be an issue to configure from the format files in GME to get the appropriate mappings...

I'm now been investigating another interesting and promising project called Crate.io, built on top of ES, but which is more SQL-esque. It allows me to use ES but also to query some data in a structured fashion where it makes more sense. I'm looking into making a driver for GME for crate. It shouldnt be too difficult... crate has a bulk api too, it differs a bit in format from ES, but I'm going to poke around a bit of GO and see if I can eventually try something...

In theory, one can insert straight to ES, but Crate does its own internal type, id, etc mapping which may be difficult or impossible to reproduce statically, so it may be best to try to do it straight to the crate api... https://crate.io/docs/stable/sql/rest.html#bulk-operations

anyways, I am thinking crate may be useful... I'm thinking abt using it as a "bridging gap" for when some things need to be quickly shifted from mysql into a more search-esque backend :) GME might be a great tool to keep the data in sync. At the same time we can still use logstash and other great tools on ES, as well as having a running ES cluster... win win 👍

from go-mysql-elasticsearch.

Thanks @RVN-BR, I will look it later. :-)

from go-mysql-elasticsearch.