How to get started? about mongoshake HOT 19 CLOSED

Please be careful if you want to build a two-way sync tunnel, the data may be moved from the source to the target, and then from the target to the source again.
Check out this question(How to build active-active replication(双活) in the current opensource version without gid support?) in the faq to break the loop.

from mongoshake.

1. No. If you use MongoShake in two-way sync, the consistency can't be guaranteed.
2. 9100 port is only used when running incr-sync stage, so if your MongoShake is running full-sync stage, you should use 9101 port, however, mongoshake-stat doesn't support 9101. You can curl the 9101 port directly to get the monitor data by: curl 127.0.0.1:9101.

from mongoshake.

Currently, we don't have the tutorial. You can check out the usage to start MongoShake, if you want to set up with detail information, read the comment of configuration and the wiki faq

from mongoshake.

O.K. when I have a few questions about it:

• How do I run the packages file on mongo-shake/src/vendor?
• Is there also a script to install govendor and is it correct to set the path to GOPATH=pwd/../..;?
• Is there an example config for a two way sync of mongodb between two instances? I'm looking at the two config files and I'm having a hard time figuring out which options I need to modify.

from mongoshake.

1. You should install govendor first and then use this to download all dependency packages which locate on mongo-shake/src/vendor.
2. govendor is an opensource tool, GOPATH should be set before using. Generally speaking, this should point to the MongoShake root path.
3. Currently, no. You can try to build one-way sync first, after that, two-way sync is simple.

from mongoshake.

I read the How to build active-active replication in the FAQ but it sounds like that active-active is not supported since I need to define a one way sync on collection level. Or do I miss something?

from mongoshake.

Yes, the granularity of two-way syncing is collection level, so that you should filter some database/collection on one side, and filter the other db/collections on the other side.
It is because mongodb doesn't support gid, so it's the only way to prevent data replication from forming a loop.

from mongoshake.

So basically that means I can't keep a collection in sync if I make changes on both servers to this collection?

from mongoshake.

yes

from mongoshake.

That's unfortunate. Then I need to look for another solution. Thank you for the clarification.

from mongoshake.

That's unfortunate. Then I need to look for another solution. Thank you for the clarification.

Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?

from mongoshake.

That's unfortunate. Then I need to look for another solution. Thank you for the clarification.

Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?

We plan to build an app that connects to a remote instance, listens to the events and publishes them in the local instance. This gives us also the benefit to filter what we like to keep in sync.

from mongoshake.

That's unfortunate. Then I need to look for another solution. Thank you for the clarification.

Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?

We plan to build an app that connects to a remote instance, listens to the events and publishes them in the local instance. This gives us also the benefit to filter what we like to keep in sync.

So you are developing the sync app, right?

Have you checked https://github.com/yougov/mongo-connector, our team prefers to use mongo-connector, but I need to figure out the feasibility of these tools.
I would appreciate it if you could give any advise.

from mongoshake.

I would appreciate it if you could give any advise.

We have the development of the tool in the backlog. So far I can't help u with expertise. Depending on the prioritization I can answer your questions in a couple of weeks.

from mongoshake.

I would appreciate it if you could give any advise.

We have the development of the tool in the backlog. So far I can't help u with expertise. Depending on the prioritization I can answer your questions in a couple of weeks.

OK I got it. Thanks a lot.

from mongoshake.

hi @vinllen, I am start using MongoShake. can I have some question about lastest version release-v2.4.9-20200806:

1. Is this version support two-way sync tunnel without using GID? I have 2 replicaset clusters, each have 3 node: one primary and two secondary, and 2 MongoShake to sync data between them. I write data to one cluster in one time (active-standby) and I saw that MongoShake run properly.
2. how to use mongoshake-stat to monitor? I using command: ./mongoshake-stat --port=9100, but failed with error: ValueError: No JSON object could be decoded.
   Thanks a lot.

from mongoshake.

1. No. If you use MongoShake in two-way sync, the consistency can't be guaranteed.
2. 9100 port is only used when running incr-sync stage, so if your MongoShake is running full-sync stage, you should use 9101 port, however, mongoshake-stat doesn't support 9101. You can curl the 9101 port directly to get the monitor data by: curl 127.0.0.1:9101.

1. Note that I write data to only one cluster in one time, mean active-standby, not active-active. Assuming that I have 2 clusters A and B, mongoshake 1 replicates data from A to B and mongoshake 2 replicates data from B to A. I insert 10 document to A, then B has 10 document replicated from A by mongoshake 1, and no data replicated back from B to A by mongoshake 2. No infinite data replication loop occur as you mention in GID feature. So why consistency can't be guaranteed? Do I miss something? Please explain to me.

from mongoshake.

You can't break the sync loop without GID supported. Checkout this scenario, I just update {_id:1, a:1} to {_id:1, a:2} and generate an oplog: {o2:{_id:1}, o:{_id:1, a:2}}, when this oplog sync from A to the B, the data with {_id:1} will be updated and generate another oplog: {o2:{_id:1}, o:{_id:1, a:2}}. So once this oplog synced back to A, the {_id:1} will be updated again even the result is not changed. So there will be a loop: A->B->A->B->...
As a result, when you update {_id:1} again to {_id:1, a:3}, the result of {_id:1} on A and B will switch back and forth between {a:2} and {a:3}.

from mongoshake.

Please open a new issue if anyone has a question, this issue should be frozen.

from mongoshake.