Possible to synchronise changes between a remote database and Typesense Database about typesense HOT 4 CLOSED

@akbt There is currently no built-in synchronization mechanism. Is there a reason why you're doing a periodic dump of your data to a Typesense collection instead of propagating the changes in real-time? i.e. when you add a record to your primary database, add it to Typesense there and then (similarly for update and deletion)?

from typesense.

@kishorenc According to me (and correct me if I'm wrong) I'm afraid it might give rise to some inconsistencies & also I'll have to make changes to the primary server to update each collection. What happens when I have numerous collections?

I would like typesense to be more of a plug and play model where my main server should not know about typesense server.

from typesense.

Usually there is no 1:1 mapping between a DB table and a Typesense collection. The records in a typical table will be normalized and will have mostly id fields linking to other tables, while on Typesense you will generally write a denormalized object. For e.g. if you are storing a bunch of books, it's likely that that author fields in the DB table would be ids, while in Typesense you would want the actual author names so that they are searchable.

This makes a connector-like project not very useful in practical scenarios. That's one of the reasons why Elasticsearch discontinued their River plugin, and I see that mongo-connector is also deprecated and not supported by Mongo anymore.

If consistency is absolutely critical, my suggestion is to temporarily write all operations to an intermediate collection (essentially treating it as a queue) and have a background job to read from the queue and write to Typesense. Your temporary collection could have the following fields: timestamp, operation (insert/delete/update), data_to_write. This way, you can ensure that even if Typesense is down, you are not losing the synchronization.

In the future, this might be something we could support out of the box, and I think this is a far better model than trying to do a 1:1 mapping.

from typesense.

An intermediate collection does seem to be a good idea . Thanks for the suggestion.

from typesense.