Redis indexation about node-eventstore HOT 9 OPEN

@TyGuy hi!

Sorry, but I couldn't take some time to start something on this. You can do it, if you have time for it.

The solution I suggested could be a good one, but as usual, it must first be validated on the field.

I hop you'll find more time than me 😅 Good luck !

from node-eventstore.

Sounds like a breaking change for the redis implementation... (have to write this at least in the release notes)
As far as I know your production data is the biggest for redis... so you're the expert ;-)

from node-eventstore.

I managed to make something work, but I need to do more tests.
I was also wondering if I should modify the existing redis implementation, or if I should apply Open Closed Principle and extend the existing implementation, and add a new database type (called indexedRedis ?), compatible with the existing implementation.
What do you think about it ?

from node-eventstore.

I think it's ok to replace the current implementation when we can proof the new one to be better in form of performance etc...

from node-eventstore.

Sure, but the store needs to be indexed, before any usage. It is done on init.
And it can take quite a long time (I've indexed my 122K+ events store in 170 sec on my Mac Book Pro 😅, because of the full scan made on all events), and might be done outside of the application (using a dedicated node task), depending on the size of the db.
So I was thinking about the best option not to break compatibility (or at least, here, not to cause an overhead when the application starts the first time with this option). It could be done using a dedicated option, or using another implementation.

from node-eventstore.

But is the already stored data compatible with the new implementation?
So really only this indexing task is what it's needed?
So perhaps the indexing in the init is ok... but we can offer a script to execute to do this indexing on an other process?

from node-eventstore.

Yes sure, it's fully compatible.
The events are not updated. Only some keys are added to better index events and snapshots.
But once you switch to an indexed redis store, the indexation is done. You can go back to a not indexed redis store, but you will have to do it manually (by removing a dedicated key).

What I can do, when you want to use an indexed redis store, is:

• checking if the database is already indexed (using a dedicated key). if not, you will have to do it manually using a call to an index() method on the store (maybe in a script).
• if you don't do the indexation, the common scan() method will be used, using SCAN redis command
• once the indexation is done, the scan() method will use indexes

Why not offering a script for this, but I was wondering how to discover all the options needed to connect to the redis instance. Maybe giving a common example in the documentation would be a first step forward.

You're OK with that?

from node-eventstore.

from node-eventstore.

@rehia do you still have interest in this, or plans to execute on it?

We are sitting with around 30k events in redis, and noticing it slowing down as well. I think this solution (indexing on aggregateId and using sorted sets) would be awesome!

If you don't have plans to continue it or work on it, would you be willing to share a WIP branch with your implementation? If/when I find some time, I could pick up where you left off, and carry this forward.

from node-eventstore.