ramezrafla / redis-oplog Goto Github PK

View Code? Open in Web Editor NEW

28.0 28.0 5.0 238 KB

Using Redis for pub / subs in Meteor for infinite scalability

License: MIT License

JavaScript 100.00%

redis-oplog's People

Contributors

Stargazers

Watchers

Forkers

weisheng-p ignl evolross ubesthelp fducom

redis-oplog's Issues

Trying to understand Watchers

Just want to wrap my head around this...

If I'm using two devices -- a desktop and a phone -- from the same user account, does that mean that updating something in device #1 will not automatically flow through to device #2 unless I use Watchers?

TypeError: this.mainCollection.cache is not a function

Im getting this error that might be caused when having cached links and using collection.startCaching()

W20201109-19:39:41.499(-5)? (STDERR) TypeError: this.mainCollection.cache is not a function
W20201109-19:39:41.499(-5)? (STDERR)     at Linker._initDenormalization (packages/cultofcoders:grapher/lib/links/linker.js:456:33)
W20201109-19:39:41.500(-5)? (STDERR)     at new Linker (packages/cultofcoders:grapher/lib/links/linker.js:28:14)
W20201109-19:39:41.500(-5)? (STDERR)     at packages/cultofcoders:grapher/lib/links/extension.js:23:28
W20201109-19:39:41.500(-5)? (STDERR)     at Function._.each._.forEach (packages/underscore.js:147:22)
W20201109-19:39:41.500(-5)? (STDERR)     at ns.Collection.addLinks (packages/cultofcoders:grapher/lib/links/extension.js:14:11)
W20201109-19:39:41.500(-5)? (STDERR)     at module (imports/api/assemblyAssistants/links.js:18:1)
W20201109-19:39:41.500(-5)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)
W20201109-19:39:41.500(-5)? (STDERR)     at Module.require (packages/modules-runtime.js:238:14)
W20201109-19:39:41.500(-5)? (STDERR)     at Module.moduleLink [as link] (/home/wanchope/.meteor/packages/modules/.0.16.0-beta200.4.1rhjm7w.g8iy++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/reify/lib/runtime/index.js:52:22)
W20201109-19:39:41.501(-5)? (STDERR)     at module (imports/startup/server/registerApi.js:1:1)
W20201109-19:39:41.501(-5)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)
W20201109-19:39:41.501(-5)? (STDERR)     at Module.require (packages/modules-runtime.js:238:14)
W20201109-19:39:41.501(-5)? (STDERR)     at Module.moduleLink [as link] (/home/wanchope/.meteor/packages/modules/.0.16.0-beta200.4.1rhjm7w.g8iy++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/reify/lib/runtime/index.js:52:22)
W20201109-19:39:41.501(-5)? (STDERR)     at module (imports/startup/server/index.js:3:1)
W20201109-19:39:41.503(-5)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)
W20201109-19:39:41.503(-5)? (STDERR)     at Module.require (packages/modules-runtime.js:238:14)
W20201109-19:39:41.504(-5)? (STDERR)     at require (packages/modules-runtime.js:258:21)
W20201109-19:39:41.504(-5)? (STDERR)     at /home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/app/app.js:75550:15
W20201109-19:39:41.505(-5)? (STDERR)     at /home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/boot.js:401:38
W20201109-19:39:41.505(-5)? (STDERR)     at Array.forEach (<anonymous>)
W20201109-19:39:41.505(-5)? (STDERR)     at /home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/boot.js:226:21
W20201109-19:39:41.505(-5)? (STDERR)     at /home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/boot.js:464:7
W20201109-19:39:41.505(-5)? (STDERR)     at Function.run (/home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/profile.js:280:14)
W20201109-19:39:41.505(-5)? (STDERR)     at /home/wanchope/Proyectos/propiedata/propiedata-meteor/.meteor/local/build/programs/server/boot.js:463:13

[Question] Memory usage

I am wondering how you got a lower memory usage with this solution, when it caches potentially a lot of documents? From what I can read in the original, it does not cache the docs in this manner, leading to more queries but lower memory usage. Is that correct?

Issue when using $unset on updates

I think there is probably an error in how $unset is handled, maybe because it has no value and now we are not doing a find after update so $unset updates are not reactive.

Assemblies.update({ _id: poll.assemblyId }, { $unset: { currentActivePollId: true } });

This piece of code is not reactive using this package, but it is when using cultofcoders one.

Will it work with multiple instances of redis?

Hello Ramez,

My client is running their Meteor app on multiple server communicating with a MongoDB replica set.

Will redis-oplog work if there is a separate instance of Redis on each server, or do all instances of Meteor need to communicate with the same instance of Redis?

Issue with cache and undefined selectors

When you dont pass a selector to a find or findOne it doesnt has the same behaviour as normal meteor collections and it will return no documents.

I think this is an issue with MiniMongo which you are using for caching. The workaround is to just use {} as the selector.

Collection.startCaching();
const docs = Collections.find(); // No documents
const docs2 = Collections.find({}); // n documents in collection

Infinity scroll pagination question

What I get with this version of redis-oplog is that when I scroll to the next page and reactively update limit parameter subscription is stopped and resubscribed which causes all records to be removed and re-added instead of just adding records for new page. Is it expected?

Behaviour in fields on find is different than in Mongo Collection

When you set a projection (fields) option in a find or findOne query having the following schema and projecting the parent field the nested fields wont be fetched.

// Schema
{
  _id: String,
  someObject: {
    _id: String,
    other: Number
  }
}

// Mongo Collection find
Collection..findOne({}, { fields: { someObject: true } }); // someObject contains _id and other fields

// Cached collection
CachedCollection.findOne({}, { fields: { someObject: true } }): // someObject is empty
CachedCollection.findOne({}, { fields: { someObject: { _id: true, other: true } } }): // someObject contains _id and other fields

Lacking 'transform' support in find options

When trying to integrate this package with jagi:astronomy, I encountered an issue that this fork, when retrieving from the cache, it does not apply the transform function which is an optional function passed to find and findOne. This makes the package not fully compatible with the official docs, or with Astronomy which relies on transforms.

https://docs.meteor.com/api/collections.html#Mongo-Collection-find

reactivity issue

Hi,

we faced an issue when using this package in our blazed based, multi-service project.
When updating the user profile from the client side (hence using minimongo), we found no reactivity.
We fall back on /cult-of-coders/redis-oplog which seems to fix our main issue though (cpu issues).
Is updating from minimongo a supported use case? This seems related to race condition detection as increasing it to high value seems to resolve a bit the issue, but with 10s delay, this won't cut it for our users experience.
If you need more information, don't hesitate.
regards,
Hervé

collection.startCaching() on Meteor.users

Should I enable collection.startCaching() on Meteor.users? A bit unclear on this.

Document of what is being published to redis

I'm the middle of writing a changestream to redis-oplog nodejs process, and decided to go with this fork.
I noticed that there is a bit of change on the data that is being published to redis compared to the original fork.
Do you have a full document of what is being published?

TypeError: Collection.startCaching is not a function

I understand this might be an issue with the extending of mongo collection, but I have zegenie:redis-oplog as my first package and still have the issue.

Is there something that need to be setup for the caching to work?

Error: Unrecognized logical operator: $text

Here's the error:

33jma
2021-02-11 19:40:47+02:00Exception while invoking method 'getResetPasswordEmail' Error: Unrecognized logical operator: $text33jma
2021-02-11 19:40:47+02:00 at packages/minimongo/common.js:579:1533jma
2021-02-11 19:40:47+02:00 at Array.map (<anonymous>)33jma
2021-02-11 19:40:47+02:00 at compileDocumentSelector (packages/minimongo/common.js:572:48)33jma
2021-02-11 19:40:47+02:00 at packages/minimongo/common.js:560:1233jma
2021-02-11 19:40:47+02:00 at Array.map (<anonymous>)33jma
2021-02-11 19:40:47+02:00 at compileArrayOfDocumentSelectors (packages/minimongo/common.js:555:20)33jma
2021-02-11 19:40:47+02:00 at Object.$and (packages/minimongo/common.js:260:7)33jma
2021-02-11 19:40:47+02:00 at packages/minimongo/common.js:583:1433jma
2021-02-11 19:40:47+02:00 at Array.map (<anonymous>)33jma
2021-02-11 19:40:47+02:00 at compileDocumentSelector (packages/minimongo/common.js:572:48)33jma
2021-02-11 19:40:47+02:00 at Matcher._compileSelector (packages/minimongo/matcher.js:116:12)33jma
2021-02-11 19:40:47+02:00 at new Matcher (packages/minimongo/matcher.js:52:29)33jma
2021-02-11 19:40:47+02:00 at hook.findOne (packages/zegenie:redis-oplog/lib/mongo/extendMongoCollection.js:154:27)33jma
2021-02-11 19:40:47+02:00 at MethodInvocation.getResetPasswordEmail (server/methods/users.jsx:534:37)33jma
2021-02-11 19:40:47+02:00 at MethodInvocation.methodMap.<computed> (packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:161:30)33jma
2021-02-11 19:40:47+02:00 at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)33jma
2021-02-11 19:40:47+02:00 at packages/ddp-server/livedata_server.js:719:1933jma
2021-02-11 19:40:47+02:00 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)33jma
2021-02-11 19:40:47+02:00 at packages/ddp-server/livedata_server.js:717:4633jma
2021-02-11 19:40:47+02:00 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)33jma
2021-02-11 19:40:47+02:00 at packages/ddp-server/livedata_server.js:715:4633jma
2021-02-11 19:40:47+02:00 at new Promise (<anonymous>)33jma
2021-02-11 19:40:47+02:00 at Session.method (packages/ddp-server/livedata_server.js:689:23)33jma
2021-02-11 19:40:47+02:00 at packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:52:3833jma
2021-02-11 19:40:47+02:00 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)33jma
2021-02-11 19:40:47+02:00 at Session.sessionProto.protocol_handlers.method (packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:51:44)33jma
2021-02-11 19:40:47+02:00 at packages/ddp-server/livedata_server.js:559:43

And here's the code firing that error:

Meteor.users.findOne({
  $and:[
    {$text:{$search:string}},
    {$or:[
      {'username':{$regex:value, $options: 'i'}},
      {'profile.name':{$regex:value, $options: 'i'}},
      {'registered_emails.address':{$regex:value, $options: 'i'}},
     ]}
  ]
})

Minimongo doesn't support operators in projections yet

Another minimongo one. Here's the offending code:

Chats.find(query,{
  fields:{messages:{$slice:-limit}}, // slice is the problem
  limit:limit,
  sort:sort,
}).fetch()

And logs.

Exception while invoking method 'getChats' MinimongoError: Minimongo doesn't support operators in projections yet.
I20210211-21:28:23.585(2)?     at MinimongoError (packages/minimongo/common.js:1087:17)
I20210211-21:28:23.585(2)?     at packages/minimongo/local_collection.js:847:13
I20210211-21:28:23.585(2)?     at Array.forEach (<anonymous>)
I20210211-21:28:23.586(2)?     at Function.LocalCollection._checkSupportedProjection (packages/minimongo/local_collection.js:834:23)
I20210211-21:28:23.586(2)?     at Function.LocalCollection._compileProjection (packages/minimongo/local_collection.js:868:19)
I20210211-21:28:23.586(2)?     at Mongo.Collection.find (packages/zegenie:redis-oplog/lib/mongo/extendMongoCollection.js:213:62)
I20210211-21:28:23.586(2)?     at MethodInvocation.getChats (server/methods/chats.jsx:14:31)
I20210211-21:28:23.586(2)?     at MethodInvocation.methodMap.<computed> (packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:161:30)
I20210211-21:28:23.586(2)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)
I20210211-21:28:23.586(2)?     at packages/ddp-server/livedata_server.js:719:19
I20210211-21:28:23.586(2)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
I20210211-21:28:23.586(2)?     at packages/ddp-server/livedata_server.js:717:46
I20210211-21:28:23.586(2)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
I20210211-21:28:23.586(2)?     at packages/ddp-server/livedata_server.js:715:46
I20210211-21:28:23.586(2)?     at new Promise (<anonymous>)
I20210211-21:28:23.586(2)?     at Session.method (packages/ddp-server/livedata_server.js:689:23)
I20210211-21:28:23.586(2)?     at packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:52:38
I20210211-21:28:23.586(2)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
I20210211-21:28:23.586(2)?     at Session.sessionProto.protocol_handlers.method (packages/mdg:meteor-apm-agent/lib/hijack/wrap_session.js:51:44)
I20210211-21:28:23.587(2)?     at packages/ddp-server/livedata_server.js:559:43

MongoError: cursor id not found

This one's a bit hard to wrap my head around, but maybe you'll see something obvious here. When I deployed with this package everything hit 98%. I was able to refresh once or twice and get some publications to load, but most of the time I just got a spinner (which would imply the publications weren't returning properly).

2021-02-11 19:38:25+02:00Exception in setInterval callback: MongoError: cursor id 184180801930 not found3m582
2021-02-11 19:38:25+02:00 at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61)3m582
2021-02-11 19:38:25+02:00 at Connection.emit (events.js:314:20)3m582
2021-02-11 19:38:25+02:00 at Connection.EventEmitter.emit (domain.js:483:12)3m582
2021-02-11 19:38:25+02:00 at processMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10)3m582
2021-02-11 19:38:25+02:00 at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15)3m582
2021-02-11 19:38:25+02:00 at TLSSocket.emit (events.js:314:20)3m582
2021-02-11 19:38:25+02:00 at TLSSocket.EventEmitter.emit (domain.js:483:12)3m582
2021-02-11 19:38:25+02:00 at addChunk (_stream_readable.js:297:12)3m582
2021-02-11 19:38:25+02:00 at readableAddChunk (_stream_readable.js:272:9)3m582
2021-02-11 19:38:25+02:00 at TLSSocket.Readable.push (_stream_readable.js:213:10)3m582
2021-02-11 19:38:25+02:00 at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)3m582
2021-02-11 19:38:25+02:00 => awaited here:3m582
2021-02-11 19:38:25+02:00 at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)3m582
2021-02-11 19:38:25+02:00 at packages/mongo/mongo_driver.js:1073:143m582
2021-02-11 19:38:25+02:00 at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:403m582
2021-02-11 19:38:25+02:00 => awaited here:3m582
2021-02-11 19:38:25+02:00 at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)3m582
2021-02-11 19:38:25+02:00 at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1122:38)3m582
2021-02-11 19:38:25+02:00 at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1136:22)3m582
2021-02-11 19:38:25+02:00 at SynchronousCursor.map (packages/mongo/mongo_driver.js:1146:10)3m582
2021-02-11 19:38:25+02:00 at SynchronousCursor.fetch (packages/mongo/mongo_driver.js:1170:17)3m582
2021-02-11 19:38:25+02:00 at Cursor.<computed> (packages/mongo/mongo_driver.js:918:44)3m582
2021-02-11 19:38:25+02:00 at Cursor.cursorProto.<computed> (packages/mdg:meteor-apm-agent/lib/hijack/db.js:131:32)3m582
2021-02-11 19:38:25+02:00 at Cursor.kadira_Cursor_fetch [as fetch] (packages/mdg:meteor-apm-agent/lib/hijack/set_labels.js:72:32)3m582
2021-02-11 19:38:25+02:00 at Cursor.cursor.fetch (packages/zegenie:redis-oplog/lib/mongo/extendMongoCollection.js:256:51)3m582
2021-02-11 19:38:25+02:00 at Cursor.cursor.count (packages/zegenie:redis-oplog/lib/mongo/extendMongoCollection.js:258:39)3m582
2021-02-11 19:38:25+02:00 at packages/natestrauser_publish-performant-counts.js:38:293m582
2021-02-11 19:38:25+02:00 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)3m582
2021-02-11 19:38:25+02:00 at packages/meteor.js:550:253m582
2021-02-11 19:38:25+02:00 at runWithEnvironment (packages/meteor.js:1286:24)

The positional operator did not find the match needed from the query

When using this package im getting this error. If I roll back to the master parent repo I dont get the issue.

I think I get this error on the following update operation

Polls.update({ _id: object.pollId, 'answers._id': object.answerId }, {
      $inc: {
        'answers.$.votesCount': votesCount,
        'answers.$.coeficientTotal': coeficientTotal
      }
    });

message: The positional operator did not find the match needed from the query.
stack:
MongoError: The positional operator did not find the match needed from the query.
    at Function.create (/var/app/current/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/error.js:57:12)
    at toError (/var/app/current/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:123:22)
    at /var/app/current/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/common_functions.js:379:39
    at handler (/var/app/current/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset.js:1204:22)
    at /var/app/current/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:405:18
    at processTicksAndRejections (internal/process/task_queues.js:79:11)

MinimongoError: The positional operator did not find the match needed from the query

Here's the code throwing this error:

        Meteor.defer(() => {
            Campaigns.update(
                {_id:nowPlaying,detail:{$elemMatch:detail}},
                {$set:{'detail.$.listened_for':length}}
            )
        })

This is executed server-side, not on client.

Here's the full log:

gxbsm
2021-02-11 19:40:53+02:00Exception in defer callback: MinimongoError: The positional operator did not find the match needed from the querygxbsm
2021-02-11 19:40:53+02:00 at MinimongoError (packages/minimongo/common.js:1087:17)gxbsm
2021-02-11 19:40:53+02:00 at findModTarget (packages/minimongo/local_collection.js:1947:17)gxbsm
2021-02-11 19:40:53+02:00 at packages/minimongo/local_collection.js:1160:24gxbsm
2021-02-11 19:40:53+02:00 at Array.forEach (<anonymous>)gxbsm
2021-02-11 19:40:53+02:00 at packages/minimongo/local_collection.js:1144:28gxbsm
2021-02-11 19:40:53+02:00 at Array.forEach (<anonymous>)gxbsm
2021-02-11 19:40:53+02:00 at Function.LocalCollection._modify (packages/minimongo/local_collection.js:1134:27)gxbsm
2021-02-11 19:40:53+02:00 at packages/zegenie:redis-oplog/lib/mongo/mutator.js:119:27gxbsm
2021-02-11 19:40:53+02:00 at Array.forEach (<anonymous>)gxbsm
2021-02-11 19:40:53+02:00 at Mongo.Collection.update (packages/zegenie:redis-oplog/lib/mongo/mutator.js:117:14)gxbsm
2021-02-11 19:40:53+02:00 at Mongo.Collection.update (packages/zegenie:redis-oplog/lib/mongo/extendMongoCollection.js:60:29)gxbsm
2021-02-11 19:40:53+02:00 at server/methods/listens.jsx:129:23gxbsm
2021-02-11 19:40:53+02:00 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)gxbsm
2021-02-11 19:40:53+02:00 at packages/meteor.js:550:25gxbsm
2021-02-11 19:40:53+02:00 at runWithEnvironment (packages/meteor.js:1286:24)

Any hints about how to integrate Redis oplog with Elastic Beanstalk?

Hey, I have successfully implemented redis-oplog using mup and mup-redis. Works just fine, so thanks for the great work.

I now also want to scale things horizontally, so I am looking into implementing Elastic Beanstalk using mup-aws-beanstalk. However, it seems as if this combination is not supported by mup.

I understand that running multiple Redis servers across the cluster won't work, so I attempted to set-up a single EC2 instance just for Redis, in addition to the beanstalk configuration. However, mup-redis tells me that only one server is supported. It seems as if it also looked at the beanstalk config and saw multiple instances there. This is although I configured the Redis part separatly, with onle one server.

So, if mup won't support it, I would have to find a tutorial on how to set things up manually, which I haven't done before. So far, I always relied on mup and was pretty happy with it. Any hints about how to setup such a scenario (redis-oplog plus beanstalk) would be appreciated.

oplogtoredis support

Hi Ramez - another question:

I note this fork of your fork that claims to have implemented a "fix to work with oplogtoredis".

https://github.com/crisis24/redis-oplog/commit/fc77c9bf7125d8cc6b11980be643eab7cdd1b836

This sounds positive. Do you have any opinions about the modifications in this commit?

findOne query on server collections with startCaching() returns doc when it shouldn't

On the Mongo console:

meteor:PRIMARY> db.users.findOne({_id:"ZqRoBNapiGcQTtTBT", nonExistentField: "nonExistentValue"})
null

Which is expected. But in the Meteor shell and in server side app code:

> Meteor.users.findOne({_id:"ZqRoBNapiGcQTtTBT", nonExistentField: "nonExistentValue"})
{
  _id: 'ZqRoBNapiGcQTtTBT',
  ...
}

Which shouldn't be happening. This only happens on collections that have startCaching() on (e.g. Meteor.users.startCaching()) in the server code.

As long as there's an _id that matches, it doesn't matter what other fields there are, the findOne method returns the document. This is not the case for find, which works as expected.

This unexpected behaviour of the findOne method only happens on the server for collections with caching started; it works as expected on the client.