Git Product home page Git Product logo

meteor-mongodb-mapreduce-aggregation's People

Contributors

jhoxray avatar lalomartins avatar vraptor75011 avatar zvictor avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

laboratory gopidon tsega simonsimcity soaresa

meteor-mongodb-mapreduce-aggregation's Issues

Cannot read property 'collectionName' of undefined

A quick background to understand the situation:
A project of ours uses the mapreduce available in MongoDB through this package.
Until now, we only had the issue when we attempted a mapreduce on an empty collection...

Since then, I have deployed the project bundle on its own server with its own instance of MongoDB with replica sets. The project bundle is launched using node as is explained in the documentation.

Our application generates 6-7 events every 2 minutes on schedule. These events are stored in a collection called events updated by triggerJob. At the end of the triggers loop, a map reduce is ran. It acts on the collection of events and replaces the collection called eventStats.

The bug occurs only after a while; approximatly 30 events or so. I suspect it might be due to the replacement of the out collection.

// scheduler calls:
jobs.forEach(function(element, index, array) {
  triggerJob(element, null);
});
Meteor.call("updateEventStats", function(err, data) {
  if (err) debug("updateEventStats error: " + err);
});

// updateEventStats def:
Meteor.methods({
  updateEventStats: function () {
    if (0 < Events.find().count()) {
      Events.mapReduce(map, reduce, {
        query: {},
        out: {replace: "eventStats"},
        finalize: finalize,
        scope: {quartileValue: quartileValue}
      });
    }
  }
});

I get the following error in the meteor application:

.../app/programs/server/packages/monbro_mongodb-mapreduce-aggregation.js:57
      collectionName: result.collectionName,
                            ^
TypeError: Cannot read property 'collectionName' of undefined
    at packages/monbro:mongodb-mapreduce-aggregation/server.coffee:42:36
    at .../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/collection/aggregation.js:86:20
    at [object Object].<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/db.js:1903:9)
    at [object Object].g (events.js:199:16)
    at [object Object].emit (events.js:110:17)
    at [object Object].Base.__executeAllServerSpecificErrorCallbacks (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/base.js:373:29)
    at Server.<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/repl_set/repl_set.js:347:14)
    at Server.emit (events.js:110:17)
    at [object Object].<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/server.js:609:84)
    at [object Object].emit (events.js:104:17)

And it actually kills the instance of MongoDB:

2015-09-16T14:05:10.660+0200 F -        [conn2] Invalid access at address: 0
2015-09-16T14:05:10.671+0200 F -        [conn2] Got signal: 11 (Segmentation fault).

 0x1150d6b 0x1150523 0x1150907 0x7ffad0e80d60 0x11ec2a6 0x10a2dad 0x109553a 0x109a76b 0x109a7d9 0x108ceea 0x109421a 0x10947a9 0x9c3804 0x9c70d7 0xa4550c 0xa466c2 0xa475
79 0xc8fa87 0xb6c4b9 0x7d3579 0x10f65e5 0x7ffad0e774a4 0x7ffacf80913d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"D50D6B"},{"b":"400000","o":"D50523"},{"b":"400000","o":"D50907"},{"b":"7FFAD0E70000","o":"10D60"},{"b":"400000","o":"DEC2A6"},{"b":"400
000","o":"CA2DAD"},{"b":"400000","o":"C9553A"},{"b":"400000","o":"C9A76B"},{"b":"400000","o":"C9A7D9"},{"b":"400000","o":"C8CEEA"},{"b":"400000","o":"C9421A"},{"b":"400000","o":"C947A9"},{"b":"400000","o":"5C3804"},{"b":"400000","o":"5C70D7"},{"b":"400000","o":"64550C"},{"b":"400000","o":"6466C2"},{"b":"400000","o":"647579"},{"b":"400000","o":"88FA87"},{"b":"400000","o":"76C4B9"},{"b":"400000","o":"3D3579"},{"b":"400000","o":"CF65E5"},{"b":"7FFAD0E70000","o":"74A4"},{"b":"7FFACF720000","o":"E913D"}]
,"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.7-2-ARCH", "version" : "#1 SMP PREEMPT Tue
 Jun 30 07:50:21 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D7EE62E8D9A44AA66F8BEE93EA5C281EA42BB95A" }, { "b" : "7FFE
1774A000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "A320F98DE311139B6ABEFFB2A7109994EDA4CB5D" }, { "b" : "7FFAD1F90000", "path" : "/usr/lib/libpcre.so.1"
, "elfType" : 3, "buildId" : "88D40283EB5177DCE76EAAD3B6E8A8B2EA6BB66E" }, { "b" : "7FFAD1D80000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "1AC3
A716BAF09BC2191EA2B75EE6CE8CE36C3433" }, { "b" : "7FFAD1B78000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "1717DA41B48DD7305D95423F7B181961BD9662D
A" }, { "b" : "7FFAD18F8000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "EFB841BBCA7F10BE4B3ECBBA7E43E0593ACD5515" }, { "b" : "
7FFAD16E0000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "7939ED3E6D025517862CB78D23BD7C4BFD1E8918" }, { "b" : "7FFAD14B8000", "path
" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "3033771E63857EF953278CEDA804C3493E896836" }, { "b" : "7FFAD12B0000", "path" : "/usr/lib/libboost_s
ystem.so.1.58.0", "elfType" : 3, "buildId" : "250CA1E0DC473E1E54BEA3E13B28ED550F7206F4" }, { "b" : "7FFAD1090000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "bu
ildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7FFAD0E70000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "284BF58119A6255AFBA45689
40CE655ECDFF1881" }, { "b" : "7FFAD0BF0000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "4E6FE13AD63FE3F0F3279F9BE077586E2BADF443" }, { "b" : "7FFA
D0778000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E72E20F8FD3FB8B2C3EC7ADB5956E99D16AF15A4" }, { "b" : "7FFAD0570000", "path" : "/usr/lib/l
ibrt.so.1", "elfType" : 3, "buildId" : "6C39F2195A3E5D3A6A1FAEF570D0B83AF2184E3D" }, { "b" : "7FFAD0368000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : 
"4C55749EE719F7CE38F5078CE3197091D24EFEDE" }, { "b" : "7FFACFFE0000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "6C6DEA88072C530D0FC9F347CF115B722C
8BE8BF" }, { "b" : "7FFACFCE0000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "1C20A8053268B03D81D85235D26B14F4D24DBC40" }, { "b" : "7FFACFAC8000", "path
" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "F4AD36B00DF033915EEEB0CD89B9C5F66AF864B3" }, { "b" : "7FFACF720000", "path" : "/usr/lib/libc.so.6", "elfType" 
: 3, "buildId" : "34701C7AA9501F113595BA20117D03FBAB4A7EDF" }, { "b" : "7FFAD2200000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B3B6F65FD2224
CB6D7FD34FCBEC82CDEE0F62A8B" }, { "b" : "7FFACF508000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "544EDFD9674594504CC408B41C1AAC2F3CF9ED63" }, { "
b" : "7FFACF2F0000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "82A1A5B6D7F272E064155BB64576230AEFAFA7B8" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x2B) [0x1150d6b]
 mongod(+0xD50523) [0x1150523]
 mongod(+0xD50907) [0x1150907]
 libpthread.so.0(+0x10D60) [0x7ffad0e80d60]
 mongod(_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x16) [0x11ec2a6]
 mongod(_ZN5boost6detail17sp_counted_impl_pIN5mongo10BSONHolderEE7disposeEv+0x3D) [0x10a2dad]
 mongod(+0xC9553A) [0x109553a]
 mongod(_ZN5mongo7V8ScopeD1Ev+0x34B) [0x109a76b]
 mongod(_ZN5mongo7V8ScopeD0Ev+0x9) [0x109a7d9]
 mongod(+0xC8CEEA) [0x108ceea]
 mongod(_ZN5mongo11PooledScopeD1Ev+0x81A) [0x109421a]
 mongod(_ZN5mongo11PooledScopeD0Ev+0x9) [0x10947a9]
 mongod(_ZN5mongo2mr5StateD1Ev+0x104) [0x9c3804]
 mongod(_ZN5mongo2mr16MapReduceCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x10F7) [0x9c70d7]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2C) [0xa4550c]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xF22) [0xa466c2]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x529) [0xa47579]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_+0x677) [0xc8fa87]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xA69) [0xb6c4b9]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xD9) [0x7d3579]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0x10f65e5]
 libpthread.so.0(+0x74A4) [0x7ffad0e774a4]
 libc.so.6(clone+0x6D) [0x7ffacf80913d]
-----  END BACKTRACE  -----

A method named 'XXX' is already defined

The .mapReduce works great on first call, but on subsequent calls I get the following error:

I20150608-11:11:05.294(-4)? Exception while invoking method 'exportDonations' Error: A method named '/donation_totals_by_donor/insert' is already defined
I20150608-11:11:05.294(-4)?     at packages/ddp/livedata_server.js:1461:1
I20150608-11:11:05.294(-4)?     at Function._.each._.forEach (packages/underscore/underscore.js:113:1)
I20150608-11:11:05.294(-4)?     at [object Object]._.extend.methods (packages/ddp/livedata_server.js:1459:1)
I20150608-11:11:05.294(-4)?     at [object Object].Mongo.Collection._defineMutationMethods (packages/mongo/collection.js:904:1)
I20150608-11:11:05.294(-4)?     at new Mongo.Collection (packages/mongo/collection.js:209:1)
I20150608-11:11:05.295(-4)?     at [object Object].ns.Collection (packages/matb33:collection-hooks/collection-hooks.js:190:1)
I20150608-11:11:05.295(-4)?     at [object Object].Meteor.methods.exportDonations (app/server/index.js:44:19)
I20150608-11:11:05.295(-4)?     at [object Object].methodMap.(anonymous function) (packages/meteorhacks:kadira/lib/hijack/wrap_session.js:160:1)
I20150608-11:11:05.295(-4)?     at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1617:1)
I20150608-11:11:05.295(-4)?     at packages/ddp/livedata_server.js:648:1

Support for incremental mapReduce

Hi,

This module looks like its exactly what I'm after, its somewhat surprising its not in the core.

I'd like to do an incremental map reduce for two reasons

  1. Smaller workload on subsequent runs
  2. Perform a 'join' between tables

Incremental Map-Reduce
http://docs.mongodb.org/master/tutorial/perform-incremental-map-reduce/

Example 'join' using map reduce
http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/

From the documentation it looks like an incremental map-reduce not possible.

// keep in mind that executing the mapReduce function will override every time the collection Tags

Does this module support incremental reductions at the moment or would it be possible to include it in a future release?

Thanks

Still actual?

Hello.

Does this package still actual since we can access node MongoDB driver via rawCollection and rawDatabase?

For example:

MyCollection.rawCollection().aggregate pipeline, (err, res) ->
  if err
    console.error err
  else
    console.log res
  return

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.