monbro / meteor-mongodb-mapreduce-aggregation Goto Github PK
View Code? Open in Web Editor NEWThis project forked from zvictor/meteor-mongo-server
Very simple implementation of some of mongodb aggregation framework functions for Meteor
This project forked from zvictor/meteor-mongo-server
Very simple implementation of some of mongodb aggregation framework functions for Meteor
A quick background to understand the situation:
A project of ours uses the mapreduce available in MongoDB through this package.
Until now, we only had the issue when we attempted a mapreduce on an empty collection...
Since then, I have deployed the project bundle on its own server with its own instance of MongoDB with replica sets. The project bundle is launched using node as is explained in the documentation.
Our application generates 6-7 events every 2 minutes on schedule. These events are stored in a collection called events updated by triggerJob. At the end of the triggers loop, a map reduce is ran. It acts on the collection of events and replaces the collection called eventStats.
The bug occurs only after a while; approximatly 30 events or so. I suspect it might be due to the replacement of the out collection.
// scheduler calls:
jobs.forEach(function(element, index, array) {
triggerJob(element, null);
});
Meteor.call("updateEventStats", function(err, data) {
if (err) debug("updateEventStats error: " + err);
});
// updateEventStats def:
Meteor.methods({
updateEventStats: function () {
if (0 < Events.find().count()) {
Events.mapReduce(map, reduce, {
query: {},
out: {replace: "eventStats"},
finalize: finalize,
scope: {quartileValue: quartileValue}
});
}
}
});
I get the following error in the meteor application:
.../app/programs/server/packages/monbro_mongodb-mapreduce-aggregation.js:57
collectionName: result.collectionName,
^
TypeError: Cannot read property 'collectionName' of undefined
at packages/monbro:mongodb-mapreduce-aggregation/server.coffee:42:36
at .../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/collection/aggregation.js:86:20
at [object Object].<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/db.js:1903:9)
at [object Object].g (events.js:199:16)
at [object Object].emit (events.js:110:17)
at [object Object].Base.__executeAllServerSpecificErrorCallbacks (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/base.js:373:29)
at Server.<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/repl_set/repl_set.js:347:14)
at Server.emit (events.js:110:17)
at [object Object].<anonymous> (.../app/programs/server/npm/mongo/node_modules/mongodb/lib/mongodb/connection/server.js:609:84)
at [object Object].emit (events.js:104:17)
And it actually kills the instance of MongoDB:
2015-09-16T14:05:10.660+0200 F - [conn2] Invalid access at address: 0
2015-09-16T14:05:10.671+0200 F - [conn2] Got signal: 11 (Segmentation fault).
0x1150d6b 0x1150523 0x1150907 0x7ffad0e80d60 0x11ec2a6 0x10a2dad 0x109553a 0x109a76b 0x109a7d9 0x108ceea 0x109421a 0x10947a9 0x9c3804 0x9c70d7 0xa4550c 0xa466c2 0xa475
79 0xc8fa87 0xb6c4b9 0x7d3579 0x10f65e5 0x7ffad0e774a4 0x7ffacf80913d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"D50D6B"},{"b":"400000","o":"D50523"},{"b":"400000","o":"D50907"},{"b":"7FFAD0E70000","o":"10D60"},{"b":"400000","o":"DEC2A6"},{"b":"400
000","o":"CA2DAD"},{"b":"400000","o":"C9553A"},{"b":"400000","o":"C9A76B"},{"b":"400000","o":"C9A7D9"},{"b":"400000","o":"C8CEEA"},{"b":"400000","o":"C9421A"},{"b":"400000","o":"C947A9"},{"b":"400000","o":"5C3804"},{"b":"400000","o":"5C70D7"},{"b":"400000","o":"64550C"},{"b":"400000","o":"6466C2"},{"b":"400000","o":"647579"},{"b":"400000","o":"88FA87"},{"b":"400000","o":"76C4B9"},{"b":"400000","o":"3D3579"},{"b":"400000","o":"CF65E5"},{"b":"7FFAD0E70000","o":"74A4"},{"b":"7FFACF720000","o":"E913D"}]
,"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.7-2-ARCH", "version" : "#1 SMP PREEMPT Tue
Jun 30 07:50:21 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D7EE62E8D9A44AA66F8BEE93EA5C281EA42BB95A" }, { "b" : "7FFE
1774A000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "A320F98DE311139B6ABEFFB2A7109994EDA4CB5D" }, { "b" : "7FFAD1F90000", "path" : "/usr/lib/libpcre.so.1"
, "elfType" : 3, "buildId" : "88D40283EB5177DCE76EAAD3B6E8A8B2EA6BB66E" }, { "b" : "7FFAD1D80000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "1AC3
A716BAF09BC2191EA2B75EE6CE8CE36C3433" }, { "b" : "7FFAD1B78000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "1717DA41B48DD7305D95423F7B181961BD9662D
A" }, { "b" : "7FFAD18F8000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "EFB841BBCA7F10BE4B3ECBBA7E43E0593ACD5515" }, { "b" : "
7FFAD16E0000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "7939ED3E6D025517862CB78D23BD7C4BFD1E8918" }, { "b" : "7FFAD14B8000", "path
" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "3033771E63857EF953278CEDA804C3493E896836" }, { "b" : "7FFAD12B0000", "path" : "/usr/lib/libboost_s
ystem.so.1.58.0", "elfType" : 3, "buildId" : "250CA1E0DC473E1E54BEA3E13B28ED550F7206F4" }, { "b" : "7FFAD1090000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "bu
ildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7FFAD0E70000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "284BF58119A6255AFBA45689
40CE655ECDFF1881" }, { "b" : "7FFAD0BF0000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "4E6FE13AD63FE3F0F3279F9BE077586E2BADF443" }, { "b" : "7FFA
D0778000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E72E20F8FD3FB8B2C3EC7ADB5956E99D16AF15A4" }, { "b" : "7FFAD0570000", "path" : "/usr/lib/l
ibrt.so.1", "elfType" : 3, "buildId" : "6C39F2195A3E5D3A6A1FAEF570D0B83AF2184E3D" }, { "b" : "7FFAD0368000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" :
"4C55749EE719F7CE38F5078CE3197091D24EFEDE" }, { "b" : "7FFACFFE0000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "6C6DEA88072C530D0FC9F347CF115B722C
8BE8BF" }, { "b" : "7FFACFCE0000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "1C20A8053268B03D81D85235D26B14F4D24DBC40" }, { "b" : "7FFACFAC8000", "path
" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "F4AD36B00DF033915EEEB0CD89B9C5F66AF864B3" }, { "b" : "7FFACF720000", "path" : "/usr/lib/libc.so.6", "elfType"
: 3, "buildId" : "34701C7AA9501F113595BA20117D03FBAB4A7EDF" }, { "b" : "7FFAD2200000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B3B6F65FD2224
CB6D7FD34FCBEC82CDEE0F62A8B" }, { "b" : "7FFACF508000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "544EDFD9674594504CC408B41C1AAC2F3CF9ED63" }, { "
b" : "7FFACF2F0000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "82A1A5B6D7F272E064155BB64576230AEFAFA7B8" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x2B) [0x1150d6b]
mongod(+0xD50523) [0x1150523]
mongod(+0xD50907) [0x1150907]
libpthread.so.0(+0x10D60) [0x7ffad0e80d60]
mongod(_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x16) [0x11ec2a6]
mongod(_ZN5boost6detail17sp_counted_impl_pIN5mongo10BSONHolderEE7disposeEv+0x3D) [0x10a2dad]
mongod(+0xC9553A) [0x109553a]
mongod(_ZN5mongo7V8ScopeD1Ev+0x34B) [0x109a76b]
mongod(_ZN5mongo7V8ScopeD0Ev+0x9) [0x109a7d9]
mongod(+0xC8CEEA) [0x108ceea]
mongod(_ZN5mongo11PooledScopeD1Ev+0x81A) [0x109421a]
mongod(_ZN5mongo11PooledScopeD0Ev+0x9) [0x10947a9]
mongod(_ZN5mongo2mr5StateD1Ev+0x104) [0x9c3804]
mongod(_ZN5mongo2mr16MapReduceCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x10F7) [0x9c70d7]
mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2C) [0xa4550c]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xF22) [0xa466c2]
mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x529) [0xa47579]
mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_+0x677) [0xc8fa87]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xA69) [0xb6c4b9]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xD9) [0x7d3579]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0x10f65e5]
libpthread.so.0(+0x74A4) [0x7ffad0e774a4]
libc.so.6(clone+0x6D) [0x7ffacf80913d]
----- END BACKTRACE -----
The .mapReduce works great on first call, but on subsequent calls I get the following error:
I20150608-11:11:05.294(-4)? Exception while invoking method 'exportDonations' Error: A method named '/donation_totals_by_donor/insert' is already defined
I20150608-11:11:05.294(-4)? at packages/ddp/livedata_server.js:1461:1
I20150608-11:11:05.294(-4)? at Function._.each._.forEach (packages/underscore/underscore.js:113:1)
I20150608-11:11:05.294(-4)? at [object Object]._.extend.methods (packages/ddp/livedata_server.js:1459:1)
I20150608-11:11:05.294(-4)? at [object Object].Mongo.Collection._defineMutationMethods (packages/mongo/collection.js:904:1)
I20150608-11:11:05.294(-4)? at new Mongo.Collection (packages/mongo/collection.js:209:1)
I20150608-11:11:05.295(-4)? at [object Object].ns.Collection (packages/matb33:collection-hooks/collection-hooks.js:190:1)
I20150608-11:11:05.295(-4)? at [object Object].Meteor.methods.exportDonations (app/server/index.js:44:19)
I20150608-11:11:05.295(-4)? at [object Object].methodMap.(anonymous function) (packages/meteorhacks:kadira/lib/hijack/wrap_session.js:160:1)
I20150608-11:11:05.295(-4)? at maybeAuditArgumentChecks (packages/ddp/livedata_server.js:1617:1)
I20150608-11:11:05.295(-4)? at packages/ddp/livedata_server.js:648:1
Hi,
This module looks like its exactly what I'm after, its somewhat surprising its not in the core.
I'd like to do an incremental map reduce for two reasons
Incremental Map-Reduce
http://docs.mongodb.org/master/tutorial/perform-incremental-map-reduce/
Example 'join' using map reduce
http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/
From the documentation it looks like an incremental map-reduce not possible.
// keep in mind that executing the mapReduce function will override every time the collection Tags
Does this module support incremental reductions at the moment or would it be possible to include it in a future release?
Thanks
Hello.
Does this package still actual since we can access node MongoDB driver via rawCollection
and rawDatabase
?
For example:
MyCollection.rawCollection().aggregate pipeline, (err, res) ->
if err
console.error err
else
console.log res
return
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.