• Model One-to-Many Relationships with Document References: Presents a data model that uses references to describe one-to-many relationships between documents.

http://docs.mongodb.org/manual/applications/data-models-applications/

Now that we have completed #26, we're ready to integrate -datasets into mongoscope! Because mongoscope-server has to manage all of the connections, we'll need to decouple from the mongo driver here.

• Merge populator.js and operations/mongo.js
• Move populator options/cli to a new bin in mongoscope-importer
• Use -importer as a devDependency in -datasets so that tests and their helpers do not need to be rewritten

The docs define 3 examples for modeling relationships between documents, detailed here http://docs.mongodb.org/manual/applications/data-models-relationships/.

• Model One-to-One Relationships with Embedded Documents: Presents a data model that uses embedded documents to describe one-to-one relationships between connected data.
• Model One-to-Many Relationships with Embedded Documents: Presents a data model that uses embedded documents to describe one-to-many relationships between connected data.

as discussed, how could we support

• versioning of tpl language
• multiple collections
• indexes
• schema -> joi validation object

http://docs.mongodb.org/manual/applications/data-models-tree-structures/

This will require:

• Generator can accept multiple schemas
• Update interpolators to check a cross schema dependency as we already do for cross field dependencies to generate or fetch the ref

With numbers in string format, queries using commands such "$gt" would be futile. The package should support inserting various BSON types or extend JSON types to mongodb.

• enable array config object (the second element of a configurable array) to have access to other fields. e.g. employees: [ "{{_$config}}", {size: "{{this.numOfEmployees}}"}, "{{name()}}" ].
• support non-repetitive enumeration. e.g. [ "{{_$config}}", { size: 2, enum: ['a', 'b', 'c'] }, "{{enum}}" ] may produce ['b', 'a']

A TransformStream takes a ReadableStream + configuration and emits the transformed result. Consider the following bash expressions

cat user-schema.json | mongodb-datasets  -n 10 > user-data.json

mongodb-datasets  -n 10 < user-schema.json > user-data.json

mongodb-datasets  -n 10 < user-schema.json > mongodb-importer mongodb://localhost:27017/demo.users --

export TOKEN=`curl -X POST http://localhost:29017/api/v1/token -d seed="localhost:27017" -H "Accept: text/plain"`;
mongodb-datasets  -n 10 < user-schema.json > curl -X POST http://localhost:29017/api/v1/import/demo.users -H "Authorization: Bearer ${TOKEN}" 

What we're doing in everyone of these cases is exactly the same: inserting mongodb-datasets as a transform! And because transform streams are just plumbing, adding support for this means all of the above are trivial.

The simplest way to test:

var fs = require('fs'), 
  md = require('mongodb-datasets');

fs.createReadStream('./examples/user-schema.json')
  .pipe(md.createGeneratorStream(10))
  .pipe(fs.createWriteStream('./examples/user-data.json');

After npm install -g mongodb-datasets, any use of mongodb-datasets return such error.

Concretely, the following template produces { doc: { counter: 0 }, wrong: 1 }

{
  "doc": {
    "counter": "{{Double(counter())}}"
  },
  "wrong": "{{this.doc.counter}}"
}

For now, all the generated content is in string format, whereas ISODate is a very useful datatype for MongoDB users. We need a way to generate Date object so that it is stored as ISODate in db.

• 1. Add support for generating array of arrays. This can be very useful when handling coordinates . 2d-sphere index.
• 2. Add new utility method for generating two-item-long array representing coordinates. This representation seems to be very popular (at least in the mongodb docs)

After #10 #11 #12 are complete, generate a reasonable set size (~1000 documents), run mongoexport for each subsection of the docs the schemas map to (so 10 tarballs in total), and upload them to a release of mongodb-datasets. We'll then coordinate with the docs team for integration of these into the public docs.mongodb.org.