Git Product home page Git Product logo

diskdb's Introduction

diskDB Build Status NPM version Gitter

NPM

A Lightweight Disk based JSON Database with a MongoDB like API for Node.

You will never know that you are interacting with a File System

Contents

Getting Started

Install the module locally :

$ npm install diskdb
var db = require('diskdb');
db = db.connect('/path/to/db-folder', ['collection-name']);
// you can access the traditional JSON DB methods here

Documentation

Connect to DB

db.connect(pathToFolder, ['filename']);

Filename will be the name of the JSON file. You can omit the extension, diskDB will take care of it for you.

var db = require('diskdb');
db = db.connect('/examples/db', ['articles']);
// or simply
db.connect('/examples/db', ['articles']);

This will check for a directory at given path, if it does not exits, diskDB will throw an error and exit.

If the directory exists but the file/collection does not exist, diskDB will create it for you.

Note : If you have manually created a JSON file, please make sure it contains a valid JSON array, otherwise diskDB will return an empty array.

[]

Else it will throw an error like

undefined:0

^
SyntaxError: Unexpected end of input

Load Collections

Alternatively you can also load collections like

var db = require('diskdb');
// this
db = db.connect('/examples/db');
db.loadCollections(['articles']);
//or
db.connect('/examples/db');
db.loadCollections(['articles']);
//or
db.connect('/examples/db')
  .loadCollections(['articles']);
//or
db.connect('/examples/db', ['articles']);

Load Multiple Collections

var db = require('diskdb');
db.connect('/examples/db', ['articles','comments','users']);

Write/Save to Collection

db.collectionName.save(object);

Once you have loaded a collection, you can access the collection's methods using the dot notation like

db.[collectionName].[methodname]

To save the data, you can use

var db = require('diskdb');
db.connect('db', ['articles']);
var article = {
    title : "diskDB rocks",
    published : "today",
    rating : "5 stars"
}
db.articles.save(article);
// or
db.articles.save([article]);

The saved data will be

[
    {
        "title": "diskDB rocks",
        "published": "today",
        "rating": "5 stars",
        "_id": "0f6047c6c69149f0be0c8f5943be91be"
    }
]

You can also save multiple objects at once like

var db = require('diskdb');
db.connect('db', ['articles']);
var article1 = {
    title : 'diskDB rocks',
    published : 'today',
    rating : '5 stars'
}

var article2 = {
    title : 'diskDB rocks',
    published : 'yesterday',
    rating : '5 stars'
}

var article3 = {
    title : 'diskDB rocks',
    published : 'today',
    rating : '4 stars'
}
db.articles.save([article1, article2, article3]);

And this will return the inserted objects

[ { title: 'diskDB rocks',
    published: 'today',
    rating: '4 stars',
    _id: 'b1cdbb3525b84e8c822fc78896d0ca7b' },
  { title: 'diskDB rocks',
    published: 'yesterday',
    rating: '5 stars',
    _id: '42997c62e1714e9f9d88bf3b87901f3b' },
  { title: 'diskDB rocks',
    published: 'today',
    rating: '5 stars',
    _id: '4ca1c1597ddc4020bc41b4418e7a568e' } ]

Read from Collection

There are 2 methods available for reading the JSON collection

  • db.collectionName.find(query)
  • db.collectionName.findOne(query)

db.collectionName.find()

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.find();

This will return all the records

[{
    title: 'diskDB rocks',
    published: 'today',
    rating: '5 stars',
    _id: '0f6047c6c69149f0be0c8f5943be91be'
}]

You can also query with a criteria like

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.find({rating : "5 stars"});

This will return all the articles which have a rating of 5.

Find can take multiple criteria

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.find({rating : "5 stars", published: "yesterday"});

This will return all the articles with a rating of 5, published yesterday.

Nested JSON :

var articleComments = {
    title: 'diskDB rocks',
    published: '2 days ago',
    comments: [{
        name: 'a user',
        comment: 'this is cool',
        rating: 2
    }, {
        name: 'b user',
        comment: 'this is ratchet',
        rating: 3
    }, {
        name: 'c user',
        comment: 'this is awesome',
        rating: 2
    }]
}
var savedArticle = db.articles.save([articleComments);
foundArticles = db.articles.find({rating : 2});

Since diskDB is mostly for light weight data storage, avoid nested structures and huge datasets.

db.collectionName.findOne(query)

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.findOne();

If you do not pass a query, diskDB will return the first article in the collection. If you pass a query, it will return first article in the filtered data.

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.findOne({_id: '0f6047c6c69149f0be0c8f5943be91be'});

Update Collection

db.collectionName.update(query, data, options);

You can also update one or many objects in the collection

options = {
    multi: false, // update multiple - default false
    upsert: false // if object is not found, add it (update-insert) - default false
}

Usage

var db = require('diskdb');
db.connect('/examples/db', ['articles']);

var query = {
  title : 'diskDB rocks'
};

var dataToBeUpdate = {
  title : 'diskDB rocks again!',
};

var options = {
   multi: false,
   upsert: false
};

var updated = db.articles.update(query, dataToBeUpdate, options);
console.log(updated); // { updated: 1, inserted: 0 }

Remove Collection

db.collectionName.remove(query, multi);

You can remove the entire collection (including the file) or you can remove the matched objects by passing in a query. When you pass a query, you can either delete all the matched objects or only the first one by passing multi as false. The default value of multi is true.

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.remove({rating : "5 stars"});
var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.remove({rating : "5 stars"}, true); // remove all matched. Default - multi = true
var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.remove({rating : "5 stars"}, false); // remove only the first match

Using remove without any params will delete the file and will remove the db instance.

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.remove();

After the above operation db.articles is undefined.


Count

db.collectionName.count();

Will return the count of objects in the Collection

var db = require('diskdb');
db.connect('/examples/db', ['articles']);
db.articles.count(); // will give the count

Examples

Refer to the examples folder.

Performance

To validate diskDB's performance and to check if it meets your needs, you can clone this repo and run

$ node performance/time.js

An average of few tests (run on OS X - 10.9.3 | 2.9GHZ i7 | 8GB 1600MHz DDR3) can be found below

Time taken to process x number of objects (in ms) vs Action Performed

# of objects 1 1000 10000 100000 1000000
Save 1 ms 15 ms 137 ms 1728 ms 14425 ms
Find all without query 0 ms 2 ms 12 ms 204 ms 2923 ms
Find all with query 0 ms 2 ms 17 ms 738 ms 1985 ms
Find one without query 0 ms 1 ms 9 ms 791 ms 1676 ms
Find one with query 0 ms 1 ms 8 ms 219 ms 1410 ms
Update all records 1 ms 7 ms 61 ms 206 ms 48035 ms
Get count 0 ms 3 ms 11 ms 260 ms 2420 ms
Remove with query 0 ms 7 ms 59 ms 984 ms 48191 ms
Remove collection 0 ms 1 ms 4 ms 52 ms 154 ms
File size 0.000111 MB 0.116671 MB 1.196671 MB 12.26667 MB 125.66667 MB

Contributing

See the CONTRIBUTING Guidelines

Release History

  • 0.1.x
    • Base Module with
      • Connect to a Folder
      • Access a Collection/File
      • Create Read Update Delete on JSON object
      • Minor fixes and tests
      • Performance improvements

License

Copyright (c) 2014 Arvind Ravulavaru. Licensed under the MIT license.

diskdb's People

Contributors

arvindr21 avatar bcolucci avatar evanx avatar gitter-badger avatar marcbachmann avatar patchfx avatar udaykanthr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

diskdb's Issues

update throws error "no such file"

when trying to update (or upsert) data instead of using .save an error get thrown :

fs.js:432
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
             ^
Error: ENOENT, no such file or directory './/home/larsbrix/Projects/league/app/db/teams.json'
at Object.fs.openSync (fs.js:432:18)
at Object.fs.readFileSync (fs.js:289:15)
at Object.util.readFromFile (/home/larsbrix/Projects/league/app/node_modules/diskdb/lib/util.js:26:15)
at Object.coltn.update (/home/larsbrix/Projects/league/app/node_modules/diskdb/lib/collection.js:63:42)
at IncomingMessage.<anonymous> (/home/larsbrix/Projects/league/app/server.js:114:26)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:929:16
at process._tickCallback (node.js:419:13)

the line that trows the error:

db.teams.update({fullId: team.fullId}, team, options);

with options

var options = {
     multi: false, 
     upsert: true
 }

where i try to find a team by id and insert/update my data.

EDIT: forgot to mention that my json existes and the content is the empty brackets "[]"

unable to save record with different path to data file

If i will give some other path to data file than within nwjs application something as db.connect('/home/user/Videos/myDB', [ 'articles']), then I can't able to save/update records to diskdb file in nw.js application once i built it. What is the workaround?

Vs. NeDB

Does it have some advantages comparing to NeDB? From the first quick look they are very similar to me.

Dynamic collection name based on date "YYYY-MM-DD"

Hi, i love using your library for my projects it is simply awesome and everything always works like it should. Now i got a project where it would be realy ql if i could load/save collection named dynamicly by date... i've done some testing. Loading collection works ... but i'm having trouble/error when trying to save it ... any idea?

Loading (it is working):
var currentDateFileName = moment().format('YYYY-MM-DD');
... dbconn.connect('/', ['stuff', currentDateFileName ]);

Saveing (not working):
var saveToFile = conn.currentDateFileName.save(logRecord);

/test5.js:214
var saveToFile = conn.currentDateFileName.save(logRecord);
^
TypeError: Cannot read property 'save' of undefined
at logRegs (/home/pi/ctc2/test5.js:212:37)
at Transaction. (/home/pi/ctc2/test5.js:183:6)
at emitOne (events.js:77:13)
at Transaction.emit (events.js:169:7)
at /home/pi/node_modules/h5.modbus/lib/Transaction.js:320:19
at doNTCallback0 (node.js:417:9)
at process._tickCallback (node.js:346:13)

How to use multiple condition in find?

In users.json data is
[{"email":"[email protected]","password":"123","status":"1","_id":"e39a62a9c7b94d76892f9c3bdfa22715"}]

I am using it as follows:

        db.loadCollections(['users']);

        var preCheck = db.users.find({"email" : req.body.email,"password" : req.body.password});

        if(preCheck !== undefined){
           
            if( (preCheck.email == req.body.email) && (preCheck.password == req.body.password) ){ 
                //logged in
                
                req.session.email = req.body.email;
                req.session.userId = preCheck._id;
                req.flash('success','Successful Login');
                res.redirect('/option');
            }
            else{
               
                req.flash('error','Email Id or Password is Incorrect');
                res.redirect('/login');
            }
        }

But it returns invalid login always.

Save appears broken as of node 0.10.29

npm installed and did as the example said.

var db = require('diskdb');
db = db.connect('./assets/db', ['apikey']);

var data = { key: 'placeholder', vcode: 'placeholder' };
db.apikey.save(data);

Here's what's outputted on the terminal

Successfully connected to : ./assets/db

undefined:0

^
SyntaxError: Unexpected end of input
    at Object.parse (native)
    at Object.coltn.save (/home/nino/Desktop/test/node_modules/diskdb/lib/collection.js:37:31)
    at Object.<anonymous> (/home/nino/Desktop/test/test.js:5:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

db.collection.find({})

Hi

db.collection.find({}) 

should work like

db.collection.find() 

not it returns nothing

Joining two collections

Is it possible to join two collections say user and user details with a common id something similar to sql join.

Seems a bug in update()

please take a look at below error.
When I tried to update a obj which had name "mic", it actually did on another which had name "mic2".

source code:

let query1={name:user.name,password:user.password};
let query2={openid:user.openid};
let updatedata={openid:'o7nyGwoX5e7GYH86m8nlZpP6IkwI'}
var userdb = require('diskdb');
userdb.connect('db', ['userdb']);
console.log(query1);
console.log(query2);
userdb.userdb.update(query2,{openid:''},{multi: false});
console.log(userdb.userdb.find(query1));
console.log(userdb.userdb.find(query2));
userdb.userdb.update(query1,updatedata,{upsert: true});
console.log(userdb.userdb.find(query1));
console.log(userdb.userdb.find(query2));

log:

Successfully connected to : db
{ name: 'mic', password: '' } --- query1
{ openid: 'o7nyGwoX5e7GYH86m8nlZpP6IkwI' } --- query2
[ { name: 'mic',
password: '',
_id: '4d066e36b9844277abafac602570904e',
openid: ''
} ] --- userdb.userdb.find(query1)
[] --- userdb.userdb.find(query1)
[ { name: 'mic',
password: '',
_id: '4d066e36b9844277abafac602570904e',
openid: '', } ] --- userdb.userdb.find(query1)
[ { name: 'mic2',
password: '',
_id: 'ead56db65a5e41bb86ce2831a914e4cf',
openid: 'o7nyGwoX5e7GYH86m8nlZpP6IkwI' } ] ---userdb.userdb.find(query2)

JSON file requires "_id" for find(querry) to return all filtered results

I was using a custom made JSON file with "id" key instead of "_id". This caused the find(query) method to return only the first matched result.

example articles.json

[
{
"title": "diskDB rocks again",
"published": "3 days ago",
"rating": 3,
"id": "1"
},
{
"title": "diskDB rocks again",
"published": "3 days ago",
"rating": 2,
"id": "2"
},
{
"title": "diskDB rocks again",
"published": "3 days ago",
"rating": 4,
"id": "3"
},
{
"title": "diskDB rocks again",
"published": "3 days ago",
"rating": 2,
"id": "4"
},
{
"title": "diskDB rocks",
"published": "2 days ago",
"rating": 2,
"id": "5"
}
]

var db = require('diskdb');
db.connect('.', ['articles']);
articleComments4, articleComments5]);
var foundArticles = db.articles.find();
var foundArticles = db.articles.find({rating : 2});

console.log(foundArticles);
console.log(foundArticles.length);

Let's get this Async started

So, to continue to our comments on "The Jackal of Javascript Blog" (http://thejackalofjavascript.com/ionic-twilio-and-node-scheduler-part-1/), let's start and plan our naming convention for the async mehtods.

I say, to not make a braking change keep the current methods as is,
Give them new aliases like: "newMethodNameSync()"
and add the async version without prefix: "newMethodName()" to keep it in the node convention way.

We just need to think about new method names for each method.

Removal (and I think updating) does not work as expected with multiple criteria

Similar to #22, if you do a .remove() with multiple criteria, each thing that matches one of the criteria is removed, not everything that matches all the criteria.

Right now I've gotten around it by doing a find with the criteria, looping those and doing a removal via the _id, but it feels like the remove should work the same way as the find.

I mentioned update in the title, because looking at util.js where it does the updateFiltered, it has the same logic as remove. Howeve I've not needed to use update yet to confirm for sure.

Typescript Flexablity

I tried to create a type definition for this lib to easily implement it in typescript with loadCollections method just adding the collection object in the dbDisk object in way that I can't add a type for it in typescript
please check the following line:
this[collectionName] = new Collection(this, collectionName);

Save Error

hi i am new to this development. i am using diskdb for saving data. it works fine but when i close the app and reopen, all data disappears seems not saved in JSON file. anyone please help me.

Option to Suppress Console Logs on diskDB.connect()

Feature Request Summary

I think the option to suppress success/error console log messages upon connecting to a collection would be useful. This would be useful in cases where disk DB is implemented in production / command-line applications where you want to suppress these logs (successful or not) from users.

Example of Issue

For example, I use Disk DB in a CLI App I develop, and when the app connects to Disk DB it breaks up the console.log interface that the user interacts with:

? Which template would you like to use? Daily Notes
? What's the file name? (Leave blank to use the template default) : 
? Where should the file go? (Leave blank for current directory) : 
Generating Note...
Successfully connected to : C:\Users\Mykal\Documents\GitHub\NoteGen\src
Note Generated!

Proposed Solution

I propose adding in an optional object param that contains a named key supressLogs. The proposed new API would be ๐Ÿ‘

db.connect('path/to/collections', ['myCollection'], {supressLogs: true})

This would then be used in the connect method to determine whether or not console logs should be printed.

Action Items

  • Update connect function to accept optional parameter
  • Update documentation to explain the new parameter
  • Ensure all tests still pass
  • Submit PR & make necessary changes ๐ŸŽ‰

Find the collections with two conditions

How to find the collections with two conditions something equivalent to sql and conditions (where Userid=12 and updated=0).

I tried with db.settings.find({"Userid": 12, "updated": 0}) but it returns all the data.

find method in diskdb does not return correct result when query contains more than one property

I saved two objects like this

db.sam.save({name: "kiran",phno:"123"});
db.sam.save({name: "kiran",phno:"123456"});

and i wrote a query to find obj with {name: "kiran",phno:"123"}

db.sam.find({name: "kiran",phno:"123"});

the result that i get is

[ {name: "kiran",
phno:"123",
_id: "997044f2d16d440aa3b6588d4d228bea"},
{name: "kiran",
phno:"123456",
_id: "26b670304d844d9b957abf367e47f153"}
]

but the answere is supposed to be

[ {name: "kiran",
phno:"123",
_id: "997044f2d16d440aa3b6588d4d228bea"}
]

update/upsert: _id is generated even if you send a pre-generated _id the json object

Consider this lines of code with an empty db file:

[...]

//I need to pre-generate the _id of this object and not use a auto-generated _id 
var obj = {
  "_id": "xxxxxxx",
  "name":  "test"
};

console.log("Before: ", obj);
db.records.update({ _id : "xxxxxxx" },  obj, {multi: false, upsert: true});
console.log("After: ", obj);

[...]

The output will be:

Before: { "_id": "xxxxxxx", "name":  "test"}
After: { "_id": "6aec8d48d5284def9c3c85428b506d95", "name":  "test"}

//being '6aec8d48d5284def9c3c85428b506d95' the new generated _id when it should be 'xxxxxxx'

After looking into the module code, I noticed that you don't verify if the _id already exists:

filename: collection.js

coltn.update = function(query, data, options) {
        var ret = {},
            collection = JSON.parse(util.readFromFile(this._f)); // update
        var records = util.finder(collection, query, true);
        if (records.length) {
            if (options && options.multi) {
                collection = util.updateFiltered(collection, query, data, true);
                ret.updated = records.length;
                ret.inserted = 0;
            } else {
                collection = util.updateFiltered(collection, query, data, false);
                ret.updated = 1;
                ret.inserted = 0;
            }
        } else {
            if (options && options.upsert) {
                //It would be nice if you verify if the _id exists and only create a new one if not 
                data._id = uuid.v4().replace(/-/g, ''); 
                collection.push(data);
                ret.updated = 0;
                ret.inserted = 1;
            } else {
                ret.updated = 0;
                ret.inserted = 0;
            }
        }
        util.writeToFile(this._f, collection);
        return ret;
    };

Reason behind saving an array backward

I'm just curious what is the reason behind saving an array backward

var articles = [article1, article2, article3]
db.articles.save(articles);
articles = db.articles.find({})

the order of articles returned will be [article3, article2, article1]

I was expected to be the same order when I get them as the same when I saved them.
Thank

[PROPOSAL] Sorting

Sorting would be awesome feature for this awesome library. I mean like those ORDER BY ASC / DESC in MySQL.

Running performance tests returns an error

When running the performance tests with node performance/test.js it returns an error about the path for the database:

The DB Path [db] does not seem to be valid. Recheck the path and try again

This happens on my Macbook, but seems to stem from not having a full path.

[Proposal] Consider Adding a Silence mode of Logging?

I would like to have controls on whether or not to log messages out.
Those messages are useful when it is in development, but in production mode I would rather disable or store them into a log file or even the database itself as a collection.

What do you think?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.