The workerdb from bkniffler

workerdb's Introduction

WorkerDB

Performant, Reactive, offline-first, syncable Database in WebWorker based on PouchDB/RxDB, great for electronJS and web-applications

Bindings

react
vanilla: The database can be used with or without any framework

Why

PouchDB is a great database that, out of the box, enables you to build your sync-enabled apps easily. With RxDB on top you have better performance on bigger datasets as well as reactivity using RxJS and even more features like field-based encryption, schema validation/migration and more.

But if you want to build an app with lots of data, you may hit some issues with your app getting laggy due to the database querying, realtime syncing etc. all sogging up your UI threads' resources. You can try and use different native pouchdb adapters like leveldb or sqlite, but this may only make the queries go a bit faster, but the database communcation, offline-sync, etc. is still expensive. This is where WorkerDB comes into the game.

WorkerDB sits inside a WebWorker, which will, if possible, use a different CPU thread than your main app. All the heavy DB lifting happens inside the webworker, while the UI thread will only send commands to and receive results from the webworker.

Install

Yarn

yarn add workerdb react-workerdb

NPM

npm i workerdb react-workerdb

How to Use

Wether you use it with or without react, you will need to create a worker.js. The worker.js will define your database schema and host the database. Then you may use the react bindings or vanilla to instanciate the worker and query your data.

Worker.Js

For seeing all possible schema options, please check out RxDB docs here. There is just one additional property added sync. If this is defined, the collection will use the provided options to start syncing.

import { worker } from 'workerdb';

worker([
  {
    name: 'bird',
    schema: {
      version: 0,
      properties: {
        name: { type: 'string' },
        age: { type: 12 }
      }
    },
    sync: {
      remote: `https://my-server.com/bird`,
      waitForLeadership: false,
      direction: {
        pull: true,
        push: false
      },
      options: {
        live: true,
        retry: true
      }
    }
  }
]);

React (Hooks)

import * as React from 'react';
import { WorkerDB, useInsert, useFind } from 'react-workerdb';

const worker = new Worker('./worker.js');
export default () => (
  <WorkerDB worker={worker} loading={() => 'Is loading ...'}>
    <Birds />
  </WorkerDB>
);

function Birds() {
  // Search
  const [searchText, changeSearchText] = React.useState('');

  // WorkerDB insert
  const insert = useInsert('bird');
  // WorkerDB find (this query is reactive/live by default)
  const [birds, error, loading] = useFind('bird', {
    // Re-querying will automatically happen if these options change
    title: searchText ? { $regex: new RegExp(value, 'i') } : undefined
  });

  return (
    <div>
      <input
        value={searchText}
        onChange={e => changeSearchText(e.target.value)}
      />
      <button onClick={() => insert({ name: 'Filou' })}>Add Bird</button>
      <span>There is {birds.length} birds</span>
    </div>
  );
}

React

You need to wrap your app with the WorkerDB provider. You may then use Find, FindOne and Database components.

import { WorkerDB, Find, Collection } from 'react-workerdb';

const worker = new Worker('./worker.js');
export default () => (
  <WorkerDB worker={worker} loading={() => 'Is loading ...'}>
    <div>
      <Collection
        name="bird"
        render={collection => (
          <button onClick={() => collection.insert({ name: 'Filou' })}>
            Add Bird
          </button>
        )}
      />
      <Find
        live
        collection="bird"
        render={docs => <span>There is {docs.length} birds</span>}
      />
      <Find
        live
        collection="bird"
        name={{ $regex: /f/i }}
        render={docs => <span>There is {docs.length} birds</span>}
      />
    </div>
  </WorkerDB>
);

Vanilla

import { WorkerDB, WorkerDBWorker, WorkerDBProxy } from 'workerdb';
const worker = new Worker('./worker.js');

const stuff = async () => {
  const db = await WorkerDB.create(worker, {
    name: 'db',
    onSyncing: active => console.log(active),
    onError: err => console.error(err)
  });
  const collection = db.collection('bird');

  await collection.insert({ name: 'Filou' });
  const birds = await collection.find({});

  db.close();
};

stuff();

workerdb's People

Stargazers

Watchers

workerdb's Issues

Help wanted - background sync with worker thread

Hey 👋
thanks for this cool library. It is really nice to work with.

At my current project I'm planning to split the database handling into two. One part should run in a concurrent worker thread. Its purpose is to do the heavy lifting with all the network syncing (not database syncing). I probably will probably use this in combination with the periodic background sync API. Now I need a second part in the main thread of the client to get read access of the database and manipulate the DOM with the data.
In the docs I read a little about leader election etc. Is it possible to say that one specific database should be always the leader? Does that even make sense?

Eventually it makes sense to put some illustrations here. So I have already all the declarations for the schemas, collections, etc. I assume they are in the directory database/. The index file provides a function to create/initialize the database. Now I would like to use that function in a the main thread and the worker thread. I try to put a scaffold below. I'll reduce the imports to a minimum, just to show the relations. Not that I use a VueJS application as example here. But that should just work for any other framework as well.

src/database/index.ts:

export async function initializeDatabase(): Promise<RxDatabase<MyCollections>> {
    addRxPlugin(IndexeddbAdapter);

    const database = await createRxDatabase<MyCollections>({
      name: DATABASE_NAME,
      adapter: DATABASE_ADAPTER,
    });

    await database.addCollections(myCollections);
    return database;
}

src/plugins/database.ts:

import Vue from "vue";
import { initializeDatabase } from "@/database";

export async function DatabasePlugin(vue: typeof Vue): void {
  vue.prototype.$database = await initializeDatabase();
}

src/workers/database.ts:

import { initializeDatabase } from "@/database";

self.database = undefined;
initializeDatabase().then(database => self.database = database);

async function syncData() {
  const data = await fetch(...);
  // extract and transform data...
  self.database.myCollection.bulkInsert(transformedData);
}
 
self.onperiodicsync = (event) => event.waitUntil(syncData);

src/main.ts:

import Vue from "vue";
import App from "@/App.vue";
import { DatabasePlugin } from "@/plugins/database";

window.onload =  () => {
  const worker = new Worker('worker.ts');
  worker.periodicSync.register('syncData', {
    minInterval: 60 * 1000,
  });
};

Vue.use(DatabasePlugin);

new Vue({
  render: (h) => h(App),
}).$mount("#app");

I don't write the App.vue file now, since this already pretty extensive. In fact there would be then just something like this.$library.myCollection.find().$.subscribe((data) => ...);

Any tips or hints for such an approach? Would it work just like that (with proper code)? It is just quite some work to get there. And I would like to make sure beforehand that this is a solution approach that could work at the end with these two instances of the database.

Recommend Projects

bkniffler / workerdb Goto Github PK