Git Product home page Git Product logo

mongo-airtable's Introduction

Mongo Airtable

Project Status

Sadly, this project is officially "archived". I lost interest in working on this project, mostly because I just don't use MongoDB much on personal projects these days.

If you make conflicting changes, one will overwrite the other. Use this tool at your own risk!

Motivation

Airtable is great database/spreadsheet hybrid for holding your data. The problem is that sometimes you don't have internet access, so pulling and saving data using their API isn't feasible. This tool allows you to get and set your data to a MongoDB database, whether or not you have internet access, and then sync whenever is convenient. That way, Airtable is still your ultimate source of truth, but your app's proxy source of truth is MongoDB.

Getting started

Install MongoDB using brew install mongodb or similar.

Let's say you wanted your mongodb database to be called "main" and your collection (which maps to a table in Airtable) to be called "songs". The Airtable view is called "Main View".

Clone this repo, and then create a file called .mongo-airtable.yaml. Make it look like the following.

auth:
  airtable: <your airtable api key (starts with 'key')>
sync:
  - base_name: <your airtable base key (starts with 'app')>
    primary: <your airtable table name (e.g. 'Songs')>
    view: <your airtable view name (e.g. 'Main View')>
    database: <a mongo db name (e.g. 'main')>
    collection: <a mongo db collection name (e.g. 'songs')>
  - base_name: <your airtable base key (starts with 'app')>
    primary: <another airtable table name (e.g. 'Artists')>
    view: <your airtable view name (e.g. 'Main View')>
    database: <another mongo db name (e.g. 'main')>
    collection: <another mongo db collection name (e.g. 'songs')>

Create a file called .env. Make it look like the following.

  MONGO_URL=mongodb://localhost:27017/my_db
  PATH_MONGO_AIRTABLE_YAML=path/to/.mongo-airtable.yaml

Run npm install (or yarn install). Now you're mostly ready to go.

Run the tests by doing npm test. This is important, as it acts as diagnostics for your local setup.

When you want to pull from airtable down to your mongodb collection, run:

npm run pull

When you want to push any changes to your airtables, run:

npm run push

For right now (this will be fixed soon), after doing a push, you should afterward do a pull:

npm run pull

TODO:

  • make this tool use the "modified date" on airtable records
  • make this tool use the "modified date" of mongo records
  • configuration via airtable
  • progress bars
  • confirmation dialogues for conflicts
  • 100% test coverage after the above are complete

mongo-airtable's People

Contributors

arro avatar dependabot[bot] avatar gaetancottrez avatar phildeschaine0 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

mongo-airtable's Issues

bases with multiple tables and a lot of records take a lot of time to sync

I tried mongo-airtable to pull my base to mongoDB, my Airtable base has 12 tables and more than 6500 records...
it took 190 seconds to finish pulling and inserting
I tried tp modify some functions to make it work async, all tables start pulling at once but as soon as any table finish pulling it will start inserting into mongodb directly ... this improved the performance and took 92 seconds instead of 190 seconds
this still too much as I am intending to fire this syncing process every 5 minutes intervals or less to always keep mongodb synchronized with airtable

here is my edits to reach 92 secs... added this function to airtable.js:

 export async function initialPullAndInsert(config) {
   for (const table_to_sync of config.sync) {
     pullTable({
       ...table_to_sync,
       auth_key: config.auth.airtable
     }).then(() => {
       console.log(`end sync of ${table_to_sync.primary}`)
       const last_pulled_filename = path.resolve(`${__dirname}/../build/last-pulled.txt`);
       writeFile(last_pulled_filename, moment().format('ddd MMM D YYYY h:mm A ZZ'), 'utf-8').then(() => {
         initialInsertTable(table_to_sync);
       })
     })
   }
 }

and added this to mongodb.js:

 export async function initialInsertTable(table_to_sync) {
   const { primary, collection, database } = table_to_sync;
     await putIntoDB({
       primary,
       collection,
       database
     })
   }

and in pull.js I called await initialPullAndInsert(config);

Error running push after tests have passed

Hi
I ran the tests and all 6 passed. When I run "npm run push", I get the follow error. I tried both simple and complex mongodb collections
(node:8198) UnhandledPromiseRejectionWarning: TypeError: (0 , _mongodb.lookForChanges) is not a function

Suitable for production?

I'm considering to use your tool to pull our entire Airtable base to our Staging webserver's MongoDB, from which I will then generate pages for our website. We then deploy all this static content as complete stand-alone ZIP to our Live web server.

I already use the official Airtable API JS client, but with your tool I hope to avoid problems with the Airtable API limitations (5 requests per second).

This won't be a big improvement to using the Airtable API directly, because I can delay the API requests and we are OK with waiting 10-30min. However, I'm wondering if at some point I can have the MongoDB directly on our Production and threfore skip generating the static pages on Staging server and then read/write dynamic content (e.g. Orders, User profiles) directly from the MongoDB, which in background will be synced by your tool every 10min or so.

Are you using it for scenarios like this?
Or only to sync our offline copy of the base, like for local development for example?

Asking because I've read this:
The problem is that sometimes you don't have internet access, so pulling and saving data using their API isn't feasible.

Thanks for your input!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.