bbc / dropbox-paper-to-json Goto Github PK

A Node module to import data from a dropbox paper document and convert it into a json data structure.

• create app from developer console
  • chose Dropbox API
  • chose Full Dropbox
  • give it a Name: eg News Mixer
    • click on the newly created app
      • and Generate an access token for your own account

eg if the url of your dropbox paper is something like

https://paper.dropbox.com/doc/Main-Title-vJdrjMJAHdgfHz0rl83Z

Then the last string element after the last -, reading from left to right, is your document id.

In this ficticious example it would be: vJdrjMJAHdgfHz0rl83Z.

The project uses dotenv to deal with credentials and enviroment variables.

In the root of the folder repo create a .env file, this is excluded from the github repo by .gitignore to avoid leaking credentials.

Here's an examples format of .env file, with some fictitious credentials

# Dropbox credentials
DROPBOX_ACCESS_TOKEN=vJdrjMJAHdgfHz0rl83ZvJdrjMJAHdgfHz0rl83Z
DROPBOX_DOC_ID=vJdrjMJAHdgfHz0rl83Z

clone this repo

git clone [email protected]:bbc/dropbox-paper-to-json.git

cd into folder

cd dropbox-paper-to-json

npm install

npm start

This will save a data.json file in the root of the project.

npm install

npm install dropbox-paper-to-json@git+ssh://[email protected]/bbc/dropbox-paper-to-json.git#master -save

Add to your code base

//if using dotenv for environment variable credentials for dropbox paper
require('dotenv').config();
// optional if you want to write the resulting json
const fs = require('fs');
// require module
const dbpMdToJson = require('dropbox-paper-to-json');

dbpMdToJson({
    accessToken: process.env.DROPBOX_ACCESS_TOKEN,
    dbp_doc_id: process.env.DROPBOX_DOC_ID,
    // default for nested === true
    nested: true
}).then((data) => {
    console.log(`done Dropbox Paper to JSON conversion`);
    // optional: now do something with the data
    fs.writeFileSync('./data.json', JSON.stringify(data, null, 2));
});

High level overview of system architecture

The module uses dpb-download-md node module to get a dropbox paper as markdown given a dropbox paper id and access token.

As the official SDK didn't seem to have a straightforward way to get to a dropbox paper document content.

The submodule md-to-json/linear.js takes the content of a markdown file as a string and converts it into an array of objects, representing markdown elements.

it's a flat data structure, with no nesting, hence why sometimes refered to as linear.

[
    {
      "text": "Chapter 1",
      "type": "h1"
    },
    {
      "text": "Text",
      "type": "h2"
    },
    {
      "text": "vitae elementum velit urna id mi. Sed sodales arcu mi, eu condimentum tellus ornare non. Aliquam non mauris purus. Cras a dignissim tellus. Cras pharetra, felis et convallis tristique, sapien augue interdum ipsum, aliquet rhoncus enim diam vitae eros. Cras ullamcorper, lectus id commodo volutpat, odio urna venenatis tellus, vitae vehicula sapien velit eu purus. Pellentesque a feugiat ex. Proin volutpat congue libero vitae malesuada.",
      "type": "p"
    },
    {
      "text": "Video ",
      "type": "h2"
    },
...
]

For some use cases it might be heplfull to nest all the elments between an h1 tag to the next h1 take as siblings/childres/elements of that tag.

Eg h1 tag could contain h2, p tag, link etc..

Likewise h2 tag could contain all other elements up to the next h2 or h1 tag.

NOTE dropbox paper flavour of markdown only properly reppresents H1 and H2 tags hence why we stopped the nesting only at two levels for this use case. But it could be nested further should there be a use case for it.

This is done in md-to-json/index.js

{
  "title": "TEST CMS",
  "elements": [
    {
      "text": "Chapter 1",
      "type": "h1",
      "elements": [
        {
          "text": "some text element between h1 and h2 tags",
          "type": "p"
        },
        {
          "text": "text",
          "type": "h2",
          "elements": [
            {
              "text": "vitae elementum velit urna id mi. Sed sodales arcu mi, eu condimentum tell.",
              "type": "p"
            }
          ]
        },
       ...
}

For full example see md-to-json/examples/example_output.json.

How to run the development environment

Coding style convention ref optional, eg which linter to use

Linting, github pre-push hook - optional

• node
• npm
• eslint see .eslintrc.json

How to run build

NA ?

How to carry out tests

Minimal test coverage using jest for testing, to run tests:

npm test

How to deploy the code/app into test/staging/production

NA, it's a node module.

• Pull requests are welcome.
• For questions, bugs, ideas feel free to raise a github issue.

Unforntunatelly, Dropbox paper has it's own flawour of markdown. Some of the most relevant and notable difference are:

• Title of the doc and first heading 1 element, are both marked has h1 / #.
• Heading 3 is represented as bold ** instead of h3/###.
• There's no Heading 4, 5 or 6.

see md-to-json/examples/test.md as an example of dropbox flavour markdown file.

• H3 tag,since dropbox paper markdown represents it as bold **
• Parsing markdown github flavour tags h3 to h6 as not generated by dropbox paper markdown.

• Parsing markdown github flavour tags for images eg ![alt text](link url). These appear on their own line.

  • NOTE luckily even when displayed on the same line in dropbox paper, the images are still represented on individual lines when exported as markdown. Which makes it easier to identify as separate from other elements and parse.

• Parsing markdown github flavour tags for links eg [text](link url) these generally appear as part of a paragraph, but could also appear in their own line, or as part of a heading etc..