Git Product home page Git Product logo

roots-records's Introduction

Roots Records

NPM version tests coverage dependencies

Load remote data into a roots project and make it available in your views as locals.

Installation

  • Make sure you are in your roots project directory
  • Run npm install roots-records --save
  • Modify your app.coffee as such:
    records = require('roots-records')
    
    # ...
    
    module.exports =
      extensions: [
        records(
          books: { url: 'http://www.google.com/books' },
          shoes: { file: 'data/books.json' },
          scores: { data: { home: 1, away: 0 } }
        )
      ]
    
    # ...

Usage

Roots Records is a extension that you can use to fetch data from a url, file, or JSON object, manipulate it, and inject into your view templates. It can also be used to compile out additional templates for each item in a collection (like single blog posts for a blog), as discussed below in the "options" section.

This extension is a great way to separate markup and content. With Roots Records, you can keep your content in an API, but still compile it out as static content through Roots. In addition, if there's a section of the site you need to be more dynamic, you can still access the exact same data through an ajax request.

Configuration

You should pass a single options object to the roots-records function. This object can have as many keys as you'd like, each key specifying the name of a set of data you'd like to pull into your views, and with the value of each key an object allowing you to configure from where you'd like to fetch this data and modify it before being injected into your views.

The following options are available:

url, file, or data

A resource returning JSON that you would like the data from to be included in your views. URLs should be absolute paths including http, files should be a path relative to the root of your project, and data is simply a direct JSON object. For example:

# using a URL
records(
  books: { url: 'http://www.google.com/books' }
)

# using a file path
records(
  books: { file: 'data/books.json' }
)

# using JSON data
records(
  books: { data: [{ title: 'Wow', author: 'Doge' }] }
)

For the url option, if you need a bit more control, you can also pass an object instead of a string. The object can have the following values, all optional except for path:

records(
  books:
    url:
      path: 'http://www.google.com/books'
      method: 'GET'
      params: { year: '1980' }
      headers: { Security: 'secret' }
      entity: { authorName: 'Darwin' }
)

For more details on what each option here means and a couple more obscure options, feel free to check the rest documentation, which is the library internally handling this request.

Note that it is possible to pass multiple different keys to roots-records that fetch data in the same or different ways. For example:

records(
  books: { url: 'http://www.google.com/books' }
  sodas: { data: ['Dr. Pepper', 'Coke', 'Sprite'] }
)
hook

An optional function that receives the JSON response and allows you to make modifications to it before it is injected into your views. For example:

records(
  data: { foo: 'bar' }
  hook: (data) ->
    data.foo = 'changed!'
)

With this modification, the value of foo in you views would now be changed! rather than bar, as it was in the original data passed. This is of course a contrived example, for more complex data, more interesting transformations can be done here.

template and out

So imagine this. You are making a blog, and roots-records is pulling from your blog's API and returning an array of blog posts. In addition to using the locals to list out an index of your posts, you also want to compile a standalone single page view for the each full blog post. Roots records can still handle this with a set of two options, You can use the template option to specify a single view template which will be used to render out each single item in your collection. This can be a string or a function. The out option is a function that receives each individual option in your collection as an argument. You should return the path, relative to the project root, to which you'd like to write your single view. For example:

records(
  books:
    url: 'http://www.google.com/books',
    hook: (res) -> res.response.books,
    template: "views/_book.jade",
    out: (book) -> "/books/#{slugify(book.title)}"
)

Note that the slugify function in the last piece is fictional, although you can find similar string transformation functionality in the underscore.string library if you need it. Also note that in order for this single page views to work correctly, the data you are returning must be an array -- you can also use the hook option to make this transformation if necessary, as shown in the example above.

Inside your single view templates, the data for your item can be accessed with the item key. In addition, the full contents of the records locals will still be available inside all single view templates.

Accessing Data In Your Templates

In your view templates, all roots-records data will be available under the records local. So, for example, if you have a books key that's pulling some book data as shown in many examples above, in your views you would be able to access this data under records.books, as such:

!= JSON.stringify(records.books)

In a jade template, this example would simply print out the contents of the books data that you pulled. Of course you can chop up and iterate this data further in any way you need using jade, or whatever other templating engine you are using. And of course if you are fetching multiple data sources, each one will be found on records under what you named it when passing options to roots-records.

If you are using single view templates through the template and out options, you can access all data as usual through records, and additionally will find an item local, through which you can access the data for the specific item in the collection that the view is for.

License & Contributing

roots-records's People

Contributors

nporteschaikin avatar joshrowley avatar zspecza avatar knupska avatar hhsnopek avatar kylemac avatar i8ramin avatar

Stargazers

Ikaika Hussey avatar Rob McCardle avatar Edin Abazi avatar Jay Zeschin avatar Vivek Dhar avatar Kevin Bohinski avatar Reuben Cummings avatar Lynsei avatar Krispin Schulz avatar Jon-Kyle avatar Duncan avatar Marcos Sader avatar Joe Vallender avatar  avatar Nickolas Kenyeres avatar Branko Vukmirovic avatar Marcel Reschke avatar Thiago de Bastos avatar M Haidar Hanif avatar Bud Parr avatar  avatar Sean Lang avatar Jeff Escalante avatar  avatar

Watchers

 avatar Jeff Escalante avatar James Cloos avatar Carrot avatar Bruno Germansderfer avatar Henry C. Dickson Jr. avatar Elvin G. Marmol avatar Mike Adamitey avatar  avatar Aleksandra Matiyev avatar Michael Duke avatar  avatar

roots-records's Issues

[FEATURE REQUEST] Add single view compiling support to roots-records

I would like to serve Roots JSON API data from our custom CMS and be able to create the pages from the pages generated from our API.

This feature would allow me to utilize Roots sort of how you utilize Roots-Wordpress or Roots-Contentful, and allow me to continue to use our CMS for managing the content.

Thank you so much for your time!!!

Single view template files require being ignored, results in lack of live reload

If one does not prepend an underscore to a single view roots-records template, the template is compiled as a regular view and this sometimes causes error because the plain roots context does not provide any of the roots-records variable locals. However, prepending an underscore so that roots ignores the file will also prevent live reload, forcing the developer to restart the roots watch process after making any changes to layout.

I have a hunch this is a simple fix that involves setting the extract option to true and matching the file name against the name provided to the options parameter. This fix could also include any local data changes to JSON files.

Something real weird going on with single views

I am not sure what exactly is happening, but something is up. If you use a single view template, it compiles on the first pass. But as soon as you change and livereload triggers, it errors out with an error that is incorrect and makes no sense (for me, it's looking to extend a jade layout at a nonexistant path from a css file).

I'm fairly confident this is something going on with roots-records. Going to look into it further soon.

No livereload on local data update

It seems that I can't have livereload on updating local data in a .json file. It seems like the server does notice the saving, as there's a short flicker in the browser, but the data in the page is not updated.

New API call for Single Item

I have the following situation:

  • the records call is one API call, which returns a list of items, with some information
  • for each item in the records, there is an additional API call available at (DIFFERENT_API_URL/ + item.url), which serves richer information

Can I use the roots records to populate this?
Use the hook to add the additional information?

Ideally, I would like to serve static pages,
rather than loading in the extra information via JS.

Thanks

Error: URL did not return JSON

Having some issues using this extension when trying to compile for production. My API endpoint is secure and requires Authorization headers. I've verified that the tokens are correct and the endpoint works when I use Postman.. and does indeed return JSON.

  extensions: [
    records(
      locations:
        url:
          path: 'https://www.site.com/api/v1/locations.json'
          headers:
            method: 'GET'
            Authorization: 'Token token=*****************************'
    ),
  ]

Error from the compile:

โ€บ roots compile --verbose -e production
compiling...
[Error: URL did not return JSON]

API response in Postman:
screen shot 2015-09-17 at 10 30 08 am-1

Any flags I can pass into roots compile to see more details about the error? Any way I can debug this by adding a breakpoint or something in the node_modules folder for roots? Any help/clues is greatly appreciated.

`records` local is empty array inside layout and single-view template files

๐Ÿ’ฉ

index.jade - records = [...] - local behaves as expected.
layout.jade - records = []

With settings:

categories:
        url: retrieve_models 'Category'
        template: 'views/_category.jade'
        collection: (categories) ->
          categories = categories.filter filter_current_locale
        out: (category) ->
          "/articles/categories/#{category.title.toLowerCase().replace(/ /g, '-')}"
        hook: (categories) ->
          categories = categories.filter filter_current_locale

Inside _page.jade: records.categories === undefined.

SyntaxError: Unexpected token u

I am trying to make roots-records work but I cannot get it to read the file output by roots-yaml. It works fine if I pass in json key value pairs. Any pointers on what I am doing wrong here?

The error

compiling... โœ˜ SyntaxError: Unexpected token u

Config

# app.coffee
    extensions: [
        yaml(),
        dynamic_content(write: {'posts.json': 'posts', 'portfolio.json': 'portfolio'}),
        js_pipeline(files: ['assets/js/*.coffee', 'assets/js/*.js']),
        css_pipeline(files: 'assets/css/*.styl'),
        records({
             posts: {file: 'posts.json'}, # also tried {file: 'posts.json', path: 'code'}
            }),
        config(related_items: 3)
    ]

The posts.json file

{

    "items": 

[

    {
        "_render": false,
        "_content": false,
        "_url": "/posts/posts.html"
    }

],
"code": 
{

    "items": 

[

{

    "date": "2014-01-15T00:00:00.000Z",
    "title": "JSON Test",
    "description": "Creative journey and case study for the Dawn Lister Therapy Centre website.",
    "category": "code",
    "tags": 

    [
        "webdesign"
    ],
    "relatedimageurl": "http://lorempixel.com/325/250/abstract/1",
    "googlewebfonts": "Lato|Open+Sans:400,700,800",
    "height": "125px",
    "_content": false,
    "_url": "/posts/code/JSON-test.html"

},
{

    "date": "2014-01-15T00:00:00.000Z",
    "title": "Isotope Test",
    "description": "A testing page for Isotope JS",
    "category": "code",
    "tags": 

            [
                "test"
            ],
            "relatedimageurl": "http://lorempixel.com/325/250/abstract/1",
            "googlewebfonts": "Lato|Open+Sans:400,700,800",
            "height": "125px",
            "_content": false,
            "_url": "/posts/code/isotope-test.html"
        }
    ]

},
"creations": 
{

    "items": 

[

{

    "date": "2015-02-24T00:00:00.000Z",
    "title": "First Art",
    "category": "creations",
    "tags": 

            [
                "colourful",
                "lovely",
                "psychadelic"
            ],
            "height": "125px",
            "_content": false,
            "_url": "/posts/creations/01-first-art.html"
        }
    ]

},
"thoughts": 
{

    "items": 

[

{

    "date": "2014-03-03T00:00:00.000Z",
    "title": "Hello World",
    "category": "thoughts",
    "tags": 
    [
        "colourful",
        "fun",
        "happy"
    ],
    "height": "125px",
    "_content": false,
    "_url": "/posts/thoughts/01-hello-world.html"

},
{
    "date": "2015-02-23T00:00:00.000Z",
    "title": "My Second Post!",
    "category": "thoughts",
    "tags": 

                [
                    "inspiration",
                    "habit"
                ],
                "height": "125px",
                "_content": false,
                "_url": "/posts/thoughts/02-my-second-post.html"
            }
        ]
    }

}

Can the hook method return a Promise?

I am happy to submit a pull request, just want to check if this has any chance of being merged in before doing the work. I would simple Promise.resolve() the hook function, I've checked briefly and I think this would work fine because the apply_hook is passed to when.map() anyway.

Thanks.

Error: ENOENT, no such file or directory

Weird, i found this error(?) in my free testing-time.

Anyway, when i try an roots watch (v3 rc-7), it throws me:

compiling... โœ˜ Error: Error: /home/nano/Dev/test/roots-tests/roots-record-test/assets/js/main.coffee:1
  > 1| extends layout
    2| 
    3| block content
    4| 

ENOENT, no such file or directory '/home/nano/Dev/test/roots-tests/roots-record-test/assets/js/layout.jade'
  at Error (<anonymous>:null:null)
  at on_error (/usr/local/lib/node_modules/roots/lib/cli/watch.js:67:21)
  at Roots.<anonymous> (/usr/local/lib/node_modules/roots/lib/cli/watch.js:45:14)
  at Roots.EventEmitter.emit (events.js:95:17)
  at Compile.<anonymous> (/usr/local/lib/node_modules/roots/lib/api/compile.js:66:20)
  at tryCatchReject (/usr/local/lib/node_modules/roots/node_modules/when/lib/makePromise.js:790:14)
  at RejectedHandler.when (/usr/local/lib/node_modules/roots/node_modules/when/lib/makePromise.js:658:9)
  at DeferredHandler.run (/usr/local/lib/node_modules/roots/node_modules/when/lib/makePromise.js:444:13)
  at Scheduler._drain (/usr/local/lib/node_modules/roots/node_modules/when/lib/scheduler.js:56:14)
  at Scheduler.drain (/usr/local/lib/node_modules/roots/node_modules/when/lib/scheduler.js:21:9)
  at process._tickCallback (node.js:415:13)

Ahm, i'm not doing something special, just a stupid test:

axis         = require 'axis'
rupture      = require 'rupture'
autoprefixer = require 'autoprefixer-stylus'
records      = require 'roots-records'

module.exports =
  ignores: ['readme.md', '**/layout.*', '**/_*', '.gitignore']

  extensions: [
    records({
      books: {url: 'https://www.googleapis.com/books/v1/volumes?q=The+Great+Gatsby'}
      })
  ]

  stylus:
    use: [axis(), rupture(), autoprefixer()]

and, in the index.jade (/views/index.jade):

extends layout

block content
  !=JSON.stringify(records.books)

So, why it say me "i can't find nothing in assets/js/..."? I saw in documentation that references.

I may be jumping something, but really do not know, I was just testing this out of curiosity and because it can serve later.

Cheers!

Single view template ignores `hook` transform

I'm not sure if this is a bug or by design or simply just a result of limitation, although from the order in which these functions are inspected and called in lib/index.coffee suggests that this might be a bug.

When both the hook method and the single template options are present on a collection passed to roots-records (in my case via the url option), the records.<collection> local is properly transformed as per the hook, but the single view template still refers to the old data, as if the hook method were never called.

This is problematic because, in my case, I have to reference an API that is not within my control, and that API points to blog posts that are in two separate languages. I'd like to filter out only the language I need. Eventually, I'd like to separate out each language using Roots' environments feature (as I have done in the past on another project using dynamic-content).

Following is the extension code - note that retrieve_models simply returns a URL to an API endpoint and in this specific case current_locale evaluates to "eng_GB". (The other language in question is Swahili).

records
      pages: 
        url: retrieve_models 'Page'
        template: 'views/_page.jade'
        collection: (pages) ->
          pages
        out: (page) ->
          "/articles/#{page.slug}"
        hook: (pages) ->
          pages = pages.filter (page) ->
            page.language is current_locale
          pages = pages.map (page) ->
            if page.linked_pages?.length
              page.linked_pages = page.linked_pages.map (uuid) ->
                recommended = {}
                for linked_page in pages when linked_page.uuid is uuid
                  recommended.title = linked_page.title
                  recommended.slug  = linked_page.slug
                  break
                recommended
            page
      locales:
        url: retrieve_models 'Localisation'
      categories:
        url : retrieve_models 'Category'

Here's a snapshot of the generated views (notice the Swahili mixed in with the English):

If I print out JSON.stringify(records.pages, null, 2) on the root index view, none of the Swahili page items are present, which is the expected behaviour.

Data duplication ๏ผŸ

app.coffee

axis         = require 'axis'
rupture      = require 'rupture'
autoprefixer = require 'autoprefixer-stylus'
js_pipeline  = require 'js-pipeline'
css_pipeline = require 'css-pipeline'
records      = require 'roots-records'
templates    = require 'client-templates'
config       = require 'roots-config'
S            = require 'underscore.string'
moment       = require 'moment'

api_url = 'http://192.168.1.22:8080/v2/'

module.exports =
  ignores: ['readme.md', '**/layout.*', '**/_*', '.gitignore', 'ship.*conf', 'assets/*']

  extensions: [
    js_pipeline(files: 'assets/js/*.coffee'),
    css_pipeline(files: 'assets/css/*.styl'),
    templates(base: 'views/templates')
    records(
      test:
        file: "/data.json"
        template: "views/test.jade"
        out: (res) -> "/books/#{res.title}"
    )
  ]

  stylus:
    use: [axis(), rupture(), autoprefixer()]
    sourcemap: true

  'coffee-script':
    sourcemap: true

  jade:
    pretty: true

  locals:
    moment: moment

  #server:
  #  clean_urls: true

test.jade

!= console.log(item)
!= console.log('+++++++++++++++=============')
!= console.log(item.id)

data.json

[
  {
    "id": 1001,
    "title": "The Great Gatsby",
    "summary": "A green light at the end of a dock symbolizes hedonic adaptation",
    "author": "Doge",
    "description": "this is a test"
  },
  {
    "id": 1002,
    "title": "The Great Gatsby2",
    "summary": "A green light at the end of a dock symbolizes hedonic adaptation",
    "author": "Doge",
    "description": "this is a test2"
  }
]

result:

roots clean && roots watch                            
โœ“ output removed
compiling... { id: 1001,
  title: 'The Great Gatsby',
  summary: 'A green light at the end of a dock symbolizes hedonic adaptation',
  author: 'Doge',
  description: 'this is a test' }
+++++++++++++++=============
1001
{ id: 1002,
  title: 'The Great Gatsby2',
  summary: 'A green light at the end of a dock symbolizes hedonic adaptation',
  author: 'Doge',
  description: 'this is a test2' }
+++++++++++++++=============
1002
{ id: 1002,
  title: 'The Great Gatsby2',
  summary: 'A green light at the end of a dock symbolizes hedonic adaptation',
  author: 'Doge',
  description: 'this is a test2' }
+++++++++++++++=============
1002
done!

Results 1002 repeat cycles 2 times?

YAML

Hi guys, I was wondering if I could use the templating/out feature of this with YAML data. For example, in middleman, I can make proxies based on YAML data files. For example, in Middleman, I can do:

data.people.each do |name|
  proxy "/about/#{name}.html", "/about/template.html", :locals => { :person_name => name }
end

This would create a new page for each person in the YAML file. Is there a way to do this with roots-records without using JSON?

Thanks!

Error "Cannot read property 'X' of undefined" when using records-data in included jade-template-partital

Hi there,

When I try to display a JSON key/value pair in an included jade template partial, I get the following error:

Error:

compiling... โœ˜ Error: TypeError: heldenschaum-site/views/blocks/header.jade:1
  > 1| h1= JSON.stringify(block.id)
    2| 

Cannot read property 'id' of undefined

Potentially unhandled rejection [34] TypeError: heldenschaum-site/views/blocks/header.jade:1
  > 1| h1= JSON.stringify(block.id)
    2| 

Cannot read property 'id' of undefined
  at eval (<anonymous>:12:77)
  at eval (<anonymous>:17:22)
  at res (heldenschaum-site/node_modules/jade/lib/index.js:219:38)

header.jade:

h1= JSON.stringify(block.id)

index.jade:

extends layout

block content
  for block in records.blocks
    case block.type
      when 'header'
        include blocks/header
        h1= block.properties.title
        h2= block.properties.subtitle

  :markdown
  Find tutorials and documentation at http://roots.cx : )

_blocks.json:_

{ "data":
  {
    "id":"1",
    "type":"header",
    "test":"test",
    "properties":{
      "title":"Title",
      "subtitle":"Subtitle",
      "bgcolor":"blue"
    }
  }
}

_extension configuration:_

records(blocks: {file:'data/blocks.json', path:'data'})

I tried different keys, always the same error. Records data can be accessed in index.jade perfectly and when I modify header.jade to:

h1= JSON.stringify(block)

The h1-headline displays correctly:

{"id":"1","type":"header","test":"test","properties":{"title":"Title","subtitle":"Subtitle","bgcolor":"blue"}}

I'm not sure if it is a real issue or it is my fault. Please have a look.

Thanks in advance and for awesome roots generally.
Marcel Reschke

Weird issue: folder with single generated views not working on netlify

I'm not sure if this is a netlify problem or a roots-records problem but I was hoping someone would have some insight on it. When I run my site locally, roots-records works fine and I can access my generated pages using links. When I run compile on netlify - most of the pages work but I can't follow generated links. When I download the compiled source, The folders (in this case "post" and "sermons") look like exec files.
Any help would be appreciated.
screen shot 2016-06-09 at 6 25 30 am

Hook examples unclear (to me)

I want to make use of the ability to modify the JSON (from a URL) before calling it in my template, but am struggling with the examples provided.

Existing code:

  records(
    myapi: { url: http://www.path.to/api/01?key },
    anotherapi: { url: http://www.path.to/api/02?key },
  )

I have tried a bunch of stuff, all resulting in errors. The closest to the example would be something like:

  records(
    myapi: 
      url: http://www.path.to/api/01?key,
      hook: (url) ->
        url.my.value = 'changed!'
   )

or:

  records(
    myapi: 
      url: http://www.path.to/api/01?key,
      hook: (data) ->
        url.my.value = 'changed!'
   )

Any pointers appreciated.

caching

concentrate on the base implementation, but I just wanted to leave this here for when we're ready to make improvements...

It would be great that if configured to do so, the extension could store a cached json response when using the url source to reduce overhead when watching a project.

Feature proposal: Allow `all_hook` method to provide access to all records locals

I know this may seem like an edge case scenario, but I've been encountering it more and more often lately, especially on projects where my company has partnered with other companies who provide the data and we have no control over the structure of the API.

An example of a problem that this feature might solve is as follows:

Given three API endpoints -

  • /api/locales.json - contains only a UUID and name for each locale
  • /api/categories.json - contains a flat list of categories in multiple locales (referenced by their UUID)
  • /api/articles.json - contains a flat list of articles in multiple locales (referenced by their UUID)
    each article has a category property that is a UUID pointing to a category.

I might want to consolidate all of this data into one record document for easier parsing - so each locale will have a list of categories, and each category will have a list of articles.

Currently, for simple cases, there is only one workaround - to compute this data in the templating engine. However, if single template views are concerned, there is no way to attain this functionality.

So, I propose a feature, the syntax for which looks something like this:

all_hook: (records) ->
  for locale in records.locales
    locale.categories = []
    for category in records.categories
      category.articles = []
      for article in records.articles
        if article.category is category.uuid and article.locale is locale.uuid
          category.articles.push article
      if category.locale is locale.uuid
        locale.categories.push category
  return records

As a result of running this function, records.locales will now contain updated data, and in conjunction with a possible solution to #13, could make roots-records far more flexible than it is at the moment.

Generate list of records from object

I don't know how much this is a Records question and how much it's a Coffeescript one (I'm new to both), but is something like this possible?

hash = 
  somelabel: '12345'
  anotherlabel: '67890'

module.exports =
  extensions: [
    records(
        for key, value of hash
            #{key}: { url: api_root + #{value} + api_key },
    )
  ]

(assuming that api_root and api_key are defined somewhere)

The desire here is mainly to avoid having to duplicate the api_root and api_key stuff, but also potentially have that hash generated from another API.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.