broken out from #15

/pages is not accepting POST requests

it'd be soooo convenient if this service was exposed! https://github.com/nymag/amphora/blob/master/lib/services/references.js

This has something to do with leveldown, but I don't understand it.

Steps to reproduce

1. start up server
2. load components/<name>/instances in your browser
3. refresh page

Error Message

events.js:141
      throw er; // Unhandled 'error' event
            ^
Error: write after end
    at writeAfterEnd (/Users/npecora/www/byline/node_modules/through2-filter/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:144:12)
    at Through2.Writable.write (/Users/npecora/www/byline/node_modules/through2-filter/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:192:5)
    at write (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:623:24)
    at flow (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:632:7)
    at ReadStream.pipeOnReadable (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:664:5)
    at emitNone (events.js:67:13)
    at ReadStream.emit (events.js:166:7)
    at emitReadable_ (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:448:10)
    at emitReadable (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:444:5)
    at readableAddChunk (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:187:9)
    at ReadStream.Readable.push (/Users/npecora/www/byline/node_modules/levelup/node_modules/readable-stream/lib/_stream_readable.js:149:10)
    at /Users/npecora/www/byline/node_modules/levelup/lib/read-stream.js:63:12
    at /Users/npecora/www/byline/node_modules/memdown/node_modules/abstract-leveldown/abstract-iterator.js:24:16
    at Immediate.callNext [as _onImmediate] (/Users/npecora/www/byline/node_modules/memdown/memdown.js:114:5)
    at processImmediate [as _immediateCallback] (timers.js:371:17)

Right now it's putting the full url to /uris/. It should be putting the uri, pointing to the page@published

The /lists/ route is not currently working and the lists API needs unit tests.

For whenever this is brought up again, these are the reasons why there will never be a _ref at the base of any object within this particular project. This post contains logic that will hopefully be referred to from other issues to avoid repeating the same logic in other threads.

For example, we can return something like this:

{
  "a": {
    "_ref": "<someone else's resource id>" 
  }
}

When that object is composed, we will recursively add the data from _ref to the object in a. However, we shouldn't do this:

{
  "_ref": "<my own resource id>" 
}

1. The meaning of the word reference is "something that refers" and I am very hesitant to give the word special meaning for our particular project. That is, it shouldn't suddenly change from representing "that guy over there" to "who I am" because of its position in an object tree.

Reference:
a : allusion, mention
b : something (as a sign or indication) that refers a reader or consulter to another source of information (as a book or passage)
c : consultation of sources of information

2. In a RESTful API, GET <--> PUT is supposed to be commutative in that putting something somewhere should mean that getting the exact same uri should give the exact thing you put there, and vice versa. There are multiple places you could get the same information from, and assigning a ref into the data means you can't save to multiple places without modifying the data each time (i.e., versioning, publishing, etc.).

3. Having a _ref means that you are pointing somewhere else, and that anything in that ref object means that it is someone else's data. Having a _ref at the base is implying that the entire object is someone else's data. Unless you're trying to create an exception?

4. Having a _ref at the base of an object means that you can easily PUT an object with a base-level _ref to a uri that doesn't match; we often PUT to multiple places because of publishing and versioning, and having the uri in the data means we have to edit the data each time we were to do that. Performing a simple diff between objects becomes a multiple step process. Having an object at a certain uri with a base _ref pointing to a different uri sounds like a debugging nightmare.

5. Having a ref at the base of an object means that all of the code that does recursion on our objects will have to be rewritten. This includes how we do composition, how we create new pages, and how we publish data. It would have to be rewriten to ignore the first level of data and then start the recursion, which considering that most of our objects are only a single level or two levels deep, would do terrible things to our performance. For scale, that's about 60% of byline-core's codebase.

These arguments were originally posted here: clay/clay-kiln#89

This Issue is referenced in our routing documentation.

Currently the PR Blog is calling require('@nymdev/amphora/lib/files') a few places in the gulp scripts. Maybe we should expose that like we do the database, e.g. require('@nymdev/amphora').files?

• npm prepublish script?
• ideally each npm-packaged component deals with its own scripts and styles, and puts them somewhere
• should compiled component scripts + styles be version controlled? (my gut says no)
• how do we deal with custom (root-level) components?
• npm run build?
• use browserify for scripts?
• use npm-css for styles?

e.g. site.host, site.path, and state.site.name are empty for this component URL http://localhost.dev.nymag.biz:3001/components/article/instances/pr-first.html

state and site are available to the template from this page URI http://localhost.dev.nymag.biz:3001/press/2015/04/first-page.html

Perhaps asking for a URL that would render the component with the site data; something like http://localhost.dev.nymag.biz:3001/press/components/article/instances/pr-first.html

If I create a site with

host: nymag.com
path: /site1/

It won't mount a router at that path, since the root (/) has no router. Is this an ok compromise?

It works for schema.yaml in components, but not schema.yml

• components/foo is known as "the foo component"
• components/foo/bar is known as "the foo-bar component"
• node_modules/byline-foo is known as "the byline-foo component"
• node_modules/@user/byline-foo is known as "the byline-foo component"

Not sure about what we should do with the npm-scoped components. Should we use the full name (including the scope)? That would eliminate more potential naming collisions.

When a page is scheduled, and the scheduler gets around to publishing it, it doesn't do a PUT to page@published (but rather talks to db directly). This means our site-specific url generation doesn't run, and the publish fails with this error:

Unhandled rejection Error: Client: Page must have valid url to publish.
    at ~/amphora/lib/services/pages.js:172:13

This...makes sense, but maybe we should rethink how the scheduler calls publish.

broken out from #15

• Add web hooks for publishing, and unpublishing
• Sitemaps from list of /uris (assuming less than 50,000 urls)
  • sitemap.txt
  • sitemap.xml
• Use node's built-in EventEmitter instead of Eventify
• Use standard morgan and response-time response logging instead of logging ourselves in our own format. (https://scotch.io/bar-talk/expressjs-4-0-new-features-and-upgrading-from-3-0)
• Dynamic addition and removal of sites at run-time from /sites endpoint.
• Addition and removal of users at run-time from /users endpoint.
• Sites should point to themes instead of sites for styles, assets, typography, etc.
• Sites should have roles, and Users can be assigned into those roles.

Idea: Sending a PUT to @published with no data should do a cascading PUT that clones the latest version of each component. This will allow us to publish layouts (and various components) really easily without having to manually GET and PUT the data for each individual component.

Extremely useful since we don't currently have a way in Kiln to publish layouts (or individual components)

Working on this right now. It's gonna be sweet.

Nunjucks and EJS and Jade oh my!

Defining routes per site, with syntactical sugar

The ability to do tags in normal components requires the ability to store lists, and also to search through components in a breadth-first manner.

The user should be able to preload data into the cache using a bootstrap service. The data will come from either a yaml or a json file in their config.

Use case: Loading a default "base" page requires at least a few components to exist as data.

Right now, our instances are looking deep into @nymdev/amphora/lib/files in our gulp scripts, because we need to get names / directories for components.

Move files up to services and expose them in the main entrypoint, so this becomes an actually-supported part of the module.

In the demo, I'm getting the following from http://localhost.vulture.com:3001/components/story/instances

[
    "/components/story/instances/grubstreet-first",
    "/components/story/instances/grubstreet-second"
]

I should be getting a lot more, so things are getting lost.

Versioning API

GET

/components/:name --> Same as get @latest ≈ @vx.x
/components/:name@version --> Get specific version/tag/special
/components/:name/instances --> List instances
/components/:name/instances/:id --> Get @latest ≈ @vx.x
/components/:name/instances/:id@version --> Get specific version/tag/special
/pages/:name --> Same as get @latest ≈ @vx.x
/pages/:name@version --> Get specific version/tag/special
/uris/:name --> √ Already implemented; no test coverage
/uris/:name@version --> 4xx Client error; doesn't make sense

POST

/components/:name --> 4xx Client error; doesn't make sense
/components/:name@version --> 4xx Client error; doesn't make sense
/components/:name/instances --> New instance, new version, return root-level _ref
/components/:name/instances/:id --> 4xx Client error; doesn't make sense
/components/:name/instances/:id@version --> 4xx Client error; doesn't make sense
/pages/:name --> 4xx Client error; doesn't make sense
/pages/:name@version --> 4xx Client error; doesn't make sense
/uris/:name --> 4xx Client error; doesn't make sense
/uris/:name@version --> 4xx Client error; doesn't make sense

PUT

/components/:name --> Change base component, new version, return root-level _ref
/components/:name@version --> Tag new version; a "publish" if vx or vx.0
/components/:name/instances --> 4xx Client error; doesn't make sense
/components/:name/instances/:id --> New version of instance, return root-level _ref
/components/:name/instances/:id@version --> Tag new version; a "publish" if vx or vx.0
/pages/:name --> New version, return root-level _ref
/pages/:name@version --> Tag new version; a "publish" if vx or vx.0
/uris/:name --> Update / Create (no versions)
/uris/:name@version --> 4xx Client error; doesn't make sense

DELETE

• Can delete any versioned component or page instance
• Can delete any uri. Versioning give 4xx.

Testing

• Via Supertest
• Via Mocha
• Via Travis CI

In a page, the layout property points to the root-level component. We should rename that property to root to be more clear.

When publishing ✨ again ✨ it's PUTting to @published@published

Before v1.0 is released, we should establish the following features' API, so that we don't break the API immediately after "release" and have to become v2 ridiculously fast.

Config

• bootstrap.yml in components should be run in each component after startup.
• bootstrap.yml at root of project should be run after startup automatically.
• all config should be moved into /sites/<site name> folders
• api to generate components with dynamically provided data

Search / Index pages

• paging in the format of { next: "<uri>", prev: "<uri>" , results: [...] }

Maintainability

• /uris/ should have a human-readable version

broken out from #15

It works if they're properties of the page

pages:
  one:
    someComponent: /components/clay-meta-url/instances/0
components:
  clay-meta-url:
    instances:
      -
        url: 2015/09/introducing-press-room.html

that url will become <host>/<path>/2015/09/introducing-press-room.html

It does NOT work if they're in an array

pages:
  one:
    - /components/clay-meta-url/instances/0
components:
  clay-meta-url:
    instances:
      -
        url: 2015/09/introducing-press-room.html

Problem: When someone uses this project as a dependency, they get a bindings error.

Details: This project uses multiplex-templates, which uses deasync, which has an install script called build.js. When other projects run npm install, these deeper projects do not run their install scripts, and then node-gyp fails on those projects when they are referenced.

Proposed Solution: Add a package.json config to this project to run their build.js script for them.

@gloddy @yoshokatana @larrychevres @cruzanmo

When testing the redis adapter in the press blog, and logged the input coming to the batch operations, it seemed that the input was not what should come from formatBatchOperations.
As a result, I had to call formatBatchOperations within the batch function which is not how it think this is supposed to be: https://github.com/nymag/clay-pottery/blob/master/lib/adapter.js#L206

Ex here https://github.com/nymag/amphora/blob/master/lib/bootstrap.js#L196 we log the formatbatchoperations(ops) result but then we call batch directly with ops.

Can somebody clarify if that's intended?

As a user, I want to be able to create a new page (in the bootstrap config/script) that has an instance of a layout and some instances of components.

The POST request is returning 201 and has the correct data, and the entry in /schedule is correct, but page@scheduled doesn't exist

Components that occur twice on the page are getting duplicate styles, e.g. paragraph:

<link rel="stylesheet" type="text/css" href="/css/paragraph.css" />
<link rel="stylesheet" type="text/css" href="/css/paragraph.css" />
<link rel="stylesheet" type="text/css" href="/css/video-embed.css" />

301 vs 302?

Currently, multiplex-templates allows you to pass engines into it (which we're doing in composer.js).

What we can't currently do is pass those engines into amphora when it's instantiated by our clay instance (e.g. the press blog). We should allow this, so people can add their extremely-domain-specific mixins, formatters, etc.

(for example, we have a specific logic for how we display article dates in feeds, that should be attached as a formatter to our nunjucks engine)

Oh noes, they're both backwards!

<!-- Stylesheets Begin -->
<link rel="stylesheet" type="text/css" href="/css/global-footer.css" />
<link rel="stylesheet" type="text/css" href="/css/masthead.css" />
<link rel="stylesheet" type="text/css" href="/css/see-also.css" />
<link rel="stylesheet" type="text/css" href="/css/article.css" />
<link rel="stylesheet" type="text/css" href="/css/article.press.css" />
<link rel="stylesheet" type="text/css" href="/css/tags.css" />
<link rel="stylesheet" type="text/css" href="/css/share.css" />
<link rel="stylesheet" type="text/css" href="/css/mediaplay-image.css" />
<link rel="stylesheet" type="text/css" href="/css/paragraph.css" />
<link rel="stylesheet" type="text/css" href="/css/paragraph.css" />
<link rel="stylesheet" type="text/css" href="/css/video-embed.css" />
<link rel="stylesheet" type="text/css" href="/css/mediaplay-image.css" />
<link rel="stylesheet" type="text/css" href="/css/simple-header.css" />
<link rel="stylesheet" type="text/css" href="/css/byline-editor.css" />
<link rel="stylesheet" type="text/css" href="/css/global-nav.css" />
<!-- Stylesheets End -->

(from Dane's slack messages)

Proposed Task: Detach byline from leveldb/memory; create adapters for leveldb/memory, leveldb/disk, redis, riak, mongo, dynamodb.

Current State: We are hardcoded to leveldb/memory, so data resets each time the server restarts.

Goal State: We would be ready to point to an actual DB.

Duration: One-two weeks because of all the setup and testing involved. The actual coding is small.

• stabilize the api
• document everything #7
• put this on npm (public or private) so we can require it

Added unit tests a better-normalizers branch and two tests fail: Sites normalizePath never ends with / and Sites normalizeDirectory never ends with /.

If the page area points to an array (rather than just a string/ref), byline breaks when serving it:

This happens when the page data looks like:

layout: /components/layout/instances/article
    head:
      - /components/title/instances/0
      - /components/canonical-url/instances/0
    main: /components/article/instances/pr-first

Full error (the first bit is the data composer thinks it's working with.

{ _ref: 
   [ '/components/title/instances/0',
     '/components/canonical-url/instances/0' ] }
TypeError: ref.match is not a function
   at /Users/npecora/www/byline/lib/composer.js:37:64
   at baseFindIndex (/Users/npecora/www/byline/node_modules/lodash/index.js:326:11)
   at Function.<anonymous> (/Users/npecora/www/byline/node_modules/lodash/index.js:3241:23)
   at /Users/npecora/www/byline/lib/composer.js:37:19
   at tryCatcher (/Users/npecora/www/byline/node_modules/bluebird/js/main/util.js:24:31)
   at ReductionPromiseArray._promiseFulfilled (/Users/npecora/www/byline/node_modules/bluebird/js/main/reduce.js:103:38)
   at Promise._settlePromiseAt (/Users/npecora/www/byline/node_modules/bluebird/js/main/promise.js:535:26)
   at Promise._settlePromises (/Users/npecora/www/byline/node_modules/bluebird/js/main/promise.js:646:14)
   at Async._drainQueue (/Users/npecora/www/byline/node_modules/bluebird/js/main/async.js:182:16)
   at Async._drainQueues (/Users/npecora/www/byline/node_modules/bluebird/js/main/async.js:192:10)
   at Immediate.Async.drainQueues [as _onImmediate] (/Users/npecora/www/byline/node_modules/bluebird/js/main/async.js:15:14)
   at processImmediate [as _immediateCallback] (timers.js:371:17)

Getting /components/story/instances/grubstreet-first should not be published data, but is instead getting:

{
    "headline": {
        "required": {
            "social": "This is the first grub story",
            "short": "First grub story"
        }
    },
    "authors": [
        {
            "name": "Nelson Pecora",
            "twitter": "yoshokatana"
        }
    ],
    "date": "2015-04-30T16:20:00.000Z",
    "excerpt": "Nothing much to see here, folks.\n",
    "content": [
        {
            "_ref": "/components/paragraph/instances/2@published"
        },
        {
            "_ref": "/components/image/instances/2@published"
        },
        {
            "_ref": "/components/paragraph/instances/3@published"
        },
        {
            "_ref": "/components/paragraph/instances/4@published"
        }
    ],
    "sources": [
        {
            "name": "New York Times",
            "url": "http://nytimes.com/404",
            "title": "A Mote in Gods Eye"
        }
    ]
}

Currently site.path defaults to / if is undefined. This breaks the rule that site.path never ends with /. This requires if statements in amphora and in templates to only add the site.path if its length is greater than one.

for #3

e.g. http://qa.nymag.com/components/article/instances/pr-first.html is broken.

Can we allow for http://qa.nymag.com/press/components/article/instances/pr-first.html

Related to #118