mapbox / batfish Goto Github PK

View Code? Open in Web Editor NEW

126.0 126.0 17.0 5.36 MB

A static-site generator for React and Markdown

License: ISC License

JavaScript 99.75% Shell 0.09% EJS 0.15%

batfish's People

Contributors

Stargazers

Watchers

Forkers

sudhakar the-cc-dev wslerry rubythonode zonakre awchang56 ming-mapbox schnjaso2 memualexa roquestrew websiteinspiration jennifer2018cs bigemap isabella232 crocla mapclone rogermparent hackerdecoupled

batfish's Issues

Is siteOrigin required?

This configuration property is used in two places currently:

In prefixUrl.absolute(), if you use that.
In sitemap generation.

Sitemaps won't work at all without it. Should it be required?

Keep core-js polyfills out of the bundles

The Symbol polyfill from core-js creeps into the bundles. I believe this is because babel-plugin-transform-runtime adds a polyfill automatically when Symbol is mentioned; and React, at least, mentions Symbol — though it does so to check for its existence and not use it if it's not available.

Alternatives I can think of:

Don't use the plugin and don't bother with an alternative; instead, just let Babel put its helpers in each file.
Figure out something else?

Add to docs: Using Helmet to modify <head>

I forgot to document that you should use react-helmet to modify the document <head>. It is in there as a peer dependency — just forgot to document.

Planning: Markdown support

There are two kinds of Markdown support to consider: Markdown embedded in JS, and Markdown files used as pages.

Markdown embedded in JS

Looking something like this:

import md from 'some-module-name';

const text = md(`Some **markdown**`).

This would provide the flexibility we'd want to be able to use Markdown anywhere that we have extended prose.

However, we don't want to bundle a Markdown parser and send it to the browser. I think the way to do this is to build a Babel plugin that uses remark-react at compile time to transform all uses of md().

Here's an existing Babel plugin to learn from: https://github.com/threepointone/markdown-in-js

Markdown files used as pages

This is the class Jekyll format. Both Phenomic and Gatsby support it, so it should be easy to learn from their open-source code. We'd use front matter to pass props to a specified layout component.

There's a big limitation with this format: Although (I think) you could embed HTML directly into the Markdown in a document like this, you could not embed JSX and use React components. If you want to embed JSX, it makes more sense to avoid this additional level of abstraction and write the page as a (relatively simple) JS file. I think that's ok. If people need to do a minimal amount of custom styling interspersed in their prose, they can do that with HTML. If they need more, they can create a page component.

Document routeTo and routeTo.prefixed with examples

I forgot routeTo.prefixed, and also forgot to add examples.

ES2015 module syntax: is it worth it? ☉_☉

The use of ES2015 module syntax is established in Studio, and currently in dotcom and assembly-components.

With this tool we need to make prescriptions about the way users export pages. Either Node.js module syntax or ES2015 module syntax. And I'm a little bit inclined to go with Node.js module syntax, for these reasons:

One of the benefits of this build system will be that it's Node-based, and lots of folks at Mapbox are familiar with Node. Node does not have ES2015 module syntax, so most of those folks are not familiar with it.
I'm not sure there's any real benefit to using the new syntax this early, considering that Node.js module syntax will need to continue to be supported until the end of time. (Tree-shaking schmee-schmaking, that's what I say.)
I'm finding myself wanting to take advantage of require to make the system work. If we're going to use require in a bunch of places, why not everywhere?

@mapbox/studio: What do you think? Any defenders of ES2015 module syntax?

Planning: pages/ directory

I'd like to carry over from dotcom the convention that a pages/ directory determines your routes. This mimics the systems that work well in Jekyll, Next.js, and other things.

One improvement I'm thinking about, over what we have in dotcom: Let's allow people to put asset files that are not part of any processing pipeline into pages/ directories. We'd make this work the same way Jekyll does it: anything that is not processed is copied directly into the built directories, at the same location.

Planning: Image system

The dotcom repo includes the MagicImage system.

Two parts to this, both of which rely on a single config:

A Node module that reads the config and creates cropped, optimized, WEBP-plus-fallback versions of the specified images at their specified sizes.
A React component that reads the config and its input props to determine which image version to load based on the width available to the element.

I think this system has worked well for dotcom, and we should carry it over. This could be a standalone open source package — so we could put it in its own repo, or we could make this repo a Lerna MegaRepo.

Maybe we should call the image system ChunkLite, instead of MagicImage ...

@samanpwbb @mayagao @tristen: Have you run into any problems with MagicImage, or do you have ideas for how it should be improved before we generalize it?

Front Matter in React component is parsed once per build

It's not a huge deal, but it seems that changes to front matter aren't reflected in the props that get passed to their respective React components. The dev server reloads the app when the file changes, but the values in props remain the same.

Document autoprefixing

All CSS is run through Autoprefixer. Forgot to document that.

404 on production vs development

The following images are based on my example in the markdown-world branch.

Scenario 1

On development, paths that don't exist and aren't draft pages get a 404:

On production, these paths get the following:

Scenario 2

On development, draft pages that use published: false are visible by feature default:

On production, draft pages also get the following:

TODO

Ideally for production, in both the above scenarios these paths would go to the 404 page

404s

Gotta establish what happens if you hit a route that doesn't exist.

CSS, generally

Besides inlining critical CSS in the static build (#4), there are other considerations:

Authors need to be able to designate page-specific CSS, as text and as URLs.
Authors need to be able to designate site-wide CSS, as text and as URLs.
While the URLs above code be external (e.g. the Assembly CDN URL), I think we'll ultimately pull it down, bundle it up, and serve it locally. This way we can exercise some more control for optimizations, like delaying the loading of the site-wide CSS file until rendering is complete (since it's unnecessary before you change pages, since all CSS for the page will be inlined). This means we need to have users specify CSS as URLs, not as <link> elements to add to the head.
While we could include some CSS processor defaults (maybe just Autoprefixer), it'd also be nice to allow users to specify PostCSS plugins to add to pipeline.

Making it work from different subdirectories within the domain

This needs to be able to build sites that aren't at the root of the domain. Dotcom was able to cut some corners because it did not need to account for this. Making this issue so I don't forget about this vital feature.

Document batfishConfig.babelExclude

Forgot this one.

Option to exclude draft pages from the production build

We'll need this option.

Documentation around markdown options isn't clear

https://github.com/mapbox/batfish/blob/master/docs/configuration.md#jsxtrememarkdownoptions includes a list of configuration options. The list includes both https://github.com/mapbox/jsxtreme-markdown#tojsx options as well as https://github.com/mapbox/jsxtreme-markdown#tocomponentmodule – some clarification in batfish about the differences between these two types of options and how they should be used, even just an extra sentence, would be helpful for folks new to batfish.

Planning: SVG system

The dotcom repo has a module that converts raw SVGs to React components.

We can port this over.

@tristen @samanpwbb @mayagao: Do you think there are any problems with the dotcom SVG system that we should definitely address before generalizing it?

batfish.config.js doesn't seem to be optional

docs say "Create a configuration module (or don't, if you want to rely on defaults)."

Without a config file, on build, I get an error here:

batfish/lib/build.js

Line 27 in f6e4edd

if (rawConfig.production === undefined) {

, where rawConfig is undefined.

Seems related, but not quite the same as #43

Write something about differences from Gatsby

Gatsby released v1 and I'm sure a question on the minds of anybody who comes across this repo will be "Why not Gatsby?" So I should write up a document that describes the differences of the approaches.

For file-loader config, use `exclude` instead of `test`

I notice that create-react-app does this. Seems like a good way to avoid having to add forgotten extensions to the list every time you think of some other file type you'd like to use.

Create cut-the-mustard tests for outdated browsers

@davidtheclark wrote:

Let's figure out a good cut-the-mustard tests for IE<11, Android<4, Safari<7, and any other browsers like those, and show a banner to those users.

Here's some additional context about "cutting the mustard" from BBC News engineers.

Let's identify which features might be problematic in the supported browser ranges and create simple tests for them. If a test fails, we should display some warning to the user, indicating that they may experience some issues, and provide a link to help them upgrade (maybe http://outdatedbrowser.com).

Here are some implementation ideas:

Run on every page by default
- Provide mechanism for some pages to opt-out
Expose a function that returns whether the browser cuts-the-mustard or not, so pages can act on this accordingly
Allow users to dismiss the banner, and remember the dismissed state so we don't warn them repeatedly
- Maybe storing the dismissed state isn't reasonable on older browsers without localStorage
Feature detection could be a util in batfish, so consumers can handle the failed tests as they wish

Next steps:

identify potentially problematic features
create tests for each potentially problematic feature
design the banner

Ensure users can use asset optimization loaders/plugins

The file-loader is used to hash and copy images, videos, and fonts. We should make sure that Batfish is configurable such that a user could throw in an image-optimization loader of their choice, e.g. https://www.npmjs.com/package/image-webpack-loader.

Check the existence of files during config validation

This would be the best way for us to provide helpful clear error messages if, for example, your wrapperPath is incorrect and does not actually point to a file.

Webpack v3

Webpack v3 came out. Let's see if everything here is compatible with it, and if so upgrade.

Change tagline

"The React-powered static-site generator you didn't know you wanted."

There are good reasons for batfish! The project readme should lead with a clearer statement of purpose / value

Collections derived from build

Jekyll's collections include some derived from the build, such as pages and posts, that can be handy — mostly when creating lists.

Ideally all data is available at compilation time, but somehow limited in the client bundle.

(This is what Gatsby v1 is using GraphQL for. I don't want to introduce another hurdle with GraphQL. So let's figure out another way.)

Add option for inline JS

I don't think there's a good way right now to inject some JS that will end up being inlined in the HTML. We'll want this.

Cross-browser testing

Once we have a fully-functional prototype, we need some thorough cross-browser testing.

Sitemap!

We need to build a sitemap.

Add more info for Draft pages in README

Draft Pages in README could have a little bit more information on

Draft pages are built during development but are not included in production builds.

and should point to /404 in production.

API for page-specific CSS

I forgot to create a way to add CSS that is only going to be used on one page, so should not be added to the full site bundle.

Draft API review!

@samanpwbb and @tristen, I'd love it if you could try this out in its current state and give me feedback about the API.

I wrote up some documentation in the README. I think I captured the core features and configuration options.

One obviously missing feature right now is Markdown documents, as we discussed this morning. I'll add that soon.

There is a demo/ directory where I've been experimenting. You can try that out to start, to see how the CLI works and look at some simple examples. Then I think it'd be great if you tried creating a new directory for a fresh start on your own experimental demo.

Changes to front matter or page list require server restart

Because pages are found and front matter is parsed once when the Webpack config is created, it will not update dynamically while the watching & dev server are running.

Definitely should be documented. Maybe can be improved.

Name the tool

Batfish is kind of nice. But we could consider other names, before we get too far. Please populate this list with your best ideas. Or else affirm your appreciation for the name "batfish".

Mabbux

Config validation

We should validate the config upfront, instead of allowing it to produce mysterious, less-easily-debugged errors.

Document the way to exclude pages from production publication

Just add published: false to the YAML front matter:

batfish/demo/src/pages/posts/three.js

Line 4 in f6b4811

published: false

validate-config.js should warn for unknown property

An unknown property probably means a typo, or a missed guess. It makes debugging that mistake easier if there's an error thrown about the unknown property.

Log dev server URL/info only after Webpack's first compile

This would save you the trouble of going to the URL when it's not yet ready.

Remark & Rehype plugins in React pages

Remark and rehype plugins like remark-emojis and rehype-highlight don't work as expected on a React page.

The following is an implementation of the remark and rehype plugins on a pure Markdown page. The above is expected to look similar to this below.

See examples/markdown-world for this example.

Planning: inline critical CSS

In dotcom we have a little system that parses static HTML with Cheerio, then uses PostCSS to and document.querySelector to create a CSS file that only contains selectors that match something on the page.

Because it's pretty slow, we split it across available processes.

I ran across an npm module with the same Cheerio-based system: https://github.com/tscanlin/css-razor. We could try using and contributing to that public module, instead of keeping our own. The code isn't super complicated, so I think it's perfectly fine to keep our own. But do we want to contribute to that project? @tristen what do you think?

Thoroughly test, then extract, hijack-links

If hijack-links.js works well, it is useful for any user of a client-side router. For example, you could avoid the need to use ReactRouter's Link component — something I've always wanted.

We need to thoroughly test this across browsers and devices, though, before we make presumptions.

Planning: client-side routing

We want client-side routing because it feels fast, and the feeling of speed is super important for websites that are built to impress.

I want to avoid using ReactRouter if possible, because it is a very big dependency (much bigger, I think, than is justified by the problems we need it to solve) and it does not actually provide several features that we'd like to have. As I've thought about this more, I've realized that client-side routing is a collection of problems that could (probably should) be broken into different pieces and put together according to the needs and constraints of particular situations. I think we can build something pretty small that fits our needs and constraints pretty well.

Requirements:

Primary use-case (maybe exclusive use-case?) is to deal with a bunch of pre-determined routes. These will be determined by the contents of a pages/ directory. (I suppose it may not be much trouble at all to include the ability to dynamically add and remove routes.)
We don't need to support old browsers with this, so I think we can use the History API directly.
Callback-based response to route changes (a downside of ReactRouter v4). When the user clicks a link, we want the next page's JS bundle to download before we make a change to the React tree. We'll need the ability to add a little loading bar at the top (as on the current site, or GitHub); but we don't want the whole screen to go white right away.
Expose a routeTo function that can be used to programmatically change routes.
Hijack links, instead of requiring that people use a Link components. There's a bunch of prior art trying to do this, so hopefully we can make it work well.
Manage scroll position, mimicking browser behavior with regular pages/links/refreshes. Most of the browsers we support can do this with the scrollRestoration API.
Handles fragment identifiers to scroll to sections of the page. Here's how that's done right now in dotcom.

Key question: Do we care about using the Express-style route definition syntax (e.g. people/:id)? Or can we just use regular expressions, instead, to avoid the extra dependency (e.g. something like people\/([^\/]*)$? Or should we create a more simple little domain-specific-language for our limited needs that would be easy to convert to regexps (e.g. people/{id})?

Here's some prior discussion of ReactRouter, motivations and ideas for a more minimalistic router:

Here are some open source libraries we can use as sources of ideas:

Thoroughly test and extract scrollRestoration

scroll-restoration.js has generic utility for client-side-routed sites, not just Batfish ones. So we should make it into its own package.

Copy files in pages/ that aren't .js or .md directly into static build

This will allow you to do things like:

Add a favicon.
Add some images for social media sharing.
And so on.

Move configuration documentation into a separate doc, with generated table of contents

It would be great to have a table of contents for the configuration docs, and it would also be nice to move them out of the README so it doesn't overwhelm the thing. These goals go hand-in-hand: once it's in its own doc, we can use https://github.com/jonschlinkert/markdown-toc (or something similar) to generate a table of contents.

Planning: API brainstorm

Writing out some initial thoughts about the API. Adding to this list as more things come to mind.

Most importantly, there needs to be a CLI with start and build commands.
build needs to be able to distinguish between production and non-production builds.
There will need to be some kind of configuration, where source and output directories (at least) are specified.
Users will need to be able to modify the Webpack and Babel configurations as needed.
dotcom currently relies on some dynamically generated JS files that are committed to the repo (e.g. routes). Ideally, we don't need to do that. The files can be built to a temporary directory when Webpack starts and somehow exposed to modules during compilation. I'm thinking we could make these things available at build time via require('batfish/context') or something like that.
Expose a global batfish variable. This will have a routeTo function for programmatic routing. It could also contain other stuff.

Test if users can throw in React Router or other nested client-side routing

I would really like for this to be possible. I have no idea if it is. This will take some experimentation.

To be clear about the intention: It would be great if a user could nest React Router inside a page for their own ends.

Examples

We should upgrade the demo directory to be a viable (and not super ugly) example. We might want to add a few examples. We should make sure examples illustrate all of the functionality. Here's a checklist (all of these need to work for both dev and static builds):

Add HTML validation

It might be worth adding a step in here to complain about invalid HTML. The reason in my mind is that interpolation within Markdown can make this especially easy to do — e.g. nest a <div> inside a <p>.