stencila / encoda Goto Github PK

The start up time of the CLI is curiously high. On Linux, using the binary it takes about 1s:

bash -c "time for i in {1..10}; do bin/stencila-convert --version; done"
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0
0.33.0

real    0m9.936s
user    0m12.604s
sys     0m0.868s

In comparison, nixster takes 0.1s and dockter takes 0.25s (both are built using pkg).

I experimented with commenting out import './boot' from cli.ts and the time (with a binary) was almost exactly the same. However when I commented out import { convert } from './index' it fell to 0.15s. So it may simply be the amount of code that needs to be interpreted by Node at startup time that is causing the slowness. Would using dynamic require() calls be a way around this?

Note sure if we should address this now - that could be premature optimisation.

To make it easy to serve and use the Convert CLI without requiring an installation of Node, we'd like to package it up as a binary package using PKG

This will be needed fairly soon to be used from Stencila Hub.

DSV (Delimiter Separated Values)
The most common and popular are comma and tab separated values. However, the converter should support other delimiters as well.

Currently live citations in Jupyter are done via the extension.

The Word and HTML styling is too garish for my tastes and as a user the very first thing I would do is go and change the styles - particularly of the headings. But I think we should discuss that in another PR.
#59 (review)
:)

The current default themes are based on the Stencila website which, while great for marketing materials, is a little too harsh and distracting for consumption of prose/long-form content.

We'd like to turn down the volume on the branding, the objective being for generated documents to remain identifiable as coming from Stencila without distracting from the content.

Currently we are building MacOS, Windows and Linux binaries all on Linux (Travis CI). We shouldn't be doing this because we are now using native binaries (Chromium and Pandoc at present).

FWIW, the change to pkg options needed is:

pkg --targets=host --out-path=bin .

This is a placeholder issue to discuss points raised in #65 (comment):

Re. Type safety and compile time guarantess:

Type: It's impossible to correctly define a type signature for a function which throws an error in TS. Return type of true | never is as close as we can get to give a hint to users who're reading this code, but when used the type signature hint gets reduced to just true by the compiler, so it's not tremendously helpful.

Re: Where to throw exceptions:

Really depends on where we assign value, but for me an important consideration for composeable/functional paradigms is being able to trust a function to execute and pass along values, or lack thereof.
Exception at a utility/low function level go against that dependability, and undermines the compile time guarantees provided by a type system.
Some ways of designing within these constraints include using things like Either and a "Functional Core, Reactive Shell" architecture.
But this is a broad topic unrelated to this specific PR, so I will go ahead and merge, and create a separate issue to discuss.

Originally posted by @alex-ketch in #65

Related to #43

When converting from plain text formats (Markdown) to rich text formats (Word/etc), we would like to style the output for better readability and aesthetics.
Initially we will have a Stencila branded style by default, but should eventually allow selecting from various themes and publication formatting templates.

DSV (Delimiter Separated Values)
The most common and popular are comma and tab separated values. However, the converter should support other delimiters as well.

Once #45 is completed we should publish the build artifacts, both the standalone binary and the JS packages, to NPM and GitHub releases.

For automating the release process we'd like to use Semantic Release.

For example:

node dist/cli "<p>An emoji: 🎉</p>" --from html --to rpng ./temp.png

Throws the exception:

Error: Only Latin-1 characters are permitted in PNG tEXt chunks. You might want to consider base64 encoding and/or zEXt compression

I suggest we go with zEXt since it has the additional advantage of keeping images smaller.

👍 This will require an change in the xlsx converter. https://github.com/stencila/convert/blob/master/src/xlsx.ts#L121-L161

Originally posted by @nokome in stencila/schema#59

The following properties have been removed from Schemas:

• Table: cells
• TableCell: position

Currently the CLI is pretty bare bones, which is fine for now. But we may want to spruce it up a little using Pastel (React on the command line!).

When we generate an rPNG, we use a headless browser to render an HTML representation of the contents and then capture the result as a screenshot.

For better UX, readability, and branding, we should style the generated images to match the corresponding styles in Stencila Hub.

At the moment there is no feedback whilst for example for ipynb conversion
Success: import from "hello-world.ipynb" to "hello-world.dar"

The attached Google Docs JSON has an unordered list in it that gets converted to an ordered list when converting to markdown.

Output is:

---
title: real converter test
authors: []
---

# This is a file created in the HUB

A para inserted by Nokome

## It will upload to GOOGLE DOCS

1.  test
2.  test
3.  and final test

Expected output is:

---
title: real converter test
authors: []
---

# This is a file created in the HUB

A para inserted by Nokome

## It will upload to GOOGLE DOCS

-  test
-  test
-  and final test

gdoc.json.txt

One of the current testing approaches is to to have many fixtures and automatically iterate through them, doing conversions and comparisons between formats as implied by file name extensions. That's OK but it's cumbersome maintaining an adhoc set of fixtures. Particulary when there are changes to encodings.

We have recently added a custom jest matcher which tests for invert-ability: https://github.com/stencila/convert/blob/90586a9820598ebb36c45169377c24777a001b89/tests/matchers.ts#L9-L30

Currently we are just passing that some sample node trees. However, give we have a schema, we could use json-schema-faker, to generate many node trees and check that the compiler can invert them (ie. unparse and then parse them and get the same node back) e.g.

await expect(jats).toInvert(sampleNode)

As requested by Jean

One vote for tex_math_dollars, so that we can use $ and $$, which is a widely supported syntax.

Downloaded the latest stencila CLI and attempted a convert to docx on the example hello-world.md file. The resulting file is rejected by MS Word for Mac 16.14.1:

Importing in Pages also fails with an "Pages can't read the file" error.

Here is the actual file:
output.docx

One of the things we have been thinking about recently is how to model transclusion of various types of content into documents. In JSON, a transclusion node within a document might look something like:

{
    "type": "Include",
    "source": "my-code-example.py"
}

My initial thought on how to represent an Include node in Markdown was to use Commonmark's generic extension (using remark-generic-extensions) i.e.

!include[my-code-example.py]

However, @alex-ketch pointed out that iA writer takes an elegant approach and here (inspired by a John Gruber tweet) of simply parsing URLs / filesystem paths and including the content e.g.

./my-code-example.py

Or, using a URL

http://github.com/.../my-code-example.py

Of course both could be supported (one explicit, the other more implicit).

$ npx ts-node --files src/cli --from docx --to html upload-test-new.docx

Error: write EPIPE
    at WriteWrap.afterWrite (net.js:799:14)

upload-test-new.docx

An R Markdown cell with the fig.cap option set should be converted into a JATS4M fig[fig-type=repro-fig]

e.g.

```{r fig.width=7,fig.height=6,fig.cap="Figure caption"}
hist(rnorm(min(1000, 1e5)), breaks=40, col=hsv(0.6, 0.9, 1), xlab="Value", main="")
```

to

<fig id="f1" fig-type="repro-fig">
  <caption>
    <title>Figure caption</title>
  </caption>
  <alternatives>
    <code specific-use="source" language="r">#: fig.width=7,fig.height=6
hist(rnorm(min(1000, 1e5)), breaks=40, col=hsv(0.6, 0.9, 1), xlab="Value", main="")</code>
    <code specific-use="output" language="json">{}</code>
  </alternatives>
</fig>

It seems that Texture now requires a <abstract> element e.g.

<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <article-meta>
      <abstract>
        <p>Abstract paragraph.</p>
      </abstract>
    </article-meta>
  </front>

Currently, the R Markdown converter (DocumentRmdConverter) supports importing document title and authors from YAML metadata headers. It would be great to have similar functionality for Jupyter documents.

The Jupyter spec mentions an authors field in the notebook's metadata but seems to be permissive with respect to how that is represented (i.e. string v more complex objects). Other relevant discussions include:

• Fields for author/creator and title
• Support setting metadata like title, author and date
• Notebook metadata proposal (title, subtitle, authors, affiliations)

At this stage, propose to add simple support for importing title and authors from the notebook metadata by generating a Pandoc YAML header. Once that is implemented as a 'proof-of-concept' consult with Jupyter community on long term preferred options for more detailed structure e.g. affiliations.

Currently the <description> tags in the FunctionSchema.rng over in stencila/stencilaonly allow for plain text. It would be useful to allow richer content and to introduce Markdown parsing toFunctionJsDocConverter` when that is done.

Why?

Currently, Function definitions need to be authored in XML which is unwieldy and could discourage developers from writing them.

JsDoc is a convenient, widely used format for writing function documentation. It's @tag format is used for documentation in other languages (e.g. roxygen for R).

This is a high priority because we need to build up the libcore function library.

Approach

Import : The doctrine package can parse JsDoc strings. We then convert that to XML.
Export: No plans to support export to JsDoc at this time

Currently failing because of carriage return character \r: https://ci.appveyor.com/project/nokome/convert/build/1.0.31#L75

Empty cells are ignored during the conversion which breaks the converted Stencila Sheet (Invalid dimension error). Empty cells should be converted to <cell> </cell>.

Code cells in md files aren't converted to executable cells (like is done for Rmd files by DocumentXmdConverter) but just to <code language=....> </code>

Since we may use md files as a source for conversion (esp. containing cells in multiple languages) it would be good to support conversion to code cells in Stencila and indicate that they are executable (where relevant?).

Our .docx converter uses Pandoc. It needs to be refactored to our new API and the Pandoc JSON to Stencila JSON conversion reimplemented based on our new schema.

The tests on Appeyor (Windows CI) fail due to line endings (i.e. \r\n versus \n):

https://ci.appveyor.com/project/nokome/convert

For ergonomics it would be nicer to have a more succinct name.

When parsing xlsx (and ods files) the workbook.Props should be used to populate properties of the CreativeWork created in:

https://github.com/stencila/convert/blob/3934898f58be39a91478fe21739f4726d6e28006/src/xlsx.ts#L45

https://github.com/stencila/convert/blob/ff0d789081b46788ac5df1043505300fda90c806/src/util.ts#L171-L176

Unfortunately this does not account for escaped text like:

header1, header2, header3
text, "some text, it contains a comma", another text 

Originally posted by @alex-ketch in #65

Currently, we are using -- (two dashes) to signify input from stdin, or output to stdout. e.g.

echo "# Heading" | stencila-convert --from md -- ./myheading.docx

The -- is necessary to indicate that ./myheading.docx is the desired output filename, not the input filename.

However, other CLIs, notably pandoc and tar use a single dash - to indicates this. To be consistent, we'll change to a single dash.

Currently rPNG generation is quite slow. This is not a huge issue but could become one with documents that contain many reproducible elements (e.g. lots of within text code expressions providing analysis results). The pandoc.test.ts which involves generation of six rPNGs is noticebly slow https://travis-ci.org/stencila/convert/jobs/532638335#L144

A little test

Here are Markdown document with some non-textual nodes:

A !number(1) and a !boolean(true) and an !array(1,2,3,4,5) and an !object("a":1, "b":2)

When converted to Word bin/stencila-convert temp.md ./temp.docx the "non-textual" nodes get converted to rPNGs:

Which takes an average of 3.5s (and shows spike in network traffic associated with fetching highlight.js presumably):

bash -c "time for i in {1..10}; do bin/stencila-convert temp.md ./temp.docx; done"

real    0m35.710s
user    0m21.296s
sys     0m4.856s

If you remove the exclamation marks (i.e. remove the Commonmark inline extension syntax):

A number(1) and a boolean(true) and an array(1,2,3,4,5) and an object("a":1, "b":2)

Then there are no rPNGs and the time drops to 1.1 seconds (only slightly above the base startup time, #60).

bash -c "time for i in {1..10}; do bin/stencila-convert temp.md ./temp.docx; done"

real    0m11.092s
user    0m13.652s
sys     0m1.008s

Potential solution

It seems that the first approach would to lazy, and only once, initialise a rendering page. That is, do this https://github.com/stencila/convert/blob/99975bbdcd3526d98041d98bf0f098b867d770d4/src/rpng.ts#L208-L216 outside of the unparse function, and only as needed.

I think we should provide an install script and README instructions like we have for Dockter. See the README there and https://github.com/stencila/dockter/blob/master/install.sh (we'd just need to do some renaming and add a windows case to that script and add it here).

The FunctionSchema.rng allows for listing the name of related functions:

  <define name="function:relateds">
    <element name="relateds">
      <zeroOrMore>
        <ref name="function:related"/>
      </zeroOrMore>
    </element>
  </define>

  <define name="function:related">
    <element name="related">
      <text/>
    </element>
  </define>

JsDoc function documentation should support this using @related, and alias @see, tags.

When converting this Markdown with four consecutive code blocks...

```
date
```

```bash
date
```

```sh
date --utc
```

```bash pause=2
date --help
```

using...

npx ts-node --files src/cli convert ./test.md - --to json

I got...

{
  "type": "Article",
  "authors": [],
  "content": [
    {
      "type": "CodeBlock",
      "language": null,
      "value": "date"
    },
    {
      "type": "CodeBlock",
      "language": "bash",
      "meta": {
        "--utc": ""
      },
      "value": "date"
    },
    {
      "type": "CodeBlock",
      "language": "sh",
      "meta": {
        "--utc": ""
      },
      "value": "date --utc"
    },
    {
      "type": "CodeBlock",
      "language": "bash",
      "meta": {
        "pause": "2"
      },
      "value": "date --help"
    }
  ]
}

There are several issues:

• the first block has "language": null instead of not having that property at all
• the second and third blocks have meta that has 'leaked' from the code of the third block

Reported by burque505 - Winter Laite

Even a spreadsheet with only 6 cells, each containing only text, fails with the error mentioned. The json created, however, contains all the data.(...)
EDIT:
Code problem solved with bbcode tag (trying spoiler out also). By the way, I get a useable conversion by first converting xlxs -> md and then md -> pdf, but no charts so far.

EDIT: problematic file attached

TM.xlsx

Given that we integrate Puppeteer, there is an opportunity to generate PDFs using that, rather than requiring users to have a Latex tool like TeXLive installed. This could be combined with Paged.js to create beautiful paginated PDFs of documents.

Relevant links:

• https://www.pagedmedia.org/a-year-in-the-paginated-world/
• https://gist.github.com/glenhallworthreadify/d447e9d6b1fc9cb807b46f952236d4bc
• https://medium.com/@raphaelstaebler/advanced-pdf-generation-for-node-js-using-puppeteer-e168253e159c