stencila / dockta Goto Github PK

Look for .xml files with <!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) ... elements and detect any package import statements (e.g. Python import, R library, or Node.js require) in any elements matching:

• code[specific-use=cell] > named-content > alternatives > code
• fig[fig-type=repro-fig] > alternatives > code

To be consistent with Docker we could only COPY and ADD files if they have changed. This would be an optimisation and would be non-trivial to implement so is probably low priority.

It's best practice to run processes in Docker containers as a non-root user

So we should add something like the following to generated Docker files.

RUN useradd --create-home --uid 1001 -s /bin/bash user
USER user
WORKDIR /home/user

Currently we send all files in the folder to the Docker daemon as a build context, with a few exceptions:

https://github.com/stencila/dockter/blob/v0.2.6/src/DockerBuilder.ts#L61

It would also be sensible to ignore node_modules etc. See https://codefresh.io/docker-tutorial/not-ignore-dockerignore/

Add publishing commands to npm scripts, Makefile and travis.yml. See:

• https://github.com/stencila/cli/blob/master/.travis.yml
• https://www.npmjs.com/package/travis-deploy-example
• https://tailordev.fr/blog/2018/03/15/automated-npm-releases-with-travis-ci/

The .travis.yml file is already setup to push code coverage to Codecov, it just needs to be turned on.

Once this is done, set tslint.json to:

{
  "extends": "tslint-config-standard",
  "rules" : {
    "completed-docs": true
  }
}

Meta-data can be added to Dockerfiles using the LABEL and the, now deprecated, MAINTAINER directive. Dockter extracts this meta-data when "compiling" a handwritten Dockerfile. This allows project authors to publish a codemeta.json file for their Docker based project.

We should define and document the meta-data conversion that Docker does using a crosswalk table that correlates common Dockerfile labels (https://docs.docker.com/engine/reference/builder/#label) and http://label-schema.org/rc1/ labels to https://schema.org and https://codemeta.github.io terms.

It might look something like this:

See https://github.com/codemeta/codemeta/blob/master/crosswalk.csv (be sure to scroll right!) for similar crosswalk table for other software packaging systems.

Add some notes in CONTRIBUTING.md on how to know when you need to, and how to, add entries into PythonSystemDependencies.json. Doesn't need a lot of explanation, just some quick pointers.

Currently RGenerator creates a Dockerfile that downloads src/install.R from Github pages i.e. https://stencila.github.io/dockter/install.R.

Once this is published as an NPL package #17 it should use https://unpkg.com/@stencila/dockter/src/install.R (which will redirect to the latest version)

Currently, Dockter requires Docker to be installed on the system. That's because we're using:

const docker = new Docker() # Connects to local Docker daemon via '/var/run/docker.sock'

But, in theory, we could allow for a remote Docker daemon to build images (it would require that the folder contents be zipped up and send across the network). e.g.

const docker = new Docker({
  protocol: 'https',
  host: 'some.docker-server.org',
  port: process.env.DOCKER_PORT || 2375,
  ca: fs.readFileSync('ca.pem'),
  cert: fs.readFileSync('cert.pem'),
  key: fs.readFileSync('key.pem')
})

Not a high priority, but could be useful for users who don't have Docker installed and don't want to.

We're trying to make the generated Dockerfile conform to best practices and to be useful for learning how to write your own Dockerfiles. So adding comments to each section of the generated file could be useful. e.g.

# This section installs required system packages into the image and then cleans up
# If you need extra system packages add them here
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y \
      libxml2-dev \
      r-base \
...

# It's best practice to run Docker images as a non-root user
# This section creates a new user, sets it as the user for the image and their
# home directory as the working directory
RUN useradd --create-home --uid 1001 -s /bin/bash dockteruser
USER dockteruser
WORKDIR /home/dockteruser

Docker image builds are currently broken because the WORKDIR instruction which we added for #25 is not fulfilled. Need to keep track of last WORKDIR and set path here https://github.com/stencila/dockter/blob/v0.2.6/src/DockerBuilder.ts#L188

The PythonParser class currently has an empty parse() method. As a start it should parse requirements.txt files in a similar way to how the RParser handles DESCRIPTION files.

See https://warehouse.pypa.io/api-reference/json/ for obtaining package meta-data including system requirements.

We should use https://github.com/r-hub/sysreqs to install system dependencies for R packages.

See also:

• https://github.com/r-hub/sysreqsdb
• https://github.com/rstudio/shinyapps-package-dependencies

Additional, relevant discussions include:

• ropensci/codemetar#147
• https://github.com/o2r-project/containerit/blob/master/vignettes/containerit.Rmd#7-challenges
• jlisic/sysdep#1

Currently, we wait forever for the RUN instruction to finish: https://github.com/stencila/dockter/blob/v0.2.6/src/DockerBuilder.ts#L208

Probably should have some sort of maximum time limit.

An animated GIF of a console session helps give a quick illustration of what a CLI tool like this one does. A good example is https://github.com/pypa/pipenv#readme

Currently, the datePublished of the generated SoftwarePackage is the current date. @finlay suggested it could be useful to allow users to override that at the command line jinstead of editing any generated files).

The README says:

Dockter does this by looking for a special # dockter comment in a Dockerfile. Instead of throwing away layers, it executes all instructions after this comment in the same layer - thus reusing packages that were previously installed.

This explanation was a bit difficult to understand, as I thought it was an alternate implementation of the --squash option. That would create a new single layer in which to execute all images, but a new one for each build. This instead will reuse an existing layer, overwriting the contents.

Some thoughts to polish this feature:

• # dockter does not convey the meaning of this, a more specific annotation like # dockter: stateful could make it easier to search for documentation about this option
• clarify in README this is going to overwrite an existing, single layer
• clarify where this state is kept, in the Docker daemon's state files, I guess
• clarify rolling back to a previous version of the dependencies will be slower as the trade-off, having to modify the existing, single layer

There's already a .travis.yml file but we need to activate this project and add a GITHUB_TOKEN env var so that docs can get pushed to GithubPages.

Our standard CoC from https://github.com/stencila/policies/blob/master/CONDUCT.md should be added to this repo and linked to from README in the Develop section.

Currently, we install Stencila packages for R, Python, Node.js so that editor UI clients can execute code within the container. This is done independently of the project's package dependencies.

However, not all users may need this functionality so a --stencila=false option would be useful to speed up builds and reduce image sizes.

Currently, the RParser scans for .R and .Rmd files only. It should scan for lower case equivalents too.

When compiling R lesson for Software Carpentry I get an error:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at CachedRequest.parseHeaders (/snapshot/dockter/node_modules/cached-request/lib/cached-request.js:112:15)
    at ReadStream.<anonymous> (/snapshot/dockter/node_modules/cached-request/lib/cached-request.js:194:37)
    at emitNone (events.js:111:20)
    at ReadStream.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at CachedRequest.parseHeaders (/snapshot/dockter/node_modules/cached-request/lib/cached-request.js:112:15)
    at ReadStream.<anonymous> (/snapshot/dockter/node_modules/cached-request/lib/cached-request.js:194:37)
    at emitNone (events.js:111:20)
    at ReadStream.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

See TODO at https://github.com/stencila/dockter/blob/98b882fd26f31b44631f054e0bf85d7e4c6e23cc/src/JavascriptParser.ts#L61

crandb provides plenty of meta data for each package e.g. http://crandb.r-pkg.org/ggplot2

Currently, we don't handle every field or could handle them better e.g.:

• Title: assign to CreativeWork.headline ?
• Licence : should be parse the raw string into a schema.org URL or CreativeWork
• URL : if the URLS which point to a repo e.g. github.com then should be assigned to SoftwareSourceCode.codeRepository

Currently the file src/context.ts has temporary Typescript representations of JSON-LD schema types. This should be split off into the stencila/specs repo (to be renamed stencila/schema) so that it can be reused in other projects as well.

But while we are still working out what those types should be and what they will look like we'll leave them in here.

The JsParser class should load any package.json files in a folder and create a SoftwarePackage instance from it using the CodeMeta crosswalk: https://codemeta.github.io/crosswalk/node/

The PythonParser class scan for parse Python files in a folder and extract a list of unique (non-standard-library) package dependencies from import statements.

See browserify/detective#8 as an example of using JS regexes to parse for requires. A similar approach could be used for Python import statements (since we wont have a python AST to walk in TypeScript) - there are probably already regexes out there for parsing Python import statements

See https://warehouse.pypa.io/api-reference/json/ for obtaining Python package meta-data including system requirements.

See https://github.com/stencila/dockter/blob/e8521a30e0e069cdfacd46a758370c36ffb24a69/src/RParser.ts#L103 for how we use this data to populate softwareRequirements property.

The devDependency @types/dockerode was updated from 2.5.5 to 2.5.6.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/dockerode is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

When compiling a project folder with name starting with uppercase, dockter gives an error:

server error - invalid reference format: repository name must be lowercase 

This is probably related to: https://stackoverflow.com/questions/48522615/docker-error-invalid-reference-format-repository-name-must-be-lowercase

Changing the name of the project folder fixed it but we should reflect that in the documentation or deal with the naming.

We should improve test coverage prior to 1.0.0. I'd suggest we target 95% (noting of course that there is more to test quality than the %!)

@beneboy: can you work on the parser related tests and I can do the generator and builder tests

The PythonParser should be able to parse Pipfiles https://github.com/pypa/pipfile

We support both ADD and COPY in the # dockter section of a Dockerfile

https://github.com/stencila/dockter/blob/v0.2.6/src/DockerBuilder.ts#L167

Docker's ADD and COPY instructions are very similar but COPY is generally recommended. Nonetheless, if we support ADD in the # dockter section of a Dockerfile then we should provide consistent behaviour.

Current draft in how-to.md at top level.

(Just creating a list of issues to be completed for https://github.com/stencila/dockter/milestone/2)

There are quite a lot of TODOs and FIXMEs in the code. They should be removed into Github issues.

Implement a JupyterParser class which scans the folder for any ipynb files, scan the code cells in those files for any Python import or R library statements, extract a list of package dependencies, and create a SoftwarePackage instance for it.

Should also extract relevant meta-data from the .ipynb files and add it to the SoftwarePackage. There currently isn't any CodeMeta crosswalk for ipynb meta data, https://codemeta.github.io/crosswalk/, maybe we could contribute one.

Windows gives warning when trying to run the Docker binary which may prevent some users from actually launching the app.

As per result of the discussion in #37 we could add a command to dockter - suggestion by @nokome

*But how about we leverage the fact that we generate a JSON-LD document with all the software requirements for a project...including their authors. The who subcommand could be a "standing on the shoulders of giants" list of the authors of the current project (if available) and all of the software packages it relies on. e.g. *

$ dockter who
Roger Bivand (rgdal, sp), Tim Keitt (rgdal), Barry Rowlingson (rgdal), Edzer Pebesma (sp)

Which for reference is based on filtering the .environ.jsonld file for one of our test fixtures

jq '.softwareRequirements[0].softwareRequirements[] | .author[] | .name' tests/fixtures/r-spatial/.environ.jsonld

Running build results in an error:

 Error: (HTTP code 400) unexpected - OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown

It is probably because the location of bash in the container is in /usr/local/bin/bash
see here

Looks like the solution is to replace

RUN useradd --create-home --uid 1001 -s /bin/bash dockteruser

with

RUN useradd --create-home --uid 1001 -s  /usr/local/bin/bash dockteruser

@nokome Can you have a look? I can change that and push.

Surely, we gotta have a cute 🐳 whale + doctor (medical or PhD) combination logo at the top of the README: a whale with a graduation cap, a whale with a stethoscope, a whale with a test tube?

In CONTRIBUTING.md (but linked to under Contribute in README.md) we should have a brief guide to getting started on adding support for a new language.

• how to write a Parser and add to parsers.ts
• how to write a Generator and add to generators.ts
• note on adding tests based on existing ones

@apawlik : I'll start on this next week and then pink you to review it.

The JsParser class should be able to parse any .js files in the folder and extract a list of package dependencies from import or requires statements.

Use https://github.com/browserify/detective for extracting required packages from source code.

See #5 and #6 for corresponding issues for other languages.

When trying to compile folder with only one R file https://raw.githubusercontent.com/swcarpentry/r-novice-inflammation/master/tic.R (not your typical R file but the error should not occur)

TypeError: Cannot read property 'split' of undefined
    at Parser_1.default.parse.environ.softwareRequirements.Promise.all.packages.map (/snapshot/dockter/dist/RParser.js:122:27)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
undefined

The RParser class should scan for .R and .Rmd files in the folder and extract a list of package dependencies from library (and requires?) statements.

See #5 for the corresponding PythonParser issue.

The current name of this tool is Dockter. It's a portmanteau of doctor (either the medical or PhD kind) and Docker (the container platform) because it's aim is to make it easier for researchers (either the medical or PhD kind 🙂 ) to create Docker containers.

But there are five issues with the name:

• 🗣️ Hard to distinguish in verbal communications: "Was that Docker or Dockter?"
• ✍️ Ambiguity in spelling: "Dockter" or "Docktor"?
• 🖥️ Confusion at the command line: dockter build . is very similar docker build . and looking for that extra t can consume an extra brain cycle or two to figure out which tool you are using.
• 💻 Autocomplete at the command line: many people use autocomplete in the shell and having the first four letters the same makes this less useful.

As the famous joke says (well actually a riff on the original joke):

There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors.
Leon Bambrick

Please add your thoughts and suggestions in comments below 🙏. If you like someone's suggested name please 👍 it

Implement a JupyterGenerator class which, if there is a .ipynb file in the folder, in the generated Dockerfile installs the jupyter Python package and runs the notebook server (CMD instruction).