Git Product home page Git Product logo

doctr's Introduction

Doctr

A tool for automatically deploying docs from Travis CI to GitHub pages.

Doctr helps deploy things to GitHub pages from Travis CI by managing the otherwise complicated tasks of generating, encrypting, managing SSH deploy keys, and syncing files to the gh-pages branch. Doctr was originally designed for documentation, but it can be used to deploy any kind of website to GitHub pages that can be built on Travis CI. For example, you can use Doctr to deploy a blog or website that uses a static site generator.

Contribute to Doctr development on GitHub.

Installation

Install Doctr with pip

pip install doctr

or conda

conda install -c conda-forge doctr

Note that Doctr requires Python 3.5 or newer.

Usage

Run Doctr configure

First use Doctr to generate the necessary key files so that travis can push to your gh-pages (or other) branch.

Run

doctr configure

and enter your data. You will need your GitHub username and password, and the repo organization / name for which you want to build the docs.

Note: That repo should already be set up with Travis. We recommend enabling branch protection for the gh-pages branch and other branches, as the deploy key used by Doctr has the ability to push to any branch in your repo.

Edit your travis file

Doctr will output a bunch of text as well as instructions for next steps. You need to edit your .travis.yml with this text. It contains the secure key that lets travis communicate with your GitHub repository, as well as the code to run (in script:) in order to build the docs and deploy Doctr.

Your .travis.yml file should look something like this:

See the travis config file used by Doctr itself for example.

You can deploy to a different folder by giving it a different path in the call to deploy. E.g., doctr deploy docs/.

If you don't already have a gh_pages branch Doctr will make one for you.

Warning

Be sure to add set -e in script, to prevent doctr from running when the docs build fails.

Put doctr deploy . in the script section of your .travis.yml. If you use after_success, it will not cause the build to fail.

Commit your new files and build your site

doctr configure will create a new file that contains your key. Commit this as well as the changes to .travis.yml. Once you push to GitHub, travis should now automatically build your documentation and deploy it.

Notes

Doctr requires Python 3.5 or newer. Be sure to run it in a Python 3.5 or newer section of your build matrix. It should be in the same build in your build matrix as your docs build, as it reuses that.

Doctr does not require Sphinx. It will work with deploying anything to GitHub pages. However, if you do use Sphinx, Doctr will find your Sphinx docs automatically (otherwise use doctr deploy . --built-docs <DOCS PATH>).

FAQ

  • Why did you build this?

    Deploying to GitHub pages from Travis is not amazingly difficult, but it's difficult enough that we wanted to write the code to do it once. We found that Travis docs uploading scripts are cargo culted and done in a way that is difficult to reproduce, especially the do-once steps of setting up keys. The doctr configure command handles key generation automatically, and tells you everything you need to do to set Doctr up. It is also completely self-contained (it does not depend on the travis Ruby gem). The doctr deploy command handles key decryption (for deploy keys) and hiding tokens from the command output (for personal access tokens).

    Furthermore, most Travis deploy guides that we've found recommend setting up a GitHub personal access token to push to GitHub pages. GitHub personal access tokens grant read/write access to all public GitHub repositories for a given user. A more secure way is to use a GitHub deploy key, which grants read/write access only to a single repository. Doctr creates a GitHub deploy key by default (although the option to use a token exists if you know what you are doing).

  • Why not Read the Docs?

    Read the Docs is great, but it has some limitations:

    • You are limited in what you can install in Read the Docs. Travis lets you run arbitrary code, which may be necessary to build your documentation.
    • Read the Docs deploys to readthedocs.io. Doctr deploys to GitHub pages. This is often more convenient, as your docs can easily sit alongside other website materials for your project on GitHub pages.

    In general, you should already be building your docs on Travis anyway (to test that they build), so it seems natural to deploy them from there.

  • Why does Doctr require Python 3.5 or newer?

    There are several language features of Python that we wanted to make use of that are not available in earlier versions of Python, such as keyword-only arguments, subprocess.run, and recursive globs. These features help keep the Doctr code cleaner and more maintainable.

    If you cannot build your documentation in Python 3, you will need to install Python 3.6 in Travis to run Doctr.

  • Is this secure?

    Doctr creates an encrypted SSH deploy key, which allows any Travis build on your repo to push to the deploy repo. The deploy key is encrypted using Fernet encryption from the Python cryptography module. The Fernet key is then encrypted to a secure environment variable for Travis using the Travis public key.

    Travis does not make secure environment variables available to pull requests builds. Furthermore, Doctr itself does not push from any branch other than master by default, although this can be changed <any-branch>.

    By default, Doctr uses deploy keys, but it can also use a GitHub personal access token, using the --token flag. However, this is not recommended, as a GitHub personal access token grants access to your entire account, whereas a deploy key only grants push access only to a single repository.

    Both Doctr and Travis CI itself take measures to prevent the private encryption key from leaking in the build logs.

    At any time, you can revoke the deploy key created by Doctr by going to the deploy key settings for the repository in GitHub at https://github.com/{org}/{repo}/settings/keys. Personal access tokens can be revoked at https://github.com/settings/tokens. If you revoke a key, you will need to rerun doctr configure to generate a new one to continue using Doctr.

  • Can Doctr do X?

    See the recipes page for many common use case recipes for Doctr. Doctr supports virtually anything that involves pushing from Travis CI to GitHub automatically.

  • I would use this, but it's missing a feature that I want.

    Doctr is still very new. We welcome all feature requests and pull requests.

  • Why is it called Doctr?

    Because it deploys documentation from Travis. And it makes you feel good.

Projects using Doctr

Are you using Doctr? Please add your project to the list!

doctr's People

Contributors

asmeurer avatar bnaul avatar cam-gerlach avatar carreau avatar choldgraf avatar cj-wright avatar dan-blanchard avatar danielballan avatar gforsyth avatar moorepants avatar ocefpaf avatar r-b-g-b avatar scopatz avatar techgaun avatar thequackdaddy avatar willingc avatar ylemkimon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

doctr's Issues

Check for travis repo before generating keys.

Otherwise it will just fail and you'll have to regenerate.

It's a small optimisation though, that will mostly affect users that make typos.

Note that github is not case sensitive for usernames, travis is so for example, I regularly get into trouble when I write my username lowercase and it involves travis.

Project using doctr

Could be useful to add a list to the docs. Right now, SymPy and conda are using it.

.travis.yml as a config file ?

Yo,

Not that I don't like passing lots of option to doctr on .travis.yml nor that I dislike things that are not meta enough... but...

It occurred to me that doctr run on travis, so there is a high chance that a .travis.yml file is present.
Travis ignore keys it does not know so what about a doctr key that would store config ?

Latest Travis failure (Permission denied (publickey))

I'm not sure why Travis is failing now https://travis-ci.org/gforsyth/doctr/jobs/149058779

The error is

Build finished. The HTML pages are in _build/html.
Setting git attributes
git config --global user.name 'Travis docs builder (Travis CI)'
Adding doctr remote
git remote add doctr_remote [email protected]:gforsyth/doctr.git
Fetching doctr remote
git fetch doctr_remote
Warning: Permanently added the RSA host key for IP address '192.30.253.113' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

So something is wrong with the deploy ssh key. Maybe the stuff I'm putting in ~/.ssh/config, which was taken from https://github.com/alrra/travis-scripts/blob/master/doc/github-deploy-keys.md, is wrong. Other guides that I've seen use ssh-add. Do we need to use that. Is it the same?

@gforsyth can you verify that the deploy key is there on GitHub?

Run almost all doctr steps on PR

The only thing doctr shouldn't do is upload to gh-pages.

If, as part of your build, you modify your source tree, for example, doctr will refuse to do checkouts for it. The same happens for incorrect file permissions (say arising from running sphinx inside a docker container).

Right now doctr just skips when doing PR builds it seems. This means that these kinds of failures are not detected at the PR level and happen only when TravisCI is doing the push build.

For example

By running everything but the actual git push that doctr does we can minimize this friction.

delete public Key by default ?

The configure process says:

================== You should now do the following ==================

1. Commit the file github_deploy_key.enc.
The file github_deploy_key.pub contains the public deploy key for GitHub. It
does not need to be committed.

Is there any reason to keep it ? Would that be confusing to users ?

Would it be better to delete, leave as an option to keep it (it's cheap to regenerate), and say something along the lines:

  • The public deploy key has been activated on github (see <url>), and the local version erased.

Default should be the root, not /docs

The default should be the root, not /docs, since this is supposed to be for simple projects where the docs are the homepage, or, alternately, a docs repo in an org.

It's harder, though, because you have to do an rsync to update stuff. Maybe there's a simpler way to do it.

Ability to do something different on a tag

This tool is mainly designed for uploading dev docs, but it can also be used to upload release docs. In that case, we might need some way to do something different on a tag.

Or maybe the logic for uploading on a tag should just go in the individual .travis.yml files, in which case we can at least document what it could look like.

Change --gh-pages-docs default

It's a backwards incompatible change, but --gh-pages-docs defaulting to docs is not great. . is probably what people expect. Maybe we should make it a required argument.

CC @ocefpaf

Multiple deploy locations

Right now you can only deploy to one repo, because there is just one environment variable that it checks the key for (unless maybe https://github.com/gforsyth/doctr/issues/34 would let you split this into multiple builds). The exception is if you use a token.

We should allow multiple deploy keys, perhaps by adding a suffix to the DOCTR_DEPLOY_ENCRYPTION_KEY environment variable, which would be specified at the command line. The default should stay as that, to keep backwards compatibility. Or maybe we should break compatibility and include the reponame in the environment variable.

This would let us test doctr by deploying to itself, to a separate repo, and to a user.github.io style page.

"summary" and "description" fields on pypi should be switched around

A package's summary should be a short one-liner which gives people an idea what the package is about. However, when you do pip search doctr, you get a unreadable wall of text, with the formatting broken and the newlines removed. The wall of text should go into the description field, which preserves the formatting.

Release

Anything else that you want to be done before another release? I'd like to go public with the next release, so anything that should be done before that should be done first.

Detect if the repo is a fork

If I'm understanding the behavior I'm seeing correctly, encrypted environment variables aren't available on forks at all, even if they are encrypted with the key for the fork.

Automatically edit the .travis.yml

It would be nice if it automatically added the encrypted environment variable to .travis.yml. It's probably not possible to be smart enough to add doctr deploy.

I don't know how to do this well. Probably you need ruamel.yaml to round-trip parse the yaml.

git user.email failing to auto-detect

Has anyone else seen this error?

*** Please tell me who you are.
Run
  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: unable to auto-detect email address (got 'travis@testing-docker-d1d30905-1f6c-4664-aa3b-7945fba5627e.(none)')

excerpted from https://travis-ci.org/danielballan/photomosaic/builds/175131428#L803

Doctr was previously working seamlessly, and I'm not sure what has changed. I can see from the log that doctr manually sets the git user.name but not the user.email. Maybe it should.

Improve interaction messages:

When ran the tool ask:

What repo do you want to build the docs for?

not obvious is <username>/<repo> is enough, or full URL.

Add ability to run a command before deploying

For the SymPy docs we run a command that post-processes the docs to add links to the other versions. Doctr needs the ability to run a command after the docs are moved in place but before they are committed.

Tool to set up gh-pages for you

Setting up gh-pages right is apparently not so easy, so we should have a tool with this that does it for you, and maybe offers to do so if it sees that gh-pages isn't there yet when you run the local (token generation) script.

It's not easy because

  • It should be an empty branch (not based on master). This isn't a big deal, but it's better this way.
  • It should have .nojekyll at the root.
  • (For the future), if the user indicates they want docs in a subdir, like docs, then you need an index.html for the root, or it will 404. See also #10.

Fail gracefully if repo does not exists

Otherwise user get the error:

Traceback (most recent call last):
  File "/Users/bussonniermatthias/anaconda/bin/doctr", line 9, in <module>
    load_entry_point('doctr', 'console_scripts', 'doctr')()
  File "/Users/bussonniermatthias/dev/doctr/doctr/__main__.py", line 198, in main
    return process_args(get_parser())
  File "/Users/bussonniermatthias/dev/doctr/doctr/__main__.py", line 92, in process_args
    return args.func(args, parser)
  File "/Users/bussonniermatthias/dev/doctr/doctr/__main__.py", line 142, in configure
    encrypted_variable = encrypt_variable(b"DOCTR_DEPLOY_ENCRYPTION_KEY=" + key, repo=repo)
  File "/Users/bussonniermatthias/dev/doctr/doctr/local.py", line 44, in encrypt_variable
    public_key = r.json()['key']
KeyError: 'key'

Because r.status is not 200...

configure -> init ?

Hey guys !

Finally get some time to play along with doctr.

Is there a reason why you choose configure over init, most of tool these days une init: the travis gem, git, mercurial, flit, npm, Cargo (rust) ...

I think that having init (at least as an alias) would be nice.

auth issue when creating gh-pages branch

travis.create_gh_pages creates an orphan gh-pages branch with a .nojekyll file but it can't push this to the remote unless it's authenticated.

if there is no gh-pages branch then doctr will return a nonzero exit code when built on a PR since canpush will be set to False.

probably can just wrap all of the creation stuff in an if canpush block

Catch exception if travis isn't set up

Just caught this when I tried to regenerate the deploy key without first enabling travis

Key generation went fine but then requests barfed when it couldn't talk to travis

Traceback (most recent call last):
  File "/home/gil/anaconda/bin/doctr", line 9, in <module>
    load_entry_point('doctr==1.1.1+42.g9c5359b', 'console_scripts', 'doctr')()
  File "/home/gil/anaconda/lib/python3.5/site-packages/doctr-1.1.1+42.g9c5359b-py3.5.egg/doctr/__main__.py", line 210, in main
    return process_args(get_parser())
  File "/home/gil/anaconda/lib/python3.5/site-packages/doctr-1.1.1+42.g9c5359b-py3.5.egg/doctr/__main__.py", line 96, in process_args
    return args.func(args, parser)
  File "/home/gil/anaconda/lib/python3.5/site-packages/doctr-1.1.1+42.g9c5359b-py3.5.egg/doctr/__main__.py", line 153, in configure
    encrypted_variable = encrypt_variable(b"DOCTR_DEPLOY_ENCRYPTION_KEY=" + key, build_repo=build_repo)
  File "/home/gil/anaconda/lib/python3.5/site-packages/doctr-1.1.1+42.g9c5359b-py3.5.egg/doctr/local.py", line 45, in encrypt_variable
    r.raise_for_status()
  File "/home/gil/anaconda/lib/python3.5/site-packages/requests/models.py", line 837, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.travis-ci.org/repos/drdoctr/doctr/key

We can just catch this exception and gently remind the user to go to travis and enable the repository

Travis script doesn't stop on failed command

See travis-ci/travis-ci#1066.

The result is that if people write their .travis.yml as our example shows, it will run doctr even if the docs fail, which can result in the dev docs being deleted (if the build fails and no docs are generated, it will think that all files should be removed).

You can fix this either with set -e or by putting the command in a yaml block. We should update the docs, and our own .travis.yml.

Support username.github.io style GitHub pages

There are two kinds of GitHub pages, user/org pages, and project pages. User/org pages are in repos named username.github.io, and are deployed to the master branch. Project pages are in any repo, and are deployed to the gh-pages branch. We currently support the latter. Supporting the former wouldn't be hard, maybe even detecting it automatically.

This is low priority (I don't personally have a need for it).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.