Git Product home page Git Product logo

python-project-template's Introduction

Python Project Template

https://travis-ci.org/seanfisk/python-project-template.png

This project provides a best-practices template Python project which integrates several different tools. It saves you work by setting up a number of things, including documentation, code checking, and unit test runners.

As it is an all-in-one solution, the tools used are rather opinionated. They include:

  • Paver for running miscellaneous tasks
  • Setuptools for distribution (Setuptools and Distribute have merged)
  • Sphinx for documentation
  • flake8 for source code checking
  • pytest for unit testing
  • mock for mocking (not required by the template, but included anyway)
  • tox for testing on multiple Python versions

If you are new to Python or new to creating Python projects, see Kenneth Reitz's Hitchhiker's Guide to Python for an explanation of some of the tools used here.

Project Setup

This will be the README for your project. For now, follow these instructions to get this project template set up correctly. Then, come back and replace the contents of this README with contents specific to your project.

Instructions

  1. Clone the template project, replacing my-project with the name of the project you are creating:

    git clone https://github.com/seanfisk/python-project-template.git my-project
    cd my-project
    
  2. Edit the metadata file my_module/metadata.py to correctly describe your project.

  3. Generate files based upon the project metadata you just entered:

    python internal/generate.py
    

    The generation script will remove all the template files and generate real files, then self-destruct upon completion.

  4. Delete the old git history and optionally re-initialize the repository:

    rm -rf .git # or `ri -recurse -force .git' for PowerShell
    git init
    
  5. Change the license in setup.py and replace the generated LICENSE file with the one of your choice. If you would like to use the MIT license, no change is necessary.

  6. Change the classifiers keyword in setup.py. This will require modification.

  7. Replace this README with your own text.

  8. (Optional, but good practice) Create a new virtual environment for your project:

    With pyenv and pyenv-virtualenv:

    pyenv virtualenv my-project
    pyenv local my-project
    

    With virtualenvwrapper:

    mkvirtualenv my-project
    

    With plain virtualenv:

    virtualenv /path/to/my-project-venv
    source /path/to/my-project-venv/bin/activate
    

    If you are new to virtual environments, please see the Virtual Environment section of Kenneth Reitz's Python Guide.

  9. Install the project's development and runtime requirements:

    pip install -r requirements-dev.txt
    
  10. Install argparse package when developing for Python 2.6:

    pip install argparse
    
  11. Run the tests:

    paver test_all
    

    You should see output similar to this:

    $ paver test_all
    ---> pavement.test_all
    No style errors
    ========================================= test session starts =========================================
    platform darwin -- Python 2.7.3 -- pytest-2.3.4
    collected 5 items
    
    tests/test_main.py .....
    
    ====================================== 5 passed in 0.05 seconds =======================================
      ___  _   ___ ___ ___ ___
     | _ \/_\ / __/ __| __|   \
     |  _/ _ \\__ \__ \ _|| |) |
     |_|/_/ \_\___/___/___|___/
    

    The substitution performed is rather naive, so some style errors may be reported if the description or name cause lines to be too long. Correct these manually before moving to the next step. If any unit tests fail to pass, please report an issue.

Project setup is now complete!

Using Paver

The pavement.py file comes with a number of tasks already set up for you. You can see a full list by typing paver help in the project root directory. The following are included:

Tasks from pavement:
lint             - Perform PEP8 style check, run PyFlakes, and run McCabe complexity metrics on the code.
doc_open         - Build the HTML docs and open them in a web browser.
coverage         - Run tests and show test coverage report.
doc_watch        - Watch for changes in the Sphinx documentation and rebuild when changed.
test             - Run the unit tests.
get_tasks        - Get all paver-defined tasks.
commit           - Commit only if all the tests pass.
test_all         - Perform a style check and run all unit tests.

For example, to run the both the unit tests and lint, run the following in the project root directory:

paver test_all

To build the HTML documentation, then open it in a web browser:

paver doc_open

Using Tox

Tox is a tool for running your tests on all supported Python versions. Running it via tox from the project root directory calls paver test_all behind the scenes for each Python version, and does an additional test run to ensure documentation generation works flawlessly. You can customize the list of supported and thus tested Python versions in the tox.ini file.

Pip requirements[-dev].txt files vs. Setuptools install_requires Keyword

The difference in use case between these two mechanisms can be very confusing. The pip requirements files is the conventionally-named requirements.txt that sits in the root directory of many repositories, including this one. The Setuptools install_requires keyword is the list of dependencies declared in setup.py that is automatically installed by pip or easy_install when a package is installed. They have similar but distinct purposes:

install_requires keyword
Install runtime dependencies for the package. This list is meant to exclude versions of dependent packages that do not work with this Python package. This is intended to be run automatically by pip or easy_install.
pip requirements file
Install runtime and/or development dependencies for the package. Replicate an environment by specifying exact versions of packages that are confirmed to work together. The goal is to ensure repeatability and provide developers with an identical development environment. This is intended to be run manually by the developer using pip install -r requirements-dev.txt.

For more information, see the answer provided by Ian Bicking (author of pip) to this StackOverflow question.

Supported Python Versions

Python Project Template supports the following versions out of the box:

  • CPython 2.6, 2.7, 3.3
  • PyPy 1.9

CPython 3.0-3.2 may also work but are at this point unsupported. PyPy 2.0.2 is known to work but is not run on Travis-CI.

Jython and IronPython may also work, but have not been tested. If there is interest in support for these alternative implementations, please open a feature request!

Licenses

The code which makes up this Python project template is licensed under the MIT/X11 license. Feel free to use it in your free software/open-source or proprietary projects.

The template also uses a number of other pieces of software, whose licenses are listed here for convenience. It is your responsibility to ensure that these licenses are up-to-date for the version of each tool you are using.

Project License
Python itself Python Software Foundation License
argparse (now in stdlib) Python Software Foundation License
Sphinx Simplified BSD License
Paver Modified BSD License
colorama Modified BSD License
flake8 MIT/X11 License
mock Modified BSD License
pytest MIT/X11 License
tox MIT/X11 License

Issues

Please report any bugs or requests that you have using the GitHub issue tracker!

Development

If you wish to contribute, first make your changes. Then run the following from the project root directory:

source internal/test.sh

This will copy the template directory to a temporary directory, run the generation, then run tox. Any arguments passed will go directly to the tox command line, e.g.:

source internal/test.sh -e py27

This command line would just test Python 2.7.

Authors

  • Sean Fisk
  • Benjamin Schwarze

python-project-template's People

Contributors

benjixx avatar brianray avatar garnertb avatar jklukas avatar joaosalless avatar seanfisk avatar vojtechjelinek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-project-template's Issues

setup step 11 (paver) fails

---> pavement.test_all


Captured Task Output:
---------------------

---> pavement.test_all
Traceback (most recent call last):
  File "/home/flip111/python/myproject/venv/lib64/python3.6/site-packages/paver/tasks.py", line 195, in _run_task
    return do_task()
  File "/home/flip111/python/myproject/venv/lib64/python3.6/site-packages/paver/tasks.py", line 192, in do_task
    return func(**kw)
  File "pavement.py", line 114, in test_all
    retcode = _test_all()
  File "./setup.py", line 196, in _test_all
    return _lint() + _test()
  File "./setup.py", line 169, in _lint
    project_python_files = [filename for filename in get_project_files()
  File "./setup.py", line 170, in <listcomp>
    if filename.endswith(b'.py')]
TypeError: endswith first arg must be str or a tuple of str, not bytes

python 3.6.2

paver test_all fails

Hi,

I am trying out your package in order to create templates for my own work. I followed the instructions, but when I execute step 11 paver test_all I get the following error:

usage: paver [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: paver --help [cmd1 cmd2 ...]
or: paver --help-commands
or: paver cmd --help

error: invalid command 'test_all'

I tried using paver test, which causes a different error, namely

Traceback (most recent call last):
File "/usr/local/bin/paver", line 11, in <module>
 sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/paver/tasks.py", line 878, in main
  _launch_pavement(args)
File "/usr/local/lib/python2.7/site-packages/paver/tasks.py", line 846, in _launch_pavement
  exec(compile(source, environment.pavement_file, 'exec'), mod.__dict__)
File "pavement.py", line 12, in <module>
  from setup import (
ImportError: cannot import name setup_dict

Any idea what is going on? I see the functions and the required data in the scripts. Does it need to run in order to use the template (I wasn't sure if the pave command would do additional changes or not)? Does my packages code go in main.py or in another file (e.g.__main__.py or my package.py) and the is called through __init__.py?

Thanks for the help!

use pipenv instead of requirements.txt

pipenv[1] is a tool recommended by Python Packaging Authority that deprecates requirements.txt. The biggest added value is separating direct and indirect dependencies and really reproducible builds. Also has very convenient workflow regarding virtual environments. And it was written by the guy who wrote requests if you know...

[1] https://docs.pipenv.org/

Remove hard-dependency on git as version control system

Currently there is a hard-dependency on git in pavement.py:get_project_files().

There could be reasons to use another version control system (e.g. company policy), or project is probably not yet under version control at all (fresh project).

I would expect that "all" paver commands work without any dependencies on a specific version control system.

Change entry point format.

In main.py.tpl, rename main to entry_point and _main to main. The justification here is that Python code that calls the program's main will almost always want to pass arguments, in which case it would call the old _main. However, having a leading underscore indicates a private function.

This will also help when main is dependency injected, in which case we can have three functions: A zero-arg entry_point for setuptools, a main which is passed argv, and _main which is passed argv and the dependency-injected functions.

tox leading to bad marshal data ValueError

________________________ ERROR collecting tests/scheduler/test_scheduler.py ________________________
.tox/py27/lib/python2.7/site-packages/py/_path/local.py:620: in pyimport
>           __import__(modname)
.tox/py27/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:134: in find_module
>       co = _read_pyc(fn_pypath, pyc)
.tox/py27/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:311: in _read_pyc
>           co = marshal.load(fp)
E           ValueError: bad marshal data (unknown type code)
===================================== 12 error in 0.46 seconds =====================================
  ___ _   ___ _    ___ ___
 | __/_\ |_ _| |  | __|   \
 | _/ _ \ | || |__| _|| |) |
 |_/_/ \_\___|____|___|___/

I found a discussion of this problem, and the corresponding solution.

Error running `pip install -r requirements-dev.txt`

When trying to install the dev requirements using pip install, I run into the following error:

$ pip install -r requirements-dev.txt
Collecting pytest==2.5.1 (from -r requirements-dev.txt (line 5))
Collecting py==1.4.19 (from -r requirements-dev.txt (line 6))
Collecting mock==1.0.1 (from -r requirements-dev.txt (line 7))
Collecting flake8==2.1.0 (from -r requirements-dev.txt (line 10))
Collecting mccabe==0.2.1 (from -r requirements-dev.txt (line 11))
Collecting pep8==1.4.6 (from -r requirements-dev.txt (line 12))
Collecting pyflakes==0.7.3 (from -r requirements-dev.txt (line 13))
Collecting Sphinx==1.2 (from -r requirements-dev.txt (line 16))
  Using cached Sphinx-1.2-py33-none-any.whl
Collecting docutils==0.11 (from -r requirements-dev.txt (line 17))
Collecting Jinja2==2.7.1 (from -r requirements-dev.txt (line 18))
Collecting MarkupSafe==0.18 (from -r requirements-dev.txt (line 19))
Collecting Pygments==1.6 (from -r requirements-dev.txt (line 20))
Collecting Paver==1.2.1 (from -r requirements-dev.txt (line 23))
  Using cached Paver-1.2.1.tar.gz
Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
  File "/Users/stew/.envs/rummage/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/Users/stew/.envs/rummage/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ImportError: No module named 'six'

This is fixed by changing the Paver version in requirements-dev.txt from 1.2.1 to the current 1.2.4. It's possible I'm doing something wrong--I'm not clear on why this was happening in the first place--but this change resolves the issue. I'm happy to provide further context if you tell me what info would be helpful.

It looks like the requirements-dev.txt on the master branch is 3 years old--should it be updated? I'll make a pull request so you can update it easily if you think it should be.

Create a template update script.

I have a number of projects on which work is currently being done that use this template project. It would be nice if I could run a script to update the project to the current version of the template project.

A possible solution would involve the following steps:

  • Create a new "fake" project based on the template project.
  • Replace the fake project's metadata.py with the old project.
  • Run generation on the fake project.
  • Diff the output on relevant files from the old project to the fake project and save to a patch file.
  • Use git apply or plain patch to apply the patch.

Some conflict resolution would likely be necessary for most projects, but it would be much better than doing a manual merge.

Consider `pylint` instead of `flake8`

I think—and I could be wrong, but I’m pretty sure about this—pylint can do everything flake8 does, and more.

As usual for this sort of tool, it’s configurable on a user- and project-basis, and the checks it performs can be further refined on the command line. Multiple output formats exist, including configuration options for tailoring the existing outputs to your liking.

Plus, I just like tools with lint in their name better than tools with flake in their name.

(“Flakes?” Yuck! I’ll admit my code has lint, but I never want to hear that it has flakes! 👅)

Clarify out-of-the-box supported Python version(s)

I did a quick analysis on a fresh generated project using pyqver, a tool for querying the required Python version.

bird:src benjixx$ cp -r python-project-template/ pytemp

bird:src benjixx$ cd pytemp/

bird:pytemp benjixx$ python GENERATE.py
Substituting ./pavement.py.tpl -> ./pavement.py
Substituting ./docs/make.bat.tpl -> ./docs/make.bat
Substituting ./docs/Makefile.tpl -> ./docs/Makefile
Substituting ./docs/source/conf.py.tpl -> ./docs/source/conf.py
Substituting ./docs/source/index.rst.tpl -> ./docs/source/index.rst
Substituting ./docs/source/README.tpl -> ./docs/source/README
Substituting ./my_module/__init__.py.tpl -> ./my_module/__init__.py
Substituting ./my_module/main.py.tpl -> ./my_module/main.py
Substituting ./tests/test_main.py.tpl -> ./tests/test_main.py
Renaming the package: my_module -> my_module
Generation script imploding...

bird:pytemp benjixx$ find . -name "*.py" | xargs python ../pyqver/pyqver2.py  -v
./docs/source/conf.py
./my_module/__init__.py
./my_module/main.py
    2.7 argparse
    2.6 string literal .format(), string literal .format(), string literal .format()
./my_module/metadata.py
./pavement.py
    2.6 string literal .format()
    2.5 with statement
    2.4 subprocess, set, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator
./tests/test_main.py
    2.6 string literal .format()
    2.5 with statement, with statement, with statement, with statement
    2.4 function decorator, function decorator

The argparse mentioned requiring Python 2.7 is already included in requirements/run.txt as a compatibility module for older Python versions.
So, looks like we currently require at least Python 2.6.

What do you think? Is this sufficient? Or should it be Python>=2.5?

I guess Python 3 is something that should be equally well supported.

Add pip requirements file.

In #3, we decided to take out the runtime requirements.txt file in favor of having libraries in the setup.py file. However, I've been doing some reading on pip requirements files and have a new idea for what they are actually used.

According to the aforementioned docs, the requirements.txt file should be used to create a completely reproducible environment, and these steps should be followed for reproducibility. Ian Bicking (author of pip) mentions in this StackOverflow response that "install_requires is to keep people away from things that you know don't work, while requirements files to lead people towards things you know do work."

What I'm planning to do is create a clean virtual environment and generate a requirements.txt using pip freeze. I will also add a section to the README explaining the difference between requirements files and install_requires. Opinions welcome.

Make tox Sphinx warnings into errors.

This will allow us to be more strict in making sure the docs build successfully. This can be done with the -W flag to sphinx-build, but we'll probably have to override paver.doctools.html() to do it.

How to run project during development

I write here to document my trials and errors how to run this template in case someone else run to this problem

After changing my_module/metadata.py as suggested and running internal/generate.py, to my surprise both <my_named_module>/main.py and tests/test_main.py complained, that they can not find <my_named_module> (note, "paver test_all" worked).

The solution was to add path/to/project to PYTHONPATH

Support multiple kinds of project templates

I have different kind of python projects in mind that require slightly different templates:

  • lib (library that is used in other Python projects),
  • cli (application that is used -- probably interactively -- from command line),
  • daemon (application that is prepared to run as a daemon, including init.d scripts, etc.)

I'm already working on a prototype that uses Jinja (on which we already depend through Sphinx) together with template inheritance to support multiple project templates without (too much) code duplication.

Also it would be nice to get this template generation process integrated with virtualenvwrapper's mkproject command. Ideas and thoughts about this probably should go into a separate ticket.

Clarify licenses.

  • Add LICENSE file (says ISC in setup.py.tpl, but I want to change to MIT).
  • Add table of licenses of dependencies and check that they're compatible.

issue while running tests

I cloned the repo and trying to run the tests for which i'm getting the following error.
can you please help me out here.

Here i'm trying to run it from a virtualenv

 Traceback (most recent call last):
  File "/home/someuser/virtualenvs/python/2.7/bin/tox", line 9, in <module>
    load_entry_point('tox==1.6.1', 'console_scripts', 'tox')()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 26, in main
    retcode = Session(config).runcommand()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 301, in runcommand
    return self.subcommand_test()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 433, in subcommand_test
    if self.setupenv(venv):
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 358, in setupenv
    status = venv.update(action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 142, in update
    self.install_deps(action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 272, in install_deps
    self._install(deps, action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 334, in _install
    extraenv=extraenv)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 303, in run_install_command
    extraenv=env, action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 382, in _pcall
    return action.popen(args, cwd=cwd, env=env, redirect=redirect)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 96, in popen
    popen = self._popen(args, cwd, env=env, stdout=f, stderr=STDOUT)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 153, in _popen
    stdout=stdout, stderr=stderr, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I also tried to install tox and run the tests but same error is occurring again.

Upgrade to Setuptools.

Distribute and Setuptools have merged into Setuptools, so it looks like Setuptools is the future. We should upgrade.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.