Git Product home page Git Product logo

make-env's Introduction

Makefile for create (conda) environments

Build Status License

This repository contains a Makefile to make it easy to set up a conda environment and then run commands inside the environment.

Using

Step 1 - Add make-env to your repository

Option 1a -- git submodules

The make-env git repository can be added as a git submodule of your own git repository.

Adding using git submodule

git submodule add https://github.com/SymbiFlow/make-env.git third_party/make-env

Updating using git submodule

It is recommended you use an "auto rolling" dependency bot which sends you a pull request every time the upstream module moves forward.

# TODO: Add stuff here.

Option 1b -- git subtrees

The make-env git repository can be directly imported into your repository using git subtree.

git subtrees offer a number of advantages over git submodules, they include;

  • The contents of the subtree are directly included in your own tree, meaning no extra clone nor recursive clone is needed. If the upstream source disappears, you still have a complete copy of the repository contents.

  • Edits can be done and sent upstream easily.

Adding using git subtree

git remote add -f make-env https://github.com/SymbiFlow/make-env.git
git subtree add --prefix third_party/make-env make-env master
git fetch make-env master

Updating using git subtree

git subtree pull --prefix third_party/make-env make-env master

Step 2 - Create a Makefile

Create a Makefile which sets;

Include the third_party/make-env/conda.mk file.

Make your targets depend on $(CONDA_ENV_PYTHON) -- probably as an "order only dependency".

Use $(IN_CONDA_ENV) before the commands you wish to run inside the conda environment.

A template Makefile is provided, see below;

# The top directory where environment will be created.
TOP_DIR := $(realpath $(dir $(lastword $(MAKEFILE_LIST))))

# A pip `requirements.txt` file.
# https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
REQUIREMENTS_FILE := requirements.txt

# A conda `environment.yml` file.
# https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
ENVIRONMENT_FILE := environment.yml

include third_party/make-env/conda.mk

# Example make target which runs commands inside the conda environment.
test-command: | $(CONDA_ENV_PYTHON)
	@$(IN_CONDA_ENV) echo "Python is $$(which python)"
	@$(IN_CONDA_ENV) python --version

Testing

To make sure that make-env continues to work correctly and the template Makefile is up to date, an example environment can be found in the test directory.

# core.symlinks=true is needed so windows creates the symlinks.
git clone -c core.symlinks=true https://github.com/SymbiFlow/make-env.git
cd make-env/test
make test-command

Todo list

  • Make sure the conda can be disabled and system tooling be used directly.
  • Add support for other types of environments;
    • Docker provided dependencies.
    • System python + virtualenv.
    • (maybe?) nix-os provided dependencies

Contributing

There are a couple of guidelines when contributing to this project which are listed here.

Sending

All contributions should be sent as GitHub Pull requests.

License

All software (code, associated documentation, support files, etc) in the Project X-Ray repository are licensed under the very permissive ISC Licence. A copy can be found in the LICENSE file.

All new contributions must also be released under this license.

Code of Conduct

By contributing you agree to the code of conduct. We follow the open source best practice of using the Contributor Covenant for our Code of Conduct.

Sign your work

To improve tracking of who did what, we follow the Linux Kernel's "sign your work" system. This is also called a "DCO" or "Developer's Certificate of Origin".

All commits are required to include this sign off and we use the Probot DCO App to check pull requests for this.

The sign-off is a simple line at the end of the explanation for the patch, which certifies that you wrote it or otherwise have the right to pass it on as a open-source patch. The rules are pretty simple: if you can certify the below:

    Developer's Certificate of Origin 1.1

    By making a contribution to this project, I certify that:

    (a) The contribution was created in whole or in part by me and I
        have the right to submit it under the open source license
        indicated in the file; or

    (b) The contribution is based upon previous work that, to the best
        of my knowledge, is covered under an appropriate open source
        license and I have the right under that license to submit that
        work with modifications, whether created in whole or in part
        by me, under the same open source license (unless I am
        permitted to submit under a different license), as indicated
        in the file; or

    (c) The contribution was provided directly to me by some other
        person who certified (a), (b) or (c) and I have not modified
        it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

then you just add a line saying

Signed-off-by: Random J Developer <[email protected]>

using your real name (sorry, no pseudonyms or anonymous contributions.)

You can add the signoff as part of your commit statement. For example:

git commit --signoff -a -m "Fixed some errors."

Hint: If you've forgotten to add a signoff to one or more commits, you can use the following command to add signoffs to all commits between you and the upstream master:

git rebase --signoff upstream/master

make-env's People

Contributors

daniellimws avatar litghost avatar mglb avatar mithro avatar tcal-x avatar umarcor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

make-env's Issues

Document how to implement a user-level 'make clean'

conda.mk defines a clean:: target. This is a 'double-colon' rule.

In any user Makefile that includes conda.mk (which they must include in order to use $(IN_CONDA_ENV)), there cannot be a clean target unless it is also a double-colon target. And in this case, both the user's new clean:: rule as well as make-env's clean:: rule are executed. The latter removes the Conda env.

Is there a way the user can implement their own make clean which does not delete the Conda env?

Solve disk space reporting on Mac OS X

The output of make env-info has a weird /U after the disk space usage. Presumably this is caused by the sed regex not handling the du -h -s output.

$ make env-info
               Currently running on: 'MacOSX (x86_64)'
         Conda environment is named: ' conda-env'
   Conda Env Top level directory is: '/Users/travis/build/mithro/conda-env-make/test'
         Git top level directory is: '/Users/travis/build/mithro/conda-env-make'
              The version number is: 'v0.0-21-gb326bed'
            Git repository is using: 120K	/U
     Environment setup directory is: '$TOP_DIR/env'
    Download and cache directory is: '$TOP_DIR/env/downloads' (using 258M	/U)
               Conda's directory is: '$TOP_DIR/env/conda' (using 374M	/U)
 Conda's packages download cache is: '$TOP_DIR/env/downloads/conda-pkgs' (using 205M	/U)
           Conda's Python binary is: '$TOP_DIR/env/conda/envs/ conda-env/bin/python'

Add tests

Add test which checks that the Makefile.template works.

Check on CI conda hasn't "infected" the system

The CI system should check that the conda install hasn't done any of the following;

  • Modified the registry
  • Modified the environment (path, etc)
  • Registered shortcuts in the startup menu.

Make providing requirements.txt optional

For conda you only need to provide an environment.yml file if there are no Python dependencies.

Maybe the requirements.txt could be extracted from the environment.yml file?

Ubuntu 20.10 build failing on CI

See log @ https://github.com/SymbiFlow/make-env/actions/runs/3778151316/jobs/6422390105

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: The repository 'http://security.ubuntu.com/ubuntu groovy-security Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu groovy Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu groovy-updates Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu groovy-backports Release' does not have a Release file.
Error: Process completed with exit code 100.

Allow usage of `mamba` instead of `conda`

mamba was/is a rewrite of the conda tool to be a lot more memory efficient when using the extremely large conda-forge repository. It should be a drop in replacement for conda.

Fix `make env-info` on Windows

The current output of make env-info on Windows looks like this;

$ make env-info
OS_TYPE=Windows CPU_TYPE=x86_64
'name: make-env' 'make-env'
"               Currently running on: 'Windows (x86_64)'"
ECHO is off.
"         Conda environment is named: 'make-env'"
"   Conda Env Top level directory is: 'C:\Users\travis\build\mithro\conda-env-make\test'"
"         Git top level directory is: '$(git rev-parse --show-toplevel)'"
"              The version number is: '$(git describe)'"
"            Git repository is using: $(du -h -s $(git rev-parse --show-toplevel)/.git | sed -e's/\s.*//')" 	
ECHO is off.
"     Environment setup directory is: 'C:\Users\travis\build\mithro\conda-env-make\test\env'" 	
"    Download and cache directory is: 'C:\Users\travis\build\mithro\conda-env-make\test\env\downloads' (using $(du -h -s C:\Users\travis\build\mithro\conda-env-make\test\env\downloads | sed -e's/\s.*//'))" 	
"               Conda's directory is: 'C:\Users\travis\build\mithro\conda-env-make\test\env\conda' (using $(du -h -s C:\Users\travis\build\mithro\conda-env-make\test\env\conda | sed -e's/\s.*//'))" 	
" Conda's packages download cache is: 'C:\Users\travis\build\mithro\conda-env-make\test\env\downloads\conda-pkgs' (using $(du -h -s C:\Users\travis\build\mithro\conda-env-make\test\env\downloads\conda-pkgs | sed -e's/\s.*//'))" 	
"           Conda's Python binary is: 'C:\Users\travis\build\mithro\conda-env-make\test\env\conda\envs\make-env\python.exe'"	
The command "make env-info" exited with 0.

It should look like this;

$ make env-info
OS_TYPE=Linux CPU_TYPE=x86_64
'name: make-env' 'make-env'
               Currently running on: 'Linux (x86_64)'
         Conda environment is named: 'make-env'
   Conda Env Top level directory is: '/home/travis/build/mithro/conda-env-make/test'
         Git top level directory is: '/home/travis/build/mithro/conda-env-make'
              The version number is: 'v0.0-31-gf7d4e6c'
            Git repository is using: 200K
     Environment setup directory is: '$TOP_DIR/env'
    Download and cache directory is: '$TOP_DIR/env/downloads' (using 420M)
               Conda's directory is: '$TOP_DIR/env/conda' (using 571M)
 Conda's packages download cache is: '$TOP_DIR/env/downloads/conda-pkgs' (using 335M)
           Conda's Python binary is: '$TOP_DIR/env/conda/envs/make-env/bin/python'
The command "make env-info" exited with 0.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.