Git Product home page Git Product logo

devshell's Introduction

devshell - like virtualenv, but for all the languages

STATUS: unstable

Devshell Dev Environment Support room on Matrix

The goal of this project is to simplify per-project developer environments.

Imagine, a new employee joins the company, or somebody transfers teams, or somebody wants to contribute to one of your Open Source projects. It should take them 10 minutes to clone the repo and get all of the development dependencies.

Documentation

See docs (docs source)

Features

Compatible

Keep it compatible with:

  • nix-shell
  • direnv
  • nix flakes

Clean environment

pkgs.stdenv.mkDerivation and pkgs.mkShell build on top of the pkgs.stdenv which introduces all sort of dependencies. Each added package, like the pkgs.go in the "Story time!" section has the potential of adding new environment variables, which then need to be unset. The stdenv itself contains either GCC or Clang which makes it hard to select a specific C compiler.

This is why mkShell builds its environment from a builtins.derivation.

direnv loads will change from:

direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +HOST_PATH +IN_NIX_SHELL +LD +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_STORE +NM +OBJCOPY +OBJDUMP +RANLIB +READELF +RUSTC +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +TEMP +TEMPDIR +TMP +TMPDIR +buildInputs +buildPhase +builder +builtDependencies +cargo_bins_jq_filter +cargo_build_options +cargo_options +cargo_release +cargo_test_options +cargoconfig +checkPhase +configureFlags +configurePhase +cratePaths +crate_sources +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +docPhase +dontAddDisableDepTrack +dontUseCmakeConfigure +installPhase +name +nativeBuildInputs +out +outputs +patches +preInstallPhases +propagatedBuildInputs +propagatedNativeBuildInputs +remapPathPrefix +shell +src +stdenv +strictDeps +system +version ~PATH

to:

direnv: export +DEVSHELL_DIR +PRJ_DATA_DIR +PRJ_ROOT +IN_NIX_SHELL +NIXPKGS_PATH ~PATH

There are new environment variables useful to support the day-to-day activities:

  • DEVSHELL_DIR: contains all the programs.
  • PRJ_ROOT: points to the project root.
  • PRJ_DATA_DIR: points to $PRJ_ROOT/.data by default. Is used to store runtime data.
  • NIXPKGS_PATH: path to nixpkgs source.

Common utilities

The shell comes pre-loaded with some utility functions. I'm not 100% sure if those are useful yet:

  • menu - list all the programs available

MOTD

When entering a random project, it's useful to get a quick view of what commands are available.

When running nix-shell or nix develop, mkShell prints a welcome message:

๐Ÿ”จ Welcome to devshell

[[general commands]]

  hello         - prints hello
  menu          - prints this menu

[formatters]

  nixpkgs-fmt   - Nix code formatter for nixpkgs

[linters]

  golangci-lint - golang linter

[utilites]

  hub           - github utility

[devshell]$ 

Configurable with a TOML file

You might be passionate about Nix, but people on the team might be afraid of that non-mainstream technology. So let them write TOML instead. It should handle 80% of the use-cases and falling back on Nix is always possible.

Bash completion by default

Life is not complete otherwise. Huhu.

Packages that contain bash completions will automatically be loaded by mkShell in nix-shell or nix develop modes.

Capture development dependencies in CI

With a CI + Binary cache setup, one often wants to be able to capture all the build inputs of a shell.nix. With mkShell capturing all of the development dependencies is as easy as:

nix-build shell.nix | cachix push <mycache>

Runnable as a Nix application

Devshells are runnable (via nix run). This makes it possible to run commands defined in your devshell without entering a nix-shell or nix develop session:

nix run '.#<myapp>' -- <devshell-command> <and-args>

This project itself exposes a Nix application; you can try it out with:

nix run 'github:numtide/devshell' -- hello

See here for more details.

TODO

A lot of things!

  • Documentation
    • Explain how all of this works and all the use-cases.
  • Testing
    • Write integration tests for all of the use-cases.
  • Lazy dependencies
    • This requires some coordination with the repository structure. To keep the dev closure small, it would be nice to be able to load some of the dependencies on demand.
  • Doctor / nix version check
    • Not everything can be nicely sandboxed. Is it possible to get a fast doctor script that checks that everything is in good shape?
  • Support other shells
    • What? Not everyone is using bash? Right now, support is already available in direnv mode.

Contributing

Docs

  1. Change files in docs/
  2. Run nix run .#docs
  3. Open docs

Benchmark

  1. See benchmark/README.md
  2. Run nix run .#bench

Commercial support

Looking for help or customization?

Get in touch with Numtide to get a quote. We make it easy for companies to work with Open Source projects: https://numtide.com/contact

devshell's People

Contributors

amarshall avatar blaggacao avatar booniepepper avatar bors[bot] avatar brianmcgee avatar chvp avatar davhau avatar deemp avatar dependabot[bot] avatar happysalada avatar i077 avatar jfroche avatar johnae avatar kalekseev avatar karolisl avatar malob avatar mic92 avatar nrdxp avatar pacman99 avatar peterbecich avatar roberth avatar tennox avatar tfadeyi avatar thenonameguy avatar tomeon avatar ursi avatar wolfangaukang avatar yajo avatar zimbatm avatar zoonfafer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

devshell's Issues

I had expected env to support env variable substituion

${envToBash env}

I might miss a deeper reason, but as a naive user, I would have expected:

{

  env = {
    KUBECONFIG="$HOME/.kube/config";
  };

}

to work. Instead it evaluates to $HOME/.kube/config literally. Although the workaround is straight forward, I want to ensure with this issue it is intentional.

work around

{
  bash = {
    extra = ''
    export KUBECONFIG="$HOME/.kube/config"
    '';
  };
}

Propagating build inputs

My use case is being able to run bundle install in the shell for projects using Ruby gems with native dependencies that need to be compiled with gcc (say, libxml or Postgres client). This works when I specify those pacakges as buildInputs in pkgs.mkShell, but there's no option to do so with devshell and even when I hacked the source a bit to pass it down to buildEnv it doesn't help either โ€“ the culprit seems to be the list of environment variables passed down to the env (it's lacking the compiler-related envvars).

I'm not sure what's the best direction to take here (I'm a bit of a newb when it's coming to using nix as something other than a glorified ansible for my home infrastructure) but I think it's an useful use-case to cover.

Support pre-push githook natively

#18 provides for use cases of pre-push hooks specifically. #16 (comment) provides analysis for why a lightweight implementation is superior beyond the mere benefit of being leightweight.

That means, the consolidated intended use case does not require git workdir management. What's worse, providing such would induce misinterpretations about the consolidated intended use case.

Therefore, it is proposed for devshell to arrange for:

  • A simple command sequence

    • that is defined in devshell's standard interfaces
    • and that is executed in order
    • in the context of the repo's root and within debshell env
    • by a devshell library facility
    • which is symlinked to .git/hooks/pre-push.
  • The facility is capable of detecting modified files against the reference git ref,

    • which in the case of pre-push
    • is origin's HEAD.
  • The facility passes the modified files list

    • as last variadic argument
    • to each command in the list.
  • The command itself needs to arrange for

    • handling this variadic last argument (discarding if necessary),
    • filtering on file types against this variadic last argument
      • devshell may provide a configurable generic wrapper script for filtering, that supports configurable matching through:
        • regex include
        • regex exclude
        • as well as identify

Managing local dns (/etc/hosts) & development certificates

https://github.com/guumaster/hostctl is a great tool for manipulating local /etc/hosts for development.

But it also defies the very principles of isolated devshell environments.

The way it usually works is:

  • a .etchosts file checked into version control
  • this is then loaded with sudo $(which hostctl) add $profilename --from .etchosts
  • and removed with sudo $(which hostctl) remove $profilename

It also can be activated and deactivated, once loaded, with:

  • ... enable
  • ... disable

But those latter commands seem more useful for interactive usage for special requirements or temporary profile switches, and are not relevant to the genuine setup of a devshell.

Though in order to accommodate for this local /etc/hosts management through devshell, I think two things would have to be achieved.

  • acquire sudo privileges during setup (well... ๐Ÿ˜‰ I'm already running away)
  • provide finalizer hooks

Or take a completely other approach, like: https://github.com/bAndie91/libnss_homehosts (at least for nixos users that might be a hidden goodie, if packaged there)

Or just acknowledge it would be cool, but no thanks! Too complicated. ๐Ÿ˜„

Document how to share devshell modules

I think flakes are an excellent code / knowledge sharing platform infrastructure.

I would like to seek structural guidance (later on: eg. examples / templates) on how to combine those ideas in a proper way.

I'm also wondering, how user provided devshells would / could / should interact with project provided devshells.

pkg generator

Based on real experienced frustration, it would be extremely powerful to have a wrapper to go get, so that when you go get something within a devshell environment, it automatically generates a valid nix package descriptor.

Similar for other languages.

Alternative (less intrusive) UX:

  • devshell go get ...
  • devhsell pip install ... (with mach-nix)
  • ...

Bonus:

-devshell upstream -> generates a PR to nixpkgs with custom packages ๐Ÿ˜‰ โ€” problem nixpkgs folder "chaos" maybe does not provide sufficiently clear metadata

My experience is that it took a very long time to get all tooling set up (one time effort, but who knows if I decide to change tooling tomorrow, on the go).

Compatibility with nix - the package manager (w/o nixos)

Before

NIX_PATH=
  /home/blaggacao/.nix-defexpr/channels
PATH=
  /home/blaggacao/.nix-profile/bin:
  /home/blaggacao/.cargo/bin:
  /home/blaggacao/.npm-global/bin:
  /home/blaggacao/.poetry/bin:
  /home/blaggacao/go/bin:
  /home/blaggacao/bin:
  /home/blaggacao/.local/bin:
  /usr/local/bin:
  /usr/bin:
  /bin:
  /usr/games:
  /usr/local/go/bin:
  /snap/bin:
  /home/blaggacao/.fzf/bin

Do

$ nix-shell

After

NIX_PATH=
      # I don't have those...
  nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs:
          /nix/var/nix/profiles/per-user/root/channels:
                # Only this is valid
          /home/blaggacao/.nix-defexpr/channels


PATH=
      # Those would shadow devshell packages
  /home/blaggacao/.nix-profile/bin:
  /home/blaggacao/.nix-profile/bin:
  /nix/var/nix/profiles/default/bin:
  /nix/store/979a6pz5pcvlqv41k0wq91gbbqaz0ws4-neptun.nix-env/bin:
  /nix/store/xadrr3l5jvkkm3g3lb2g81j5wz51zqdv-bash-interactive-4.4-p23/bin:
  /nix/store/sdvgyx4635dwbmiqwmc8zhsj92zd7456-bash-interactive-4.4-p23/bin:

      # Refrain
  /home/blaggacao/.nix-profile/bin:
  /home/blaggacao/.cargo/bin:
  /home/blaggacao/.npm-global/bin:
  /home/blaggacao/.poetry/bin:
  /home/blaggacao/go/bin:
  /home/blaggacao/bin:
  /home/blaggacao/.local/bin:
  /usr/local/bin:
  /usr/bin:
  /bin:
  /usr/games:
  /usr/local/go/bin:
  /snap/bin:
  /home/blaggacao/.fzf/bin:

      # Repeat everybody!
  /home/blaggacao/.nix-profile/bin:
  /home/blaggacao/.cargo/bin:
  /home/blaggacao/.npm-global/bin:
  /home/blaggacao/.poetry/bin:
  /home/blaggacao/go/bin:
  /home/blaggacao/bin:
  /home/blaggacao/.local/bin:
  /usr/local/bin:
  /usr/bin:
  /bin:
  /usr/games:
  /usr/local/go/bin:
  /snap/bin:
  /home/blaggacao/.fzf/bin

Furthermore would not have any effect:

{
  env = {
    NIX_PATH = "nixpkgs=${pkgs.path}";
  };
}

Flexible environment variable update

First of all, thank you again in the open for this strategic piece of work!

I want to report a use case we hit:

  • we need to modify KUBECONFIG to switch environments quickly in a k8s-nix combined devops project.
  • any task runner (just) would not be able to export this environment out of his process scope
  • we don't want to tell people to manually eval something additionally (this can get quickly messy)
  • so we kind of settled on using devshell (still the tradicional one) to modify the environment and reload it

Is it legit to thing of a way to export environment variables structurally, not within a '' literal string part ''?


Maybe it's not legit, though, and we should use overlays to modify the invoking commands to behave differently in the context of the nix shell.

'goodbye' to complement 'startup' sequence

While refactoring and playing on #28 according to the new module structure, I was feeling a slight need for tearing down the custom certificate upon leaving the devshell (so in order to safely "clean up after myself").

Hence I thought, maybe a goodbye sequence similar to #62 could be helpful.

On the other hand, this might be a dangerous feature (in the sense of a powerful door that can never be shut again).

Allow to use arbitrary scripts

There are various places where we convert strings to shell scripts. And those assume that the script will be using bash.

Add a bit of logic in nix that checks if the string starts with a shebang. If there is a shebang, write the script as-is. Otherwise, add the bash shebang.

Tracking windows support

I hope devshell will grow into an universal tool, if wsl2 is properly supported for nix.

It would be interesting to hear, if somebody has already experience to share, so that I can provide instructions to windows users for a repository I maintain.

Handle dynamic environment variables

[env]
BAR = "123"
FOO = "$BAR"

This construct has the advantage of being straight-forward and easy to understand. But it has one issue: values are always escaped and the user has to use the [bash.extra] section in order to do more advanced stuff. Which in turn is problematic because bash.extra is not easily extensible by other modules.

If we want to support dynamic values, we will have to introduce an array like this:

[[envs]]
name = "BAR"
value = "123"
[[envs]]
name = "FOO"
dyn_value = "$BAR"

We can draw inspiration on Kubernetes to see how they are doing this on pods.

Another useful option would be a depends = [ "BAR" ] key that would allow to re-shuffle the list. Essentially making this a DAG.

As you can see, this makes the whole setup more verbose.

Open questions

Should we keep both constructs or is one enough?

Document how to get started

The current documentation is not quite correct. The CLI is not ready and will not be for a while. Instead, document how to integrate using the nix-shell.

devshell.toml.local

Users might want to extend or override some of the project devshell.toml settings with their own.

If a devshell.local.toml exists, also load that.

Use case: 2 nixpkgs versions

I had a use case where I needed packages from two nixpkgs version:

  • haskell packages from the verion for which haskell.nix has a relatively decent cache
  • freecad / kicad from latest (due to a bug in freecad)

My solution so far, no idea if it's optimal, but wanted to share:

# flake.nix
{
  inputs.nixpkgsUnstable.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  outputs = { self, nixpkgs, flake-utils, devshell, nixpkgsUnstable }:
    let
      name = "CashBoxy";
      config = {};
    in flake-utils.lib.simpleFlake {
      inherit self name nixpkgs config;
      overlay = import ./overlay.nix {inherit nixpkgsUnstable config;};
      shell = ./devshell.nix;
    };
}

# overlay.nix
{ nixpkgsUnstable, config }: final: prev:
{
  CashBoxy =
    let
      pkgsUnstable = import nixpkgsUnstable {
        inherit config;
        localSystem = prev.stdenv.buildPlatform;
        crossSystem = prev.stdenv.hostPlatform;
      };
    in
      {
        kicad = pkgsUnstable.kicad;
        freecad = pkgsUnstable.freecad;
      };
}

# devshell.nix
{ pkgs }:
with pkgs; mkDevShell {
  name = "CashBoxy";
  packages = [
    CashBoxy.kicad
    CashBoxy.freecad
  ];
}

I found it tricky, so I wanted to share one option, if somebody is looking for how to do it.

Please don't regard this at any rate authoritative. It's just me playing around... (!)

Generate the Go datastructure

It would be good to be able to generate the Go struct from the Nix module to guarantee that they stay in sync.

The schema defined in mkDevShell/options.nix is the canonical version and devshell/config.go needs to be kept in sync with it. To avoid unnecessary discrepancy, it would be great if the latter could be derived from the former.

inNixShell not working as expected

It looks like this variable is not being injected in older versions of Nix.

TODO: find out at which version of Nix this started working.

adding package's to the environment

sometimes I want to put rustPlatform.rustcSRC into RUST_SRC_PATH, so it will return the evaluated path.

maybe the implementation is like:

[env]
RUST_SRC_PATH="(rustPlatform.rustcSrc)"

I'm not sure it is possible in toml.

Udev rules for dev environments

For some dev envronments, there is a use case to install udev rules for non-root device access (MODE="0666") to specific devices such as a particular arduino board.

The current nixos-specific workarround is to use services.udev.package = [ pkgs.platformio ];.

However, this does not properly encapsulate the desired state of the development shell.
Maybe this ultimately falls into the same problem category as managing PKI or DNS for development environments on the host system.

Further links:

  The udev rules are read from the files located in the system rules
  directories /usr/lib/udev/rules.d and /usr/local/lib/udev/rules.d,
  the volatile runtime directory /run/udev/rules.d and the local
  administration directory /etc/udev/rules.d. All rules files are
  collectively sorted and processed in lexical order, regardless of the
  directories in which they live. However, files with identical
  filenames replace each other. Files in /etc/ have the highest
  priority, files in /run/ take precedence over files with the same
  name under /usr/. This can be used to override a system-supplied
  rules file with a local file if needed; a symlink in /etc/ with the
  same name as a rules file in /usr/lib/, pointing to /dev/null,
  disables the rules file entirely. Rule files must have the extension
  .rules; other extensions are ignored.

To manipulate potential candidate destination at /run/udev/rules.d, root access seems required (who would have thought ๐Ÿ˜‰):

โžœ  ~ ls -la /run/udev/
drwxr-xr-x root root 160 B  Mon Dec 21 11:15:39 2020 ๏„• .

/cc @carlosluquec

packagesFrom

sometimes one wants to get all the build dependencies of a package into the environment

DevShell native pre commit hooks

My first intuition was to claim a tight integration with pre-commit would be the way to go. Sleeping over it I think differently, hence this very issue comes to be. It proposes to consider the possibility of a shortcut and aspire a clear cut, simple and clean re-implementation.

pre-commit essentially does three things:

Alarm! Doesn't nix package tools? โ€” And is arguably better at doing so?!? Since, we are left with git hooks and workdir management. A typical pre-commit pre-push hook is relatively straight forward:

pre-push
#!/nix/store/n8nviwmllwqv0fjsar8v8k8gjap1vhcw-python3-3.7.6/bin/python3
"""File generated by pre-commit: https://pre-commit.com"""
from __future__ import print_function

import distutils.spawn
import os
import subprocess
import sys

# work around https://github.com/Homebrew/homebrew-core/issues/30445
os.environ.pop('__PYVENV_LAUNCHER__', None)

HERE = os.path.dirname(os.path.abspath(__file__))
Z40 = '0' * 40
ID_HASH = '138fd403232d2ddd5efb44317e38bf03'
# start templated
CONFIG = '.pre-commit-config.yaml'
HOOK_TYPE = 'pre-push'
INSTALL_PYTHON = '/nix/store/n8nviwmllwqv0fjsar8v8k8gjap1vhcw-python3-3.7.6/bin/python3.7'
SKIP_ON_MISSING_CONFIG = False
# end templated


class EarlyExit(RuntimeError):
    pass


class FatalError(RuntimeError):
    pass


def _norm_exe(exe):
    """Necessary for shebang support on windows.

    roughly lifted from `identify.identify.parse_shebang`
    """
    with open(exe, 'rb') as f:
        if f.read(2) != b'#!':
            return ()
        try:
            first_line = f.readline().decode('UTF-8')
        except UnicodeDecodeError:
            return ()

        cmd = first_line.split()
        if cmd[0] == '/usr/bin/env':
            del cmd[0]
        return tuple(cmd)


def _run_legacy():
    if __file__.endswith('.legacy'):
        raise SystemExit(
            "bug: pre-commit's script is installed in migration mode\n"
            'run `pre-commit install -f --hook-type {}` to fix this\n\n'
            'Please report this bug at '
            'https://github.com/pre-commit/pre-commit/issues'.format(
                HOOK_TYPE,
            ),
        )

    if HOOK_TYPE == 'pre-push':
        stdin = getattr(sys.stdin, 'buffer', sys.stdin).read()
    else:
        stdin = None

    legacy_hook = os.path.join(HERE, '{}.legacy'.format(HOOK_TYPE))
    if os.access(legacy_hook, os.X_OK):
        cmd = _norm_exe(legacy_hook) + (legacy_hook,) + tuple(sys.argv[1:])
        proc = subprocess.Popen(cmd, stdin=subprocess.PIPE if stdin else None)
        proc.communicate(stdin)
        return proc.returncode, stdin
    else:
        return 0, stdin


def _validate_config():
    cmd = ('git', 'rev-parse', '--show-toplevel')
    top_level = subprocess.check_output(cmd).decode('UTF-8').strip()
    cfg = os.path.join(top_level, CONFIG)
    if os.path.isfile(cfg):
        pass
    elif SKIP_ON_MISSING_CONFIG or os.getenv('PRE_COMMIT_ALLOW_NO_CONFIG'):
        print(
            '`{}` config file not found. '
            'Skipping `pre-commit`.'.format(CONFIG),
        )
        raise EarlyExit()
    else:
        raise FatalError(
            'No {} file was found\n'
            '- To temporarily silence this, run '
            '`PRE_COMMIT_ALLOW_NO_CONFIG=1 git ...`\n'
            '- To permanently silence this, install pre-commit with the '
            '--allow-missing-config option\n'
            '- To uninstall pre-commit run '
            '`pre-commit uninstall`'.format(CONFIG),
        )


def _exe():
    with open(os.devnull, 'wb') as devnull:
        for exe in (INSTALL_PYTHON, sys.executable):
            try:
                if not subprocess.call(
                        (exe, '-c', 'import pre_commit.main'),
                        stdout=devnull, stderr=devnull,
                ):
                    return (exe, '-m', 'pre_commit.main', 'run')
            except OSError:
                pass

    if os.path.isfile('/nix/store/1kfw43by83rd2ri483vqbd32srm4v45d-pre-commit-1.21.0/bin/pre-commit') and os.access('/nix/store/1kfw43by83rd2ri483vqbd32srm4v45d-pre-commit-1.21.0/bin/pre-commit', os.X_OK):
        return ('/nix/store/1kfw43by83rd2ri483vqbd32srm4v45d-pre-commit-1.21.0/bin/pre-commit', 'run')
    if distutils.spawn.find_executable('pre-commit'):
        return ('pre-commit', 'run')

    raise FatalError(
        '`pre-commit` not found.  Did you forget to activate your virtualenv?',
    )


def _rev_exists(rev):
    return not subprocess.call(('git', 'rev-list', '--quiet', rev))


def _pre_push(stdin):
    remote = sys.argv[1]

    opts = ()
    for line in stdin.decode('UTF-8').splitlines():
        _, local_sha, _, remote_sha = line.split()
        if local_sha == Z40:
            continue
        elif remote_sha != Z40 and _rev_exists(remote_sha):
            opts = ('--origin', local_sha, '--source', remote_sha)
        else:
            # ancestors not found in remote
            ancestors = subprocess.check_output((
                'git', 'rev-list', local_sha, '--topo-order', '--reverse',
                '--not', '--remotes={}'.format(remote),
            )).decode().strip()
            if not ancestors:
                continue
            else:
                first_ancestor = ancestors.splitlines()[0]
                cmd = ('git', 'rev-list', '--max-parents=0', local_sha)
                roots = set(subprocess.check_output(cmd).decode().splitlines())
                if first_ancestor in roots:
                    # pushing the whole tree including root commit
                    opts = ('--all-files',)
                else:
                    cmd = ('git', 'rev-parse', '{}^'.format(first_ancestor))
                    source = subprocess.check_output(cmd).decode().strip()
                    opts = ('--origin', local_sha, '--source', source)

    if opts:
        return opts
    else:
        # An attempt to push an empty changeset
        raise EarlyExit()


def _opts(stdin):
    fns = {
        'prepare-commit-msg': lambda _: ('--commit-msg-filename', sys.argv[1]),
        'commit-msg': lambda _: ('--commit-msg-filename', sys.argv[1]),
        'pre-merge-commit': lambda _: (),
        'pre-commit': lambda _: (),
        'pre-push': _pre_push,
    }
    stage = HOOK_TYPE.replace('pre-', '')
    return ('--config', CONFIG, '--hook-stage', stage) + fns[HOOK_TYPE](stdin)


if sys.version_info < (3, 7):  # https://bugs.python.org/issue25942
    def _subprocess_call(cmd):  # this is the python 2.7 implementation
        return subprocess.Popen(cmd).wait()
else:
    _subprocess_call = subprocess.call


def main():
    retv, stdin = _run_legacy()
    try:
        _validate_config()
        return retv | _subprocess_call(_exe() + _opts(stdin))
    except EarlyExit:
        return retv
    except FatalError as e:
        print(e.args[0])
        return 1
    except KeyboardInterrupt:
        return 1


if __name__ == '__main__':
    exit(main())

It corresponds to a per-hook-configuration-interface roughly in the lines of:

entry: /nix/store/s4vwm64km5xc6glq69fa2dn6s0fl57dv-nixpkgs-fmt-0.6.1/bin/nixpkgs-fmt
exclude: ^$
files: \\.nix$  # include filter regex - include when a certain git diff touches one of those files
id: nixpkgs-fmt
name: nixpkgs-fmt
pass_filenames: true  # whether to pass touched filenames as argument to the tool
types:  # include filter using identity lib - this is interesting and convenient
 - file

In addition to this, one can selectively skip hooks by passing SKIP="nixpkgs-fmt,...".

It is also a handy feature to manually execute those configurations against all files (irrespective of them being touched by a certain commit range, but respecting the include / exclude filters) in order to do ad-hoc or general cleanups. The CLI interface for this task is bad, though: devshell cold attempt to do better.


[pre-commit]
[[nixpkgs-fmt]]
entrypoint = "$DEVSHELL_DIR/bin/nixpkgs-fmt ..."
exclude = ["^.ignore.me.nix$"]
include = ["(\\w|\\/)+?\\.nix$"]
types = [ "files", "nix" ]
pass_filenames = true

[pre-push]
[[go-lint]]  # name
entrypoint = "$DEVSHELL_DIR/bin/go-lint? ..."
types = [ "go" ]
### ๐Ÿ”จ Welcome to mkDevShell ####

# Commands

devshell-menu - print this menu
devshell-root - change directory to root
pre-commit.nixpkgs-fmt   - used to format Nix code
pre-push.go-lint   - used to format Go code

Localhost HTTPS and domains

Related to #74 and #75 .

This is not directly a responsibility of devshell but is related to making a comfortable development environment.

Oftentimes, developers want to be able to test one or more services locally, with HTTPS enabled, in order to replicate the production environment as much as possible. This ticket contains some notes on what I think is the ideal setup.

Localhost HTTPS interface

We need the cooperation of the host on two fronts:

  • dynamic hostnames that point to localhost
  • TLS certificate acquisition

*.test TLD for localhost

The .test TLD is reserved and is therefore guaranteed to never be clashing with external domains. https://en.wikipedia.org/wiki/.test

The developer can either edit their /etc/hosts, or have a local DNS resolver that maps all the *.test domains to localhost.

Open question: how to handle port-mapping so that multiple services can bind? On Linux, 127.X.X.X is all reserved for localhost so each service could technically get their own IPs. Maybe we can use a dictionary to IP mapping or something like that.

Let's encrypt for localhost

Ideally, a localhost service is running that talks ACME protocol.

The CA public certificate would be installed in the computer's trust cert.

The CA private certificate is installed in the ACME service and used to issue new certificates.

The services themselves would query the ACME protocol to acquire new certs and use http01 for validation.

Ideally, the CA can be pinned to it can only issue certs for the .test top-level. That way if the private certificate gets leaked, the impact would still be limited.

Prompt with custom shell functions

{
  bash = {

    extra = ''
    # kube-ps1 context & namespace prompt
    source $(which kube-ps1)  # doesn't run in the final shell
    echo Run: $(tput bold; tput setaf 3)source '$(which kube-ps1)'$(tput sgr0)
    export RPROMPT='$(kube_ps1)'
    '';
  };
}

I want to elide to ask the user to manually run:

$ source $(which kube-ps1)

The shell function kube_ps1 is not loaded into the final shell using either direnv or plain nix-shell.
Is this only supposed to work with nix develop ?

It looks as if the following trampoline was arranged for this use case, though:

devshell/default.nix

Lines 213 to 227 in 8850638

# $stdenv/setup is loaded by nix-shell during startup.
# https://github.com/nixos/nix/blob/377345e26f1ac4bbc87bb21debcc52a1d03230aa/src/nix-build/nix-build.cc#L429-L432
stdenv = writeTextFile {
name = "devshell-stdenv";
destination = "/setup";
text = ''
# Fix for `nix develop`
: ''${outputs:=out}
runHook() {
eval "$shellHook"
unset runHook
}
'';
};

โ†’ kube-ps1

Native Shift Left goodies

This issue aims to replace #16 with a more concise and well defined generic use case instead of calling for particular tooling, see #16 (comment).

Shift left is a strategy to move CI and CD components into the โ€” figuratively โ€” "airplane mode".

This typically involves a range of things like:

  • Code formatting
  • Code linting
  • Code generators
  • Integration testing
  • Deploy spec overlay rendering
  • Even artifact generation (and pushing once off the plane)

In my interpretation, this approach mainly avoids the ownership mismatch of central tooling.

Typical tools involed in the solution are:

  • formatting โ†’ $EDITOR hooks + .editorconfig
  • linting โ†’ pre-push โ€” cave: not pre-commit โ€” hooks.
  • code generators โ†’ pre-push or task runner
  • e2e, etc. โ†’ task runner (e.g. make or just)

make the menu optional

When there are a lot of dependencies, printing all the binaries takes more than one shell page, which is not super helpful. Would it be better if the menu was constructed manually?

warning: shell level (1000) too high, resetting to 1

Hi @zimbatm , I encountered this error when using indirect package definitions via [[commands]]. In turn using plain package, the problem is not observable.

Do you have any clou?

devshell git:(err-shell-level) โœ— stack
/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash: warning: shell level (1000) too high, resetting to 1
/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash: warning: shell level (1000) too high, resetting to 1
^C

Reproducible on my machine on this minimal branch: https://github.com/blaggacao/devshell/tree/err-shell-level

NIXPKGS_PATH is defined after the user variables

Because of how the module system works, the values in the top module are set before anybody else.

This causes an issue because a user might want to set the NIX_PATH to nixpkgs:$NIXPKGS_PATH, which is set after.

provide slots for nix * native commands

Coming from devshell and integrating it with nix flakes, I can perceive how it would be useful if devshell could expose a flakify helper function, so that:

  • a devshell command can be marked as part of nix check
  • a devshell command can be marked as part of nix build

This would make flakes integration more elegant. CI and people not familiar with devshell could do nix check / nix build .#..., but developers could still do menu and then let's say test-all within a nix shell.

Surely, this use case is easily handled coming from nix flakes transition into devshell. (command = "${pkgs.myproject.my_local_script}/bin/my-local-scipt ...";)
So I'm not sure if it is concern of devshell to shovel people into nix flakes via the proposed transition facilities...

Add a [nixpkgs] module

NixOS allows the user to override nixpkgs from within the module system. What happens under the hood is quite magic really.

In order to minimize the amount of nix plumbing needed, we could introduce something like that that allows a poor-man's nixpkgs pinning and configuration:

[nixpkgs.src]
url = "git+https://github.com/NixOS/nixpkgs"
rev = "<commit-id>"
[nixpkgs.config]
allowBroken = true
allowUnfree = true

How to handle custom versions of packages

I do have some custom packages for dev shells. For example kustomize is a central devshell component to us, but it's been under ongoing heavy development and release activity for the past year or so. Sometimes the last monday's version does not include the latest required bux fixes.

The pace at which nixpkgs catches up โ€” even though extraordinarily up to date โ€” is still sometimes too slow for our use case.

It would be interesting to think of ways how plain nix packages could be plugged into a toml config-based approach and work together with this cli tool seamlessly.

I think devops-responsible persons do have sometimes similar problems: how to ship latest critical bugfixes in devop tools to all developers quickly

Originally posted by @blaggacao in #13 (comment)

Current default MOTD causes errors

I get the following error with the default motd: error: \u characters in JSON strings are currently not supported
Probably the hammer

${ansiOrange}๐Ÿ”จ Welcome to ${config.name}${ansiReset}

I'm guessing this is a nix v2 limitation

Setting a custom motd fixed the issue.

Is this fix sufficient for now? Remove emoji until nix v3?

Allow selecting different packages on different platforms

For example on Linux, cabal needs the binutils package, but not on macOS. Some packages are also not supported on all platforms.

It would be good if there was a mechanism to restrict the addition of certain packages on a per-platform basis.

Target shell transparency

I decided to close #22 as a special case of the need for user shell transparency.

What I mean by that is that devshell scripts should/might gain a notion of the underlying user shell.

The current use case I can think of is prompt modifications, such as with jlesquembre/kubeprompt#13 that need to have awareness of the target shell to choose correct escape codes.

But also the possibility to declare shell functions that are brought over to the target shell might have it's use case, if I'm not mistaken and that's already the case.

This is a relatively barren issue since I have not a lot of context on the underlying issues, but I wanted to log this question for further development.

/cc @jlesquembre

Support for architecture-dependent packages

Sometimes, some packages only make sense on one architecture. For example binutils is needed for GHC on Linux but breaks builds on macOS.

The current workaround is to add a shell.nix and inject the dependency there.

let
  pkgs = import <nixpkgs> { overlays = [ 
    # replace this with your own method of loading the devshell dependency
    ./devshell/overlay.nix
   ]; };
  importTOML = file: {
    _file = file;
    config = builtins.fromTOML (builtins.readFile file);
  };
in
pkgs.mkDevShell {
  imports = [
    # First load the TOML
    (importTOML ../devshell.toml)
  ];

  config = {
    packages = pkgs.lib.optionals pkgs.stdenv.isLinux [
       # `ar` is needed on Linux for `cabal build`
      pkgs.binutils
    ];
 };
}

Track evaluation speed

Let's keep an eye on how fast this is compared to pkgs.mkShell. The nix module system is relatively complicated and might take more time to evaluate.

convert to module system

As mkDevShell gains more attributes, it becomes more and more useful to build types and documentation into the function.

Badge

Devshell Dev Environment

[![Devshell Dev Environment](https://img.shields.io/badge/nix-devshell-success?logo=NixOS)](github.com/numtide/devshell)

Maybe users want to label their repository as supporting devshell development environments. Should it be blue?

Strange issue on a Mac

I'm testing this on a Mac (nix flakes enabled), and I'm getting this output:

% nix develop
mktemp: illegal option -- -
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
### ๐Ÿ”จ Welcome to devshell ####
[commands]
devshell-menu - print this menu
devshell-root - change directory to root
[devshell]$ 

I've been unable to find any mention of mktemp in this repo, however it seems to be related to mkDevShell since I can use mkShell from nixpkgs without ever seeing the above output. On NixOS everything is fine.

Fix `nix develop`

For some reason, nix develop stopped working:

$ nix develop
error: --- Error --------------------------------------------------------------------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/vfc8ydhi6sacga3j677ww6djzi6cfqfh-mkDevShell-env.drv' failed to produce output path '/nix/store/28vlpzyvc8i5c22v9hws8nfwds4537sh-mkDevShell-env'

The mkDevShell-env.drv has different builder args than what is defined in the nix code:

    "builder": "/nix/store/sdvgyx4635dwbmiqwmc8zhsj92zd7456-bash-interactive-4.4-p23/bin/bash",
    "args": [
      "/nix/store/0c88xw36hn0wzmnlyai9s3sibbnq3pnd-get-env.sh"
    ],

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.