Git Product home page Git Product logo

jsonnet-bundler's Introduction

jsonnet-bundler

NOTE: This project is alpha stage. Flags, configuration, behavior and design may change significantly in following releases.

The jsonnet-bundler is a package manager for Jsonnet.

Install

go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

NOTE: please use a recent Go version to do this, ideally Go 1.13 or greater.

This will put jb in $(go env GOPATH)/bin. If you encounter the error jb: command not found after installation then you may need to add that directory to your $PATH as shown in their docs.

Package Install

  • Arch Linux AUR
  • Mac OS X via Homebrew: brew install jsonnet-bundler
  • Fedora (>= 32): sudo dnf install golang-github-jsonnet-bundler

Features

  • Fetches transitive dependencies
  • Can vendor subtrees, as opposed to whole repositories

Current Limitations

  • Always downloads entire dependent repositories, even when updating
  • If two dependencies depend on the same package (diamond problem), they must require the same version

Example Usage

Initialize your project:

mkdir myproject
cd myproject
jb init

The existence of the jsonnetfile.json file means your directory is now a jsonnet-bundler package that can define dependencies.

To depend on another package (another Github repository): Note that your dependency need not be initialized with a jsonnetfile.json. If it is not, it is assumed it has no transitive dependencies.

jb install https://github.com/anguslees/kustomize-libsonnet

Now write myconfig.jsonnet, which can import a file from that package. Remember to use -J vendor when running Jsonnet to include the vendor tree.

local kustomize = import 'kustomize-libsonnet/kustomize.libsonnet';

local my_resource = {
  metadata: {
    name: 'my-resource',
  },
};

kustomize.namePrefix('staging-')(my_resource)

To depend on a package that is in a subtree of a Github repo (this package also happens to bring in a transitive dependency):

jb install https://github.com/prometheus-operator/prometheus-operator/jsonnet/prometheus-operator

Note that if you are copy pasting from the Github website's address bar, remove the tree/master from the path.

If pushed to Github, your project can now be referenced from other packages in the same way, with its dependencies fetched automatically.

All command line flags

$ jb -h
usage: jb [<flags>] <command> [<args> ...]

A jsonnet package manager

Flags:
  -h, --help     Show context-sensitive help (also try --help-long and
                 --help-man).
      --version  Show application version.
      --jsonnetpkg-home="vendor"  
                 The directory used to cache packages in.
  -q, --quiet    Suppress any output from git command.

Commands:
  help [<command>...]
    Show help.

  init
    Initialize a new empty jsonnetfile

  install [<flags>] [<uris>...]
    Install new dependencies. Existing ones are silently skipped

  update [<uris>...]
    Update all or specific dependencies.

  rewrite
    Automatically rewrite legacy imports to absolute ones

Design

This is an implemention of the design specified in this document: https://docs.google.com/document/d/1czRScSvvOiAJaIjwf3CogOULgQxhY9MkiBKOQI1yR14/edit#heading=h.upn4d5pcxy4c

jsonnet-bundler's People

Contributors

alonsomoya avatar bgagnon avatar brancz avatar c0ffeec0der avatar cboggs avatar davidovich avatar discordianfish avatar duologic avatar gliptak avatar hangxie avatar jacobstr avatar klavsklavsen avatar metalmatze avatar mhuxtable avatar olivierlemasle avatar paulfantom avatar pgier avatar rollandf avatar sh0rez avatar sparkprime avatar suriya-ganesh avatar thesuess avatar velmoga avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jsonnet-bundler's Issues

Github ETag doesn't seem to provide the sha1 anymore

The tar file download optimization seems to be broken:

GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/6a0450b32a0f3b25299dcca7f40157e21c50d023.tar.gz 200
archive install failed: etag value "W/"4c236536ad4b8b15f4daa7ceddac1960d016a60b1f39c3e21d83eada0cc71081"" does not look like a SHA1
retrying with git...
...

#38 added an optimization to download a tar from the github server. This optim uses the Etag to resolve the sha1 of the dependency. Since it seems that github is no longer returning the sha in it's Etag, the optimisation is broken.

As we know the sha1 once it is downloaded (first through a slower git operation), we could use this info to re-enable the optimisation (on locked dependency).

We can also use the github api to resolve the ref (but this is rate-limited to 60 req/hour/incoming IP).

jsonnetfiles not updated when installing new bundles

Steps to reproduce

jb init
jb install https://github.com/bitnami-labs/kube-libsonnet
jb install https://github.com/ksonnet/ksonnet-lib/ksonnet.beta.3

Expected results

Both jsonnetfile.json and jsonnetfile.lock.json includes both bundles.

Actual results

Only the first installed bundle is present in jsonnetfile.json and jsonnetfile.lock.json

Version

Experienced using current master, commit 080f157

panic: runtime error: slice bounds out of range

$ jb
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
main.Main(0xc420078058)
	/Users/twilkie/Documents/src/github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb/main.go:71 +0x255
main.main()
	/Users/twilkie/Documents/src/github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb/main.go:202 +0x22

vendor directory structure like golang

I'm curious that why jb does not do vendor directory like golang does, i.e. vendor/github.com/anguslees/kustomize-libsonnet? Then import can just do golang way like import 'github.com/anguslees/kustomize-libsonnet/kustomize.libsonnet' with help of -J vendor to guarantee uniqueness.

Or there is already a way to reach this? I tried to change name in jsonnetfile.json but it does not work.

JB Update Failing

Hi Guys,

im having a strange issue with JB.

When i run a JB Update i get an error like this

$jb update Error parsing commandline arguments: expected command but got "update" jb: error: expected command but got "update"

Ive already tried a jb reinstall?
Any ideas on how i could debug this error?

Regards Michael

[Feature] Keep installation path of local sources in jsonnetfile.json

local source has only directory parameter, it would be great to have an optional parameter to indicate where the lib should be deployed, like:

    {
      "source": {
        "local": {
          "directory": "pkg",
          "targetPath": "vendor/github.com/hangxie/jsonnet-sample/"
        }
      },
      "version": ""
    },

Long story (https://github.com/hangxie/jsonnet-sample shows what I want to achieve) - we expect unit tests of the repo use the same import format as actual clients like:

local lib = import 'github.com/hangxie/jsonnet-sample/pkg/main.libsonnet',

however I didn't find a way to do this, the closest one is --jsonnetpkg-homeoption for both install and update, but it affects all imports.

Our current workaround is creating a symlink as shown in the Makefile:

ln -rfs pkg vendor/github.com/hangxie/jsonnet-sample/

SIGSEGV installing kube-prometheus

I get a seg fault trying to install kube-prometheus:

❯ jb install https://github.com/coreos/[email protected]
GET https://github.com/coreos/kube-prometheus/archive/a4a143a9ab770d72eb0f7dd44bf8e72184347944.tar.gz 200
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11eab1d]

goroutine 1 [running]:
github.com/jsonnet-bundler/jsonnet-bundler/spec/v1.FromV0(0xc0000125d0, 0x104, 0x304, 0x13dd680, 0xc0001521d0)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/spec/v1/v0.go:40 +0x27d
github.com/jsonnet-bundler/jsonnet-bundler/pkg/jsonnetfile.Unmarshal(0xc0001ddc00, 0x104, 0x304, 0x104, 0x304, 0x0, 0x0)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/jsonnetfile/jsonnetfile.go:71 +0x2c5
github.com/jsonnet-bundler/jsonnet-bundler/pkg/jsonnetfile.Load(0xc000226380, 0x39, 0x3, 0xc000226380, 0x39, 0xc000012030)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/jsonnetfile/jsonnetfile.go:44 +0xa8
github.com/jsonnet-bundler/jsonnet-bundler/pkg.ensure(0xc0001223f0, 0x144da24, 0x6, 0xc000122750, 0xc000095520, 0xc000165ad8, 0x11e81e2)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/packages.go:246 +0xc98
github.com/jsonnet-bundler/jsonnet-bundler/pkg.Ensure(0xc0001223f0, 0xc000122701, 0x144da24, 0x6, 0xc000122750, 0x0, 0x0, 0x0)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/packages.go:55 +0x67
main.installCommand(0xc00001c004, 0x21, 0x144da24, 0x6, 0xc000090fb0, 0x1, 0x1, 0x0)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/install.go:71 +0x68b
main.Main(0xc000086058)
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/main.go:77 +0x860
main.main()
	/Users/jason/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/main.go:34 +0x22

jb init on existing project

Running jb init on existing project completely overrides the jsonnetfile.json with an empty one.

Should it just fail or rather ask for permission if the file already exists?

filesystem error when downloading into vendor directory with 0.2.0

with version 0.2.0 whenever i run a jb install or jb update i get

jb: error: failed to install packages: downloading: failed to create tmp dir: stat vendor/.tmp: no such file or directory

I thought this could be related to me running on btrfs but i also tried on a tmpfs filesystem and it fail the same

Setting the jsonnetpkg-home fail as well

with 0.1.0 it works as before

Relative local path dependencies are interpreted relative to cwd and not the containing jsonnetfile location

$ cd ~
$ mkcd a/b
$ pwd
/home/intelfx/a/b
$ jb init
$ GIT_TRACE=1 jb install github.com/coreos/prometheus-operator/contrib/kube-prometheus
19:35:32.168355 git.c:415               trace: built-in: git clone https://github.com/coreos/prometheus-operator vendor/.tmp/jsonnetpkg-kube-prometheus-master407984102
Cloning into 'vendor/.tmp/jsonnetpkg-kube-prometheus-master407984102'...
19:35:32.170174 run-command.c:637       trace: run_command: git-remote-https origin https://github.com/coreos/prometheus-operator
19:35:33.193653 run-command.c:637       trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://github.com/coreos/prometheus-operator/
19:35:33.195016 git.c:415               trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://github.com/coreos/prometheus-operator/
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
19:35:33.384312 run-command.c:637       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 23200 on intelfx-laptop' --check-self-contained-and-connected --pack_header=2,38625
19:35:33.385578 git.c:415               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 23200 on intelfx-laptop' --check-self-contained-and-connected --pack_header=2,38625
remote: Total 38625 (delta 0), reused 0 (delta 0), pack-reused 38622
Receiving objects: 100% (38625/38625), 53.01 MiB | 8.04 MiB/s, done.
Resolving deltas: 100% (21532/21532), done.
19:35:40.531724 run-command.c:637       trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
19:35:40.532658 git.c:415               trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
19:35:41.389250 git.c:415               trace: built-in: git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.
19:35:41.616175 git.c:415               trace: built-in: git clone ../../ vendor/.tmp/jsonnetpkg-kube-prometheus-50038889fade4d7a17352592cd268d9d1e709b93371312653
Cloning into 'vendor/.tmp/jsonnetpkg-kube-prometheus-50038889fade4d7a17352592cd268d9d1e709b93371312653'...
19:35:41.618256 run-command.c:637       trace: run_command: unset GIT_DIR; 'git-upload-pack '\''/home/intelfx/a/b/../../'\'''
19:35:41.619862 git.c:415               trace: built-in: git upload-pack /home/intelfx/a/b/../../
done.
19:35:41.657116 git.c:415               trace: built-in: git checkout 50038889fade4d7a17352592cd268d9d1e709b93
fatal: reference is not a tree: 50038889fade4d7a17352592cd268d9d1e709b93
jb: error: failed to install: failed to install package: exit status 128

As you can see, jb tries to clone /home/intelfx/a/b/../../ which is my dotfiles repo instead of interpreting ../../ relative to github.com/coreos/prometheus-operator/contrib/kube-prometheus and doing the right thing.

jsonnetpkg-home not working

When the jsonnetpkg-home command is specified installing of packages is failing. During the creation of the symlink it expects the vendor folder to be present in the folder where the jsonnetfile.json is present.

command jb install --jsonnetpkg-home="_tmp/vendor" fails with the following error:

jb: error: failed to install packages: symlink github.com/grafana/jsonnet-libs/mixin-utils vendor/mixin-utils: no such file or directory

When jb install is run without the jsonnetpkg-home command, everything installs fine.

[Feature request] jb version

Hello,

I wanted to know is there a way to check which version of jb we are using. Is there a way to check it ? Something like jb -v.

Also, what is the way to update the jb version itself ?
I have installed jb using

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

PS : New to golang (go get) so not sure about this.

jsonnet-bundler not installing on ubuntu-18.04

The following go command was documented here for installing jsonnet-bundler; however, the go command returns no error indication, but the jb command appears unavailable despite the apparent success. Any ideas?

ubuntu % go get -v github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
ubuntu % pwd
/home/bwillcox/jsonnet-bundler
ubuntu %
ubuntu % go get -v github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
ubuntu % jb
jb: command not found
ubuntu % cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
ubuntu % go version
go version go1.10.4 linux/amd64

install/update does not support tag

It works fine with branch though, to reproduce the problem:

  1. fork https://github.com/anguslees/kustomize-libsonnet
  2. create branch test-branch and tag test-tag without code change, so they are all the same as master
  3. try to install with different "versions":
  • master works fine:
jb install git+ssh://[email protected]:hangxie/kustomize-libsonnet.git
...
Branch 'master' set up to track remote branch 'master' from 'origin'.
Already on 'master'
CLEAN vendor/terraform-jsonnet
  • branch works fine:
jb install git+ssh://[email protected]:hangxie/kustomize-libsonnet.git@test-branch
...
Branch 'test-branch' set up to track remote branch 'test-branch' from 'origin'.
Switched to a new branch 'test-branch'
  • tag does not work:
jb install git+ssh://[email protected]:hangxie/kustomize-libsonnet.git@test-tag
...
From github.com:hangxie/kustomize-libsonnet
 * tag               test-tag   -> FETCH_HEAD
error: pathspec 'test-tag' did not match any file(s) known to git
jb: error: failed to install packages: downloading: exit status 1

Just in case it matters:

$ git version
git version 2.23.0

I went through source code quickly, and found https://github.com/jsonnet-bundler/jsonnet-bundler/blob/master/pkg/git.go#L244, git fetch --depth 1 origin <version> handles branch only, and I need to use git fetch --depth 1 origin --tags <version> to handle tag. The --tags/--no-tags behavior is part of git-config so I believe it needs to use the explicit value here (https://git-scm.com/docs/git-fetch, search for tagOpt).

Granular update

We often want to update only one specific dependency, but what we end up having to do is run jb update, then git add -p and revert the rest, and make sure we didn't make a mistake. It would be much more convenient if this was built in.

@sh0rez @metalmatze

Panic on `jb install` due to permission denied

I'm getting a panic when trying to run jb install:

$ jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]
GET https://github.com/coreos/kube-prometheus/archive/ee8077db04965d6b4d9e6a328d5283dd6ba71d33.tar.gz 200
panic: rename vendor/.tmp/jsonnetpkg-github.com-coreos-kube-prometheus-jsonnet-kube-prometheus-release-0.4678299929/jsonnet/kube-prometheus vendor/github.com/coreos/kube-prometheus/jsonnet/kube-prometheus: permission denied

goroutine 1 [running]:
github.com/jsonnet-bundler/jsonnet-bundler/pkg.(*GitPackage).Install(0xc0000b2268, 0x8e0ce0, 0xc0000ae038, 0xc0000d4840, 0x39, 0x84f5f3, 0x6, 0x7fffc9faddbc, 0xb, 0x0, ...)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/git.go:212 +0x24ac
github.com/jsonnet-bundler/jsonnet-bundler/pkg.download(0xc0000d6370, 0x0, 0x7fffc9faddbc, 0xb, 0x0, 0x0, 0x0, 0x0, 0x84f5f3, 0x6, ...)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/packages.go:284 +0x192
github.com/jsonnet-bundler/jsonnet-bundler/pkg.ensure(0xc0001140c0, 0x84f5f3, 0x6, 0xc000114120, 0xc000099040, 0xc000157ad8, 0x5e9042)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/packages.go:233 +0x60e
github.com/jsonnet-bundler/jsonnet-bundler/pkg.Ensure(0xc0001140c0, 0xc000114101, 0x84f5f3, 0x6, 0xc000114120, 0x0, 0x0, 0x0)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/pkg/packages.go:55 +0x67
main.installCommand(0xc00001e004, 0x2d, 0x84f5f3, 0x6, 0xc000094ee0, 0x1, 0x1, 0x0)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/install.go:71 +0x68b
main.Main(0xc00008a058)
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/main.go:77 +0x860
main.main()
        /home/sarah/go/pkg/mod/github.com/jsonnet-bundler/[email protected]/cmd/jb/main.go:34 +0x22
$ jb --version
v0.3.1
$ go version
go version go1.13.5 linux/amd64
$ git --version
git version 2.17.1
$ ls -Rla vendor/
vendor/:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 .
drwxrwxrwx 1 sarah sarah 512 Apr 11 17:47 ..
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:51 .tmp
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 github.com

vendor/.tmp:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:51 .
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 ..

vendor/github.com:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 .
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 ..
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 coreos

vendor/github.com/coreos:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 .
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 ..
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 kube-prometheus

vendor/github.com/coreos/kube-prometheus:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 .
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 ..
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 jsonnet

vendor/github.com/coreos/kube-prometheus/jsonnet:
total 0
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 .
drwxr-xr-x 1 sarah sarah 512 Apr 11 17:47 ..

I am using WSL, so it's possible that there's some weird interaction there? Otherwise I'm not sure what could be causing permissions errors...

local directory as package source

While developing it would be nice to be able to have a package from a local directory, so changes don't have to be committed in order to try them. Currently git repositories can be local remotes, but that requires each change to be committed in order to try it.

ssh: Could not resolve hostname

Broken in master:

$ rm -rf *
$ GO111MODULE="on" go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@master
go: finding github.com master
go: finding github.com/jsonnet-bundler/jsonnet-bundler master
go: finding github.com/jsonnet-bundler/jsonnet-bundler/cmd master
go: finding github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb master
go: finding github.com/jsonnet-bundler master
$ jb --version
v0.2.1-0.20200124070234-7b8a7836a495
$ jb init
$ jb install git+ssh://[email protected]:hangxie/kustomize-libsonnet.git@test-branch
Initialized empty Git repository in /home/xiehang/tmp/jb/vendor/.tmp/jsonnetpkg-github.com-hangxie-kustomize-libsonnet-test-branch532767166/.git/
ssh: Could not resolve hostname github.com:hangxie: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could not resolve hostname github.com:hangxie: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
jb: error: failed to install packages: downloading: exit status 128

Things worked with 0.2.0:

$ rm -rf *
$ GO111MODULE="on" go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
$ jb --version
v0.2.0
$ jb init
$ jb install git+ssh://[email protected]:hangxie/kustomize-libsonnet.git@test-branch
Initialized empty Git repository in /home/xiehang/tmp/jb/vendor/.tmp/jsonnetpkg-kustomize-libsonnet-test-branch247024765/.git/
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 11 (delta 0), reused 7 (delta 0), pack-reused 0
Unpacking objects: 100% (11/11), done.
From github.com:hangxie/kustomize-libsonnet
 * branch            test-branch -> FETCH_HEAD
 * [new tag]         test-tag    -> test-tag
 * [new branch]      test-branch -> origin/test-branch
Branch 'test-branch' set up to track remote branch 'test-branch' from 'origin'.
Switched to a new branch 'test-branch'

nil pointer error when refering to local git directory

I'm seeing a nil pointer error trying to run jb install using a file with a dependency pointing to a local filesystem path. It seems this use to work in 1.0, but is not working in 2.0 and master.
Example jsonnetfile.json

{
    "dependencies": [
        {
            "name": "prometheus-operator",
            "source": {
                "git": {
                    "remote": "../../",
                    "subdir": "jsonnet/prometheus-operator"
                }
            },
            "version": "."
        }
    ]
}

Output:

cd scripts/generate; /home/pgier/go/bin/jb install;                                                                                                           
panic: runtime error: invalid memory address or nil pointer dereference                                                                                       
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5e21a1]                                                                                        
                                                                               
goroutine 1 [running]:                                                                                                                                        
github.com/jsonnet-bundler/jsonnet-bundler/spec/deps.(*Git).UnmarshalJSON(0xc0000f0280, 0xc000112afc, 0x77, 0x2b4, 0x7ffbb328f220, 0xc0000f0280)
        /home/pgier/go/src/github.com/jsonnet-bundler/jsonnet-bundler/spec/deps/git.go:71 +0xb1     
encoding/json.(*decodeState).object(0xc0000fa370, 0x7f6300, 0xc00012c000, 0x196, 0xc0000fa398, 0x7b)

Make updating master packages easier

Say I installed github.com/whatever@master, or github.com/whatever@branch and wanted jb to pull in the latest changes of that branch, a simple jb install github.com/whatever@master is not sufficient, because the contents of jsonnetfile.json and the supplied URI match, thus jb does not take action.

The only fix is to use the commit hash once and then use master again, so that jb is forced to install that specific commit.

Instead, jb should recognize that we are trying to update (or have an update flag).

Lock file incorrectly built

Transitive dependencies are not correctly propagated to the lock file, only the original dependency currently ends up in the lock file with its latest version.

how to achieve .jbignore

hello - as a fan & user, wondering if this functionality exists:
in some cases, having certain files omitted from the bundle is desirable. An ignore file -a la .gitignore seems useful & general purpose enough. If this is doable today, i'm listening!
If it should be a feature request, let us know!

panic when executing `jb install`

I used jb from master branch (commit c734431) and got following error:

/home/paulfantom/.go/bin/jb install
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x582ad3]

goroutine 1 [running]:
github.com/jsonnet-bundler/jsonnet-bundler/pkg.Install(0x6a6a20, 0xc0000ac018, 0x101, 0xc0000ae300, 0x15, 0xc00010a000, 0x9, 0x9, 0x65b705, 0x6, ...)
        /home/paulfantom/.go/src/github.com/jsonnet-bundler/jsonnet-bundler/pkg/packages.go:58 +0x353
main.installCommand(0x65b705, 0x6, 0x0, 0x0, 0x0, 0x7)
        /home/paulfantom/.go/src/github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb/main.go:295 +0xa5e
main.Main(0xc00008e058)
        /home/paulfantom/.go/src/github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb/main.go:93 +0x724
main.main()
        /home/paulfantom/.go/src/github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb/main.go:61 +0x22
make: *** [Makefile:35: vendor] Error 2

It doesn't happen when using release binary.

jb update on an existing jsonnetfile.json (but created with master) fails

I believe the upgrade path of the new jb is broken for old (v0.2.0) jb users with operating on an inited jsonnetfile.json from a master jb.

To reproduce: install jb at latest (v0.2.0):

GO111MODULE="on" go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

clone tanka (https://github.com/grafana/tanka)

cd examples/prom-graphana

david$ jb update
jb: error: failed to install packages: downloading: failed to create tmp dir: stat vendor/.tmp: no such file or directory

works when I update jb to master.

schemaVersion

It seems reasonable to add an ever increasing uint number to jsonnetfile.json, that can be increased to mark breaking changes to the format of that file, so that older jb versions can request the user to update

Make a release

I'm doing a training and it would be super useful to have ready made binaries for those that don't have go installed.

Latest master from #44 breaks jsonnet-bundler

I haven't investigated very far yet, but the current master 7fc7c31 breaks jsonnet-bundler.

previous commit 766a0c7 works.

jb install produces an empty vendor directory.

As for jb update:

$ jb update
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200

The archive is downloaded (to .tmp), then wiped, and the vendor dir stays empty.

Will try to dig deeper in the next few days.

Unable to install packages - pathspec 'master' did not match any file(s) known to git

On a CentOS 7 machine with the most recent version of jb installed through 'go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb', with git version 2.16.5

Trying to install kube-prometheus in this case, via

#jb install github.com/coreos/prometheus-operator/contrib/kube-prometheus/jsonnet/kube-prometheus

I get

 jb install github.com/coreos/prometheus-operator/contrib/kube-prometheus/jsonnet/kube-prometheus
GET https://github.com/coreos/prometheus-operator/archive/master.tar.gz 200
archive install failed: etag value "W/"1b6e9e79ab596d977e77fd7ca473b19cd12a1c594f6d14360e136666d32e29d6"" does not look like a SHA1
retrying with git...
Initialized empty Git repository in /root/code/my-kube-prom/vendor/.tmp/jsonnetpkg-kube-prometheus-master316687774/.git/
remote: Enumerating objects: 3075, done.
remote: Counting objects: 100% (3075/3075), done.
remote: Compressing objects: 100% (2379/2379), done.
remote: Total 3075 (delta 846), reused 1528 (delta 495), pack-reused 0
Receiving objects: 100% (3075/3075), 7.32 MiB | 13.21 MiB/s, done.
Resolving deltas: 100% (846/846), done.
From https://github.com/coreos/prometheus-operator
 * branch            master     -> FETCH_HEAD
error: pathspec 'master' did not match any file(s) known to git.
jb: error: failed to install: failed to install package: exit status 1

Same error with every other bundle I've tried installing, where the pathspec did not match any files known to git.

Is there something I'm doing wrong, in this case?

feature: allow cloning operations to be parallel

Git clones of many repositories is a good candidate for parallelization; they are distinct and don't need inter-repo synchronization.

Using parallel git clones would enhance download performance.

Unable to install bare files

jb installed from

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

# find .
.
./environments
./environments/default
./environments/default/spec.json
./environments/default/main.jsonnet
./jsonnetfile.json
./lib
./vendor
# cat jsonnetfile.json
{}
# jb install github.com/ksonnet/ksonnet-lib/ksonnet.beta.3/k.libsonnet
GET https://github.com/ksonnet/ksonnet-lib/archive/master.tar.gz 200
>>> Installed k.libsonnet version master

# jb install github.com/ksonnet/ksonnet-lib/ksonnet.beta.3/k8s.libsonnet
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
>>> Installed k.libsonnet version 0d2f82676817bbf9e4acf6495b2090205f323b9f
GET https://github.com/ksonnet/ksonnet-lib/archive/master.tar.gz 200
>>> Installed k8s.libsonnet version master

# jb install github.com/grafana/loki/production/ksonnet/promtail
GET https://github.com/grafana/loki/archive/master.tar.gz 200
>>> Installed promtail version master
GET https://github.com/grafana/jsonnet-libs/archive/master.tar.gz 200
>>> Installed ksonnet-util version master
# cat environments/default/main.jsonnet
local promtail = import 'promtail/promtail.libsonnet';

promtail + {
  _config+:: {
    namespace: 'loki',

    promtail_config+: {
      clients: [
        {
          scheme:: 'https',
          hostname:: 'logs-us-west1.grafana.net',
          username:: 'user-id',
          password:: 'password',
          external_labels: {},
        }
      ],
      container_root_path: '/var/lib/docker',
    },
  },
}
# tk show environments/default
Evaluating jsonnet: RUNTIME ERROR: couldn't open import "promtail/promtail.libsonnet": no match locally or in the Jsonnet library paths
	main.jsonnet:1:18-54	thunk <promtail> from <$>
	main.jsonnet:3:1-9	$
	During evaluation	

# find .
.
./environments
./environments/default
./environments/default/spec.json
./environments/default/main.jsonnet
./jsonnetfile.lock.json
./jsonnetfile.json
./lib
./vendor
./vendor/.tmp
# cat jsonnetfile.json
{
    "dependencies": [
        {
            "name": "k.libsonnet",
            "source": {
                "git": {
                    "remote": "https://github.com/ksonnet/ksonnet-lib",
                    "subdir": "ksonnet.beta.3/k.libsonnet"
                }
            },
            "version": "master"
        }
    ]
}
# cat jsonnetfile.lock.json
{
    "dependencies": [
        {
            "name": "k.libsonnet",
            "source": {
                "git": {
                    "remote": "https://github.com/ksonnet/ksonnet-lib",
                    "subdir": "ksonnet.beta.3/k.libsonnet"
                }
            },
            "version": "0d2f82676817bbf9e4acf6495b2090205f323b9f"
        }
    ]
}

Creating a new release?

Would it be possible to have a new release?

There's quite a few changes since v0.1.0 and pre-built binaries are really nice :)

Many thanks

Don't change/modify jsonnetfile.json jsonnetfile.lock.json if there is nothing to do

When running jb install on a given project multiple times with no changes, both the change as well as modify file timestamp of jsonnetfile.json and jsonnetfile.lock.json are changed. The content as well as meta-data of these files stay the same.

How to reproduce:

stat jsonnetfile.json jsonnetfile.lock.json
jb install
stat jsonnetfile.json jsonnetfile.lock.json

Why would this be needed:

GNU Make uses timestamps of a recipes prerequisites in order to determine whether to rebuild the given target or not. Given jsonnetfile.json and jsonnetfile.lock.json as prerequisites and jb install as the recipe, the target will always be rebuild if jsonnetfile.json and jsonnetfile.lock.json change on every execution.

Tests are not being run currently

The current makefile uses go test -i $(PKGS) which disables the running of tests.

From go help test:

...
        -i
            Install packages that are dependencies of the test.
            Do not run the test.
...

Fail on installing plain files

Users try to install files directly from jb, but this is not supported.

jb should (one of):

  • support the use-case.
  • error out with an informative message.

Furthermore, in an invesitgation, I found that some errors are shadowed in the unarchiving code path. I have a branch (later to be PRed) that enhances the error reporting, will submit later.

jb fails to install dependency after PR #56

As far as I can tell, #56 broke jb, at least with some dependencies like https://github.com/ksonnet/ksonnet-lib/.

On the latest master:

~/p/poetry czibolski ❯❯❯ jb install github.com/ksonnet/ksonnet-lib/ksonnet.beta.4/k.libsonnet
Cloning into 'vendor/.tmp/jsonnetpkg-k.libsonnet-master812140101'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 2505 (delta 0), reused 3 (delta 0), pack-reused 2500
Receiving objects: 100% (2505/2505), 2.48 MiB | 5.75 MiB/s, done.
Resolving deltas: 100% (933/933), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
>>> Installed k.libsonnet version master
jb: error: failed to install: stat vendor/k.libsonnet/jsonnetfile.lock.json: not a directory

I mentioned this to @brancz in the k8s slack when I hit the issue and I looked at recent commits touching the area of code producing the error I see and checked out the commit before this (7fc7c31) and the issue is gone.

Running the same jb install as above using my locally build jb from commit 7fc7c31:

~/p/poetry czibolski ❯❯❯ /Users/czibolski/projects/jsonnet-bundler/_output/darwin/amd64/jb install github.com/ksonnet/ksonnet-lib/ksonnet.beta.4/k8s.libsonnet                                                   ⏎
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
CLEAN vendor/k.libsonnet

Originally posted by @chancez in #56 (comment)

git output is verbose about detached head

The current implementation checkouts a specific version which results in a detached HEAD git mode. Git is very verbose about it.

This is an implementation detail and IMHO should not be output when issuing jb install.

There is a simple solution with a PR coming up.

Support other hosts than GitHub for https

The introduction of the "absolute imports" feature broke the ability to use a generic (non-github) git repo as a package URI.

Given that we have private Gitlab repos that house dependencies for our projects, this is a blocker. :-)

I'll have a PR up for it shortly.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.