Git Product home page Git Product logo

jill.py's Introduction

JILL.py

The enhanced Python fork of JILL -- Julia Installer for Linux (and every other platform) -- Light

py version version Actions Status codecov OSCS release-date 中文README

Features

  • download Julia releases from the nearest mirror server
  • support all platforms and architectures
  • manage multiple julia releases
  • easy-to-use CLI tool

asciicast

Install JILL

For the first time users of jill, you will need to install it using pip: pip install jill --user -U. Also use this to upgrade JILL version.

Python >= 3.6 is required. For base docker images, you also need to make sure wget and gnupg are installed.

Installing Julias

When you type jill install, it does the following things:

  1. query the latest version
  2. download, verify, and install julia
  3. make symlinks, e.g., julia, julia-1, julia-1.6

For common Julia users:

  • Get the latest stable release: jill install
  • Get the latest 1.y.z release: jill install 1
  • Get the latest 1.6.z release: jill install 1.6
  • Get the specific version: jill install 1.6.2, jill install 1.7.0-beta3
  • Get the latest release (including unstable ones): jill install --unstable

Note that for Julia 1.10, you'll have to install it with jill install '"1.10"' because of the python-fire limit.

For Julia developers and maintainers:

  • Get the nightly builds: jill install latest. This gives you julia-latest.
  • Checkout CI build artifacts of specific commit in the Julia Repository: jill install 1.8.0+cc4be25c (<major>.<minor>.<patch>+<build> with at least the first 7 characters of the hash). This gives you julia-dev.

Some flags that can be useful:

  • No confirmation before installation: jill install --confirm
  • Download from Official source: jill install --upstream Official
  • Keep downloaded contents after installation: jill install --keep_downloads
  • Force a reinstallation: jill install --reinstall

The symlinks

To start Julia, you can use predefined JILL symlinks such as julia. jill install uses the following rule makes sure that you're always using the latest stable release.

Stable releases:

  • julia points to the latest Julia release.
  • julia-1 points to the latest 1.y.z Julia release.
  • julia-1.6 points to the latest 1.6.z Julia release.

For unstable releases such as 1.7.0-beta3, installing it via jill install 1.7 --unstable or jill install 1.7.0-beta3 will only give you julia-1.7; it won't make symlinks for julia or julia-1.

To dance on edge:

  • julia-latest points to the nightly build from jill install latest
  • julia-dev points to the julia CI build artifacts from, for example, jill install 1.8.0+cc4be25c.

List symlinks and their target versions

jill list [version] gives you every symlinks and their target Julia versions.

list

Change symlink target

For non-windows system, you are free to use ln command to change the symlink targets. For Windows it uses an entry .cmd file for this so you'll need to copy them. In the meantime, jill switch provides a simple and unified way to do this:

  • jill switch 1.6: let julia points to the latest julia 1.6.z release.
  • jill switch <path/to/my/own/julia/executable>: let julia points to custom executables.
  • jill switch 1.6 --target julia-1: let julia-1 points to the latest julia 1.6.z release.

About downloading upstreams

By default, JILL tries to be smart and will download contents from the nearest upstream. You can get the information of all upstreams via jill upstream. Here's what I get in my laptop, I live in China so the official upstreams aren't so accessible for me :(

upstream

To temporarily disable this feature, you can use flag --upstream <server_name>. For instance, jill install --upstream Official will faithfully download from the official julialang s3 bucket.

To permanently disable this feature, you can set environment variable JILL_UPSTREAM.

Note that flag is of higher priority than environment variable. For example, if JILL_UPSTREAM is set to mirror server "TUNA", you can still download from the official source via jill install --upstream Official.

About installation and symlink directories

Here's the default JILL installation and symlink directories:

system installation directory symlink directory
macOS /Applications ~/.local/bin
Linux/FreeBSD ~/packages/julias ~/.local/bin
Windows ~\AppData\Local\julias ~\AppData\Local\julias\bin

For example, on Linux jill install 1.6.2 will have a julia folder in ~/packages/julias/julia-1.6 and symlinks julia/julia-1/julia-1.6 created in ~/.local/bin.

Particularly, if you're using jill as root user, you will do a system-wide installation:

  • Installation directory will be /opt/julias for Linux/FreeBSD.
  • Symlink directory will be /usr/local/bin for Linux/FreeBSD/macOS.

To change the default JILL installation and symlink directories, you can set environment variables JILL_INSTALL_DIR and JILL_SYMLINK_DIR.

(Deprecated) jill install also provides two flag --install_dir <dirpath> and --symlink_dir <dirpath>, they have higher priority than the environment variables JILL_INSTALL_DIR and JILL_SYMLINK_DIR.

JILL environment variables

jill is made as a convenient tool and it can sometimes be annoying passing flags to it. There are some predefined environment variables that you can use to set the default values:

  • Specify a default downloading upstream JILL_UPSTREAM: --upstream
  • Override default symlink directory JILL_SYMLINK_DIR: --symlink_dir
  • Override default installation directory JILL_INSTALL_DIR: --install_dir

The flag version has higher priority than the environment variable version.


Advanced: Example with cron

If you're tired of seeing (xx days old master) in your nightly build version, then jill can make your nightly build always the latest version using cron:

# /etc/cron.d/jill
PATH=/usr/local/bin:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# install a fresh nightly build every day
* 0 * * * root jill install latest --confirm --upstream Official

Advanced: Registering a new public releases upstream

If it's an public mirror and you want to share it worldwide to other users of JILL. You can add an entry to the public registry, make a PR, then I will tag a new release for that.

Please check the sources.json format for more detailed information on the format.

Advanced: Specifying custom (private) downloading upstream

To add new private upstream, you can create a file ~/.config/jill/sources.json (fow Windows it is ~/AppData/Local/julias/sources.json) and add your own upstream configuration just like the JILL sources.json does. Once this is done JILL will recognize this new upstream entry.

Please check the sources.json format for more detailed information on the format.

Advanced: make a Julia release mirror

There are two ways to do so:

  • use aws s3 sync, this should be the easiest way to do so I highly recommend this.
  • (Deprecated) use jill mirror command with mirror config example. I didn't know about the aws s3 sync stuff when I implemented this.

The Julia release mirror does not contain Julia package contents, to mirror all the Julia packages and artifacts (which requires >1.5Tb storage), you can use StorageMirrorServer.jl.

Advanced: The Python API

jill.py also provides a set of Python API:

from jill.install import install_julia
from jill.download import download_package

# equivalent to `jill install --confirm`
install_julia(confirm=True)
# equivalent to `jill download`
download_package()

You can read its docstring (e.g., ?install_julia) for more information.

FAQs

Why you should use JILL?

Distro package managers (e.g., apt, pac) is likely to provide a broken Julia with incorrect binary dependencies (e.g., LLVM ) versions. Hence it's recommended to download and extract the Julia binary provided in Julia Downloads. jill.py doesn't do anything magical, but just makes such operation even stupid.

Why I make the python fork of JILL?

At first I found myself needing a simple tool to download and install Julia on my macbook and servers in our lab, I made my own shell scripts and I'd like to share it with others. Then I found the jill.sh project, Abel knows a lot shell so I decide to contribute my macOS Julia installer to jill.sh.

There are three main reasons for why I decided to start my Python fork:

  • I live in China. Downloading resources from GitHub and AWS s3 buckets is a painful experience. Thus I want to support downloading from mirror servers. Adding mirror server support to jill.sh is quite complicated and can easily become a maintenance nightmare.
  • I want to make a cross platform installer that everyone can use, not just Linux/macOS users. Shell scripts doesn't allow this as far as I can tell. In contrast, Python allows this.
  • Most importantly, back to when I start this project, I knew very little shell, I knew nothing about C/C++/Rust/Go and whatever you think a good solution is. I happen to knew a few Python.

For some "obvious" reason, Julia People don't like Python and I understand it. (I also don't like Python after being advanced Julia user for more than 3 years) But to be honest, revisiting this project, I find using Python is one of the best-made decision during the entire project. Here is the reason: no matter how you enjoy Julia (or C++, Rust), Python is one of the best successful programming language for sever maintenance purpose. Users can easily found tons of "how-to" solutions about Python and it's easy to write, deploy, and ship Python codes to the world via PyPI.

And again, I live in China so I want to rely on services that are easily accessible in China, PyPI is, GitHub and AWS S3 bucket aren't. A recent Julia installer project juliaup written in Rust solves the Python dependency problem very well, but the tradeoff is that juliaup needs its own distributing system (currently GitHub and S3 bucket) to make sure it can be reliably downloaded to user machine. And for this it just won't be as good as PyPI in the foreseeable future.

Is it safe to use jill.py?

Yes, jill.py use GPG to check every tarballs after downloading. Also, *.dmg/*.pkg for macOS and .exe for Windows are already signed.

What's the difference between jill.sh and jill.py

jill.sh is a shell script that works quite well on Linux x86/x64 machines. jill.py is an enhanced python package that focus on Julia installation and version management, and brings a unified user experience on all platforms.

Why julia fails to start

The symlink julia are stored in JILL predefined symlinks dir thus you have to make sure this folder is in PATH. Search "how to add folder to PATH on xxx system" you will get a lot of solutions.

How do I use multiple patches releases (e.g., 1.6.1 and 1.6.2)

Generally, you should not care about patch version differences so jill.py make it explicitly that only one of 1.6.x can exist. If you insist to have multiple patch versions, you could use jill install --install_dir <some_other_folder> to install Julia in other folder, and then manually make a symlink back. As I just said, in most cases, common users should not care about this patch version difference and should just use the latest patch release.

How to only download contents without installation?

Use jill download [version] [--sys <system>] [--arch <arch>]. Check jill download --help for more details.

Linux with musl libc

For Julia (>= 1.5.0) in Linux with musl libc, you can just do jill install and it gives you the right Julia binary. To download the musl libc binary using jill download, you will need to pass --sys musl flag.

MacOS with Apple silicon (M1)

Yes it's supported. Because macOS ARM version is still of tier-3 support, jill.py will by default install the x86_64 version. If you want to use the ARM version, you can install it via jill install --preferred-arch arm64.

CERTIFICATE_VERIFY_FAILED error

If you're confident, try jill install --bypass-ssl.

Skip symbolic links generation

If for some reason you prefer to download julia without generating symbolic links jill install --skip-symlinks

jill.py's People

Contributors

carstenbauer avatar dependabot[bot] avatar imgbot[bot] avatar johnnychen94 avatar kerim371 avatar kronosthelate avatar skyzh avatar yemeng77 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jill.py's Issues

Does not work on Raspberry Pi Zero W

Getting this error:

pi@raspberrypi:~ $ jill install 1.6.2
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia 1.6.2 for linux-armv6l into /home/pi/packages/julias
  2) make symlinks in /home/pi/.local/bin
You may need to manually add /home/pi/.local/bin to PATH
Continue installation? [Y/n] y
----- Download Julia -----
downloading Julia release for 1.6.2-linux-armv6l
failed to find 1.6.2-linux-armv6l in available upstreams. Please try it later.
False

Presumably because there is no armv6l version in https://julialang-s3.julialang.org/bin/versions.json

How to fix this?

jill fails on MacOS

somehow this fails on MacOS

~ jill download julia
Traceback (most recent call last):
  File "/Users/roger/miniconda3/bin/jill", line 11, in <module>
    load_entry_point('jill==0.6.5', 'console_scripts', 'jill')()
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/__main__.py", line 21, in main
    }, name="jill")
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/download.py", line 112, in download_package
    version = latest_version(version, system, architecture)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/utils/version_utils.py", line 242, in latest_version
    return f_list[idx](version, system, architecture, **kwargs)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/utils/version_utils.py", line 188, in latest_minor_version
    **kwargs)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/utils/version_utils.py", line 134, in _latest_version
    last_version = Version(version)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/utils/version_utils.py", line 48, in __init__
    self.major_version = f_major_version(version_string)
  File "/Users/roger/miniconda3/lib/python3.7/site-packages/jill-0.6.5-py3.7.egg/jill/utils/filters.py", line 131, in __call__
    raise ValueError(msg)
ValueError: validation fails:
  - args: ('julia.0.0',)
  - kwargs: {}

appropriately handle registry source with no versions section

Old jill versions don't require versions section for registry:

# ~/.config/jill/sources.json
{
    "upstream": {
        "LFLab": {
            "name": "LFLab@Math",
            "urls": [
                "https://mirrors.lflab.cn/julia/releases/$vminor_version/$filename"
            ],
            "latest_urls": [
                "https://mirrors.lflab.cn/julia/releases/latest/$latest_filename"
            ]
        }
    }
}

and it outputs:

jill install 1 --upstream LFLab --confirm
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light
querying release information from None
Traceback (most recent call last):
  File "/usr/local/bin/jill", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/jill/__main__.py", line 19, in main
    }, name="jill")
  File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/jill/install.py", line 387, in install_julia
    raise(ValueError(msg))
ValueError: wrong version(>= 0.6.0) argument: 1
Example: `jill install 1`

This bug should only exist in v0.9.0 and v0.9.1

Can not install nightly version with unofficial mirrors

It seems that currently most julia release mirrors do not host nightly builds. As a result, jill install nightly errors if I am not using the official mirror.

PS D:\code> jill install nightly --upstream SJTUG
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia latest for winnt-x86_64 into C:\Users\melonedo\AppData\Local\julias
  2) make symlinks in C:\Users\melonedo\AppData\Local\julias\bin
You may need to manually add C:\Users\melonedo\AppData\Local\julias\bin to PATH
Continue installation? [Y/n]
----- Download Julia -----
downloading Julia release for latest-winnt-x86_64
failed to find latest-winnt-x86_64 in available upstream. Please try it later.
False

Option to copy default environment to new Julia version

It might be nice to give the user the (command-line) option to automatically copy the Project.toml of the default environment, say, ~/.julia/environments/v1.5/Project.toml to ~/.julia/environments/v1.6/ when updating (i.e. installing a newer version of) Julia.

add freebsd CI

The installation script for Linux should work properly for Freebsd as well. But since I don't have access to FreeBSD system, unless we add a CI for it, I can say it for sure.

Issue when upgrading julia

I am trying to use jill to upgrade to julia 1.5.1 using jill install --upgrade and am getting an error. The traceback is

Traceback (most recent call last):
 File "/home/ptiede/.local/bin/jill", line 8, in <module>
   sys.exit(main())
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/__main__.py", line 15, in main
   fire.Fire({
 File "/home/ptiede/.local/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
   component_trace = _Fire(component, args, parsed_flag_args, context, name)
 File "/home/ptiede/.local/lib/python3.8/site-packages/fire/core.py", line 463, in _Fire
   component, remaining_args = _CallAndUpdateTrace(
 File "/home/ptiede/.local/lib/python3.8/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
   component = fn(*varargs, **kwargs)
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 372, in install_julia
   installer(package_path, install_dir, symlink_dir, version, upgrade)
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 201, in install_julia_linux
   copy_root_project(version)
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 160, in copy_root_project
   old_ver = last_julia_version(version)
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 77, in last_julia_version
   proj_versions = sorted(filter(lambda ver: sort_key(ver) < version,
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 77, in <lambda>
   proj_versions = sorted(filter(lambda ver: sort_key(ver) < version,
 File "/home/ptiede/.local/lib/python3.8/site-packages/jill/install.py", line 72, in sort_key
   return float(ver.lstrip("v"))
ValueError: could not convert string to float: '1.5.bak'

It looks like the version isn't formatted as you expect which is causing the conversion from string to float to fail.

I believe I am using version 0.7.0 of jill and python 3.8

jill install 1.6.5 : failed to download (caused by wget SSL: CERTIFICATE_VERIFY_FAILED )

python version : Python 3.8.5
os : ms window 10 x64

%windir%\System32\cmd.exe "/K" d:\Miniconda3_py38\Scripts\activate.bat d:\Miniconda3_py38

execute : pip install jill

execute : jill install 1.6.5
get error :

[1mJILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light [0m

jill will:

  1. install Julia 1.6.5 for winnt-x86_64 into �[4mC:\Users\enmonster\AppData\Local\julias�[0m
  2. make symlinks in �[4mC:\Users\enmonster\AppData\Local\julias\bin�[0m
    You may need to manually add �[4mC:\Users\enmonster\AppData\Local\julias\bin�[0m to PATH
    Continue installation? [Y/n] y
    �[1m----- Download Julia -----�[0m
    downloading Julia release for 1.6.5-winnt-x86_64
    downloading from https://mirrors.sjtug.sjtu.edu.cn/julia-releases/bin/winnt/x64/1.6/julia-1.6.5-win64.exe
    �[91mfailed to download julia-1.6.5-win64.exe�[0m
    Traceback (most recent call last):
    File "d:\miniconda3_py38\lib\urllib\request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
    File "d:\miniconda3_py38\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
    File "d:\miniconda3_py38\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
    File "d:\miniconda3_py38\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    File "d:\miniconda3_py38\lib\http\client.py", line 1010, in _send_output
    self.send(msg)
    File "d:\miniconda3_py38\lib\http\client.py", line 950, in send
    self.connect()
    File "d:\miniconda3_py38\lib\http\client.py", line 1424, in connect
    self.sock = self._context.wrap_socket(self.sock,
    File "d:\miniconda3_py38\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
    File "d:\miniconda3_py38\lib\ssl.py", line 1040, in _create
    self.do_handshake()
    File "d:\miniconda3_py38\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Feature request: probing installed versions

I'd like to be able to do something like query for the path to the installation of the most recent stable version. For example, under linux, this might return

/home/username/packages/julias/julia-1.6/

I looked through the jill source and it seems the functions to do this are not immediately available. It may be out of scope for the jill project. But, I am thinking of using jill in conjunction with pyjulia and in particular this: julia_project. With a little more functionality in jill I could do this.

  1. Make a python package that uses julia_project (or a similar tool, I think there may be others) to manage the Julia dependency. The python package is, say, mypackage.
  2. Provide a function mypackage.install_julia() (Perhaps make it automatic, but I don't favor that.) that uses julia_project and jill to download and install julia. New feature --> Use jill to find the installation directory. Then julia_project uses this path to configure pyjulia.

There are two advantages. First, the user does not need ensure that the symlinked directory is in their PATH. It requires no user intervention outside of Python. Second, some people, like myself use a Julia with a custom system image as their default Julia. But, for mypackage I can't use this, I need a Julia with this stock image. In fact, via, julia_project, mypackage will manage it's own custom system image.

`jill install` is broken in Alpine Linux

Alpine uses musl as its C runtime library, and by default jill selects glibc version of Julia when installing, which leads to a broken installation of Julia.

/ # ldd /usr/local/bin/julia
        /lib64/ld-linux-x86-64.so.2 (0x7f2fb4b00000)
Error loading shared library libjulia.so.1: No such file or directory (needed by /usr/local/bin/julia)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f2fb4b00000)
        librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f2fb4b00000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f2fb4b00000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f2fb4b00000)
Error relocating /usr/local/bin/julia: ios_write: symbol not found
Error relocating /usr/local/bin/julia: jl_enter_handler: symbol not found
Error relocating /usr/local/bin/julia: jl_apply_generic: symbol not found
Error relocating /usr/local/bin/julia: jl_static_show: symbol not found
Error relocating /usr/local/bin/julia: ios_readline: symbol not found
Error relocating /usr/local/bin/julia: jl_symbol: symbol not found
Error relocating /usr/local/bin/julia: jl_get_ptls_states: symbol not found
Error relocating /usr/local/bin/julia: jl_call2: symbol not found
Error relocating /usr/local/bin/julia: jl_exception_occurred: symbol not found
Error relocating /usr/local/bin/julia: jl_lisp_prompt: symbol not found
Error relocating /usr/local/bin/julia: jl_current_exception: symbol not found
Error relocating /usr/local/bin/julia: jl_stderr_obj: symbol not found
...

And it seems that -sys musl not working:

/ # jill install --sys musl
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia latest stable release for linux-x86_64 into /opt/julias
  2) make symlinks in /usr/local/bin
You may need to manually add /usr/local/bin to PATH
Continue installation? [Y/n] y
query the latest  version, it may take seconds...
----- Download Julia -----
downloading Julia release for 1.5.1-linux-x86_64
julia-1.5.1-linux-x86_64.tar.gz already exists, skip downloading
----- Install Julia -----
install Julia to /opt/julias/julia-1.5
make new symlink /usr/local/bin/julia-1
make new symlink /usr/local/bin/julia-1.5
make new symlink /usr/local/bin/julia
----- Post Installation -----
remove downloaded files...
remove /julia-1.5.1-linux-x86_64.tar.gz
remove /julia-1.5.1-linux-x86_64.tar.gz.asc
Done!
ERROR: Could not consume arg: --sys
Usage: jill install -

For detailed information on this command, run:
  jill install - --help

A (naive) solution is to choose musl version of Julia when it detects its running on Alpine, or provide an option for user to select.

How to Reproduce:

  1. Use Alpine Docker image: sudo docker run -it alpine sh
  2. Install Python3, Pip3 and Gnupg: apk add python3 py3-pip gnupg
  3. Install Jill: pip install jill
  4. Install Julia: jill install, and the installed Julia is not runnable.

Error installing Julia 1.7.0-beta3 on an M1 Mac

On an Apple Mac Mini with M1 CPU, I tried to use Jill to install Julia 1.7.0-beta3. This is with Jill 0.9.5, installed via pip3, using Python 3.9 as provided by homebrew. This is what happened:

$ python3 Library/Python/3.9/bin/jill download 1.7.0-beta3
downloading Julia release for 1.7.0-beta3-mac-arm64
failed to find 1.7.0-beta3-mac-arm64 in available upstreams. Please try it later.

The actual archive on the website is https://julialang-s3.julialang.org/bin/mac/aarch64/1.7/julia-1.7.0-beta3-macaarch64.dmg

I wonder if the mismatch arm64 versus aarch64 (resp. mac-arm64 versus macaarch64) might be related?

add `a+x` to system-wide installation

/opt/julias/julia-1.3/bin/julia
fish: The file “/opt/julias/julia-1.3/bin/julia” is not executable by this user
root@mathai2:/opt/julias# ls -l
total 16
drwx------ 7 root root 4096 Feb  2 18:04 julia-1.0
drwx------ 7 root root 4096 Feb  2 18:05 julia-1.1
drwx------ 7 root root 4096 Feb  2 18:05 julia-1.2
drwx------ 8 root root 4096 Feb  2 18:05 julia-1.3

Options to avoid bashrc editing and adjust symlink creation

Feature requests

  • An option to skip the modification of .bashrc file would be useful. I sync my .bashrc between different machines, and keep a .bashrc_local file (called from .bashrc) for local changes like this. So every time after running jill, I have to undo the changes to .bashrc. If there's a --skip-bashrc option, this could be avoided.
  • When installing unstable versions, I'd prefer that the julia symlink isn't changed. For eg., currently I have Julia 1.6 installed, and now installed Julia 1.7-beta2. I'd prefer if the julia and julia-1 symlinks keeps pointing to the stable Julia 1.6 version, and only julia-1.7 was created to point to Julia 1.7-beta2. An option so that only this full versioned link is created would be great, but if that sounds complicated, just an option to skip creating any symlinks at all would be fine.

Thanks for this really useful tool!

Improve printing output of `jill update`

Currently, you only get

➜  ~/repos/jill.py git:(master) jill update
check new Julia release info, it will take a while

Would be great if it would also tell the user whether it has found new versions and perhaps even what the latest stable version is.

impropriate cleanup in gpg checks

This happens occasionally in a minimal docker setup:

# base docker image nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
apt-get install -y -qq python3-pip python3-wheel python3-setuptools gnupg wget --no-install-recommends
pip3 install jill --quiet
 $ jill install $julia_version --upstream LFLab --confirm
 JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light
 query the latest 1 version, it may take seconds...
 ----- Download Julia -----
 downloading Julia release for 1.5.3-linux-x86_64
 downloading from https://mirrors.lflab.cn/julia/releases/v1.5/julia-1.5.3-linux-x86_64.tar.gz
 finished downloading julia-1.5.3-linux-x86_64.tar.gz
 downloading from https://mirrors.lflab.cn/julia/releases/v1.5/julia-1.5.3-linux-x86_64.tar.gz.asc
 finished downloading julia-1.5.3-linux-x86_64.tar.gz.asc
 Traceback (most recent call last):
   File "/usr/local/bin/jill", line 8, in <module>
     sys.exit(main())
   File "/usr/local/lib/python3.6/dist-packages/jill/__main__.py", line 21, in main
     }, name="jill")
   File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 138, in Fire
     component_trace = _Fire(component, args, parsed_flag_args, context, name)
   File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 468, in _Fire
     target=component.__name__)
   File "/usr/local/lib/python3.6/dist-packages/fire/core.py", line 672, in _CallAndUpdateTrace
     component = fn(*varargs, **kwargs)
   File "/usr/local/lib/python3.6/dist-packages/jill/install.py", line 371, in install_julia
     overwrite=overwrite)
   File "/usr/local/lib/python3.6/dist-packages/jill/download.py", line 195, in download_package
     if not verify_gpg(package_path, gpg_signature_path):
   File "/usr/local/lib/python3.6/dist-packages/jill/utils/gpg_utils.py", line 27, in verify_gpg
     return bool(_verify_gpg(gpg, datafile, signature_file))
   File "/usr/lib/python3.6/tempfile.py", line 948, in __exit__
     self.cleanup()
   File "/usr/lib/python3.6/tempfile.py", line 952, in cleanup
     _rmtree(self.name)
   File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
     _rmtree_safe_fd(fd, path, onerror)
   File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
     onerror(os.unlink, fullname, sys.exc_info())
   File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
     os.unlink(name, dir_fd=topfd)
 FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.ssh'

check if nightly is outdated

If registered mirror stops those nightly builds, there will be cases that an old version is installed. This won't be a big issue since we could always get updated nightly builds from upstream Official.

throw a warning when required version isn't available

It would be nicer to throw a warning when version search fails.

$ jill install 99.99
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia 99.99 for macos-x86_64 into /Applications
  2) make symlinks in /Users/jc/.local/bin
You may need to manually add /Users/jc/.local/bin to PATH
Continue installation? [Y/n]
query the latest 99.99 version, it may take seconds...
----- Download Julia -----
query the latest  version, it may take seconds...

The main reason I did this is that Arm platforms might not have a binary release yet(or never), so when this version/arch/os is not available, go to the fallback solution, that

  • installs the latest stable version since the specified version
  • if not available, the latest stable version
$ jill download 1.4 --arch ARMv7 --sys linux
query the latest 1.4 version, it may take seconds...
downloading Julia release for 1.4.1-linux-ARMv7

An exception is that the fully specified version does not follow this rule, that jill install 1.4.0 for ARMv7 will faithfully download version 1.4.0, which will result an error.

Why /Users/$HOME/.local/bin ?

I have some related questions about the installation of JILL.

First, why use --user in pip install? I haven't done this before. And I have to export /Users/$HOME/.local/bin/ to PATH now. Is it common or unusual? Is it safe if I have some duplicated binaries in /Users/$HOME/.local/bin/ and other paths? (I use conda env.)

Second, similarly jill install makes the symlinks to /Users/$HOME/.local/bin while the official document of Julia uses /usr/local/bin/. What's the difference of these two choices?

Third, 'jill install' add 3 symlinks, julia-1, julia-1.5, and julia to /Users/$HOME/.local/bin. They seems to link to the same Julia. So why do I need 3 versions?

Thanks

add `jill switch <version>` command

a remaining issue from #76 (comment)

add jill switch <version> [--symlink_dir SYMLINK_DIR] command to explicitly change what julia points to. For Linux/MacOS this is basically just cd SYMLINK_DIR && ln -sf julia-1.6 julia

Download only works on second attempt

At first I thought this was an internet connection thing on my end. But I consistently see this across different machines at different places (cities).

I'm running jill install something (latest for example) twice. The first time it complains ("failed to find ..."), the second time, executed immediately afterwards, works just fine.

➜  bauerc@ln-0002 ~  jill install latest
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia latest for linux-x86_64 into /upb/departments/pc2/groups/pc2-mitarbeiter/bauerc/software/julia
  2) make symlinks in /upb/departments/pc2/users/b/bauerc/.local/bin
You may need to manually add /upb/departments/pc2/users/b/bauerc/.local/bin to PATH
Continue installation? [Y/n]
----- Download Julia -----
downloading Julia release for latest-linux-x86_64
failed to find latest-linux-x86_64 in available upstreams. Please try it later.
False

➜  bauerc@ln-0002 ~  jill install latest
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia latest for linux-x86_64 into /upb/departments/pc2/groups/pc2-mitarbeiter/bauerc/software/julia
  2) make symlinks in /upb/departments/pc2/users/b/bauerc/.local/bin
You may need to manually add /upb/departments/pc2/users/b/bauerc/.local/bin to PATH
Continue installation? [Y/n]
----- Download Julia -----
downloading Julia release for latest-linux-x86_64
downloading from https://julialangnightlies-s3.julialang.org/bin/linux/x64/julia-latest-linux64.tar.gz
100% [......................................................................] 121143120 / 121143120
finished downloading julia-latest-linux64.tar.gz
downloading from https://julialangnightlies-s3.julialang.org/bin/linux/x64/julia-latest-linux64.tar.gz.asc
100% [..................................................................................] 866 / 866
finished downloading julia-latest-linux64.tar.gz.asc
verifying latest-linux-x86_64 succeed
----- Install Julia -----
remove previous Julia installation: /upb/departments/pc2/groups/pc2-mitarbeiter/bauerc/software/julia/julia-latest
install Julia to /upb/departments/pc2/groups/pc2-mitarbeiter/bauerc/software/julia/julia-latest
----- Post Installation -----
remove downloaded files...
remove /upb/departments/pc2/users/b/bauerc/julia-latest-linux64.tar.gz
remove /upb/departments/pc2/users/b/bauerc/julia-latest-linux64.tar.gz.asc
Done!

We need to figure out what's going on here.

python 3.8 using jill

(base) ➜  GitHub jill install 1.6
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

jill will:
  1) install Julia 1.6 for linux-x86_64 into /home/redy/packages/julias
  2) make symlinks in /home/redy/.local/bin
You may need to manually add /home/redy/.local/bin to PATH
Continue installation? [Y/n] y
query the latest 1.6 version, it may take seconds...
----- Download Julia -----
query the latest  version, it may take seconds...
downloading Julia release for 1.5.2-linux-x86_64
downloading from https://mirrors.tuna.tsinghua.edu.cn/julia-releases/bin/linux/x64/1.5/julia-1.5.2-linux-x86_64.tar.gz
100% [......................................................................] 105324048 / 105324048
finished downloading julia-1.5.2-linux-x86_64.tar.gz
downloading from https://mirrors.tuna.tsinghua.edu.cn/julia-releases/bin/linux/x64/1.5/julia-1.5.2-linux-x86_64.tar.gz.asc
100% [..................................................................................] 866 / 866
finished downloading julia-1.5.2-linux-x86_64.tar.gz.asc
verifying 1.5.2-linux-x86_64 succeed
----- Install Julia -----
Traceback (most recent call last):
  File "/home/redy/miniconda3/bin/jill", line 8, in <module>
    sys.exit(main())
  File "/home/redy/miniconda3/lib/python3.8/site-packages/jill/__main__.py", line 15, in main
    fire.Fire({
  File "/home/redy/miniconda3/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/redy/miniconda3/lib/python3.8/site-packages/fire/core.py", line 463, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/redy/miniconda3/lib/python3.8/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/redy/miniconda3/lib/python3.8/site-packages/jill/install.py", line 387, in install_julia
    installer(package_path, install_dir, symlink_dir, version, upgrade)
  File "/home/redy/miniconda3/lib/python3.8/site-packages/jill/install.py", line 191, in install_julia_linux
    mver = f_minor_version(version)
  File "/home/redy/miniconda3/lib/python3.8/site-packages/jill/utils/filters.py", line 130, in __call__
    if not self.validate(*args, **kwargs):
  File "/home/redy/miniconda3/lib/python3.8/site-packages/jill/utils/filters.py", line 99, in is_version
    return bool(VERSION_REGEX.match(version))
TypeError: expected string or bytes-like object

issues:

  1. install from tsinghua tuna mirror when I'm not in mainland China..
  2. seems some unhandled case for version.
  3. 1.6 got me 1.5.2

Windows下make_symlinks不正确

非常感谢这一程序方便了安装Julia。但今天发现0.9.1版本在Windows下创建symlinks调用的是mklink导致报错。检查是在make_symlinks()中110、139、168行处应由“windows”改为“winnt”,修改后可以正常安装。应该是修改了current_system()的返回值后没有完全修改对应的位置,不知道还有没有需要修改的。

advertise this package in julialang.org?

Probably I can advertise this package at the bottom of https://julialang.org/downloads/ for those who feel more comfortable in cmd? Looks like many people are still using distro managers (chocho, apt, etc) to install Julia but that’s not the recommended way and I’ve seen many related issues due to this.

One thing that might not fit your concern is it downloading binaries from the nearest upstream, is there a way to redirect statistics to julilang.org in this case? @staticfloat The downloading is implementated via wget:

wget.download(url, temp_outpath)

An "uninstalling" feature might be wanted, too.

No need to download asc files

This is a nice-to-have improvement.

We currently download the associated *.asc files for gpg verification purposes. Given that #58 brings versions.json support, which contains an asc field for all the releases, we don't need to manually download *.asc files from mirror sites.

quiet installation

So far I've seen some packages that uses the jill Python API to automate the Julia installation:

jill.py/jill/install.py

Lines 299 to 306 in 70a40d1

def install_julia(version=None, *,
install_dir=None,
symlink_dir=None,
upgrade=False,
upstream=None,
keep_downloads=False,
confirm=False,
reinstall=False):

It would be friendlier if there's a --quite, -Q keyword to silent all print command, i.e., to remove all the following output:

JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

----- Download Julia -----
downloading Julia release for latest-macos-x86_64
downloading from https://mirrors.lflab.cn/julia/releases/latest/julia-latest-mac64.dmg
100% [........................................................................] 99596927 / 99596927
finished downloading julia-latest-mac64.dmg
----- Install Julia -----
remove previous Julia installation: /Applications/Julia-1.7.app
install Julia to /Applications/Julia-1.7.app
----- Post Installation -----
remove downloaded files...
remove /Users/jc/julia-latest-mac64.dmg
Done!

Integrate with diffeqpy

Is there a way to integrate this with https://github.com/SciML/diffeqpy to give users a function that sets up a Julia installation automatically? Right now we require users go out and install Julia, but it would be better if the package worked more like Conda.jl.

bypass ssl check

Hi,

I've found strange behaviour and writing here in hope you could help me.

My app builds python 3.6.7 from source in Release mode (no matter what my app config is, python is always built in release).
Then I do ./python pip install jill
Then:
/python -m jill install 1.6.1 --confirm --install_dir ~/Documents/d for debug app
/python -m jill install 1.6.1 --confirm --install_dir ~/Documents/r for release app

This command only works for release app configuration (even in both cases python is built as release, I believe pythons are identical) and in debug app I can see the error:

./python jill install 1.6.1 --confirm --install_dir ~/Documents
JILL - Julia Installer 4 Linux (MacOS, Windows and FreeBSD) -- Light

----- Download Julia -----
downloading Julia release for 1.6.1-linux-x86_64
downloading from https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.1-linux-x86_64.tar.gz
failed to download julia-1.6.1-linux-x86_64.tar.gz
False

I've tried different julia versions, even tried sudo ./python ... but it fails anyway...

I understand that information I provide is unsufficient but I'm confused... Do you have ideas why this can possibly happen?

Ubuntu 20.04

Allow installation of release candidates

It doesn't seem to be possible to use jill to download the 1.6 release candidate:

➜  ~/.local/bin jill download 1.6
query the latest 1.6 version, it may take seconds...
version 1.6 is not available, use latest stable release.

Would be great if this were supported. Also, since we already have jill install stable and jill install latest we could also add something like jill install beta or jill install rc to install the latest rc candidate.

(Note that an explicit jill download 1.6.0-rc1 works. So it seems that jill is only checking for stable 1.6.x versions.)

`julia` not recognized in windows bash terminal

I have installed Julia 1.7.1 on Windows 10. Now when using the bash terminal I need to do julia.cmd (with the extension cmd) to run Julia.

This never happened to me before. It does not happen when I use PowerShell or cmd.

Env variable for installation dir

Apart from the --install_dir cmd option it would be nice to be able to specify a fixed installation directory via an environment variable, e.g. JILL_INSTALL_DIR.

jill list report an error on new installed jill

I new installed jill, and not jill install .
than jill list report an error :

(base) PS C:\Users\zsz61> jill list
Traceback (most recent call last):
  File "c:\users\zsz61\miniconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\zsz61\miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\zsz61\Miniconda3\Scripts\jill.exe\__main__.py", line 9, in <module>
  File "c:\users\zsz61\miniconda3\lib\site-packages\jill\__main__.py", line 23, in main
    }, name="jill")
  File "c:\users\zsz61\miniconda3\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "c:\users\zsz61\miniconda3\lib\site-packages\fire\core.py", line 471, in _Fire
    target=component.__name__)
  File "c:\users\zsz61\miniconda3\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "c:\users\zsz61\miniconda3\lib\site-packages\jill\list.py", line 79, in list_julia
    raise(ValueError(f"symlink dir {symlink_dir} doesn't exist!"))
ValueError: symlink dir C:\Users\zsz61\AppData\Local\julias\bin doesn't exist!

having issues querying versions

Things like jill install and jill install 1.5 were hanging for a long long time at the querying versions step.

jill install 1.5.0 worked without issue.

support update flag for non-tier-1 arch

The current update codes don't actually work for the arch that doesn't have Julia release in a strict semantic way. (e.g., We don't have 1.0.0 for armv8)

We can do this by checking new versions using a tier 1 arch instead of specified download/install arch.

Possibility to set `--upstream` default

Currently, as per default, we are testing the latency to mirrors. While being a reasonable default, this leads to a somewhat annoying and unnecessary delay (e.g. I know that my connection to the official upstream is best, why test it all the time?).

I can avoid the latency test by providing --upstream Official. Would be great to have a way (a config?) to set a default upstream.

respect .bash_profile

currently when jill install PATH to .bashrc, but on some machines bash will use .bash_profile, would be nice to let jill detect that

`choco` in readme?

Distro package managers (e.g., apt, pac, chocho) is likely to provide a broken Julia with incorrect binary dependencies (e.g., LLVM ) versions. Hence it's recommended to download and extract the Julia binary provided in Julia Downloads.

I believe chocho should be choco?

Also, assuming choco was meant, to be fair to Chocolatey, it does just use the official Julia .exe binary download. The MD5 checksum is the same. So in that sense alone I would not think that jill.py provides any benefit over installing Julia with choco. IMHO the benefit is that with jill.py you can easily have multiple Julia versions accessible.

support windows installation

This comes from https://github.com/julia-actions/setup-julia/blob/4e8b9a1f067a1e9dd6c86a6629beb9ba3d1d0fe5/src/installer.ts#L113-L121 So I believe this is possible, just need to find a way to have the same symlinks.

# after 1.4
powershell -Command Start-Process -FilePath ${juliaDownloadPath} -ArgumentList "/SILENT /dir=${juliaInstallationPath}" -NoNewWindow -Wait
# before 1.4
powershell -Command Start-Process -FilePath ${juliaDownloadPath} -ArgumentList "/S /D=${juliaInstallationPath}" -NoNewWindow -Wait

Someone with access to Windows can put these together.

`list`?

A list (sub)command to list the installed Julias managed by jill and where they are installed could be nice

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.