crystal-lang / distribution-scripts Goto Github PK

It would be nice to not have to install llvm-8-dev when working on Crystal itself in the docker image.

E.g. eliminate this installation step here
https://github.com/crystal-lang/crystal/runs/597246816#step:4:1
https://github.com/crystal-lang/crystal/actions/runs/81290075/workflow#L12

Only llvm-8 is provided, though:

The -dev package should be a small difference.

Not sure if this is on the team's radar or not, but increasingly a nice-to-have, would be for the official macOS builds to receive notarization and codesigning to ensure:

1. the build hasn't been modified
2. to satisfy the strict enforcement of notarization present in macOS 10.15+.

While it's relatively easy to workaround it (either removing the quarantine xattr manually, going through System Preferences to allow crystal and shards, or downloading it via something that does not set the quarantine xattr), it's not particularly convenient and it's not immediately clear to a new user that they need to do that (they'll just get the "Move to Trash" or "Cancel" options when running). You can use Homebrew builds and it won't be an issue, but not everyone uses Crystal via Homebrew on macOS

For my own Crystal projects, I've taken to using gon for automating the notarization and codesigning in CI (see example here), which works pretty smoothly and (it or something like it) could probably be integrated into release process. The tricky piece would be credential management for macOS developer certs, though it sounds like CircleCI recommends something called Fastlane Match for this.

The following build targets are supported by OBS:
https://en.opensuse.org/openSUSE:Build_Service_supported_build_targets

The following files are currently produced as build artifacts of the nightly release workflow on circleci (see https://app.circleci.com/pipelines/github/crystal-lang/crystal/8502/workflows/77e21f43-14ad-4840-91c7-53a30649ca20/jobs/68065).

crystal-1.4.0-dev-1-darwin-universal.tar.gz (46 MB)
crystal-1.4.0-dev-1-linux-x86_64-bundled.tar (95 MB) ❌
crystal-1.4.0-dev-1-linux-x86_64-bundled.tar.gz (30 MB)
crystal-1.4.0-dev-1-linux-x86_64.tar (94 MB) ❌
crystal-1.4.0-dev-1-linux-x86_64.tar.gz (29 MB)
crystal-1.4.0-dev-1-linux-x86_64.tar.xz (19 MB) ❓
crystal-1.4.0-dev-1.universal.pkg (46 MB)
crystal-1.4.0-dev-docs.tar.gz (13 MB)
crystal_1.4.0-dev_amd64.snap (24 MB) ❓
docker-1.4.0-dev-alpine-build.tar.gz (262 MB) ❓
docker-1.4.0-dev-alpine.tar.gz (173 MB) ❓
docker-1.4.0-dev-ubuntu-x86_64-build.tar.gz (370 MB) ❓
docker-1.4.0-dev-ubuntu-x86_64.tar.gz (208 MB) ❓
versions-1.4.0-dev.json (2.1 kB) ❌

I'm wondering about the purpose of some of them. I suppose some are just intermediary steps that just happen do end up being published.

• ❌ The uncompressed .tar files are redundant and more wasteful than the compressed counterparts.
• ❓ Compressed packages are usually tar.gz, but the linux build also has a tar.xz. Is there a particular reason for this? The xc compression seems quite more efficient than gz. But it's still a question whether the duplication makes sense. Or if we should consider xc compressed files for other artifacts as well.
• ❓ The docker images are already published to docker hub. The snap package is uploaded to snapcraft. Do they even need to be available as directly downloadable files?
• ❌ The json file is completely unnecessary. That's just a temporary artifact from the API docs build process.

This initiative is related to crystal-lang/crystal#11902.

Some of these artifacts probably don't need to be pushed at all, others may be useful for debugging but don't need to be made publicly available.

Release preparation

1. Announce expected release date and time span for feature freeze
   • Feature freeze is about two weeks before release
   • Set date on milestone
2. Start preparing changelog and release notes (crystal-lang/crystal-website#248)
3. Start feature freeze period
   • Either no merging of features into master or split off release branch for backporting bugfixes.
4. Publish release PR draft (crystal-lang/crystal#11744)
   • It should contain the expected date of the release (~two weeks after the PR is issued).
   • It should be populated with updates to CHANGELOG.md and VERSION.
5. Ensure documentation for language and compiler changes and other relevant changes is up to date.
   • Crystal Book
     • Update language specification
     • Update compiler manual
     • Add or update guides / tutorials?

Release process

Source release

1. Finalize the release PR
   • Make sure all changes are mentioned in the changelog
   • Check release date
   • Un-draft the PR
2. Split off release branch (release/x.y)
3. Verify Maintenance CI workflow succeeds on the HEAD of the release branch (https://app.circleci.com/pipelines/github/crystal-lang/crystal/8041/workflows/7425f8bb-f05d-433d-9740-21bbb3c34900)
4. Smoke test with test-ecosystem
   • Run *Test Crystal & Shards Workflow with the release branch as crystal_branch. (https://github.com/crystal-lang/test-ecosystem/runs/4806030075)
5. Merge the release PR
6. Tag & annotate the commit with the changelog using <M.m.p> pattern as {version} (as a pre-release directly in GH?)
   • git tag -s -a -m $VERSION $VERSION
7. Publish Github release
   1. Copy the changelog section as description
   2. Binaries are added later
8. Close milestone

Publish documentation for the release

1. Publish API docs
   1. Have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env variables defined
      • Keys can be generated at https://console.aws.amazon.com/iam/home#/security_credentials (contact a Manas admin if you don't have access).
   2. Run make -C docs publish_docs CRYSTAL_VERSION=${VERSION} to publish docs to api/${VERSION}
   3. Run make -C docs dist-redirect_latest CRYSTAL_VERSION=${VERSION} to apply redirect from api/latest to api/${VERSION}
2. Publish language reference (TBD)

Binary releases

1. Wait for the release build in circle CI
2. Smoke test with test-ecosystem (again)
3. Attach build artifacts to Github release
   • crystal-*-darwin-*.tar.gz
   • crystal-*-linux-*.tar.gz
   • crystal-*.pkg
   • crystal-*-docs.tar.gz
4. Push changes to OBS for building linux packages
   1. Checkout https://github.com/crystal-lang/distribution-scripts and go to ./packages
   2. Configure build.opensuse.org credentials in environment variables:
      • export OBS_USER=
      • export OBS_PASSWORD=
   3. Run ./obs-release.sh devel:languages:crystal crystal $VERSION
      • Uses the docker image crystallang/osc to run the CLI client for OBS.
      • The script creates a branch in you home project, updates the version and pushes it back to OBS.
      • You can also run the commands from that file manually and check build locally with
        • osc build xUbuntu_20.04 x86_64
        • osc build Fedora_Rawhide x86_64
   4. Now OBS builds the packages. It’s best to follow the build status in the browser:
      1. open https://build.opensuse.org/project/show/home:$OBS_USER:branches:devel:langauges:crystal/crystal
      2. Wait for all package build jobs to finish and succeed
   5. Verify package installation
      • OBS_PROJECT=home:$OBS_USER:branches:devel:languages:crystal bats test
   6. When everything is green, create a submit request against the original package (Submit package link in the menu bar on the package in your branch)
5. Tag latest docker images
   • Versioned docker images have been pushed to dockerhub.
   • Now just assign the latest tags:
   • $ ./docker/apply-latest-tags.sh {version}
6. Publish snap package
   1. You need to logged in via $ snapcraft login
   2. Recent tagged release is published directly to edge channel. The CI logs the snap revision number. Otherwise the .snap file is in the artifacts.
   3. Check the current status to find the revision of the tagged release otherwise:
   4. $ snapcraft status crystal
   5. $ snapcraft release crystal <revision-number> beta
   6. $ snapcraft release crystal <revision-number> stable
7. Submit a PR to update the homebrew formula in https://github.com/Homebrew/homebrew-core/blob/master/Formula/crystal.rb (Homebrew/homebrew-core#93105)
   1. Update the previous and new version (with their respective hashes).
   2. Try locally $ brew install --build-from-source <source of formula>
   3. Create PR

Release announcements

1. Publish release notes on the website
2. Post announcement in https://forum.crystal-lang.org/c/news/official
3. Tweet about the release
4. Post in Reddit
5. Update https://github.com/crystal-lang/crystal-book/blob/master/crystal-version.txt

Post-release

1. Update crystal master branch to use released version
   • Edit PREVIOUS_CRYSTAL_BASE_URL in .circleci/config.yml
   • Edit DOCKER_TEST_PREFIX in bin/ci
   • Edit prepare_build on_osx download package and folder
   • Edit .github/workflows/*.yml to point to docker image
   • Edit shell.nix latestCrystalBinary using nix-prefetch-url --unpack <url>
2. Increment VERSION file to the next minor and -dev suffix
3. Perform uncomment/todos left in the repo

The latest LLVM version 13.0.0 has been released yesterday. It includes relevant fixes we've been waiting for.

A regression in LLVM 11 prevented us from upgrading to LLVM 11 or 12 (crystal-lang/crystal#10359). The fix should be available with LLVM 13.
It has been very recently discovered that LLVM 13 should also have a fix for crystal-lang/crystal#11047 (although that's less relevant here because we don't ship Windows builds yet).

A challenge for that is that LLVM 13 is not yet available in aports edge and I'm not sure when it will be. I don't think it would be backported to Alpine 3.12, which we're still stuck with for now (#127, crystal-lang/crystal#10366). We still don't know the reasons for the issues experienced on Alpine > 3.12.

We can either pull in the llvm-13 packages from Alpine edge whenever they are available. Or we need to build LLVM 13 manually.

If we want to link the upcoming Crystal 1.2 builds with LLVM 13, we'll probably have to resort to the latter. We want to ship them next week.

This is obviously more effort. So maybe it's not worth it considering that we can stay with LLVM 10 for now to avoid crystal-lang/crystal#10359.
The most relevant fix in LLVM 13 at this point is really for Windows and we can already use that in CI.

Requires crystal-lang/crystal#11277

Perhaps we need to split the workflow maintenance_release so it only runs this step after a release/ branch. Now it also runs on a ci branch.

The current version (which I guess is 2.17.1 from Bionic) causes GitHub Actions to not actually expose the Git repo as a repo but just check out the bare files, which can be a hard-to-debug surprise when people rely on the image in GitHub Actions.

The repository will be downloaded using the GitHub REST API
To create a local Git repository instead, add Git 2.18 or higher to the PATH

https://github.com/crystal-lang/crystal/runs/597246816?check_suite_focus=true#step:3:20

I've had to work around this by not using the image and instead installing Crystal with something like snap

It seems we need to use --auto-add-directories or --directories to the fpm call to build rpm (64 and 32 bits) in https://github.com/crystal-lang/distribution-scripts/blob/master/linux/Makefile

Extracted from crystal-lang/crystal#7421

The new BinTray install.sh script fails on OpenSUSE Tumbleweed 20200817 (and, I suspect, most OpenSUSE distributions).

In _discover_distro_type, execution continues until if hash apt-get 2>/dev/null, which evaluates as true and sets DISTRO_TYPE to "deb", when, by default, OpenSUSE uses Zypper, which is RPM-based. Actually, the reason for that if evaluating to true is that, for some reason, apt-get "forwards" to Zypper by default on OpenSUSE instead of just not existing. DISTRO_TYPE equal to "deb" later causes apt-key to fail because there is no such command.

That is, of course, only the first issue. The script as it stands seems to support only Yum and Apt, whereas OpenSUSE support would likely require the use of Zypper. I hope to put in a PR for this myself, but at least this issue will exist if I don't get around to it.

Missing man pages for crystal, shards, shard.yml (5).

Steps to Reproduce

$ man shards
No manual entry for shards
$ man shard.yml
No manual entry for shard.yml
$ man 5 shard.yml
No manual entry for shard.yml in section 5

Version Info

$ which crystal
/var/lib/snapd/snap/bin/crystal
$ which shards
/var/lib/snapd/snap/bin/shards
$ snap info crystal | grep installed
installed:          0.36.1               (600) 23MB classic    
$ snap version 
snap    2.49-2.fc32
snapd   2.49-2.fc32
series  16
fedora  32

Currently snap packages are published, but the release is performed manually. Automate this step for every tagged_release.

Installation script does not work with Red Hat Enterprise Linux of version 8 and above (current 8.5) because of OBS not set up for any above 7, current workaround is to use CentOS 8 package

Right now aarc64 tasks are run by CI, but they aren't really part of the build process.

• create musl images #123
• create gnu images #124
  (WIP)

Release preparation

1. Announce expected release date and time span for feature freeze
   • Feature freeze is about two weeks before release
2. Start preparing changelog and release post
3. Start feature freeze period
   • Either no merging of features into master or split off release branch for backporting bugfixes.
4. Publish release PR draft (crystal-lang/crystal#11391)
   • It should contain the expected date of the release (~two weeks after the PR is issued).
   • It should be populated with updates to CHANGELOG.md and VERSION.

Release process

Source release

1. Finalize the release PR
   • Make sure all changes are mentioned in the changelog
   • Check release date
   • Un-draft the PR
2. Split off release branch (release/x.y)
3. Verify Maintenance CI workflow succeeds on the HEAD of the release branch
4. Smoke test with test-ecosystem
   • Run *Test Crystal & Shards Workflow with the release branch as crystal_branch.
5. Merge the release PR
6. Tag & annotate the commit with the changelog using <M.m.p> pattern as {version} (as a pre-release directly in GH?)
7. Publish Github release
   1. Copy the changelog section as description
   2. Binaries are added later

Binary releases

1. Wait for the release build in circle CI
2. Smoke test with test-ecosystem (again)
3. Attach build artifacts to Github release
   • crystal-*-darwin-*.tar.gz
   • crystal-*-linux-*.tar.gz
   • crystal-*.pkg
   • crystal-*-docs.tar.gz
4. Publish docs
   1. Have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env variables defined
      • Keys can be generated at https://console.aws.amazon.com/iam/home#/security_credentials (contact a Manas admin if you don't have access).
   2. Run make -C docs publish_docs CRYSTAL_VERSION=${VERSION} to publish docs to api/${VERSION}
   3. Run make -C docs dist-redirect_latest CRYSTAL_VERSION=${VERSION} to apply redirect from api/latest to api/${VERSION}
5. Push changes to OBS for building linux packages
   1. Checkout https://github.com/crystal-lang/distribution-scripts and go to ./packages
   2. Configure build.opensuse.org credentials in environment variables:
      • export OBS_USER=
      • export OBS_PASSWORD=
   3. Run ./obs-release.sh devel:languages:crystal crystal $VERSION
      • Uses the docker image crystallang/osc to run the CLI client for OBS.
      • The script creates a branch in you home project, updates the version and pushes it back to OBS.
      • You can also run the commands from that file manually and check build locally with
        • osc build xUbuntu_20.04 x86_64
        • osc build Fedora_Rawhide x86_64
   4. Now OBS builds the packages. It’s best to follow the build status in the browser:
      1. open https://build.opensuse.org/project/show/home:$OBS_USER:branches:devel:langauges:crystal/crystal
      2. Wait for all package build jobs to finish and succeed
   5. Verify package installation
      • OBS_PROJECT=home:$OBS_USER:branches:devel:languages:crystal bats test
   6. When everything is green, create a submit request against the original package (Submit package link in the menu bar on the package in your branch)
6. Tag latest docker images
   • Versioned docker images have been pushed to dockerhub.
   • Now just assign the latest tags:
   • $ ./docker/apply-latest-tags.sh {version}
7. Publish snap package
   1. You need to logged in via $ snapcraft login
   2. Recent tagged release is published directly to edge channel. The CI logs the snap revision number. Otherwise the .snap file is in the artifacts.
   3. Check the current status to find the revision of the tagged release otherwise:
   4. $ snapcraft status crystal
   5. $ snapcraft release crystal <revision-number> beta
   6. $ snapcraft release crystal <revision-number> stable
8. Submit a PR to update the homebrew formula in https://github.com/Homebrew/homebrew-core/blob/master/Formula/crystal.rb .
   1. Update the previous and new version (with their respective hashes).
   2. Try locally $ brew install --build-from-source <source of formula>
   3. Create PR (Homebrew/homebrew-core#89149)

Release announcements

1. Update crystal-website (crystal-lang/crystal-website#208)
   1. Post a release notes in the website
   2. Update information of latest_release in https://github.com/crystal-lang/crystal-website/blob/master/_config.yml
   3. Update link to blog in https://github.com/crystal-lang/crystal-website/blob/5905e5644fcb4863976d0ca69856edbd74091370/index.html#L26
2. Post announcement in https://forum.crystal-lang.org/c/news/official
3. Tweet about the release
4. Post in Reddit
5. Update https://github.com/crystal-lang/crystal-book/blob/master/crystal-version.txt (https://github.com/crystal-lang/crystal-book/compare/master...straight-shoota:feature/crystal-1.2.2?expand=1)

Post-release

1. Update crystal master branch to use released version (crystal-lang/crystal#11430)
   • Edit PREVIOUS_CRYSTAL_BASE_URL in .circleci/config.yml
   • Edit DOCKER_TEST_PREFIX in bin/ci
   • Edit prepare_build on_osx download package and folder
   • Edit .github/workflows/win.yml to point to docker image
   • Edit shell.nix latestCrystalBinary using nix-prefetch-url --unpack <url>
2. Increment VERSION file to the next minor and -dev suffix
3. Perform uncomment/todos left in the repo
4. Update carc.in / play.crystal-lang.org (by jhass)
   Note: This is done after Arch repos are updated

We can use alpine edge, since it has a reasonably updated version of crystal.

As ref: currently images are built using crystal-infrastructure.

Hi, as you may know, Let's Encrypt Root certificate has expired.
Because of that, ca-certificate package has to be updated.

I've run into an issue, where running docker build in my CI pipeline fails, because of that expired certificate.

dev:~# docker pull crystallang/crystal:1.1.1
1.1.1: Pulling from crystallang/crystal
Digest: sha256:ae8f53cc10b9cfdd786682cc420cce1097a06fc2e0db0ab4647965df64f25951
Status: Image is up to date for crystallang/crystal:1.1.1
docker.io/crystallang/crystal:1.1.1

dev:~# docker run --rm -ti crystallang/crystal:1.1.1 git clone https://my-le-domain.com/test/test.git
Cloning into 'test'...
fatal: unable to access 'https://my-le-domain.com/test/test.git/': server certificate verification failed. CAfile: none CRLfile: none

(If you do this via shards install, you just get:

$ shards install
Resolving dependencies
Fetching my-le-domain.com/test/test.git
Failed to clone my-le-domain.com/test/test.git

with no explanation why... I had to do manual git clone in CI config to find out the real cause.)

I have to run apt update && apt install ca-certificate for this to work.
Interestingly, curl on the same domain works (don't know why).
Updating ca-certificates package removes one CA certificate and updates none. Maybe it has to do something with that.

Anyway, the current image for current crystal/crystallang:1.1.1 is two months old now.

Are there any plans to update all (at least current, not EOL) versions of crystal images automatically?

Patches in upstream ubuntu:focal container maybe should be propagated downstream to your image... what do you think?

Base ubuntu:focal image doesn't include ca-certificates package, nor git-core package, so you install that yourself. Autobuilding your images could aleviate such problems in the future...

The /etc/* files in the crystal repo are not bundled

$ docker run --rm -ti crystallang/crystal:0.33.0-alpine
/ # echo 'puts "hello"' > a.cr
/ # crystal build -Dpreview_mt a.cr
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: G-C-.o: in function `set_stackbottom':
/usr/share/crystal/src/gc/boehm.cr:245: undefined reference to `GC_set_stackbottom'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: G-C-.o: in function `current_thread_stack_bottom':
/usr/share/crystal/src/gc/boehm.cr:236: undefined reference to `GC_get_my_stackbottom'
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o '/a'  -rdynamic  -lpcre -lm /usr/lib/libgc.a -lpthread /usr/share/crystal/src/ext/libcrystal.a -levent_pthreads -levent -lrt -L/usr/bin/../lib/crystal/lib -L/usr/lib -L/usr/local/lib`

The alpine version uses libgc 8.0.4 (latest release), but it doesn't include de needed patch.

See #42 and #47.

Since i386-linux-gnu has been demoted to a tier 2 supported platform and builds are occassionally failing, I don't think we should continue to build packages for this architecture.
We have already announced that we won't publish 32-bit binary packages for Crystal 1.2 (https://crystal-lang.org/2021/09/30/preparing-1.2.html).

Ref: #120 (comment)

After every change to distribution-scripts, that affect the release packaging process, we need to test if they work in the CI workflow of https://github.com/crystal-lang/crystal/.

This could be automated. For every PR, a corresponding test branch needs to be created at https://github.com/crystal-lang/crystal/. The branch name must start with ci/ to trigger a maintenance build. This could just be prefixed to the branch name in https://github.com/crystal-lang/distribution-scripts.
That test branch must contain a commit which updates the commit SHA in https://github.com/crystal-lang/crystal/blob/5663eb00613114a7d6a784428f1d0af030a2aa09/.circleci/config.yml#L147 to the HEAD of the distribution-scripts branch. For this purpose we should probably extract that reference to an easily modifyable value.
For every subsequent commit to the distribution-scripts branch, there should be a respective update to the test branch.
A comment should be posted to the PR with a link to the test branch and the corresponding workflow run at circleCI.

Right now it is kinda-possible to get a pre-built Crystal (.exe) for Windows, by clicking into one of the GitHub Actions runs. Unfortunately, this requires being logged in to GitHub Web (yes, no anonymous access through web and no access through APIs at all).

Of course, the end game will be to have a Crystal build for Windows as part of the Releases, but right now there isn't yet such a release to speak of (0.34.0 didn't have essential fixes for Windows yet).

My main point here is that we should be ready for that when the release comes. It would simplify a lot of workflows (including Crystal's own CI), and I'm personally very excited about it.

The minimal thing to publish might even be the artifact archive that the workflow produces (at the commit corresponding to the release tag); for start it would suffice to manually download it from there and upload it under the release. (We could also tell GitHub Actions to do that automatically when a tag appears, but it requires the release to also already be there, which might not fit with the current release workflow.)

Of course, the contents of the artifact (just crystal.exe and static libs) aren't exactly in line with the other systems' releases, which have share/crystal/src (source code) and bin/crystal (shell wrapper, to which there is currently no Windows equivalent). And there have been no discussions at all about the directory structure that Crystal should be installed as. But I would strongly prefer having some download even if these details aren't fleshed out.

Publishing the nightlies themselves somewhere would also be nice. I would appreciate having an official download URL for any crystal.exe even before the release happens, but maybe that's asking too much right now.
And I hear that the story with Mac nightlies also isn't so great as it requires hoops through CircleCI APIs, so maybe there's some common solution here.

Release preparation

1. Announce expected release date and time span for feature freeze (https://forum.crystal-lang.org/t/crystal-1-3-lets-see-what-the-three-wise-men-bring-us/4125)
   • Feature freeze is about two weeks before release
   • Set date on milestone
2. Start preparing changelog and release notes
3. Start feature freeze period
   • Either no merging of features into master or split off release branch for backporting bugfixes.
4. Publish release PR draft (crystal-lang/crystal#11599)
   • It should contain the expected date of the release (~two weeks after the PR is issued).
   • It should be populated with updates to CHANGELOG.md and VERSION.
5. Publish release notes PR (crystal-lang/crystal-website#219)
6. Ensure documentation for language and compiler changes and other relevant changes is up to date.
   • Crystal Book
     • Update language specification
     • Update compiler manual
     • Add or update guides / tutorials?

Release process

Source release

1. Finalize the release PR
   • Make sure all changes are mentioned in the changelog
   • Check release date
   • Un-draft the PR
2. Split off release branch (release/x.y)
3. Verify Maintenance CI workflow succeeds on the HEAD of the release branch (https://app.circleci.com/pipelines/github/crystal-lang/crystal/7972/workflows/72a0bc71-41ce-4dfe-86fd-05845a19ec4c)
4. Smoke test with test-ecosystem
   • Run *Test Crystal & Shards Workflow with the release branch as crystal_branch. (https://github.com/crystal-lang/test-ecosystem/actions/runs/1662578953)
5. Merge the release PR
6. Tag & annotate the commit with the changelog using <M.m.p> pattern as {version} (as a pre-release directly in GH?)
   • git tag -s -a -m $VERSION $VERSION
7. Publish Github release
   1. Copy the changelog section as description
   2. Binaries are added later
8. Close milestone

Publish documentation for the release

1. Publish API docs
   1. Have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env variables defined
      • Keys can be generated at https://console.aws.amazon.com/iam/home#/security_credentials (contact a Manas admin if you don't have access).
   2. Run make -C docs publish_docs CRYSTAL_VERSION=${VERSION} to publish docs to api/${VERSION}
   3. Run make -C docs dist-redirect_latest CRYSTAL_VERSION=${VERSION} to apply redirect from api/latest to api/${VERSION}
2. Publish language reference (TBD)

Binary releases

1. Wait for the release build in circle CI
2. Smoke test with test-ecosystem (again) (https://github.com/crystal-lang/test-ecosystem/actions/runs/1663249585)
3. Attach build artifacts to Github release
   • crystal-*-darwin-*.tar.gz
   • crystal-*-linux-*.tar.gz
   • crystal-*.pkg
   • crystal-*-docs.tar.gz
4. Push changes to OBS for building linux packages
   1. Checkout https://github.com/crystal-lang/distribution-scripts and go to ./packages
   2. Configure build.opensuse.org credentials in environment variables:
      • export OBS_USER=
      • export OBS_PASSWORD=
   3. Run ./obs-release.sh devel:languages:crystal crystal $VERSION
      • Uses the docker image crystallang/osc to run the CLI client for OBS.
      • The script creates a branch in you home project, updates the version and pushes it back to OBS.
      • You can also run the commands from that file manually and check build locally with
        • osc build xUbuntu_20.04 x86_64
        • osc build Fedora_Rawhide x86_64
   4. Now OBS builds the packages. It’s best to follow the build status in the browser:
      1. open https://build.opensuse.org/project/show/home:$OBS_USER:branches:devel:langauges:crystal/crystal
      2. Wait for all package build jobs to finish and succeed
   5. Verify package installation
      • OBS_PROJECT=home:$OBS_USER:branches:devel:languages:crystal bats test
   6. When everything is green, create a submit request against the original package (Submit package link in the menu bar on the package in your branch)
5. Tag latest docker images
   • Versioned docker images have been pushed to dockerhub.
   • Now just assign the latest tags:
   • $ ./docker/apply-latest-tags.sh {version}
6. Publish snap package
   1. You need to logged in via $ snapcraft login
   2. Recent tagged release is published directly to edge channel. The CI logs the snap revision number. Otherwise the .snap file is in the artifacts.
   3. Check the current status to find the revision of the tagged release otherwise:
   4. $ snapcraft status crystal
   5. $ snapcraft release crystal <revision-number> beta
   6. $ snapcraft release crystal <revision-number> stable
7. Submit a PR to update the homebrew formula in https://github.com/Homebrew/homebrew-core/blob/master/Formula/crystal.rb .
   1. Update the previous and new version (with their respective hashes).
   2. Try locally $ brew install --build-from-source <source of formula>
   3. Create PR

Release announcements

1. Publish release notes on the website
2. Post announcement in https://forum.crystal-lang.org/c/news/official
3. Tweet about the release
4. Post in Reddit
5. Update https://github.com/crystal-lang/crystal-book/blob/master/crystal-version.txt

Post-release

1. Update crystal master branch to use released version
   • Edit PREVIOUS_CRYSTAL_BASE_URL in .circleci/config.yml
   • Edit DOCKER_TEST_PREFIX in bin/ci
   • Edit prepare_build on_osx download package and folder
   • Edit .github/workflows/*.yml to point to docker image
   • Edit shell.nix latestCrystalBinary using nix-prefetch-url --unpack <url>
2. Increment VERSION file to the next minor and -dev suffix
3. Perform uncomment/todos left in the repo

Also, making sure it uses the binaries/images generated from there.

The newest nightly docker image of crystallang/crystal seems to have an issue.
My CI currently fails with the error message Could not execute 'crystal': Error executing process: 'crystal': No such file or directory.

Trying this on my local machine gives the following output:

[blobcodes@david-pc ~]$ podman run crystallang/crystal:nightly shards --version
Shards 0.16.0 [ff94dd2] (2021-10-06)

[blobcodes@david-pc ~]$ podman run crystallang/crystal:nightly crystal --version
{"msg":"exec container process (missing dynamic library?) `/usr/bin/crystal`: No such file or directory","level":"error","time":"2021-11-30T10:29:58.000185975Z"}

[blobcodes@david-pc ~]$ podman run crystallang/crystal:nightly nonexistent --version
Error: executable file `nonexistent` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

So there seems to be a crystal binary, but some library seems to be missing.

The following environment variables are used in shards and crystal for some workflows.

• SHARDS_OPTS
• SHARDS_OVERRIDE
• SHARDS_CACHE_PATH
• CRYSTAL_OPTS
• CRYSTAL_PATH
• CRYSTAL_LIBRARY_PATH
• CRYSTAL_CACHE_DIR

The snap package does not support them because of its sandboxing.

The right way to support them is to use snap configuration, and to use snapctl get inside the snap wrapper scripts for crystal and shards.

Ref: https://snapcraft.io/docs/adding-snap-configuration

Right now it just runs on the latest release of shards, but that doesn't guarantee that if something fails it's our fault.

Right now the changelog is built using a script per PR. Ideally, the changelog shuld be generated with a button, and then we will manually edit it to polish it.

0.28.0 32 bits .deb packages went out without static linking.

Yet nightlies worked fine https://circleci.com/gh/crystal-lang/crystal/21793#artifacts/containers/0

The tagged one are not statically linked https://circleci.com/gh/crystal-lang/crystal/21811#artifacts/containers/0

The non-static link also happened in maintenance release ci/preview.

But since last nightlies, before 0.28 worked I thought tagged version would work.

Kudos for adding snapcraft support. 🎉 👏

Any chance armhf architecture will be supported?

$sudo snap install crystal
error: snap "crystal" is not available on stable for this architecture (armhf)
       but exists on other architectures (amd64).

The VERSION file that is used by compiler/crystal/config.cr is missing from both Debian and Archlinux packaging.

This leads to errors when using the Crystal::Config package.

To reproduce (full log):

$ docker run --rm crystallang/crystal:0.28.0 sh -c 'apt-get update && apt-get install -y llvm-dev && crystal eval '"'"'require "compiler/crystal/**"; puts Crystal::Config.default_config'"'"''
...
in usr/share/crystal/src/compiler/crystal/config.cr:11: error executing command: cat /usr/share/crystal/src/compiler/crystal/../../../VERSION, got exit status 1

        {{ `cat #{__DIR__}/../../../VERSION`.stringify.chomp }}

There is no more issue when creating the missing VERSION file on the two distributions.
However, for Archlinux, this file is located at /usr/lib/VERSION which might be problematic :-)

I'm not really sure if this is a packaging or code issue. Please just let me know if I should open an issue in the core's repo.

More info about the packages:

Debian

$ pacman -Ql crystal | grep -q VERSION && echo OK || echo KO
KO

$  pacman -Qi crystal
Name            : crystal
Version         : 0.28.0-1
Description     : The Crystal Programming Language
Architecture    : x86_64
URL             : http://crystal-lang.org
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : gc  libatomic_ops  pcre  libevent  llvm6-libs
Optional Deps   : shards: crystal language package manager [installed]
                  libyaml: For YAML support [installed]
                  gmp: For BigInt support [installed]
                  libxml2: For XML support [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 95.62 MiB
Packager        : Anatol Pomozov <[email protected]>
Build Date      : Wed 17 Apr 2019 07:41:48 PM CEST
Install Date    : Sun 28 Apr 2019 08:04:36 PM CEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Archlinux

$ dpkg -L crystal | grep -q VERSION && echo OK || echo KO
KO

$ apt-cache show crystal
Package: crystal
Status: install ok installed
Priority: extra
Section: default
Installed-Size: 88303
Maintainer: Crystal Team <[email protected]>
Architecture: amd64
Version: 0.28.0-1
Depends: gcc, pkg-config, libpcre3-dev, libevent-dev
Recommends: git, libssl-dev, libz-dev
Suggests: libxml2-dev, libgmp-dev, libyaml-dev, libreadline-dev
Description: no description given
Description-md5: c0af8b65ef8df63b3bfb124d96da1778
License: APACHE-2.0
Vendor: Crystal Team <[email protected]>
Homepage: https://crystal-lang.org/

(I only tested with Crystal 0.28.0 on Debian Buster and an up to date Archlinux)

Proposing feature-thread-stackbottom.patch to https://github.com/ivmai/bdwgc would be a win for everybody:

• Crystal core developers won't have to maintain this patch
• Crystal developers won't have to recompile bdwgc to have this feature, it will be directly available in the OS package
• bdwgc users will be able to use it

Edit: the patch is already merged with ivmai/bdwgc@5668de7
Remains to remove the patch when a new release of bdwgc will be out.

Release preparation

2. Start preparing changelog and release notes (crystal-lang/crystal-website#252)
3. Publish release PR draft (crystal-lang/crystal#11751)
   • It should be populated with updates to CHANGELOG.md and VERSION.

Release process

Source release

1. Finalize the release PR
   • Make sure all changes are mentioned in the changelog
   • Check release date
   • Un-draft the PR
2. Verify Maintenance CI workflow succeeds on the HEAD of the release branch
3. Smoke test with test-ecosystem
   • Run *Test Crystal & Shards Workflow with the release branch as crystal_branch.
4. Merge the release PR
5. Tag & annotate the commit with the changelog using <M.m.p> pattern as {version} (as a pre-release directly in GH?)
   • git tag -s -a -m $VERSION $VERSION
6. Publish Github release
   1. Copy the changelog section as description
   2. Binaries are added later
7. Close milestone

Publish documentation for the release

1. Publish API docs
   1. Have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env variables defined
      • Keys can be generated at https://console.aws.amazon.com/iam/home#/security_credentials (contact a Manas admin if you don't have access).
   2. Run make -C docs publish_docs CRYSTAL_VERSION=${VERSION} to publish docs to api/${VERSION}
   3. Run make -C docs dist-redirect_latest CRYSTAL_VERSION=${VERSION} to apply redirect from api/latest to api/${VERSION}

Binary releases

1. Wait for the release build in circle CI
2. Smoke test with test-ecosystem (again)
3. Attach build artifacts from circleci and GitHub Actions (Windows) to GitHub release
   • crystal-*-darwin-*.tar.gz
   • crystal-*-linux-*.tar.gz
   • crystal-*.pkg
   • crystal-*-docs.tar.gz
   • crystal.zip (GHA) -> crystal-$VERSION-windows-x86_64-msvc-unsupported.zip
4. Push changes to OBS for building linux packages
   1. Checkout https://github.com/crystal-lang/distribution-scripts and go to ./packages
   2. Configure build.opensuse.org credentials in environment variables:
      • export OBS_USER=
      • export OBS_PASSWORD=
   3. Run ./obs-release.sh devel:languages:crystal crystal $VERSION
      • Uses the docker image crystallang/osc to run the CLI client for OBS.
      • The script creates a branch in you home project, updates the version and pushes it back to OBS.
      • You can also run the commands from that file manually and check build locally with
        • osc build xUbuntu_20.04 x86_64
        • osc build Fedora_Rawhide x86_64
   4. Run ./obs-release.sh devel:languages:crystal crystal${VERSION%.*} $VERSION
   5. Now OBS builds the packages. It’s best to follow the build status in the browser:
      1. open https://build.opensuse.org/project/show/home:$OBS_USER:branches:devel:langauges:crystal/crystal
      2. Wait for all package build jobs to finish and succeed
   6. Verify package installation
      • OBS_PROJECT=home:$OBS_USER:branches:devel:languages:crystal bats test
   7. When everything is green, create a submit request against the original package (Submit package link in the menu bar on the package in your branch)
5. Tag latest docker images
   • Versioned docker images have been pushed to dockerhub.
   • Now just assign the latest tags:
   • $ ./docker/apply-latest-tags.sh {version}
6. Publish snap package
   1. You need to logged in via $ snapcraft login
   2. Recent tagged release is published directly to edge channel. The CI logs the snap revision number. Otherwise the .snap file is in the artifacts.
   3. Check the current status to find the revision of the tagged release otherwise:
   4. $ snapcraft status crystal
   5. $ snapcraft release crystal <revision-number> beta
   6. $ snapcraft release crystal <revision-number> stable
7. Submit a PR to update the homebrew formula in https://github.com/Homebrew/homebrew-core/blob/master/Formula/crystal.rb .
   1. Update the previous and new version (with their respective hashes).
   2. Try locally $ brew install --build-from-source <source of formula>
   3. Create PR

Release announcements

1. Publish release notes on the website
2. Post announcement in https://forum.crystal-lang.org/c/news/official
3. Tweet about the release
4. Post in Reddit
5. Update https://github.com/crystal-lang/crystal-book/blob/master/crystal-version.txt

Post-release

1. Update crystal master branch to use released version
   • Edit PREVIOUS_CRYSTAL_BASE_URL in .circleci/config.yml
   • Edit DOCKER_TEST_PREFIX in bin/ci
   • Edit prepare_build on_osx download package and folder
   • Edit .github/workflows/*.yml to point to docker image
   • Edit shell.nix latestCrystalBinary using nix-prefetch-url --unpack <url>
2. Update default base version in test-ecosystem

There is official box provision located at vagrantup, but the last update is quite far from what we have right now.

As development on Docker for MacOS is barely usable, some of us would appreciate an update to above box with latest crystal-lang releases.

Thanks!

One option is to use Arch linux for this.

In the release process, create a docker image to distribute using the latest binaries generated from CI.

Related #123

My understanding from this line:

Would it be possible to bump this to something much newer, perhaps LLVM 10 (or ideally LLVM 11+)?

The reason I ask is that I believe either LLVM 10 or 11 is needed to target AArch64 (ARM64) on Darwin/macOS. If were to use a newer LLVM and also set -DLLVM_TARGETS_TO_BUILD=X86;AArch64 for the builds of LLVM used to build the distribution releases, we could use the official x86_64 binary release to bootstrap a ARM64 Homebrew build

Also, it should be part of the tagged_release workflow

The readme at https://hub.docker.com/r/crystallang/crystal is outdated.

crystal-lang/crystal-dist#14 suggested an automation to update the readme from the Crystal repository. We could to this and integrate it as a step in docker/apply-latest-tags.sh.

However, I think it would be better to provide a readme specific to the docker repository, which explains the available image types. We might still want to have that automated to always show the latest tags, though.

clock_gettime doesn't exist on macOS versions previous to Sierra (10.12).

Crystal's codebase explicitly tries to avoid it, but the builds done by the CI still depend on it, failing to run on OS X El Capitan (10.11).

Local builds (even in Sierra) don't depend on them, so there must be something odd in the omnibus script or something.

See Homebrew/homebrew-core#29527

See crystal-lang/crystal#6205.

It requires querying the binaries of the last two versions and its hashes; not 🚀 science. It should be part of a tagged_release.

A new Alpine image for 0.36.1 made it's way to docker hub last night. This is unable to statically compile crystal programs that use YAML.

Reduced example: https://gist.github.com/kimburgess/2a740af0f2c8bc5b217a9fa47638df0e.

Cause appears to be bdff883

Build output:

Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM crystallang/crystal@sha256:ee97e1ef8b609cd80a8d504fdf8c5c09f7ac2fc520d360a7b10c8d50aa21f4b2
 ---> 32474966bd9f
Step 2/3 : COPY . .
 ---> 7e65d2efd8e7
Step 3/3 : RUN crystal build --static --error-trace yaml.cr
 ---> Running in cffb373d5aed
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lyaml (this usually means you need to install the development package for libyaml)
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o /yaml  -rdynamic -static -L/usr/bin/../lib/crystal/lib -lyaml  -lpcre -lm -lgc -lpthread /usr/share/crystal/src/ext/libcrystal.a -levent -lrt`
The command '/bin/sh -c crystal build --static --error-trace yaml.cr' returned a non-zero code: 1

Release preparation

1. Start preparing changelog and release notes
2. Publish release PR draft (crystal-lang/shards#549)
   • It should be populated with updates to CHANGELOG.md, VERSION, and shard.yml.

Release process

1. Finalize the release PR
   • Make sure all changes are mentioned in the changelog
   • Check release date
   • Build man files with the release date: $ make clean docs SOURCE_DATE_EPOCH=$(gdate -d "YYYY-MM-DD" +"%s")
   • Un-draft the PR
2. Smoke test with test-ecosystem
   • Run Test Crystal & Shards Workflow with the release branch as shards_branch.
3. Merge the release PR
4. Tag & annotate the commit with the changelog using v<M.m.p> pattern as {version}
   • git tag -s -a -m v$VERSION v$VERSION
5. Publish Github release (https://github.com/crystal-lang/shards/releases/new)
   • Copy the changelog section as description
6. Close milestone (https://github.com/crystal-lang/shards/milestones)

Post-release

1. Increment VERSION file to the next minor and -dev suffix
2. Update distribution-scripts (#184)
   • Edit linux/Makefile
   • Edit omnibus/config/software/shards.rb
3. Update https://github.com/crystal-lang/crystal (crystal-lang/crystal#11930)
   • Edit .github/workflows/win.yml
4. Submit a PR to update the homebrew formula in https://github.com/Homebrew/homebrew-core/blob/master/Formula/crystal.rb . Or do it on Crystal release.
5. Perform uncomment/todos left in the repo

and make it part of the building process

The bundled libraries in the linux tar.gz (currently only libgc) depend on GNU libc. They don't work with musl libc. Thus, for the alpine docker images, we need a different libgc build that works with musl libc. This is currently done in docker/alpine.Dockerfile:

This is subpar because it's mixing concerns. The build process of binary artifacts should be enclosed in the linux stage, docker should just be responsible for putting them in a docker image.
And it's actually a duplication: We're already building a musl-compatible libgc in the build process of the tar.gz.

So I believe we should have an additional tar.gz with musl-compatible libraries as an intermediary product. Maybe even publish that, but that's an optional enhancement. Main reason is to streamline alpine image generation.

Right now nightlies images are being pushed already, the difference lies in that the images of releases (apparently) uses OBS packages instead of hard-coded binaries. The intention as we understand it is to be able to deliver updateable images. However, that doesn't seem to be super important: those willing to have such images should simply add the steps to install crystal from a base image.

docker run --rm crystallang/crystal:nightly-alpine crystal eval 'puts "foo"'

Produces:

/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/bin/../lib/crystal/libgc.a(pthread_support.o): in function `GC_thr_init':
pthread_support.c:(.text+0xe39): undefined reference to `gnu_get_libc_version'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/bin/../lib/crystal/libgc.a(mach_dep.o): in function `GC_with_callee_saves_pushed':
mach_dep.c:(.text+0x3b): undefined reference to `getcontext'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/bin/../lib/crystal/libgc.a(pthread_start.o): in function `GC_inner_start_routine':
pthread_start.c:(.text+0x44): undefined reference to `__pthread_register_cancel'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: pthread_start.c:(.text+0x6b): undefined reference to `__pthread_unregister_cancel'
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o /root/.cache/crystal/crystal-run-eval.tmp  -rdynamic -L/usr/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent -lrt`

First noticed via https://github.com/Blacksmoke16/oq/runs/3875659755?check_suite_focus=true.

Bug Report

When installing Crystal on macOS via the .pkg distributable:

This will install Crystal on your Mac.

---

This is a generic Omnibus welcome message. If you are building Mac OS X packages, you should generate a custom welcome file to suit your needs.

If you are not the creator of Crystal, the presence of this message is likely a bug and should be reported to the creator.

Similarly for the package's license:

All Rights Reserved

This is a generic Omnibus license file for Crystal. If you are building Mac OS X packages, you should generate a custom license file and description to suit your needs.

CircleCI has a field to specify the size of the computer. It's not required, but probably a good idea to set. Probably we could stick with medium+ (that's 6GB RAM).