Publish a Windows build (crystal.exe) about distribution-scripts HOT 22 OPEN

Returning here after some months to hopefully rekindle this discussion;

With 1.10 in the horizon, hoping this would get picked up. I know we are doing some dist script cleanups at
crystal-lang/crystal#10639

Just having the crystal.exe and the stdlib files as part of the release artifacts/zip would make it so much easier to use Crystal with any windows package manager. This is very low hanging fruit and IMO has great impact for people planning to use Crystal for non-web related use cases on Windows (gamedev, windows apps, GUIs etc)

from distribution-scripts.

So it would be nice to publish some kind of binary artifact for the 0.35.0 release, even just for Crystal itself to stop relying on bootstrapping in its CI setup.

I could just keep relying on the Artifacts functionality (look up the CI build that corresponds to the commit that's tagged as 0.35.0), but I'm afraid that the storage for it is not persistent, it'll just disappear eventually. So actually this is not possible, a persistent location really is needed.

Adding @bcardiff for specific questions:

1. Would you be fine to just take the binaries directly from the current continuous build, and put them in the GitHub release?
2. If not, is there anything we need to change in Crystal to make it viable to include binaries for the release?

For reference, the current binary artifacts include just these 5 files:

• crystal.exe (built without --release flag, fully statically linked)
• Static libs in release mode, /MT:
  • gc.lib
  • pcre.lib
  • yaml.lib
  • z.lib

Might be nice to also include source code in the archive.

from distribution-scripts.

I feel we're still getting off in the weeds. We're trying to sort out a CI only bootstrapping problem. This build isn't going to users, so we shouldn't worry about it's lineage. We're just trying to get rid of the "linux" part of the windows CI.

When we release we can cross-compile and think more about these details.

from distribution-scripts.

With 1.0 out, can we look back into this?
Crystal on windows is usable today for many usecases bar networking based ones, and we should make it easy for people to use Crystal for these use cases (like desktop apps, gamedev etc).

We should have an approach for easy installation of Crystal on Windows today. Anything we can do to support package managers like Winget, Chocolatey, Scoop etc will go a long way in unblocking windows user in using Crystal.

from distribution-scripts.

How about adding a scoop > Extras bundle for Crystal, while a lot of these details are being fleshed out?
Scoop can hide all the gnarly internals and provides scripting and dependency management. Here is a sample for rust-up
https://github.com/chawyehsu/dorado/blob/bf8643c053c831b3f7b394eab4edf172000bcbb9/rustup.json#L18

Being able to just do scoop install crystal on windows would be amazing.

I think, right now it is better to just package the relevant bits like cl.exe and link.exe with the crystal.exe, instead of building all dependencies from source as well.
Later when all the details have been hashed out, we can do a much more involved process.

from distribution-scripts.

This is still very relevant.

The latest Crystal tag is usable for Windows and has no advantage over Crystal nightly.

https://github.com/oprypin/install-crystal is also very content with prebuilt Crystal as it is being uploaded from the artifacts. The archive is usable with minimal setup.

However, that action is forced to offer only Crystal nightly for Windows because there is no stable location for the build of a particular tag; an artifact build will expire eventually.

A solution that would work well is to attach the archive from the artifact directly to the release. But I can see that maybe it'll confuse people.

A weird solution that I'd also accept is to get someone to bump the artifacts build on GitHub once every 3 months...

I can also very easily just copy these archives onto some website of my own, but that'll just look shady to most.

from distribution-scripts.

Well those builds are directly usable, while the ones for Windows require setup, and, you know, the actual Windows support is not "production-ready" to say lightly.

from distribution-scripts.

I think i'd prefer the code that produced any release tarball to live in here...

from distribution-scripts.

This repo is responsible of keeping track of how the releases are built. Since the sha is used in the main repo, there it is all the information on how to keep the lineage of the compiler.

I think we can publish some windows binary as long as we include in this repo some scripts that builds them.

The scripts in the different folder need to know the sha and crystal version to build plus using which previous released tar.gz should be used.

This means that for a couple of version this process implies cross compiling from linux. If at some point we want to bootstrap a windows lineage we can do it, but it will be a second phased.

I would probably delay that second phase until we discuss a windows installer.

But as a preview for windows, cross-compiling and building a .zip (?) package sound reasonable to me.

Since the windows process is run in GitHub Actions instead of CircleCI we will have some duplication when updating the distributions-scripts.

Let me know if the branch filter in circleci config need some clarification. I would expect something similar to happen on this new workflow.

from distribution-scripts.

Uhh I didn't reply for a while, but the main reason for that is I actually don't understand what you mean, especially the last sentence :s

Some inline comments:

This repo is responsible of keeping track of how the releases are built.

(Technically, both the crystal repo itself and this one are used for builds.)

I think we can publish some windows binary as long as we include in this repo some scripts that builds them.

If one added a trivial (one-line) script here to just defer to the main repo, it would technically also be "both the crystal repo itself and this one" 😋
But that's just a side note, something I found funny.

This means that for a couple of version this process implies cross compiling from linux. If at some point we want to bootstrap a windows lineage we can do it, but it will be a second phased.

Importantly, there is no way to produce a Windows build having just Linux, it's either Linux + Windows, or just Windows -- both ways are possible (and are built continuously) now, the only hurdle for the latter is that there is no permanently-stored crystal.exe to refer to. That is what I really care about, the rest is side details.

I would probably delay that second phase until we discuss a windows installer.

I don't fully understand why you want to delay switching to Windows-only, as that's so much simpler. I guess you want to get the process fully "clean" without possibly-non-reproducible steps (if you consider GitHub Actions non-reproducible)? I suppose nothing stops you from doing that later when the process is fully established, just re-publish a few prior steps.

But as a preview for windows, cross-compiling and building a .zip (?) package sound reasonable to me.

OK, done! That is already being produced by GitHub Actions, just grab it from there. I could also make some changes to the workflow if you want, e.g. it could probably be made to publish the .zip into the release by itself.

Since the windows process is run in GitHub Actions instead of CircleCI we will have some duplication when updating the distributions-scripts.

OK, I have no input there, but if this phrase has some secondary meaning to it, I don't know what it is.

Let me know if the branch filter in circleci config need some clarification. I would expect something similar to happen on this new workflow.

Umm this is where I'm entirely confused. Just to clarify, (if that's what you mean) I have no intention of learning any existing CircleCI config or adding to it. Especially if you insist on a two-stage (two-OS) config.

Ideally we'd get to a situation with just one OS before having to finalize a solution, otherwise it'd need a lot of difficult work that would be thrown out right afterwards.

Or maybe you just mean I should look at the config and make something similar with GitHub Actions but in distribution-scripts repo? Not sure what's meant here, and not sure what's best.
Anyhow, I haven't looked at the CircleCI config.

As a summary to clarify my intention,

Unfortunately, I don't intend to be the main author of a proper solution based on distribution-scripts and/or CircleCI. I also don't think we're ready for a proper solution yet. I'm just asking to publish some Crystal binaries, and ideally (but not necessarily) attached to the GitHub release of the main repo. That's why I have directly asked about the possibility of one particular temporary solution, without any intention of it becoming authoritative.

1. Would you be fine to just take the binaries directly from the current continuous build, and put them in the GitHub release?

from distribution-scripts.

I'd prefer to either publish something just for the CI and work out a temporary solution just for that (not attached to a release at all), or work out a full "extract and run on windows" .zip file, similar to the linux tarballs.

Publishing just the compiler executable somewhere where people who don't know the windows port details can find it isn't something I'd like to do.

from distribution-scripts.

work out a full "extract and run on windows" .zip file

Generally this is a longer-term solution, which I'm not focusing on. You could kinda do it now, - just directly combine the artifact archive with the source code archive. But as there's no installer or such, this always requires setting some environment variables.

I'd prefer to either publish something just for the CI and work out a temporary solution just for that (not attached to a release at all)

That's completely fine by me.
Ideally it'd be a very stable location, so perhaps https://github.com/oprypin/install-crystal could rely on it as well.

from distribution-scripts.

You could kinda do it now, - just directly combine the artifact archive with the source code archive. But as there's no installer or such, this always requires setting some environment variables.

That's totally fine. We can use a wrapper script like on POSIX to set CRYSTAL_PATH relative to the script, just like the tarball for linux.

from distribution-scripts.

Yeah - lets focus on just a s3 bucket or something simple to upload windows crystal.exes to for the short-term. The immediate problem is having a stable storage location with a stable URL. That can be solved rather simply. Lets sort the rest later.

from distribution-scripts.

Even if we publish the current CI's output in the GitHub release artifact I would not use that binary as input for future releases. We will be missing the reference on how that binary was built in the first time.

We could publish a binary as a preview, but I would avoid using it in any automated/integration until there is a more stable a feature complete zip / installer.

What I imagine is that in this repo we can have a:

• windows/prepare.sh to perform something similar to the linux-job. Having as argument the previous linux tar.gz and the crystal sha1.
• windows/build.bat to perform something similar the windows-job. Having as argument the directory of the crystal.obj file and other arguments.

We could have a windows/deps.bat that would build the dependencies and decouple that. We can store those in S3 and use them as input in windows/build.bat. That is windows/deps.bat can be manually triggered as we are doing today in this repo to update the llvm for darwind pkg.

This prepare and build would be run in GitHub action on nightlies and tagged releases. Once they are published for a first time (at least) we could drop the prepare and change the build to use an input the url of the previos windows .zip package, hence using windows to build the next windows compiler.

I know this process is not as comfortable but is a way to keey a better reproducible build script to keep the lineage of the compilers.

Think for example that rebootstraping the windows compiler directly from the github actions is something that can't be done easily in a couple of years when the services is changed in a substancial way.

from distribution-scripts.

Hmm I had covered that before.

I guess you want to get the process fully "clean" without possibly-non-reproducible steps (if you consider GitHub Actions non-reproducible)? I suppose nothing stops you from doing that later when the process is fully established, just re-publish a few prior steps.

Really the most annoying problem is the need for MSVC to be installed and the environment set up for it (which is also not a fully transparent process). So the straightforward approach would just assume that it's active. The non strightforward process would need to look for MSVC itself, and it could be in various locations... And with any kind of approach, this relies on proprietary software to be kind enough to still be installable :)

.bat

Yeahh that one is probably not expressive enough, need powershell. Which brings its own complications, like Windows not allowing you to run powershell scripts unless you reconfigure the system for it??

from distribution-scripts.

I didn't mean to force CMD. A powershell script is valid. Assuming that MSVC is installed is valid to me. As long as it is expressed in a README I'm fine.

It's is a grey area. To me it make sense to have something that almost usable in a context that is not GitHub actions. Not because GA is bad, but to ensure we can move and use the process eventually in other environments, like locally building previews, etc.

from distribution-scripts.

OK.

We could have a windows/deps.bat that would build the dependencies and decouple that. We can store those in S3 and use them as input in windows/build.bat.

How would the running and the storing happen, btw?
Because either someone would need to manually run Windows, or GitHub Actions would need to write to S3, or there'd be a manual copying step in between? How will we set this up in reality?

from distribution-scripts.

In Linux currently we build all the deps.
In osx obminus we have two main targets: crystal or llvm. The llvm is run manually in this repo in CircleCI on a specific branch name.
We can add GitHub actions to this repo to build llvm on windows and whatever other dependency we think is time consuming.

from distribution-scripts.

Since the other thread got closed, hoping to get the conversation restarted here.
Can we look at adding the crystal.zip with the windows compiled files to the release page?
That's the smallest first step we need to take. we can look to an ideal solution later.

from distribution-scripts.

But I can see that maybe it'll confuse people.

How so? Isn't that the same we're doing for linux and mac?

from distribution-scripts.

Really the most annoying problem is the need for MSVC to be installed and the environment set up for it (which is also not a fully transparent process).

Can we not look at the simplest path forward and list VC20XX as a dependency for now, and then look at removing this dependency at a later date?

A solution that would work well is to attach the archive from the artifact directly to the release.

This is the one I favor as well. Simply having access to the Crystal windows builds in the Release would allow us to setup crystal via the windows package managers.

from distribution-scripts.