fly-apps / rails-nix Goto Github PK
View Code? Open in Web Editor NEWDeploy Rails apps on Fly.io with Nix
Deploy Rails apps on Fly.io with Nix
In preparation for presenting this project, and to simplify importing values from an app spec JSON file, such as:
ffmpeg
layer to the image with an entry in the Docker image config config.history.comment
option mentioned hereCMD
This gem changed its extconf.rb
so the substitution here will no longer work. What should be the correct approach for fixing this?
[CXX] Compiling src/core/lib/profiling/stap_timers.cc
mkdir -p `dirname /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.o`
clang++ -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -DGPR_BACKWARDS_COMPATIBILITY_MODE -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="\"RUBY\"" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="\"1.42.0\"" -g -Wall -Wextra -DOSATOMIC_USE_INLINED=1 -Ithird_party/abseil-cpp -Ithird_party/re2 -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -O2 -Wframe-larger-than=16384 -fPIC -I. -Iinclude -I/nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/gens -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\" -arch arm64 -Ithird_party/zlib -std=c++11 -stdlib=libc++ -fno-exceptions -MMD -MF /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.dep -c -o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.o src/core/lib/profiling/stap_timers.cc
[AR] Creating /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a
mkdir -p `dirname /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a`
rm -f /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a
libtool -o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/adler32.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/compress.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/crc32.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/deflate.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzclose.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzlib.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzread.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzwrite.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/infback.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inffast.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inflate.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inftrees.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/trees.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/uncompr.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/zutil.o
make: libtool: No such file or directory
make: *** [Makefile:2503: /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a] Error 127
make: *** Waiting for unfinished jobs....
make: Leaving directory '/nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/nix/store/77bjl5185g1rxq0g7m868j5b7izvxanl-ruby-2.7.5/bin/$(RUBY_BASE_NAME)
extconf failed, exit code 1
Gem files will remain installed in /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0 for inspection.
Results logged to /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/grpc-1.42.0/gem_make.out
error: builder for '/nix/store/l8cgfwdl91g8gq9fgi7g22nnnwl56gd1-ruby2.7.5-grpc-1.42.0.drv' failed with exit code 1;
last 10 log lines:
> --with-make-prog
> --without-make-prog
> --srcdir=.
> --curdir
> --ruby=/nix/store/77bjl5185g1rxq0g7m868j5b7izvxanl-ruby-2.7.5/bin/$(RUBY_BASE_NAME)
>
> extconf failed, exit code 1
>
> Gem files will remain installed in /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0 for inspection.
> Results logged to /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/grpc-1.42.0/gem_make.out
For full logs, run 'nix log /nix/store/l8cgfwdl91g8gq9fgi7g22nnnwl56gd1-ruby2.7.5-grpc-1.42.0.drv'.
error: 1 dependencies of derivation '/nix/store/6wlz359basx4nx83xkwbrb9jvcgmbq7c-development.drv' failed to build
The big plan, I think, is for Fly to have a 'main' nixpkgs pinned version. Whenever we decide to change this version, we'll kick off jobs that:
Question: What's a good strategy for deciding when to move the main pin forward? Is there already a best practice out there?
Presumably, security patches and such are going into nixpkgs all the time. So there are benefits to frequently moving the pin. But it will be impossible to cover all the possible issues that may pop up.
Question: How should we manage versioning of our Nix builder code and nixpkgs?
It feels to me that these should be versioned separately.
For nixpkgs, certainly it's desirable to get security patches and other important software upgrades automatically. But this necessarily would change the contents of your build, slow down deploys, and possibly cause problems. So it makes sense to pin apps to a Nixpkgs version. But we'd also need to version our builder code as we improve it. Let's think about how this could work.
Builder images are semantically versioned, so we could continue with what we're doing now - release new builders with new builder code. New builders get new code, old ones can be upgraded. So, an app could in theory move forward in nixpkgs, but use the same builder code.
Maybe we store Nix pins in our backend so apps can compare and upgrade if they choose?
By default, only compile assets if app/assets
has changed. Otherwise, reuse the previous build's cache. If possible, make triggering if this step configurable by allowing changes in other paths to trigger it.
The image produced by the current setup weighs 340MB. This isn't terrible for a typical Rails image, but could be a lot smaller based on what's visible in the image.
Using dive we see a large 245MB layer at the bottom. This layer contains the source code, rubygems, and the rubygems dependencies. This means that every code change will require pushing a 245MB layer. Let's see how we can improve this.
Here are tickets describing specific options to explore:
Whew, the NixOS learning curve is pretty steep!
I had a crack at a basic working example integrating Yarn and Webpacker in a branch here: main...Matt-Yorkley:webpacker-trial
Issues
There were a couple of Webpacker-related issues related to the fact that the container has no internet access during the build phase. The first is that webpacker inserts itself into the precompile process in Rails and by default it checks to see if yarn
is available by calling yarn --version
and errors out if it doesn't get a result. I resolved this by including yarn
(the nix package) in the build deps.
The second issue is that Webpacker tries to run yarn install
by default before compiling, which isn't possible during the build phase and throws a fatal error (network error). There wasn't much room to manuever here as the yarn version available via nix is 1.x.x
, which doesn't have nice ENV vars for changing it's behaviour (that's in 2.x.x
), but luckily Webpacker skips calling yarn here if WEBPACKER_NODE_MODULES_BIN_PATH
is provided with a valid path to node_modules/.bin
, which was a viable option.
Notes
Since this required some quite specific interventions, I'm imagining this would need to handle things slightly differently depending on the JS bundling option used, eg: Webpacker, Vite, Esbuild, jsbundling-rails, Importmaps, etc.
Each npm package is nixified here via yarn2nix
, the command for updating yarn.nix
is: yarn2nix > yarn.nix
. A reference to the store is symlinked into the build at the end so the app can find it.
Most Gemfiles these days have the desired Ruby version embedded in Gemfile
like:
ruby "2.3.5"
Bundix incorrectly interprets this entry as a gem. We should either fix bundix, or work around it with the trick used here: nix-community/bundix#81 (comment), or remove the line temporarily when running bundix
. The line itself is useful to help us discover the target Ruby version.
I had a quick look at getting default.nix
to build with arbitrary Ruby versions here: main...Matt-Yorkley:versions
I'm not sure if that's the right way to do it, but it builds and successfully runs the app on the latest Ruby 3.1.1
(which was just released a few days ago). It feels like it should be extracted into a nice module somehow, but I'm not sure what that looks like 😅
To be useful for a range of Rails apps, it'll ultimately need the ability to specify a Ruby dot-version, specify a bundler dot-version, then build / fetch them both, then install all gems through bundix using the exact Ruby and bundler versions.
Short of fixing things upstream in Nixpkgs, are there (safe) tricks we can use to remove parts of the closures to reduce the size of the builds?
(Linked to #4)
There might be inexpensive ways to make the layering more useful to end-users. Explore quickly and see if it's a viable approach.
(For #4)
From #4:
More layers could help here, but we need to check that it works on Fly builders.
Some common gems like rbtrace
will fail to compile on arm64-darwin without this coreutils patch:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/coreutils/fix-arm64-macos.patch
So we should probably track unstable
for now, perhaps?
See:
Since the gemdir
is the src
dir, any change will also cause a minor rebuild of the gems layer. Note that this only "re-assembles" the gem directory, no gems are being rebuilt. The actual cost in build time is negligible, but still present.
The solution listed here might not be correct, as it wouldn't include local directories used as gems. Though the general approach would be correct, e.g. use strictly only the inputs required for gems. Not sure how to best approach this.
The pg gem depends on postgresql the (server), perl, among other things. This adds quite a lot of heft to the closure.
From #4:
One easyish fix would be to switch the pg
gem to depend on postgresql.lib
instead of postgresql
. It has a large tree of dependencies including systemd.
[nix-shell:~/.../rails-nix]$ nix --extra-experimental-features nix-command why-depends -f ./. packed-app pkgs.perl | cat
trace: Using default Nixpkgs revision 'c28fb0a4671ff2715c1922719797615945e5b6a0'...
warning: Git tree '/Users/samuel/DetSys/2022-fly/rails-nix' is dirty
/nix/store/2zajgla3b3bw47rbnyy10m17nkr90w8q-packed-rails-nix-0.0.0
└───tmp/cache/bootsnap/compile-cache-iseq/08/d8490ca05c810a: ….................E.../nix/store/hnm67idngkqkwg3ap02jkh142sqwvc67-ruby2.7.5-pg-1.3.1/lib/ruby/gem…
→ /nix/store/hnm67idngkqkwg3ap02jkh142sqwvc67-ruby2.7.5-pg-1.3.1
└───lib/ruby/gems/2.7.0/build_info/pg-1.3.1.info: …--with-pg-config=/nix/store/5w35mb5b4a26rwac1qa4rbwdnwfyrx1w-postgresql-13.5/bin/pg_config.…
→ /nix/store/5w35mb5b4a26rwac1qa4rbwdnwfyrx1w-postgresql-13.5
└───bin/pg_config: ….2-dev/lib/pkgconfig:/nix/store/xy347fibaxrc58xb8l09y88r6qqjyxsx-openssl-1.1.1m-dev/lib/pkgconfi…
→ /nix/store/xy347fibaxrc58xb8l09y88r6qqjyxsx-openssl-1.1.1m-dev
└───nix-support/propagated-build-inputs: … /nix/store/3vy7p7m0a38ip1dlad0pk0ydd3plrgyy-openssl-1.1.1m-bin /nix/store/6…
→ /nix/store/3vy7p7m0a38ip1dlad0pk0ydd3plrgyy-openssl-1.1.1m-bin
└───bin/c_rehash: …#!/nix/store/y84babjpa0ngq62pqvygsajscx7mlnks-perl-5.34.0/bin/perl..# WARNING…
→ /nix/store/y84babjpa0ngq62pqvygsajscx7mlnks-perl-5.34.0
Here's an issue discussing this: NixOS/nixpkgs#61580
--
We may want to consider prohibit the gem's derivation from depending on certain run-time dependencies like perl. This would make it less likely for this situation to occur in the future.
From #4:
Currently, packages are split into layers based on how often they are referred to by other packages. But the reality for Rails is that most changes happen in gems and the source code, and not in the system dependencies. The source code and gems get stuck in that big final layer.
Ideally, we could customize layers the following can get their own layers:
Here's a related promising PR: NixOS/nixpkgs#122608
Ideally, the user's source tree doesn't need to have any .nix
files in it. Their fly.toml
should contain all the info Nix needs to run a build.
The build VM will mount a persistent volume for the Nix store and any other persistent data to live. Then, flyctl deploy
will push the source code to the builder, and run a single build command there over SSH.
So some steps to make this happen:
fly.toml
From #4:
This is standard practice for Ruby deployments. It doesn't seem to be supported by Bundix directly, but probably can be done easily, maybe with bundleEnv
.
Rather than trying to hack the builder VM to build with nix, we should setup a new machine running the official nix image. We can run builds over rsync/ssh instead of relying on Docker.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.