fly-apps / rails-nix Goto Github PK

Deploy Rails apps on Fly.io with Nix

Ruby 69.52% JavaScript 3.08% CSS 1.76% HTML 17.06% Shell 2.96% Dockerfile 5.62%

rails-nix's Introduction

This is a sample Rails app for testing a Nix-based development and deployment system. It inherits most behavior from Fly's Nix modules. The following files are interesting for understanding how the module system works.

default.nix: Entry point for Nix commands. All module configuration is passed through here. This can eventually be moved to nix-base.
shell.nix: Entry point for nix-shell. It ensures all gem groups are included in the bundle environment.
nix-base.nix: Importer shim for loading Nix modules that fetches the nix-base git commit from fly.toml. A helper script in .nix/update-nix-base will update values in fly.toml HEAD of nix-base/main.
nix-build-image: Script that will build an image and push it to the Fly repository.

Building a Docker image

If you just want to build the Docker image and see the results, you can run the following on Linux. Building Docker images on Darwin is not well supported.

nix-build . -A eval.config.outputs.container.image | docker load

This will generate a Docker image tarball and load it into your local Docker. You can inspect its contents with an amazing tool called Dive.

Deploying on Fly.io

To deploy, use fly deploy -i image_name, substituting the image name from the docker load.

For Fly employees and enthusiasts: there's experimental support for running a remote build using fly deploy --nix. Using this will require a custom remote builder Docker image. Ask an admin how to get that setup. Eventually, Nix will be enabled by default on remote builder VMs.

Using Nix in Development

The same Nix environment used in production can also be run in development.

First install Nix locally, then run nix-shell in the project root. You'll be popped into a shell with all the good stuff. If you want a shell with nothing but Nix - not inheriting PATH or any other environment from your system - use nix-shell --pure.

Notes

Changing Ruby versions can mess with the Bootsnap cache. If errors are raised by bootsnap, try rm -rf tmp/cache.

Bundling gems requires an extra step to update gemset.nix. Run bin/bundix after any Gemfile updates.

rails-nix's People

Contributors

Stargazers

Watchers

Forkers

matt-yorkley

rails-nix's Issues

Gems are rebuilt on gemdir input change

See:

https://github.com/fly-apps/rails-nix/tree/fix/gems-isolation

Since the gemdir is the src dir, any change will also cause a minor rebuild of the gems layer. Note that this only "re-assembles" the gem directory, no gems are being rebuilt. The actual cost in build time is negligible, but still present.

The solution listed here might not be correct, as it wouldn't include local directories used as gems. Though the general approach would be correct, e.g. use strictly only the inputs required for gems. Not sure how to best approach this.

Javascript dependencies

Whew, the NixOS learning curve is pretty steep!

I had a crack at a basic working example integrating Yarn and Webpacker in a branch here: main...Matt-Yorkley:webpacker-trial

Issues

There were a couple of Webpacker-related issues related to the fact that the container has no internet access during the build phase. The first is that webpacker inserts itself into the precompile process in Rails and by default it checks to see if yarn is available by calling yarn --version and errors out if it doesn't get a result. I resolved this by including yarn (the nix package) in the build deps.

The second issue is that Webpacker tries to run yarn install by default before compiling, which isn't possible during the build phase and throws a fatal error (network error). There wasn't much room to manuever here as the yarn version available via nix is 1.x.x, which doesn't have nice ENV vars for changing it's behaviour (that's in 2.x.x), but luckily Webpacker skips calling yarn here if WEBPACKER_NODE_MODULES_BIN_PATH is provided with a valid path to node_modules/.bin, which was a viable option.

Notes

Since this required some quite specific interventions, I'm imagining this would need to handle things slightly differently depending on the JS bundling option used, eg: Webpacker, Vite, Esbuild, jsbundling-rails, Importmaps, etc.

Each npm package is nixified here via yarn2nix, the command for updating yarn.nix is: yarn2nix > yarn.nix. A reference to the store is symlinked into the build at the end so the app can find it.

Bundix can't handle Gemfile with the ruby version specified there

Most Gemfiles these days have the desired Ruby version embedded in Gemfile like:

ruby "2.3.5"

Bundix incorrectly interprets this entry as a gem. We should either fix bundix, or work around it with the trick used here: nix-community/bundix#81 (comment), or remove the line temporarily when running bundix. The line itself is useful to help us discover the target Ruby version.

Skip inclusion of gems in the development/test groups

From #4:

This is standard practice for Ruby deployments. It doesn't seem to be supported by Bundix directly, but probably can be done easily, maybe with bundleEnv.

Ensure nix-shell is usable and useful

Custom Ruby / Bundler versions

I had a quick look at getting default.nix to build with arbitrary Ruby versions here: main...Matt-Yorkley:versions

I'm not sure if that's the right way to do it, but it builds and successfully runs the app on the latest Ruby 3.1.1 (which was just released a few days ago). It feels like it should be extracted into a nice module somehow, but I'm not sure what that looks like 😅

To be useful for a range of Rails apps, it'll ultimately need the ability to specify a Ruby dot-version, specify a bundler dot-version, then build / fetch them both, then install all gems through bundix using the exact Ruby and bundler versions.

Update the `pg` gem in Nixpkgs to depend on `postgresql.lib` to reduce the closure size

The pg gem depends on postgresql the (server), perl, among other things. This adds quite a lot of heft to the closure.

From #4:

One easyish fix would be to switch the pg gem to depend on postgresql.lib instead of postgresql. It has a large tree of dependencies including systemd.

[nix-shell:~/.../rails-nix]$ nix --extra-experimental-features nix-command why-depends -f ./. packed-app pkgs.perl | cat
trace: Using default Nixpkgs revision 'c28fb0a4671ff2715c1922719797615945e5b6a0'...
warning: Git tree '/Users/samuel/DetSys/2022-fly/rails-nix' is dirty
/nix/store/2zajgla3b3bw47rbnyy10m17nkr90w8q-packed-rails-nix-0.0.0
└───tmp/cache/bootsnap/compile-cache-iseq/08/d8490ca05c810a: ….................E.../nix/store/hnm67idngkqkwg3ap02jkh142sqwvc67-ruby2.7.5-pg-1.3.1/lib/ruby/gem…
    → /nix/store/hnm67idngkqkwg3ap02jkh142sqwvc67-ruby2.7.5-pg-1.3.1
    └───lib/ruby/gems/2.7.0/build_info/pg-1.3.1.info: …--with-pg-config=/nix/store/5w35mb5b4a26rwac1qa4rbwdnwfyrx1w-postgresql-13.5/bin/pg_config.…
        → /nix/store/5w35mb5b4a26rwac1qa4rbwdnwfyrx1w-postgresql-13.5
        └───bin/pg_config: ….2-dev/lib/pkgconfig:/nix/store/xy347fibaxrc58xb8l09y88r6qqjyxsx-openssl-1.1.1m-dev/lib/pkgconfi…
            → /nix/store/xy347fibaxrc58xb8l09y88r6qqjyxsx-openssl-1.1.1m-dev
            └───nix-support/propagated-build-inputs: … /nix/store/3vy7p7m0a38ip1dlad0pk0ydd3plrgyy-openssl-1.1.1m-bin /nix/store/6…
                → /nix/store/3vy7p7m0a38ip1dlad0pk0ydd3plrgyy-openssl-1.1.1m-bin
                └───bin/c_rehash: …#!/nix/store/y84babjpa0ngq62pqvygsajscx7mlnks-perl-5.34.0/bin/perl..# WARNING…
                    → /nix/store/y84babjpa0ngq62pqvygsajscx7mlnks-perl-5.34.0

Here's an issue discussing this: NixOS/nixpkgs#61580

We may want to consider prohibit the gem's derivation from depending on certain run-time dependencies like perl. This would make it less likely for this situation to occur in the future.

Work out how to reduce the image size

The image produced by the current setup weighs 340MB. This isn't terrible for a typical Rails image, but could be a lot smaller based on what's visible in the image.

Using dive we see a large 245MB layer at the bottom. This layer contains the source code, rubygems, and the rubygems dependencies. This means that every code change will require pushing a 245MB layer. Let's see how we can improve this.

Here are tickets describing specific options to explore:

#6
#8
#9 and #7
#10

Look at strategies to separate runtime vs. app in containerization layers

There might be inexpensive ways to make the layering more useful to end-users. Explore quickly and see if it's a viable approach.

(For #4)

Setup a machine VM for running Nix builds

Rather than trying to hack the builder VM to build with nix, we should setup a new machine running the official nix image. We can run builds over rsync/ssh instead of relying on Docker.

Write a basic documentation page for launching Rails apps

Add examples for all configurable options in rails-nix/default.nix

In preparation for presenting this project, and to simplify importing values from an app spec JSON file, such as:

a custom ruby version
adding an ffmpeg layer to the image with an entry in the Docker image config config.history.comment option mentioned here
changing the image CMD

Try strategies to reduce closure sizes in an ad-hoc manner

Short of fixing things upstream in Nixpkgs, are there (safe) tricks we can use to remove parts of the closures to reduce the size of the builds?

(Linked to #4)

Customize the layering strategy

From #4:

Currently, packages are split into layers based on how often they are referred to by other packages. But the reality for Rails is that most changes happen in gems and the source code, and not in the system dependencies. The source code and gems get stuck in that big final layer.

Ideally, we could customize layers the following can get their own layers:

individual gems + deps
compiled assets (output can be different without changes to Ruby code)
Ruby code (which can change without modifying the asset output)

Here's a related promising PR: NixOS/nixpkgs#122608

Make assets pre-compilation a discrete cacheable build and layer

By default, only compile assets if app/assets has changed. Otherwise, reuse the previous build's cache. If possible, make triggering if this step configurable by allowing changes in other paths to trigger it.

Extract nix code to be reusable

Ideally, the user's source tree doesn't need to have any .nix files in it. Their fly.toml should contain all the info Nix needs to run a build.

The build VM will mount a persistent volume for the Nix store and any other persistent data to live. Then, flyctl deploy will push the source code to the builder, and run a single build command there over SSH.

So some steps to make this happen:

Setup a separate repo for storing our nix code
Pin that repo to a specific nixpkgs version
Extract data from nix code to be imported from fly.toml
Write a script that fetches the nix repo into the remote source tree (or equivalent) and runs the build

Bundix does not support private gem sources

See nix-community/bundix#61

nixpkgs 21.11 is missing an important coreutils patch

Some common gems like rbtrace will fail to compile on arm64-darwin without this coreutils patch:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/coreutils/fix-arm64-macos.patch

So we should probably track unstable for now, perhaps?

grpc gem build failure

This gem changed its extconf.rb so the substitution here will no longer work. What should be the correct approach for fixing this?

[CXX]     Compiling src/core/lib/profiling/stap_timers.cc
mkdir -p `dirname /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.o`
clang++ -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -DGPR_BACKWARDS_COMPATIBILITY_MODE -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="\"RUBY\""  -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="\"1.42.0\""  -g -Wall -Wextra -DOSATOMIC_USE_INLINED=1 -Ithird_party/abseil-cpp -Ithird_party/re2 -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -O2 -Wframe-larger-than=16384 -fPIC -I. -Iinclude -I/nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/gens -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\" -arch arm64  -Ithird_party/zlib   -std=c++11 -stdlib=libc++   -fno-exceptions -MMD -MF /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.dep -c -o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/src/core/lib/profiling/stap_timers.o src/core/lib/profiling/stap_timers.cc
[AR]      Creating /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a
mkdir -p `dirname /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a`
rm -f /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a
libtool -o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/adler32.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/compress.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/crc32.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/deflate.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzclose.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzlib.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzread.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/gzwrite.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/infback.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inffast.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inflate.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/inftrees.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/trees.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/uncompr.o /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/objs/opt/third_party/zlib/zutil.o
make: libtool: No such file or directory
make: *** [Makefile:2503: /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0/src/ruby/ext/grpc/libs/opt/libz.a] Error 127
make: *** Waiting for unfinished jobs....
make: Leaving directory '/nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/nix/store/77bjl5185g1rxq0g7m868j5b7izvxanl-ruby-2.7.5/bin/$(RUBY_BASE_NAME)

extconf failed, exit code 1

Gem files will remain installed in /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0 for inspection.
Results logged to /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/grpc-1.42.0/gem_make.out
error: builder for '/nix/store/l8cgfwdl91g8gq9fgi7g22nnnwl56gd1-ruby2.7.5-grpc-1.42.0.drv' failed with exit code 1;
       last 10 log lines:
       >  --with-make-prog
       >       --without-make-prog
       >    --srcdir=.
       >     --curdir
       >       --ruby=/nix/store/77bjl5185g1rxq0g7m868j5b7izvxanl-ruby-2.7.5/bin/$(RUBY_BASE_NAME)
       >
       > extconf failed, exit code 1
       >
       > Gem files will remain installed in /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/gems/grpc-1.42.0 for inspection.
       > Results logged to /nix/store/6k1xk2b77fgqlbhmgfnyy7y6197gfmjh-ruby2.7.5-grpc-1.42.0/lib/ruby/gems/2.7.0/extensions/arm64-darwin-20/2.7.0/grpc-1.42.0/gem_make.out
       For full logs, run 'nix log /nix/store/l8cgfwdl91g8gq9fgi7g22nnnwl56gd1-ruby2.7.5-grpc-1.42.0.drv'.
error: 1 dependencies of derivation '/nix/store/6wlz359basx4nx83xkwbrb9jvcgmbq7c-development.drv' failed to build

Extend Rails documentation to include Sidekiq and multiprocess apps

Choosing a strategy for changing nixpkgs versions

The big plan, I think, is for Fly to have a 'main' nixpkgs pinned version. Whenever we decide to change this version, we'll kick off jobs that:

Build and cache our custom software like Ruby patch versions, postgres gem, etc
Run CI builds against real apps to ensure they build and deploy

Question: What's a good strategy for deciding when to move the main pin forward? Is there already a best practice out there?

Presumably, security patches and such are going into nixpkgs all the time. So there are benefits to frequently moving the pin. But it will be impossible to cover all the possible issues that may pop up.

Question: How should we manage versioning of our Nix builder code and nixpkgs?

It feels to me that these should be versioned separately.

For nixpkgs, certainly it's desirable to get security patches and other important software upgrades automatically. But this necessarily would change the contents of your build, slow down deploys, and possibly cause problems. So it makes sense to pin apps to a Nixpkgs version. But we'd also need to version our builder code as we improve it. Let's think about how this could work.

Builder images are semantically versioned, so we could continue with what we're doing now - release new builders with new builder code. New builders get new code, old ones can be upgraded. So, an app could in theory move forward in nixpkgs, but use the same builder code.

Maybe we store Nix pins in our backend so apps can compare and upgrade if they choose?

Increase the layer count

From #4:

More layers could help here, but we need to check that it works on Fly builders.

fly-apps / rails-nix Goto Github PK

rails-nix's Introduction

Building a Docker image

Deploying on Fly.io

Using Nix in Development

Notes

rails-nix's People

Contributors

Stargazers

Watchers

Forkers

rails-nix's Issues

Recommend Projects

Recommend Topics

Recommend Org