goodbyekansas / nedryland Goto Github PK

Call Package Pattern: https://nixos.org/guides/nix-pills/callpackage-design-pattern.html

Also make it overridable.

We noticed that the current solution for selecting cache folder doesn't work.

https://github.com/goodbyekansas/nedryland/blob/master/languages/rust/package.nix

First use home folder cache "$HOME/.cache" if available.
If that doesn't work take the var cache folder.
If that doesn't work we take the temp cache folder in /tmp.

We expected the last option to always work which is not the case. We need to rethink this and do it over again.

When we run rust with SCcache and build it will try to search for the cache in the home folder but the home variable is set to something we do not expect which means it won't find the cache.

Yo where the gh pages hosted at?
I see this 🚀Deploy to gh-pages at branch gh-pages but can't find the url to wherever 😩
ty fam

deny(warnings) is not great since it is controlled from code and also separate ones are needed for tests (they are their own crates). We should instead set this in nedryland through env vars for the RUST helpers.

We should definitely use niv for handling our pinned version of nixpkgs and mozilla overlay (and other future sources). That will make managing much much easier.

As discussed in #288 components should not give special meanings to attribute names.

One solution mentioned in the PR is:

We could do something like tagging everything created with the base.documentation functions (and add a base.documentation.includeInCombinedDocs function to tag custom derivations), and then do all the special stuff with everything in the component that has the tag. Then it wouldn't matter where in the component they were so it would be a bit flatter.

Another solution would be to add a config setting for what the docs attribute should be called.

We should probably also try to be more consistent about using the config for default target, right now we hardcode package in some places and use the config value (which defaults to package) in other.

Add a helper function that calls a provided closure for everything that is a Nedryland component, recursively

Today when we make a rust component one target is created for the host platform (package). If you want additional targets you will have to add those to the crossTargets attribute. This works ok as long as you always want one for host platform plus others.

We have a case in firm where a rust library is windows only where this does not work nicely. Currently hacked with the following code.

{ ... }
.overrideAttrs (oldAttrs: {
  rust = oldAttrs.windows;
  package = oldAttrs.windows;
})

The package attribute needs to be overridden since it will not build on linux.
The rust attribute because it is the same as package and is used in case the user wants to build all rust packages.

Need to change how the crossTargets works so we can use it without having to wrangle with overrides.

Random ideas:

• Just like for phases the rust components could have an array of all the targets it supports. In the windows specific case it would only contain windows.

Currently we use a setup where we have something like

| component/
|___ component.nix
|___ CHANGELOG.md
|___ README.md
|___ src/
|____ stuff

where the component.nix have

  src = ./.;

A change to the readme or changelog will force a rebuild of all the stuff. Maybe they should be excluded from the source of the package? Either the language components make an explicite exclude for them or we change the structure somewhat so the package derivation does not include them. Changing the code should rebuild the package and changing markdown files should rebuild the component with the old package.

One idea is also to have some standard files like CHANGELOG.md README.md and LICENSE to be special and they are always excluded by languages by default and also added as separate attributes on the component.

This would be more neat than our own home-grown filter with builtins.path.

Today we run mypy everywhere which needs a pyi stub to validate the types we generate.

The docs attribute targets does not have a proper shell, so for example shellCommands does not get put into nativeBuildInputs. We make the mk... docs functions into other functions (for overriding name and type) but this (probably) means they are filtered out when applying the shell attributes.

How about we add a base.mkDerivation? Which would be a wrapper of pkgs.stdenv.mkDerivation that makes the src invariant and filters everything gitignored. And we can add other things we consider good practice in nedryland. It would look something like this:

mkDerivation = attrs@{ src, stdenv ? pkgs.stdenv }: stdenv.mkDerivation (attrs // { src = gitIgnoredAndInvariant src; });

With this you can override stdenv when you need to cross compile.

This is dependent on us doing #143.

In our overlays we always use pytohn3 packages which will always be 3.7, 3.8, or 3.9.
We must make overlays for all python versions.

Python packages can also be reached by doing pkgs.python3packages.{package}, pkgs.python37packages.{package}, pkgs.python38packages.{package}. There is also pkgs.python3.pkgs.

Right now we only have overlay for pkgs.python3packages.{package}. This is a problem since the packages can be reached on other paths without the overlay.

Today we have some targets such as package, packageWithChecks, docs and so on. We have to enable to add additional targets from outside Nedryland. Add extraTargets to mkGrid.

When using base.mkDerivation, .override/.overrideAttrs is the one from stdenv.mkDerivation which means that any logic in base.mkDerivation is not run when overriding.

Currently, when a Nedryland project depends on another, Nedryland is imported once per project.

Don't see why not. It can be handy in cases where you want to make a deployment based on a project.

Make a big map with
target = { nixTarget=XXX; rustTarget=YYY; }
and use it to make cross compile targets automatically for rust components.

Today our dependencies looks like this in Cargo.toml.

firm-protocols = { path = "nix-deps/rust-firm-protocols" }

The nix user needs to know that rust dependencies are automatically linked to nix-deps/something.
This is not very nice. We can hide this by creating our own local registry.

We could possibly make it look like the following.

[dependencies]
firm-protocols = { registry = "nix" }

When we get rust dependencies (preferably through one of the standard nix dependency mechanisms) we create one derivation for all of them by a symlinkJoin. This will combine the derivations into a single tree of symlinks. This directory can then be used as a source for the nix registry by generating a .cargo/config.toml in the configure phase.

Sources:
https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-other-registries

https://nixos.org/manual/nixpkgs/stable/#chap-trivial-builders - symlinkJoin

We should set up nix-linter and shellcheck on this repo.

We have some utility leftovers. We must rename it to mkLibrary because utility is such a bad name.

This tried to address this:
af4a8a0
but it had the unintended consequence that checks could not be turned on.

The problem is that we can never hit the or case in drv.doCheck or false. A component that looks like this

{ stdenv }:
let
  pkg = stdenv.mkDerivation {
    name = "example";
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      mkdir $out
    '';
  };
{
    name = "example";
    package = builtins.trace (pkg.doCheck || pkg.drvAttrs.doCheck) pkg;
}

Will print
trace: false
Probably because of https://github.com/NixOS/nixpkgs/blob/826c20dcae34f7e3be4d1a638b07fb7d95570ba0/pkgs/stdenv/generic/make-derivation.nix#L201
and
https://github.com/NixOS/nixpkgs/blob/826c20dcae34f7e3be4d1a638b07fb7d95570ba0/pkgs/stdenv/generic/make-derivation.nix#L335

Random note: a comment in there mentions doCheck to be "old semantics" and is going away.

[erjo@3d119 pipeline]$ nix-shell -A ftrackServer
⛑ Changing dir to "/mnt/home/erjo/git/pipeline/infra/ftrack-server"
🐚 Running shell hook for "ftrack-server-terraform"
cp: overwrite '/tmp/tfvars.json'?

Everything seems to work as expected but a little annoying

Currently we build component.docs.api.target, if there's only one target it becomes component.docs.api. We should instead build the docs for all targets at once so that we get a drop down where the user can choose the platform.

Should be to combine all cross targets' target attribute into one flat list and generate docs for that.

In order for nedryland to be able to test itself it would be useful to add some "hello world" level components of each language.

They could also serve as example for how to make a project with components in an example folder.

Instead of having different derivations for checks/no checks, change to have the whole matrix evaluated with checks/no checks for all targets of all components.

Also, add a all target to components which is a list of all top-level derivation attributes on the component. This can be used as a target in CI: nix build all after overriding the matrix with check = true.

Change shells to use the check variant of the matrix, making shells available for all targets on all components.

This must remove the special handling of package we do currently (to the extent of making it even optional).

Add author info on project level which python and rust can use

Utility is a terrible name. We must change it to Library.

It should say base.languages.python.mkLibrary

Change target setup to use a setup hook and to be a shell input.

Using vendoring in shells leads to a horrible developer experience when adding external dependencies.

Instead we take the hit on nix-build or the CI, which is less often.

If a Python component does not have a default setup.py we should generate one from the info in the component together with some assumptions/conventions.

E.g.

Library: Just use the name and version of the component together with find_packages
Application: Assume a console script entrypoint with the same name as the component pointing to main.py:main in a package with the same name as the component (so component_name.main:main).

shells.nix should not use mkShell, it does more harm than good.

Also there is an assumption that componentAttrs exists:

error: attribute 'componentAttrs' missing

       at /nix/store/07wzwpl5zc8qvfmhxkzlhhvjf4qa19j7-source/shells.nix:116:98:

          115|               )
          116|               (lib.filterAttrs (_n: t: lib.isDerivation t && !(t.isNedrylandComponent or false)) component.componentAttrs);
             |                                                                                                  ^
          117|

Instead of using our own home grown monster in languages/rust/package.nix.

No good reason as far as I can see to not have it.