nix-community / dream2nix Goto Github PK

Simplified nix packaging for various programming language ecosystems [maintainer=@DavHau]

Home Page: https://nix-community.github.io/dream2nix

License: MIT License

Nix 85.28% Python 11.29% XSLT 0.33% Shell 0.21% jq 0.44% CSS 0.24% JavaScript 0.91% TypeScript 1.30%

2nix dream-to-nix hacktoberfest lang2nix nix nix-community-buildbot nixos nodejs package-management reproducibility rust

dream2nix's People

Contributors

Stargazers

Watchers

Forkers

matthewcroughan happysalada l-as raghuramlavan ink-pot-monkey humancalico zachcoyle 06kellyjac mic92 0x4a6f yusdacra lassulus tomberek max-privatevoid asymmetric dieracdelta raboof tgunnoe nixinator zimbatm ryantm kylegentle william3johnson lelongg jaen zarak nrdxp olafklingt pombredanne ners aciceri blaggacao wmertens lelit a-kenji kranzes leungbk tinybeachthor ctem ngi-nix m1-s fresheyeball bezmuth caguiclajmg fufexan natsukium srhb hsjobeki impl dark-ether netpalm555 phaer antotocar34 chaoflow brainrake doronbehar hercules-ci gtrunsec zainab-ali oati pveierland moke12g alejandrosame waalge denibertovic drzln adomixaszvers rdmolony p42s-de dnswd peel jamesbucher tylerjl mangoiv socios-linux phfroidmont lainera atry t3rro akiiino ligoldragon irisjae florianfranzen niknetniko belevy romatthe mrcjkb amundhov shamrocklee 71 danlipsitt srid inlaw aszenz xyven1 moduon teto virtual-architect vincentvanlaer ryangibb

dream2nix's Issues

missing peerDependencies

here is the problem I encountered.
While trying to build this project https://github.com/happysalada/primo

building

> [email protected] build /nix/store/15q09sxbbgbca9l53cjvlg0arv5k88li-primo-server-1.2.6/lib/node_module>
> svelte-kit build

> Cannot find package 'svelte' imported from /nix/store/1h2ri6f620r47vqg54bv3iamyf360z9y-__at__sveltejs__>
Did you mean to import svelte-3.47.0/lib/node_modules/svelte/compiler.js?
Did you mean to import svelte-3.47.0/lib/node_modules/svelte/compiler.js?
    at new NodeError (internal/errors.js:322:7)

the problematic package is

    "node_modules/@sveltejs/vite-plugin-svelte": {
      "version": "1.0.0-next.41",
      "resolved": "https://registry.npmjs.org/@sveltejs/vite-plugin-svelte/-/vite-plugin-svelte-1.0.0-next.41.tgz",
      "integrity": "sha512-2kZ49mpi/YW1PIPvKaJNSSwIFgmw9QUf1+yaNa4U8yJD6AsfSHXAU3goscWbi1jfWnSg2PhvwAf+bvLCdp2F9g==",
      "dependencies": {
        "@rollup/pluginutils": "^4.2.0",
        "debug": "^4.3.4",
        "kleur": "^4.1.4",
        "magic-string": "^0.26.1",
        "svelte-hmr": "^0.14.11"
      },
      "engines": {
        "node": "^14.13.1 || >= 16"
      },
      "peerDependencies": {
        "diff-match-patch": "^1.0.5",
        "svelte": "^3.44.0",
        "vite": "^2.9.0"
      },
      "peerDependenciesMeta": {
        "diff-match-patch": {
          "optional": true
        }
      }
    },

svelte is in the peerDependencies. It looks like at the moment those are not installed in the node_module of the related dependency.
when I check dependencies installed for that package, I only see

lrwxr-xr-x    98 root 31 Dec  1969  sourcemap-codec -> /nix/store/3niwynnjsys1rizrin9nxxr10gbc1sar-sourcemap-codec-1.4.8/lib/node_modules/sourcemap-codec
dr-xr-xr-x     - root 31 Dec  1969  @rollup
lrwxr-xr-x    94 root 31 Dec  1969  estree-walker -> /nix/store/i8vq4vxa5igzlkp1z3w58rra9v3zyn7f-estree-walker-2.0.2/lib/node_modules/estree-walker
lrwxr-xr-x    93 root 31 Dec  1969  magic-string -> /nix/store/9sfjl8ixrd5sh0yks76y4r745l6xhm93-magic-string-0.25.7/lib/node_modules/magic-string
lrwxr-xr-x    89 root 31 Dec  1969  svelte-hmr -> /nix/store/y4vb39bby7rvnrvlsfg3a543vhfq7y5y-svelte-hmr-0.14.9/lib/node_modules/svelte-hmr
lrwxr-xr-x    86 root 31 Dec  1969  picomatch -> /nix/store/iqmv2i7ps8mx3x0yzraywcbm5pw4di6w-picomatch-2.3.1/lib/node_modules/picomatch
lrwxr-xr-x    78 root 31 Dec  1969  kleur -> /nix/store/kda75cwi7rn1ccb5f2ll6368389ar3bq-kleur-4.1.4/lib/node_modules/kleur
lrwxr-xr-x    78 root 31 Dec  1969  debug -> /nix/store/b878bb1lc5js7sxvzpdxhfq9qzlmw154-debug-4.3.3/lib/node_modules/debug
lrwxr-xr-x    72 root 31 Dec  1969  ms -> /nix/store/76wsm74kb9v1bxxnn848jbdpk0y0q5rj-ms-2.1.2/lib/node_modules/ms

most probably we should add peerdependencies as well if possible.

Setting `pkgs` in lib.init has no effect

Setting the pkgs argument in lib.init does not seem to impact the build.

Failed to package strapi

after checking out dreampkgs I ran the following inside the repo
commands:

nix run github:davhau/dream2nix add github:strapi/strapi/v4.0.2
nix run .#strapi

error:

warning: Git tree '/home/lass/src/dreampkgs' is dirty
error: builder for '/nix/store/bvscf68p8f1467z188zji5fw7yc858wi-strapi-unknown.drv' failed with exit code 1;
       last 8 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/6i2ljff2rgkxwm9qfshv4s3hfblxgdjw-source
       > patching sources
       > d2nPatchPhase
       > Traceback (most recent call last):
       >   File "/nix/store/skybcqxgqssjn28naxd48p5h42ngsbji-fix-package.py", line 34, in <module>
       >     if version not in ["unknown", package_json['version']]:
       > KeyError: 'version'
       For full logs, run 'nix log /nix/store/bvscf68p8f1467z188zji5fw7yc858wi-strapi-unknown.drv'.

I couldn't find a way to run dream2nix directly without dreampkg. If someone could tell me how to do that I could try a more recent version of dream2nix probably?

Improve and document dependency injection API

see: #117 (comment)

discoverer: purify paths to subpackages

I've mentioned this before:

When dream2nix parses monorepos using its discoverer mechanism, It takes the whole monorepo in as an input. There's a common situation where sub packages will have dependencies of other subpackages--many times a tree of dependencies extending from several packages around the repo. If I make a change in a root dependency sub package (eg coreutils) certainly all of its reverse dependencies should be rebuilt (as per the nix hash tree). However, dependencies should not need to be rebuilt when their reverse dependencies are updated in the monorepo.

Hope that makes sense. I think the discoverer needs to filter out the rest of the repo? It would greatly reduce build times for sub packages that have a chain of dependencies on other sub packages around the monorepo.

Has this similiarities with fetlock ?

Just stumbled over https://github.com/timbertson/fetlock

NodeJS packages do not follow the Flake output schema

Nix expects flakes to export their packages as packages.«system».«name», but dream2nix exports them as packages.«system».«name».«version»:

https://github.com/DavHau/dream2nix/blob/47287126d8ea4c5b586f38e3044f31af7578cccb/src/builders/nodejs/node2nix/default.nix#L117

This causes the nix flake show command to fail with error: expected a derivation.

simple rust package fails to build

Building a rust package with a dependency fails:

error: builder for '/nix/store/nzb7yln59r767bx7avrdpyky85g724p2-dream2nix-bug-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Executing cargoSetupPostUnpackHook
       > Finished cargoSetupPostUnpackHook
       > patching sources
       > configuring
       > building
       > Executing cargoBuildHook
       > ++ env CC_x86_64-unknown-linux-gnu=/nix/store/pja9g36cy32z3d51942jqk91a6l2d5nv-gcc-wrapper-10.3.0/bin/cc CXX_x86_64-unknown-linux-gnu=/nix/store/pja9g36cy32z3d51942jqk91a6l2d5nv-gcc-wrapper-10.3.0/bin/c++ CC_x86_64-unknown-linux-gnu=/nix/store/pja9g36cy32z3d51942jqk91a6l2d5nv-gcc-wrapper-10.3.0/bin/cc CXX_x86_64-unknown-linux-gnu=/nix/store/pja9g36cy32z3d51942jqk91a6l2d5nv-gcc-wrapper-10.3.0/bin/c++ cargo build -j 12 --target x86_64-unknown-linux-gnu --frozen --release --package dream2nix-bug
       > error: no matching package named `chrono` found
       > location searched: registry `crates-io`
       > required by package `dream2nix-bug v0.1.0 (/build/4n4yrgfjhhw8ldcqrv8w08gdnr948w0q-source)`
       For full logs, run 'nix log /nix/store/nzb7yln59r767bx7avrdpyky85g724p2-dream2nix-bug-0.1.0.drv'.

Repo for repro: https://github.com/lelongg/dream2nix-bug

Rust builds - referencing path dependencies outside of a workspace.

Hi!

While playing around using dream2nix for our monorepo, I ran into some trouble with crate dependencies that aren't in a workspace.

The monorepo contains projects in many languages, and we currently using nix for

rust builds - https://github.com/cargo2nix/cargo2nix
python builds - poetry2nix
The monorepo directory structure is something like:

root
   |
   | service
         |
         | -- > serviceA/Cargo.toml // imports ../../lib/libraryA
         | -- > serviceB/Cargo.toml // imports libraryA, but will never import serviceA
         | -- > pythonServiceA/pyproject.toml // imports pythonLibraryA
   | lib
         | -- > libraryA/Cargo.toml // may import libraryB
         | -- > libraryB/Cargo.toml
         | -- > pythonLibraryA/pyproject.toml

The python builds work well - the overlay being:

        overlay = nixpkgs.lib.composeManyExtensions [
          poetry2nix.overlay
          (final: prev: {
            devEnv = prev.poetry2nix.mkPoetryEnv {
              python = final.pkgs.python38;
              projectDir = ./.;
              overrides = prev.poetry2nix.overrides.withDefaults
                (import ../../tools/nix/python-nix-overlay.nix);
            };
          })
        ];

Cargo2nix also works well - but keeping the Cargo.nix files in sync with Cargo.lock files is somewhat of a hassle.

When trying to build a service using dream2nix it would fail with
error: could not find crate libraryA

The flake looks like

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=release-21.11";
    dream2nix.url = "github:nix-community/dream2nix";
    dream2nix.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, dream2nix, ... }@inputs:
    let
      dream2nix = inputs.dream2nix.lib.init {
        # modify according to your supported systems
        systems = [ "x86_64-linux" "aarch64-darwin" ];
        config.projectRoot = ./.;
      };
    in dream2nix.makeFlakeOutputs {
      source = ./.;
    };
}

I also tried changing config.projectRoot to ../.. and using:

      sourceOverrides = oldSources: {
        capture_storage_client = builtins.path {
          path = ../../lib/libraryA;
        };
      };

But since poetry2nix manages to work around this I reckoned that the rust builder can do it as well 😄

Thanks!

Run dream2nix on lang / package manager ecosystems

We should have a tool to run dream2nix on language / package manager ecosystems we support. Essentially https://github.com/rust-lang/crater but for dream2nix. This would help us keep track of how many projects fail to build, how many succeed and have numbers for these across time.

As for how the tool would work; considering not every translator is pure in dream2nix (and probably won't be), it makes most sense for the tool to fetch ecosystem package information, then run dream2nix CLI on them to generate dream-lock.json files, and then build. This way we can also have the tool "snapshot the world", keep these in a git repository or just archive & compress. The tool can afterwards parse the build output of Nix and figure out which ones failed with what error and write these in a machine readable format.

Add extensive example flake to readme using all features

see #112 (comment)

nodejs monorepo: error: value is a set while a list was expected

while testing out libv2 on a yarn monorepo, I encounter the error error: value is a set while a list was expected

simple entry flake to reproduce:

  {
    inputs.dream2nix.url = "github:nix-community/dream2nix";
    inputs.optimism.url = `"github:ethereum-optimism/optimism";`
    inputs.optimism.flake = false;
    outputs = { self, dream2nix, optimism }@inputs:
      let
        dream2nix = inputs.dream2nix.lib2.init {
          # modify according to your supported systems
          systems = [ "x86_64-linux" ];
          config.projectRoot = ./. ;
        };
      in dream2nix.makeFlakeOutputs {
        source = optimism;
        pname = "optimism";
      };
  }

error: value is a set while a list was expected

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:35:20:

           34|   */
           35|   forEach = xs: f: map f xs;
             |                    ^
           36|
(use '--show-trace' to show detailed location information)
❯ nix flake show --show-trace
Found existing alias for "nix". You should use: "n"
warning: Git tree '/home/tgunnoe/src/boba/test' is dirty
git+file:///home/tgunnoe/src/boba/test
error: value is a set while a list was expected

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:35:20:

           34|   */
           35|   forEach = xs: f: map f xs;
             |                    ^
           36|

       … while evaluating 'forEach'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:35:17:

           34|   */
           35|   forEach = xs: f: map f xs;
             |                 ^
           36|

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:56:10:

           55|       l.flatten
           56|         (l.forEach (packageJson.workspaces or [])
             |          ^
           57|           (glob:

       … while evaluating 'flatten'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:137:13:

          136|   */
          137|   flatten = x:
             |             ^
          138|     if isList x

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:55:7:

           54|     in
           55|       l.flatten
             |       ^
           56|         (l.forEach (packageJson.workspaces or [])

       … while evaluating 'getWorkspaces'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:51:25:

           50|   # collect project info for workspaces defined by current package.json
           51|   getWorkspaces = tree: parentInfo:
             |                         ^
           52|     let

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:119:22:

          118|
          119|         workspaces = getWorkspaces tree currentProjectInfo;
             |                      ^
          120|

       … while evaluating 'discoverInternal'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:81:5:

           80|   discoverInternal =
           81|     {
             |     ^
           82|       tree,

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:15:5:

           14|     }:
           15|     discoverInternal {
             |     ^
           16|       inherit tree;

       … while evaluating 'discover'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/nodejs/default.nix:12:5:

           11|   discover =
           12|     {
             |     ^
           13|       tree,

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/default.nix:23:24:

           22|         (l.map
           23|           (discoverer: discoverer.discover { inherit tree; })
             |                        ^
           24|           allDiscoverers);

       … while evaluating anonymous lambda

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/default.nix:23:12:

           22|         (l.map
           23|           (discoverer: discoverer.discover { inherit tree; })
             |            ^
           24|           allDiscoverers);

       … from call site

       … while evaluating 'flatten'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:137:13:

          136|   */
          137|   flatten = x:
             |             ^
          138|     if isList x

       … from call site

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:139:24:

          138|     if isList x
          139|     then concatMap (y: flatten y) x
             |                        ^
          140|     else [x];

       … while evaluating anonymous lambda

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:139:21:

          138|     if isList x
          139|     then concatMap (y: flatten y) x
             |                     ^
          140|     else [x];

       … from call site

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:139:10:

          138|     if isList x
          139|     then concatMap (y: flatten y) x
             |          ^
          140|     else [x];

       … while evaluating 'flatten'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/lists.nix:137:13:

          136|   */
          137|   flatten = x:
             |             ^
          138|     if isList x

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/default.nix:21:7:

           20|     }:
           21|       l.flatten
             |       ^
           22|         (l.map

       … while evaluating 'discoverProjects'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/discoverers/default.nix:17:5:

           16|   discoverProjects =
           17|     {
             |     ^
           18|       source ? throw "Pass either `source` or `tree` to discoverProjects",

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/default.nix:501:28:

          500|       flakeMode = ! builtins ? currentSystem;
          501|       discoveredProjects = dlib.discoverers.discoverProjects { inherit tree; };
             |                            ^
          502|

       … while evaluating 'resolveProjectsFromSource'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/default.nix:491:5:

          490|   resolveProjectsFromSource =
          491|     {
             |     ^
          492|       source ?

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:139:28:

          138|
          139|               dreamLocks = dream2nix.resolveProjectsFromSource {
             |                            ^
          140|                 inherit pname source;

       … while evaluating 'realizeProjects'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/default.nix:631:5:

          630|   realizeProjects =
          631|     {
             |     ^
          632|       dreamLocks ? resolveProjectsFromSource { inherit pname source; },

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:143:15:

          142|             in
          143|               dream2nix.realizeProjects {
             |               ^
          144|                 inherit dreamLocks source;

       … while evaluating anonymous lambda

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:135:20:

          134|         lib.mapAttrs
          135|           (system: pkgs:
             |                    ^
          136|             let

       … from call site

       … while evaluating 'optionalAttrs'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:356:25:

          355|   */
          356|   optionalAttrs = cond: as: if cond then as else {};
             |                         ^
          357|

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:61:8:

           60|     flakifyBuilderOutputs = system: outputs:
           61|       (lib.optionalAttrs (outputs ? "defaultPackage") {
             |        ^
           62|         defaultPackage."${system}" = outputs.defaultPackage;

       … while evaluating 'flakifyBuilderOutputs'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:60:37:

           59|
           60|     flakifyBuilderOutputs = system: outputs:
             |                                     ^
           61|       (lib.optionalAttrs (outputs ? "defaultPackage") {

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:150:29:

          149|         lib.mapAttrsToList
          150|           (system: outputs: flakifyBuilderOutputs system outputs)
             |                             ^
          151|           allBuilderOutputs;

       … while evaluating anonymous lambda

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:150:20:

          149|         lib.mapAttrsToList
          150|           (system: outputs: flakifyBuilderOutputs system outputs)
             |                    ^
          151|           allBuilderOutputs;

       … from call site

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:259:16:

          258|   mapAttrsToList = f: attrs:
          259|     map (name: f name attrs.${name}) (attrNames attrs);
             |                ^
          260|

       … while evaluating anonymous lambda

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:259:10:

          258|   mapAttrsToList = f: attrs:
          259|     map (name: f name attrs.${name}) (attrNames attrs);
             |          ^
          260|

       … from call site

       … while invoking 'zipAttrsWith'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:429:8:

          428|       );
          429|     in f [] [rhs lhs];
             |        ^
          430|

       … while evaluating 'recursiveUpdateUntil'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:419:37:

          418|      */
          419|   recursiveUpdateUntil = pred: lhs: rhs:
             |                                     ^
          420|     let f = attrPath:

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:155:32:

          154|         b.foldl'
          155|           (allOutputs: output: lib.recursiveUpdate allOutputs output)
             |                                ^
          156|           {}

       … while evaluating anonymous lambda

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:155:24:

          154|         b.foldl'
          155|           (allOutputs: output: lib.recursiveUpdate allOutputs output)
             |                        ^
          156|           {}

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:154:9:

          153|       flakeOutputs =
          154|         b.foldl'
             |         ^
          155|           (allOutputs: output: lib.recursiveUpdate allOutputs output)

       … while invoking 'zipAttrsWith'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:429:8:

          428|       );
          429|     in f [] [rhs lhs];
             |        ^
          430|

       … while evaluating 'recursiveUpdateUntil'

       at /nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source/lib/attrsets.nix:419:37:

          418|      */
          419|   recursiveUpdateUntil = pred: lhs: rhs:
             |                                     ^
          420|     let f = attrPath:

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:160:7:

          159|     in
          160|       lib.recursiveUpdate
             |       ^
          161|         flakeOutputs

       … while evaluating 'makeFlakeOutputsFunc'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:116:5:

          115|   makeFlakeOutputsFunc =
          116|     {
             |     ^
          117|       pname ? throw "Please pass `pname` to makeFlakeOutputs",

       … from call site

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:97:35:

           96|
           97|         makeFlakeOutputs = mArgs: makeFlakeOutputsFunc
             |                                   ^
           98|           (

       … while evaluating 'makeFlakeOutputs'

       at /nix/store/9iqiawm83jgsv9528p9qhi082pbi0m2m-source/src/libV2.nix:97:28:

           96|
           97|         makeFlakeOutputs = mArgs: makeFlakeOutputsFunc
             |                            ^
           98|           (

       … from call site

       at /nix/store/nc84p2b0fnjv6m9rdq54llj437wm4whf-source/flake.nix:12:8:

           11|       };
           12|     in dream2nix.makeFlakeOutputs {
             |        ^
           13|       source = optimism;

       … while evaluating 'outputs'

       at /nix/store/nc84p2b0fnjv6m9rdq54llj437wm4whf-source/flake.nix:5:13:

            4|   inputs.optimism.flake = false;
            5|   outputs = { self, dream2nix, optimism }@inputs:
             |             ^
            6|     let

       … from call site

       at «string»:45:21:

           44|
           45|           outputs = flake.outputs (inputs // { self = result; });
             |                     ^
           46|

       … while evaluating anonymous lambda

       at «string»:10:13:

            9|     builtins.mapAttrs
           10|       (key: node:
             |             ^
           11|         let

       … from call site

       … while evaluating anonymous lambda

       at «string»:2:23:

            1|
            2| lockFileStr: rootSrc: rootSubdir:
             |                       ^
            3|

       … from call site

Yarn translator cannot handle "uid" field in dependencies

Same file as #73.
https://github.com/TryGhost/Admin/blob/b872df2e0eee3ca07c5717a36ef1f5fa0cc97c1d/yarn.lock#L6931

$ nix run github:max-privatevoid/dream2nix/yarn-lock-empty-quoted-strings -- \
    add github:TryGhost/Admin/b872df2e0eee3ca07c5717a36ef1f5fa0cc97c1d

(...)
Translating project metadata
error: parser failed at:
         uid da580474a2c449b715444934ddb626b7c07f46a7
         r


       … while evaluating 'mapAttrsToList'

       at /nix/store/ygz58w4f2hgnrdwa657k7gkx3ajd6d8r-bs58z9v8mfd6hxh9zwhsmyzlmiamhqjj-p53cz6rh27q40g9i0q98k3vfrz6lm12w-source/lib/attrsets.nix:258:23:

          257|   */
          258|   mapAttrsToList = f: attrs:
             |                       ^
          259|     map (name: f name attrs.${name}) (attrNames attrs);

       … from call site

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/translators/nodejs/pure/yarn-lock/default.nix:104:11:

          103|         serializePackages = inputData:
          104|           lib.mapAttrsToList
             |           ^
          105|             (yarnName: depAttrs: depAttrs // { inherit yarnName; })

       … while evaluating 'serializePackages'

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/translators/nodejs/pure/yarn-lock/default.nix:103:29:

          102|
          103|         serializePackages = inputData:
             |                             ^
          104|           lib.mapAttrsToList

       … from call site

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/translator.nix:43:32:

           42|
           43|       serializedPackagesList = final.serializePackages final.inputData;
             |                                ^
           44|

       … while evaluating 'findCycles'

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/translator.nix:167:45:

          166|             let
          167|               findCycles = node: prevNodes: cycles:
             |                                             ^
          168|                 let

       … from call site

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/translator.nix:202:17:

          201|               cyclesList =
          202|                 findCycles
             |                 ^
          203|                   (utils.nameVersionPair mainPackageName mainPackageVersion)

       … while evaluating the attribute 'cyclicDependencies'

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/translator.nix:252:20:

          251|
          252|             inherit cyclicDependencies;
             |                    ^
          253|

       … while evaluating 'compressDependencyGraph'

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/dream-lock.nix:183:31:

          182|
          183|     compressDependencyGraph = decompGraph:
             |                               ^
          184|       lib.mapAttrs

       … from call site

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/dream-lock.nix:226:11:

          225|         cyclicDependencies =
          226|           compressDependencyGraph
             |           ^
          227|             uncomp.cyclicDependencies;

       … while evaluating 'toJSON'

       at /nix/store/hi9lkvpv3hdhrvnr2kl3anx46wh3g65w-1pf3i6vyis07cmghq9w0psi6hmwzc12j-dream2nix-full-src/utils/dream-lock.nix:244:14:

          243|
          244|     toJSON = dreamLock:
             |              ^
          245|       let

       … from call site

       at «string»:8:7:

            7|     in
            8|       dream2nix.utils.dreamLock.toJSON
             |       ^
            9|         # don't use nix to detect cycles, this will be more efficient in python

  Exception

  Translator failed to create dream-lock.json

  at /nix/store/ig5if3b8b7rq70x2z21nvb2k347fdlyj-cli/commands/add.py:476 in run_translate
      472│         )
      473│ 
      474│       # raise error if output wasn't produced
      475│       if not output_temp_file.read():
    → 476│         raise Exception(f"Translator failed to create dream-lock.json")
      477│ 
      478│       # read produced lock file
      479│       with open(output_temp_file.name) as f:
      480│         lock = json.load(f)

pure yarn translator: error: attribute missing

There's several quirks with dependency+version lookups in sets on the nodejs side. I found another:

error: attribute 'ethers@^5' missing

       at /nix/store/s37pa5yrkb5s8rdkn4rwl1bxx5fbsjcz-source/src/translators/nodejs/pure/yarn-lock/default.nix:63:25:

           62|             yarnName = "${depName}@${semVer}";
           63|             depObject = objectsByKey.yarnName."${yarnName}";

In this case, this happens because ethers@^5 is specified as a peerDependencies in package.json, and yarn resolves that to a specific 5.x.x version in yarn.lock. And when peerDependencies is processed in dream2nix (https://github.com/nix-community/dream2nix/blob/main/src/translators/nodejs/utils.nix#L8), it does not take the resolved package number. So ethers@^5 is not found when a lookup is attempted.

A similar issue is here #104

nodejs monorepo: .."packages/common/yarn.lock': No such file or directory"

It seems that in a nodejs monorepo that dream2nix searches for a yarn.lock file in the packages subdirectory, which should never exist in a monorepo (as the yarn.lock file should exist at the root of the monorepo instead)

Test flake:

  {
    inputs.dream2nix.url = "github:nix-community/dream2nix";
    inputs.hardhat.url = "github:nomiclabs/hardhat";
    inputs.hardhat.flake = false;
    outputs = { self, dream2nix, hardhat }@inputs:
      let
        dream2nix = inputs.dream2nix.lib2.init {
          # modify according to your supported systems
          systems = [ "x86_64-linux" ];
          config.projectRoot = ./. ;
        };
      in dream2nix.makeFlakeOutputs {
        source = hardhat;
        pname = "hardhat";
      };
  }

attribute 'result' missing

full error message is

error: attribute 'result' missing

       at /nix/store/vvcpa8awapq19wp3gix21hlrw4fwjz03-source/src/discoverers/default.nix:32:7:

           31|     in
           32|       toposorted.result;
             |       ^
           33|

I tried this on a python repo https://github.com/OpenBB-finance/OpenBBTerminal
happy to test stuff if you want.

feat: let users be able to specify a `ref` alongside a `rev` for the `git` fetcher

This is good for not using allRefs, which can slowdown the fetch if there are many refs. Translators that have access to the ref alongside a rev can use this to not require allRefs usage (for example, Cargo.lock files can include a branch or a tag for git dependencies, alongside the usual git revision).

dream2nix fully based on nixos modules?

related:
#155
#151
#153

This is a hypothetical usage example of dream2nix if the framework was fully based on the nixos module system.

This is not only useful for configuring nixos systems. Even if the intention is to just create single packages or a dev shell, dream2nix could still be invoked via a module definition and benefit from properties of the nixos module system.

Benefits I see:

Discoverable packages and package options (all derivation inputs configurable)
Plugin interface. Translators, Builders, etc. can be added by setting options
Simply make many other parts of the framework configurable via options.
Type checked interfaces

dream2nix-ripgrep.nix:

{

  imports = [
    (builtins.getFlake "github:nix-community/dream2nix").module
  ];

  # cache directory for dream-lock files
  dream2nix.packagesDir = ./dream2nix-packages;

  # extend the framework with a new translator
  drem2nix.translators.some-new-translator = ...;

  # define projects for which to generate packages
  /*
    interface like:
    dream2nix.projects.<name> = {};

    As a result, top level packages of that project will be exposed via:
    `dream2nix.projects.<name>.packages`

    And dependencies can be overridden/configured via
    `dream2nix.projects.<name>.dependencies.<depName>.`
  */
  dream2nix.projects.ripgrep = {
    src = builtins.fetchGit ...;
    settings = [
      {
        builder = "crane";
        translator = "some-new-translator";
      }
    ];
  };

  # configure package on a per project basis
  # replaces `packageOverrides`
  dream2nix.projects.ripgrep.packages.grep = {
    preBuild = ''
      echo "user defined override for grep"
    '';
  };

  # global package overrides
  # (will usually be shipped by dream2nix upstream)
  dream2nix.overrides.rust.grep.fix-build = {
    _condition = ...;
    preBuild = ''
      echo "default dream2nix override for grep"
    '';
  };

  # For example, if the intention is to configure a nixos system,
  # one could just add packages from dream2nix to the systemPackages like that
  environment.systemPackages = [
    # add the ripgrep package from dream2nix
    config.dream2nix.projects.ripgrep.packages.default
    # add the grep package as well
    config.dream2nix.projects.ripgrep.dependencies.grep.package
  ];
}

For usage with simple nix expressions, outside of a nixos configuration, a utility function could be defined, taking care of calling evalModules correctly.
This for example could be used in a flake:
(./dream2nix-ripgrep.nix corresponds to the file above)

  let
    evaluated = callDream2nix ./dream2nix-module.nix;
  in
    evaluated.projects.ripgrep.packages.default;

@yusdacra @jaen

nocodb: ts-loader failing dependency

trying to build nocodb, I get this at the end of the build (after all the overrides).

Hash: 88d3eb52fb7243dc0e86
Version: webpack 4.46.0
Time: 1375ms
Built at: 04/23/2022 11:52:54 PM
 1 asset
Entrypoint main = bundle.js
[0] ./src/lib/noco/Noco.ts 349 bytes {0} [built] [failed] [1 error]
[1] ./src/lib/noco/nc.try.ts 375 bytes {0} [built] [failed] [1 error]
[2] ./src/lib/utils/NcConfigFactory.ts 380 bytes {0} [built] [failed] [1 error]
[3] ./src/lib/index.ts 1.04 KiB {0} [built]

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value. Set 'mode' opti>
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configu>

ERROR in ./src/lib/utils/NcConfigFactory.ts 36:29
Module parse failed: Unexpected token (36:29)
You may need an appropriate loader to handle this file type, currently no loaders are configured to proce>
| };

it looks like something isn't finding properly ts-loader. Not sure I have to say.
I'm posting it here in case you've seen something similar. I think we ran into this once.

injecting a dependency does not add proper dependencies/sources for injection

original error:

error: attribute 'prom-client' missing

       at /nix/store/aamp3d520rnanjbvcx3nwcs1b2d0588f-source/src/builders/nodejs/granular/default.nix:158:13:

          157|       deps
          158|       (dep: allPackages."${dep.name}"."${dep.version}");
             |             ^
          159|

simple flake that injects a dependency for yarn pure:

  {
    inputs.dream2nix.url = "github:nix-community/dream2nix";
    outputs = { self, dream2nix }@inputs:
      let
        dream2nix = inputs.dream2nix.lib.init {
          systems = [ "x86_64-linux" ];
          config.projectRoot = ./. ;
        };
      in dream2nix.makeFlakeOutputs {
        source = ./.;
        inject = {
          express-prom-bundle."6.4.1" = [
            ["prom-client" "13.1.0"]
          ];
        };
      };
  }

package.json:

{
  "name": "test2",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "express-prom-bundle": "^6.4.1"
  }
}

result:

  {
    _generic = {
      defaultPackage = "test2";
      location = "";
      packages = {
        test2 = "1.0.0";
      };
      sourcesAggregatedHash = null;
      subsystem = "nodejs";
    };
    _subsystem = { nodejsVersion = "14"; };
    cyclicDependencies = { };
    decompressed = true;
    dependencies = {
      ee-first = { "1.1.1" = [ ]; };
      express-prom-bundle = {
        "6.4.1" = [
          { name = "on-finished"; version = "2.4.1"; }
          { name = "url-value-parser"; version = "2.1.0"; }
          { name = "prom-client"; version = "13.1.0"; }
        ];
      };
      on-finished = {
        "2.4.1" = [
          { name = "ee-first"; version = "1.1.1"; }
        ];
      };
      test2 = {
        "1.0.0" = [
          { name = "express-prom-bundle"; version = "6.4.1"; }
        ];
      };
      url-value-parser = {
        "2.1.0" = [ ];
      };
    };
    sources = {
      ee-first = {
        "1.1.1" = {
          hash = "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0=";
          type = "http";
          url = "https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz";
        };
      };
      express-prom-bundle = {
        "6.4.1" = {
          hash = "sha512-Sg0svLQe/SS5z1tHDTVfZVjNumobiDlXM0jmemt5Dm9K6BX8z9yCwEr93zbko6fNMR4zKav77iPfxUWi6gAjN  A==";
          type = "http";
          url = "https://registry.yarnpkg.com/express-prom-bundle/-/express-prom-bundle-6.4.1.tgz";
        };
      };
      on-finished = {
        "2.4.1" = {
          hash = "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+k  g==";
          type = "http";
          url = "https://registry.yarnpkg.com/on-finished/-/on-finished-2.4.1.tgz";
        };
      };
      test2 = {
        "1.0.0" = {
          path = "/nix/store/b4zlnjwd3aaph1xav090bd04zr60f23m-crh0vxqmg4zd3bj1i2g0cyc73jg9g2kg-source";
          type = "path";
        };
      };
      url-value-parser = {
        "2.1.0" = {
          hash = "sha512-gIYPWXujdUdwd/9TGCHTf5Vvgw6lOxjE5Q/k+7WNByYyS0vW5WX0k+xuVlhvPq6gRNhzXVv/ezC+OfeAet5Kc  w==";
          type = "http";
          url = "https://registry.yarnpkg.com/url-value-parser/-/url-value-parser-2.1.0.tgz";
        };
      };
    };
  }

rust: building packages from workspaces is now broken

dream2nix will fail to build any package in a workspace and output a "stack overflow (possible infinite recursion)" error. repo that showcases the issue: https://github.com/yusdacra/dream2nix-rust-discoverer-bug. The repo also has a working branch that uses an old commit before lib2 was renamed to lib, and it works fine; so I believe the issue is with the Rust discoverer?

translator creates illegal store paths [ ":" "@" "/" ]

While trying to dreamify photoview-ui I've stumbled over this issue.

dreampkgs on main-photoview-ui
❯ nix run . -- add /home/user/repositories/github.com/photoview/photoview/ui
(translator: nodejs.pure.package-lock with default values)

❯ nix build .#photoview-ui
error: store path 'pbnrirqp0ayzplh92nd2w8myz96pz6a1-tailwindcss-npm:@tailwindcss/[email protected]' contains illegal character ':'

Verify output of translators via jsonschema.

The dream2nix cli should always test the validity of the dream lock format which is produced by a translator.
There is a jsonschema library for python we can use.
The schemas are in /specifications/. Those are not 100% accurate yet and need to be fixed up as well.

zellij: fails to build by not finding a dependency

repro:

{
  inputs.dream2nix.url = "github:nix-community/dream2nix";

  outputs = { self, dream2nix }@inputs:
    let
      dream2nix = inputs.dream2nix.lib.init {
        # modify according to your supported systems
        systems = [ "x86_64-linux" ];
        config.projectRoot = ./. ;
      };
    in dream2nix.makeFlakeOutputs {
      source = ./.;
    };
}

in flake.nix and run nix build .#zellij.

The error:

error: no matching package named `pretty-bytes` found
location searched: registry `crates-io`
required by package `strider v0.2.0 (/build/pcbnlwx4a4r25h4zjkyl0gznigs93kf1-source/default-plugins/strider)`

pretty-bytes is in Cargo.lock, but it seems it's not getting vendored for some reason.

Create translator and builder for rust/cargo

Some work on the translator has already been started but is stalled: #5
We might be able to re-use some parts of import-cargo-lock.nix from nixpkgs.

exclusion for repos not resolvable on the fly seems to miss something

trying to run nix run .#resolveImpure inside of nocodb here is the output

Run `nix run .#resolveImpure` once to resolve impure projects.
The following projects cannot be resolved on the fly and are therefore excluded:
  nc-lib-gui: packages/nc-lib-gui

Resolving:: Name: nc-lib-gui; Subsystem: nodejs; relPath: packages/nc-lib-gui

is it possible that the logic is just inverted ? Instead of not resolving nc-lib-gui it only resolves that one ?

After running that command, you can see that there is only a dream.lock inside that package.

here is the url for the repo if you want to test https://github.com/nocodb/nocodb

nodejs: optional dependencies weird npm behavior

so in the package-lock.json when a dependency is marked as "dev: true", then the attribute "requires" will not be enforced.
This is a weird behavior that causes dream2nix to fail at the moment.
I've made a commit to show what happens here https://github.com/happysalada/directus
trying to build the main flake (careful I've set the system to darwin, so you'll have to change that if you're on linux). I get the following error

error: attribute 'esbuild-android-arm64' missing

       at /nix/store/y8pllkvqbpns9s37y9c9wrxr7wxx2k0n-source/src/translators/nodejs/pure/package-lock/defa
ult.nix:84:44:

           83|                       name = reqName;
           84|                       version = getVersion selfScopeDeps."${reqName}";
             |                                            ^
           85|                     });
(use '--show-trace' to show detailed location information)

but you see that esbuild-android-arm64 is not going to be present since the parent dependency esbuild is marked as "dev: true".
Just putting it here so we remember we need to do something about it.

yarn-lock: versions like `npm:[email protected]` not supported

see #149 (comment)

Move documentation to mdBook

... and create website serving it. This should probably be possible with github-pages.

stack overflow with a certain Rust project

It seems the Rust builder will stack overflow if used on the helix repo. I've tested the translator separately, which works fine. This also happens on #83 with the crane builder, so it doesn't seem to be specific to the buildRustPackage builder.

repro:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    dream2nix = {
      url = "github:nix-community/dream2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    helix = {
      url = "github:helix-editor/helix";
      flake = false;
    };
  };

  outputs = inputs@{ nixpkgs, dream2nix, helix, ... }:
    let
      system = "x86_64-linux";
      dream2nix = inputs.dream2nix.lib.init {
        pkgs = nixpkgs.legacyPackages.${system};
      };
    in dream2nix.riseAndShine { source = helix; };
}

running resolve app is broken now

With the removal of the cli 489f1ab, resolving a package no longer works.

error: attribute 'cli' missing

       at /nix/store/inyg55hlb92cg35r84xxav16jdk9knjl-source/src/utils/default.nix:242:11:

          241|         cd $WORKDIR
          242|         ${apps.cli.program} add ${source} \
             |           ^
          243|           --force \

Yarn workspaces: Hardhat unable to locate node_modules for sub packages

Steps taken:
git clone https://github.com/nomiclabs/hardhat.git && cd hardhat

Drop test flake with dream2nix in repository, then:

git add flake.nix
nix flake show

you get a long list of packages, most of them are dependencies. and one defaultPackages for the root project.

nix build

build of the default package fails:


@nix { "action": "setPhase", "phase": "unpackPhase" }
  unpacking sources
  unpacking source archive /nix/store/7yz9x41x8rvdg8dmca52rmijzi243r73-5d7bixywasyxi7wgmacbx4w0sj77zkxi-source
  @nix { "action": "setPhase", "phase": "patchPhase" }
  patching sources
  @nix { "action": "setPhase", "phase": "d2nLoadFuncsPhase" }
  d2nLoadFuncsPhase
  @nix { "action": "setPhase", "phase": "d2nPatchPhase" }
  d2nPatchPhase
  @nix { "action": "setPhase", "phase": "configurePhase" }
  configuring
  installing: @changesets/cli
  installing: prettier
  installing: shelljs
  installing: typescript
  installing: wsrun
  @nix { "action": "setPhase", "phase": "buildPhase" }
  building
   
  > [email protected] build /nix/store/lnmd1ka7rllp0p85pkchsjrr6i7phv23-root-0.0.0/lib/node_modules/root
  > tsc --build packages/hardhat-core packages/hardhat-docker packages/hardhat-ethers packages/hardhat-ethersc  an packages/hardhat-ganache packages/hardhat-solhint packages/hardhat-solpp packages/hardhat-truffle4 packag  es/hardhat-truffle5 packages/hardhat-vyper packages/hardhat-waffle packages/hardhat-web3 packages/hardhat-we  b3-legacy
   
  packages/hardhat-core/src/types/config.ts:15:25 - error TS2307: Cannot find module 'ethereumjs-util' or its   corresponding type declarations.
   
  15 import type { BN } from "ethereumjs-util";
                             ~~~~~~~~~~~~~~~~~
   
  packages/hardhat-core/src/types/config.ts:277:11 - error TS2503: Cannot find namespace 'Mocha'.
   
  277   mocha?: Mocha.MochaOptions;
                ~~~~~
   
  packages/hardhat-core/src/types/config.ts:285:10 - error TS2503: Cannot find namespace 'Mocha'.
   
  285   mocha: Mocha.MochaOptions;
               ~~~~~
   
  packages/hardhat-core/src/internal/util/abi-helpers.ts:1:22 - error TS2307: Cannot find module '@ethersproje  ct/abi' or its corresponding type declarations.
   
  1 import * as abi from "@ethersproject/abi";
                         ~~~~~~~~~~~~~~~~~~~~
   
  packages/hardhat-core/src/internal/hardhat-network/stack-traces/model.ts:1:29 - error TS2307: Cannot find mo  dule 'ethereumjs-util' or its corresponding type declarations.
   
  1 import { bufferToHex } from "ethereumjs-util";
                                ~~~~~~~~~~~~~~~~~
   
  packages/hardhat-core/src/internal/hardhat-network/stack-traces/message-trace.ts:1:30 - error TS2307: Cannot   find module '@ethereumjs/vm/dist/exceptions' or its corresponding type declarations.
   
  1 import type { VmError } from "@ethereumjs/vm/dist/exceptions";
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   
  packages/hardhat-core/src/internal/hardhat-network/stack-traces/message-trace.ts:2:25 - error TS2307: Cannot   find module 'ethereumjs-util' or its corresponding type declarations.
   
  2 import type { BN } from "ethereumjs-util";
...

On and on.

Inspecting the generated dream-lock.json file, I see that all the packages from the yarn.lock file are included in it. Even the sub-packages' dependencies. And the only repo package listed is again the root project, which only lists its devDependencies.

I think at minimum, the project's sub packages dependencies should be listed under the defaultPackage, not just the devDependencies.

Yarn translator cannot handle empty version spec

No idea if this would even be a valid yarn lockfile. From here:
https://github.com/TryGhost/Admin/blob/b872df2e0eee3ca07c5717a36ef1f5fa0cc97c1d/yarn.lock#L6224

$ nix run github:nix-community/dream2nix -- add github:TryGhost/Admin/b872df2e0eee3ca07c5717a36ef1f5fa0cc97c1d
(...)
Translating project metadata
error: parser failed at:
           ember-get-config ""
           lodash.defaults "^4.2.

       … while evaluating 'mapAttrsToList'

       at /nix/store/ygz58w4f2hgnrdwa657k7gkx3ajd6d8r-bs58z9v8mfd6hxh9zwhsmyzlmiamhqjj-p53cz6rh27q40g9i0q98k3vfrz6lm12w-source/lib/attrsets.nix:258:23:

          257|   */
          258|   mapAttrsToList = f: attrs:
             |                       ^
          259|     map (name: f name attrs.${name}) (attrNames attrs);

       … from call site

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/translators/nodejs/pure/yarn-lock/default.nix:104:11:

          103|         serializePackages = inputData:
          104|           lib.mapAttrsToList
             |           ^
          105|             (yarnName: depAttrs: depAttrs // { inherit yarnName; })

       … while evaluating 'serializePackages'

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/translators/nodejs/pure/yarn-lock/default.nix:103:29:

          102|
          103|         serializePackages = inputData:
             |                             ^
          104|           lib.mapAttrsToList

       … from call site

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/translator.nix:43:32:

           42|
           43|       serializedPackagesList = final.serializePackages final.inputData;
             |                                ^
           44|

       … while evaluating 'findCycles'

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/translator.nix:167:45:

          166|             let
          167|               findCycles = node: prevNodes: cycles:
             |                                             ^
          168|                 let

       … from call site

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/translator.nix:202:17:

          201|               cyclesList =
          202|                 findCycles
             |                 ^
          203|                   (utils.nameVersionPair mainPackageName mainPackageVersion)

       … while evaluating the attribute 'cyclicDependencies'

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/translator.nix:252:20:

          251|
          252|             inherit cyclicDependencies;
             |                    ^
          253|

       … while evaluating 'compressDependencyGraph'

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/dream-lock.nix:183:31:

          182|
          183|     compressDependencyGraph = decompGraph:
             |                               ^
          184|       lib.mapAttrs

       … from call site

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/dream-lock.nix:226:11:

          225|         cyclicDependencies =
          226|           compressDependencyGraph
             |           ^
          227|             uncomp.cyclicDependencies;

       … while evaluating 'toJSON'

       at /nix/store/pp50mg5zjm4if97n27wncqmpbj9x0j3y-49xvflbb44ffsmkjysmrkf0kvk8fmd96-dream2nix-full-src/utils/dream-lock.nix:244:14:

          243|
          244|     toJSON = dreamLock:
             |              ^
          245|       let

       … from call site

       at «string»:8:7:

            7|     in
            8|       dream2nix.utils.dreamLock.toJSON
             |       ^
            9|         # don't use nix to detect cycles, this will be more efficient in python

  Exception

  Translator failed to create dream-lock.json

  at /nix/store/ig5if3b8b7rq70x2z21nvb2k347fdlyj-cli/commands/add.py:476 in run_translate
      472│         )
      473│ 
      474│       # raise error if output wasn't produced
      475│       if not output_temp_file.read():
    → 476│         raise Exception(f"Translator failed to create dream-lock.json")
      477│ 
      478│       # read produced lock file
      479│       with open(output_temp_file.name) as f:
      480│         lock = json.load(f)

Track nodejs progress

Let's use this issue for tracking the progress on nodejs stuff.

Tanslators implemented:

yarn.lock translators
package-lock.json translator
package.json translator
support yarn v2

Other tasks:

handle missing integrity
extensively test on a number of nodejs projects

Supported Typescript workflow?

I'm trying to figure out how to build and/or run a Typescript project using dream2nix. I've created a minimal example at kylegentle/ts-dream-example, with some more context in the README; let me know if you want me to bring more of that context into the issue here.

packaging ripgrep: not all packages build

using the following flake.nix:

{
  inputs.dream2nix.url = "github:nix-community/dream2nix";
  inputs.src = { flake = false; url = "github:BurntSushi/ripgrep/13.0.0"; };

  outputs = { self, dream2nix, src, }@inputs:
    let
      dream2nix = inputs.dream2nix.lib.init {
        systems = [ "x86_64-linux" ];
        config.projectRoot = ./. ;
      };
    in dream2nix.makeFlakeOutputs {
      source = src;
    };
}

...only the ripgrep package builds. All other packages fail with various errors all related to not finding the Cargo.toml file.

example:

> nix build .#grep
error: builder for '/nix/store/pp7aqc1qnw6lzbvz5ja92230z5fhflji-grep-0.2.8.drv' failed with exit code 101;
       last 10 log lines:
       >   failed to load source for dependency `grep-cli`
       >
       > Caused by:
       >   Unable to update /build/cli
       >
       > Caused by:
       >   failed to read `/build/cli/Cargo.toml`
       >
       > Caused by:
       >   No such file or directory (os error 2)

@yusdacra

feature resolution issue with actix-tls

When compiling a project using actix-http as a dependency, actix_tls seems to be missing the connect feature.

This issue is reproducible at https://github.com/lelongg/dream2nix-bug

error[E0432]: unresolved imports `actix_tls::connect::new_connector`, `actix_tls::connect::Connect`
  --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/connector.rs:17:5
   |
17 |     new_connector, Connect as TcpConnect, ConnectError as TcpConnectError,
   |     ^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^ no `Connect` in `connect`
   |     |
   |     no `new_connector` in `connect`
   |
help: a similar name exists in the module
   |
17 |     connector, Connect as TcpConnect, ConnectError as TcpConnectError,
   |     ~~~~~~~~~
help: a similar name exists in the module
   |
17 |     new_connector, Connector as TcpConnect, ConnectError as TcpConnectError,
   |                    ~~~~~~~~~

error[E0432]: unresolved import `actix_tls::connect::Connect`
  --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/mod.rs:14:5
   |
14 |     Connect as TcpConnect, ConnectError as TcpConnectError, Connection as TcpConnection,
   |     -------^^^^^^^^^^^^^^
   |     |
   |     no `Connect` in `connect`
   |     help: a similar name exists in the module: `Connector`

error[E0432]: unresolved import `actix_service::pipeline_factory`
  --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/service.rs:13:21
   |
13 | use actix_service::{pipeline_factory, IntoServiceFactory, Service, ServiceFactory};
   |                     ^^^^^^^^^^^^^^^^ no `pipeline_factory` in the root

error[E0432]: unresolved import `actix_service::pipeline_factory`
 --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/h1/service.rs:8:21
  |
8 | use actix_service::{pipeline_factory, IntoServiceFactory, Service, ServiceFactory};
  |                     ^^^^^^^^^^^^^^^^ no `pipeline_factory` in the root

error[E0432]: unresolved import `actix_service::pipeline_factory`
  --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/h2/service.rs:10:29
   |
10 |     fn_factory, fn_service, pipeline_factory, IntoServiceFactory, Service,
   |                             ^^^^^^^^^^^^^^^^ no `pipeline_factory` in the root

error[E0282]: type annotations needed
   --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/connector.rs:267:28
    |
267 |     pub fn finish(self) -> ConnectorService<S, Io> {
    |                            ^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type for type parameter `Io`

error[E0282]: type annotations needed
   --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/connector.rs:533:13
    |
533 | impl<S, Io> Service<Connect> for TcpConnectorInnerService<S>
    |             ^^^^^^^^^^^^^^^^ cannot infer type for type parameter `Io`

error[E0282]: type annotations needed
   --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/connector.rs:548:8
    |
548 |     fn call(&self, req: Connect) -> Self::Future {
    |        ^^^^ cannot infer type for type parameter `Io`

error[E0282]: type annotations needed
   --> /build/4y8ml4p65982cqjdjjzp3by17lsmgrqn-source/../nix-vendor/actix-http-3.0.0-beta.5/src/client/connector.rs:546:5
    |
546 |     actix_service::forward_ready!(service);
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type for type parameter `Io`
    |
    = note: this error originates in the macro `actix_service::forward_ready` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0282, E0432.
For more information about an error, try `rustc --explain E0282`.
error: could not compile `actix-http` due to 9 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed

[question] How is dream2nix different than what "Status of lang2nix approaches" Discourse thread describes?

Apologies for the ignorant question (especially because Summary of the core concepts and benefits probably answers it, but I'm just unable to comprehend it yet).

Another reason for opening this issue was also to link the Status of lang2nix approaches thread in this repo as dream2nix was mentioned there as well.

Please feel free to close this, and thank you for your efforts!

Set up CI for testing

Set up a CI for continuous testing.
It should execute all different translators on a list of projects to ensure that they are working.
This could be done with nixpkgs vm tests via flake check attributes.
The individual tests just need to call the dream2nix app with different sources / configurations.

Feedback on some overrides.

Hey, this issue is a little rough because it contains various questions.

Regarding "electron" package build for darwin. The build fails with cp: cannot stat '/nix/store/q4yvhf9wy56zxzcmikb8ixaimink5gy7-electron-12.2.3/lib/electron': No such file or directory . I check in the resulting derivation and there is nothing under /lib, there is a /bin/electron though. I was just looking at the electron override inside the repo, and it seems the build step for darwin is putting something inside /bin and not /lib.
Regarding "7zip-bin" package, the install step on darwin fails with > /nix/store/m7jxsbvwvkw2zdvlqqqqdbcvr14cp6an-auto-patchelf-hook/nix-support/setup-hook: line 333: -l: command not found which is segmentHeaders="$(LANG=C $READELF -l "$file")". it looks like $READELF is not defined on darwin. @Mic92 sorry to bother you with this, but would you know what additional dependency needs to be added on darwin to make this work ?

if you are curious, I'm testing with https://github.com/BoostIO/BoostNote-App

Architecture: Project Discovery

This addresses the challenge of handling source trees that contain multiple software projects.

These are notes/ideas for a potential discovery phase, which is supposed to be executed before the translation phase and meant to determine a list of projects within the given source tree to be processed by one or more translators.

The goal is to achieve a standardized way of handling source trees containing more than one software project inside.

Glossary:

project: Consists of source code + metadata defining dependencies and how to produce various artifacts (packages, dev envs, ...). A source tree can contain multiple projects.
translate: The process of parsing the metadata of a project and generating a nix derivation out of it.

A project living alongside other projects within the same source tree can have the following properties:

independent: The project can be translated independently of other projects. Example: multiple sub-directories containing their own independent lock files.
dependent: The project shares some of its metadata with other projects. Example: A rust based source tree containing multiple projects with their own Cargo.toml but only one Cargo.lock

Current situation:
Currently dream2nix contains multiple implementations for project discovery. The rust/cargo translator does it's own scanning of projects and dream2nix itself also contains a language independent project discovery mechanism. The latter is too simple and insufficient for use with rust.
My goal is to establish a new puggable discovery mechanism which is handled by the core framework instead of the individual translator.

Problems with the current situation:

Duplicated code and maintenance overhead due to multiple implementations of project discovery.
It is difficult to build a generic UI for the multi project situation, if there is no abstraction for project discovery.
With the current builtin mechanism, projects are identified via a sub-directory path and translated in complete isolation from each other. This only works for projects which are independent. Therefore it is not applicable to rust and probably others.
The project discovery and project translation is not properly separated from each other. This limits the capability of inspecting the project list before translation and decreases the potential for advanced configuration options like, for example, selecting only some projects for translation or adapting translator parameters on a per-project basis, etc....

After adding a discovery phase, the new architecture of dream2nix would be:
discovery -> translation -> dream-lock -> building

Inputs of the discovery phase:

source tree

Outputs of the discovery phase: a list of:

project name
project subsystem (eg. python, nodejs, etc.)
list of file paths of relevant metadata files
subdirectory path the project resides in (if applicable)

The outputs of the discovery phase are the inputs for the translator(s).

If several projects are found which share some metadata (like a single lock file), those can be processed by a single translator call if the translator supports this.
Other projects might require multiple translator calls in order to be processed.

Each translator could declare a multiplexing function which receives the discovered list of projects and their properties and then decides if one or more translator calls are required to process the inputs.

Things left to do:

Allow configuring translators/builders
Create well commented template for discoverers
Find and document mechanism to feed sub-projects as a build input to parent projects (probably via override system)

rust: fails on projects with crates located in sub directories of length > 1

flake.nix to reproduce the issue:

{
  inputs.dream2nix.url = "github:nix-community/dream2nix";
  inputs.ripgrep = { flake = false; url = "github:BurntSushi/ripgrep/13.0.0"; };
  outputs = { self, nixpkgs, dream2nix, ripgrep, }@inputs:
    let
      # change according to your system
      system = "x86_64-linux";
      dream2nix = inputs.dream2nix.lib.init {
        pkgs = nixpkgs.legacyPackages."${system}";
      };
    in {
      packages = (dream2nix.riseAndShine {
        source = builtins.path {
          path = ripgrep;
          name = "ripgrep";
        };
      }).packages;
    };
}

Current error:

last 10 log lines:
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/fmt.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/fmt.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/lib.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/lib.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/macros.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/macros.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/map.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/map.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/osstringext.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/osstringext.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/strext.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/strext.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/suggestions.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/suggestions.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/src/usage_parser.rs' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/src/usage_parser.rs'
       > '/nix/store/k43y0s16wzm4h477mjkcm7p9mfkjjsab-unpack-clap-2.33.3/.cargo-checksum.json' -> '/nix/store/60vrxbl316809dxwmrazcqk8d6kdkzc4-vendor-ripgrep-13.0.0/clap-2.33.3/.cargo-checksum.json'
       > cp: cannot stat '/nix/store/plm64awd1mip3pj8lm9v030ywpdamzx9-ripgrep/grep': No such file or directory

Even after fixing the issue in the translator (done in #80 ), it still fails with another error about not finding some source.

@yusdacra

Documentation on running a translator is unclear

Issue

I don't understand what I am meant to do from this line alone:
https://github.com/nix-community/dream2nix/blob/main/docs/contributors-guide.md?plain=1#L43

This makes it seem like there is supposed to be some sort of runnable flake app but these are the only apps I see being exported here:

│   │   ├───contribute: app
│   │   ├───format: app
│   │   ├───install: app
│   │   ├───tests-all: app
│   │   ├───tests-examples: app
│   │   └───tests-unit: app

Context

As mentioned zaninime/sbt-derivation#5 (comment), I have a viable path forward for generating functional lock files for SBT based Scala projects (which so far doesn't exist in Nix), and I would very much like to implement a translator and builder here for it, but I am stuck not even understanding how I can run my in progress translator.

HTTP fetcher calculates wrong hash

The HTTP fetcher uses utils.hashPath to calculate the hash:

dream2nix/src/fetchers/http/default.nix

Lines 20 to 22 in c5f6bd5

 calcHash = algo: utils.hashPath algo (b.fetchurl { 

 inherit url; 

 });

This calculates the hash of the NAR archive, but when passing this to fetchurl it gets a hash mismatch because fetchurl uses the flat hash of the file, not the NAR:

dream2nix/src/fetchers/http/default.nix

Lines 24 to 37 in c5f6bd5

 fetched = hash: 

 let drv = 

 if lib.stringLength hash == 40 then 

 fetchurl { 

 inherit url; 

 sha1 = hash; 

 } 

 else 

 fetchurl { 

 inherit url hash; 

 }; 

 in drv.overrideAttrs (old: { 

 name = lib.strings.sanitizeDerivationName old.name; 

 });

Example command to reproduce:

$ nix run github:nix-community/dream2nix add http://example.com

Add a package to this repo

fetching source for 'http://example.com'
Failed calling nix function 'fetchers.fetchShortcut'
error: hash mismatch in fixed-output derivation '/nix/store/6rc92x0dj8x3ya86cdw9pf0x5n959nka-example.com.drv':
         specified: sha256-4QrIZZUUMPmfyDqzVdsEB60gyhZoo1wW+4XewnT4+UM=
            got:    sha256-6o+sfGX7WJsNU1YPUlH3T56bJDR43Laz6nm142RJyNk=
error: 1 dependencies of derivation '/nix/store/iwflq425x9fs6rc6ih90fdkfq9v914p9-example.comextracted.drv' failed to build

The solution could be to introduce a utils.hashFile which uses nix hash file instead of nix hash path, and use that for calcHash.

Produce some video tutorials

Content could be:

walk through the packaging workflow with dream2nix
show how to implement a new translator
show how to implement a builder

Wow!

& thanks! 😄

cargo2nix integration

I am making this issue to find aspects of similarity in our work. Currently I'm planning cargo2nix architecture. My goal is to implement independent derivation building in a way suitable for nixpkgs long-term, maximizing cache granularity.

cargo2nix in dream2nix

cargo2nix binary is a translator, generating a Cargo.nix with some feature & platform logic baked in
mkcrate.nix and associated mkcrate-utils.sh are the builder
each crate's derivation is independent

Will look at the structure of dream2nix overrides before updating cargo2nix's overrides. Same for fetching.

Questions

I'm putting my questions here.

I'm curious why you chose json format for communication. Would nix not work better in this case ? It might be easier to modify/override.
My last question is around the lack of example. It would clarify things a lot to take a dummy example of a lockfile with a single package and go through what you would like each step to do.
I mentioned it briefly in the chat, but there is naersk, that mentions no FOD and no code generation, I don't know how it works and if it could make sense to adopt a similar approach https://github.com/nix-community/naersk

esbuild: finding a solution that works for all versions

I was having some problem with a package failing on esbuild (missing the binary of course)
I was thinking we should potentially have a json file with all the versions and hashes for a couple of packages.
We could potentially make a script to populate that binary.lock file. (we could call it anything you want).

Unexplained abbreviations

Some abbreviations are used in the documentation that are never explained. What do IFD and FOD stand for? I genuinely don‘t know. Most people probably know that UX stands for “user experience” but maybe not everyone.

Could you please write out these terms at least once before you introduce an abbreviation?

	calcHash = algo: utils.hashPath algo (b.fetchurl {
	inherit url;
	});

	fetched = hash:
	let drv =
	if lib.stringLength hash == 40 then
	fetchurl {
	inherit url;
	sha1 = hash;
	}
	else
	fetchurl {
	inherit url hash;
	};
	in drv.overrideAttrs (old: {
	name = lib.strings.sanitizeDerivationName old.name;
	});