tarides / opam-monorepo Goto Github PK

Assemble dune workspaces to build your project and its dependencies as a whole

License: ISC License

Makefile 0.16% OCaml 88.49% Perl 11.35%

opam-monorepo's Introduction

`opam-monorepo`

opam-monorepo is an opam plugin designed to assemble standalone Dune workspaces with your projects and all of their dependencies, letting you build it all from scratch using only dune and ocaml.

Documentation on opam-monorepo is available in the repository as well as available online.

Installation

You can simply install it via opam in your current switch by running:

opam install opam-monorepo

Note that once it's installed you can invoke it as an opam command:

opam monorepo <subcommand> ...

Note that opam knows about available plugins and will offer to install opam-monorepo if you try to invoke it without having installed it beforehand.

Usage

The basic usage for opam-monorepo is to start by running the following command from the root of your project:

opam monorepo lock

This will generate a project-wide lock file under <project-name>.opam.locked that contains:

The full list of your direct and transitive opam dependencies, according to the specifications in the .opam files at your project's root, with hard version constraints ({= <version>) in the depends field.
A pin-depends field filled with reproducible pins, either to tarballs or to Git repos with an explicit commit hash for all the above dependencies, except for ocaml, dune, and any virtual opam package.
A few extra fields meant to be interprated by opam monorepo

This lock file can then be consumed by the following command:

opam monorepo pull

which will fetch the sources of all the locked dependencies into a duniverse/ folder at the root of your project, marking them as vendored_dirs (see Dune's documentation) so that Dune will only build the artifacts you need from that folder.

From that point you should be able to run dune build and dune runtest as you normally would and build your entire project from scratch!

The `lock` Command

It's important to note that opam monorepo lock will only succeed if all of your non-virtual and non "base" dependencies (e.g., ocaml or dune) build with Dune (i.e., directly depend on the dune or jbuilder packages). If that's not the case, the solver will report which packages don't build with dune.

We maintain a separate opam repository with Dune ports of commonly used opam packages. If you have non-Dune dependencies, we advise adding this repository before running opam monorepo lock. You can do so by running the following command:

opam repository add dune-universe git+https://github.com/dune-universe/opam-overlays.git

Note that if it's not setup, the plugin will warn you.

The lock command takes your global and switch's opam configurations into account, meaning any opam repository or pins you set up will be picked up by the solver when resolving the full set of your project's dependencies.

The generated lock file is meant to be compatible with opam in such a way that running

opam install . --locked

should give you the same versions you would using opam monorepo pull in a reproducible way (i.e., independently of any change that might have happened on the upstream opam-repository) thanks to the pin-depends. You can use that property to your advantage by allowing one to choose between a "monorepo" or a regular opam workflow depending on the situation.

You can also exclude packages from the set of packages to be vendored by opam-monorepo. To do so, specify an additional field in your opam file:

x-opam-monorepo-opam-provided: ["ocamlformat" "patdiff"]

This will exclude the packages from the list of packages opam-monorepo will pull, so they can be installed via opam manually.

`opam monorepo pull`

The pull command fetches the sources using the URLs in the lock file. It benefits from the opam cache but its outcome does not depend on your opam configuration.

Monorepo Projects

If you wish to use opam-monorepo to manage your dependencies, we suggest that you Git version the lock file but not the content of the duniverse/.

If you use ocaml-ci and have an opam-monorepo lock file at the root of your project, it will detect it is an opam-monorepo project and start a specific pipeline. It will use the plugin to assemble a Dune workspace with your dependencies rather than installing them through opam.

opam-monorepo's People

Contributors

Stargazers

Watchers

opam-monorepo's Issues

What to do when a package is not dune-compatible

When using duniverse on a mirage unikernel, a lot of base layer packages remain non-dune. These packages (solo5, solo5-bindings-hvt, ocaml-freestanding, minios-xen, ..) are quite a hard work to port to dune, and probably other problems will arise.
In the short term, I propose to keep track of them, and allow to install them using a duniverse opam-install command for example.

Improve git ref resolution

This can take an unreasonably long time for large duniverses and without user feedback it sometimes feels like the tool got stuck.

We can parallelize this step to improve speed and add some logs.

fix duniverse git cache when tags change

Right now the git cache fails when a given tag has a new commit added to it (with an "invalid branch point" failure from git).

If we can't figure out the problem quickly, them simply deleting the repository and recloning is an acceptable workaround -- stopping the build is bad.

Duniverse init seems to pull test deps of pinned packages

It appears that when pinning a package, duniverse init will add its test dependencies to the lockfile.

You can clone https://github.com/NathanReb/duniverse-transitive-test-deps-bug and follow the steps described in the README to reproduce this.

Duniverse folder breaks opam monorepo lock when recursion is enabled

The --recurse-opam option is unusable when a duniverse/ folder is populated, as it tries to include all the opam files from that folder. This has several consequences:

it tries to include test packages from the duniverse-fetched dune/ folder, leading to multiple packages having the same name and thus the same opam file name.
when there are multiple times the same opam file in the workspace, opam-monorepo fails with the exception Invalid_argument Map.of_list_exn.
when using opam monorepo lock <filter-pkg> to select a given package name, the recursion option is enabled, and thus fails with the previous issue.

As a fix, I propose to:

Ignore _build and duniverse subfolders when performing recursion.
Improve the error message when two opam files with the same name appear in the source tree
When filtering packages, apply the filter before checking for uniqueness of opam filenames.

Verify 4.12 compatibility

We should ensure opam-monorepo is compatible with 4.12 before the initial release, pending the release of compat fixes for its dependencies.

duniverse sticks on a pin

When I tried to use duniverse with mirage+dune (https://github.com/TheLortex/mirage/tree/full-dune), duniverse is stick on the pin of mirage-unikernel-pasteur-unix. Currently the work is done on pasteur (https://github.com/dinosaure/pasteur/tree/duniverse).

tracking opam metadata changes in a duniverse

dune 1.10 will hopefully be able to generate opam files from project metadata, which means that we can use a duniverse to create a bunch of opam files in one go (see ocaml/dune#2017)

In that thread, @jberdine notes:

I wonder if any thought has been given to the related problem of how to keep this metadata in sync with changes made to the opam files in the opam-repository? For example, it has happened that missing dependency version constraints were noticed and added directly to opam-repo, and then when I made a new release the added constraint was lost. Other than manually checking each package's opam file in the repo before each release, I don't know of a work flow to preserve such changes.

I'm creating this issue here to track the idea of duniverse being able to somehow spot diffs between the opam repository and the local dune file. This seems pretty tricky at first thought since it's a three way merge: the master branch of the repository may have valid divergences from the released stable package and subsequent fixes in the opam repository. However, this tool does seem like the right place to figure this out, since duniverse bridges both opam and dune together.

Duniverse gets confused when a package no longer needs the duniverse overlay

When a package used to need a duniverse fork but no longer does (e.g. hannesm/duration) the tool gets confused and tries to check out a non-existant branch from the duniverse fork.

complete dependency list per repository cloned

duniverse init currently fails out of the box on a lot of repositories because we only depend on the opam dependencies for one package. Unfortunately, when we clone the repository for an opam package, it might bring in more dependencies since there are sometimes multiple packages held inside an archive.

This then results in a dune build in the duniverse failing since the dependency cone isn't complete. A common example is depending on cstruct, and then there is an incomplete cstruct-async dependency since that's present in the cstruct archive.

We may just need to do a fixed point iteration of fetching all the archives for a dependency set and completing it. Ideally, we would delete the unnecessary packages from the duniverse, but this requires a dune dependency pass I suspect.

clean up exec.ml logging

Right now we deliberately fail some commands and those show up in mainline invocations without -vv present..

e.g.

#25 0.235 duniverse: [ERROR] ocamlc -version failed. Output was:
#25 0.235 
#25 0.236 duniverse: [ERROR] dune --version failed. Output was:
#25 0.236 
#25 0.237 duniverse: [ERROR] /src/_ocaml/bin/ocamlc -version failed. Output was:
#25 0.237 
#25 204.0 duniverse: [WARNING] (Sys_error "dune-project: No such file or directory")

when running duniverse tools

Adding depexts in lock file

At the moment the lock file does not contain the transitive depexts. This means that when looking for them (such as in CI), it's necessary to pin the package recursively. Here's an example from the RWO CI:

RUN opam pin -n add rwo . --locked
RUN opam depext --update -y rwo
RUN opam pin -n remove rwo

This is an issue because it requires access to opam-repository when building the package. Moreover, it can require a version that is as recent as the one that was used to generate the lock file. OCaml-ci does not always have that, since it assumes that it's safe to use the oldest version that knows about the locked version of ocaml, dune and opam-monorepo itself (see ocurrent/ocaml-ci#282).

duniverse needs to resolve test dependencies

Right now it only does core dependencies, needs to also opam list -t to get the test ones

duniverse goes interactive in CI

The Prompt module needs a -y flag so that it can always proceed

In drone:

+ duniverse pull
796	==> You are using version 1.8 of the dune language
797	duniverse: internal error, uncaught exception:
798	End_of_file
799	==> Duniverse requires version 1.11 or above
800	??? Should I update your dune-project? [Y/n]
801	Raised at file "stdlib.ml", line 441, characters 14-31
802	Called from file "cli/prompt.ml", line 9, characters 33-47
803	Called from file "cli/pull.ml", line 23, characters 5-31
804	Called from file "cli/pull.ml", line 82, characters 6-40
805	Called from file "src/cmdliner_term.ml", line 25, characters 19-24
806	Called from file "src/cmdliner.ml", line 33, characters 27-34
807	Called from file "src/cmdliner.ml", line 117, characters 32-39

Verify opam 2.0/2.1 compat

The tool should be compatible with both 2.0 and 2.1 one installs, even if it does require 2.1 versions of the opam libraries. On the fly in-memory config conversion in the 2.1 libraries should allow us to be backward compatible but this needs to be confirmed!

Using opam libraries to obtain the sources in pull

This would let us benefit from the opam cache and allow for support of all pining methods supported by opam natively.

lock fails if a package is pinned to a specific commit

$ opam pin csexp https://github.com/ocaml-dune/csexp.git#25dfc7b8e313adb85ea75fb0a84f59751ac5a144 --no-action
[NOTE] Package csexp is currently pinned to git+https://github.com/ocaml-dune/csexp.git#feed-api (version 1.3.2).
[csexp.1.3.2] synchronised (git+https://github.com/ocaml-dune/csexp.git#25dfc7b8e313adb85ea75fb0a84f59751ac5a144)
csexp is now pinned to git+https://github.com/ocaml-dune/csexp.git#25dfc7b8e313adb85ea75fb0a84f59751ac5a144 (version 1.3.2)
$ opam monorepo lock
==> Using locally scanned package 'test-opam-monorepo' as the root.
opam-monorepo: [WARNING] The dune-universe opam-repository isn't set in the current switch. It contains dune ports for some opam packages. Note that opam monorepo lock will fail if not all of the project dependencies use dune as their build system. Adding this opam-repository to your current switch will help with that. If you wish to do so, run the following command:
opam repository add dune-universe git+https://github.com/dune-universe/opam-overlays.git
opam-monorepo: [ERROR] Unable to classify archive https://github.com/mirage/ocaml-xenstore/tarball/ocaml-xenstore-1.0.0
==> Found 9 opam dependencies for test-opam-monorepo.
==> Querying opam database for their metadata and Dune compatibility.
==> All 2 opam packages are Dune compatible! It's a spicy miracle!
==> Calculating Git repositories to vendor source code.
opam-monorepo: No 25dfc7b8e313adb85ea75fb0a84f59751ac5a144 ref for https://github.com/ocaml-dune/csexp.git

We should at least skip the resolution in this case!

Duniverse is broken on non-English systems

When using git_default_branch, duniverse makes a call to git remote show which has an internationalized output.
HEAD branch : ... is translated to Branche HEAD : ... in French, thus breaking the following part of the code:

https://github.com/ocamllabs/duniverse/blob/07f687043867416cfe9dc7afe493ac85e44d3d69/lib/exec.ml#L130-L133

Better deal with base packages when resolving dependencies

Now that we switched to an empty opam switch to resolve dependencies, the opam command used to build the list of transitive dependencies of the root packages returns base packages.

Since we don't need to pull in those packages in the duniverse we shouldn't be bothered by their dependencies. In particular some of them depend on ocamlfind and that is an issue because there are valid cases where one might rely on ocamlfind's libraries (utop does for example) so we don't necessarily want to make it a virtual dependency. Because ocamlfind hasn't been ported to dune yet this causes duniverse init to fail when it really shouldn't.

Update and fix duniverse's duniverse

As pointed out by @MisterDA, our CI is currently broken because of what seems to be a misconfigured duniverse.

It also seems that our duniverse contains some old versions of our dependencies that could seemlessly be upgraded to allow compatibility with the latest OCaml release.

See #77 (comment) for the initial discussion

variable "doc" not found in dune-project file

I tried installing duniverse (with opam pin to master), and ran into this error:

### output ###
# File "duniverse/topkg.1.0.1+dune/src-bin/browse.ml", line 24, characters 21-32:
# 24 |   "topkg-api", `Uri "%%PKG_DOC%%", "topkg's API docs"
#                           ^^^^^^^^^^^
# Error: variable "doc" not found in dune-project file

OCaml 4.08.1
dune 2.6.1
opam 2.0.7

Pin to tarballs in lock

There should be a way to pin to tarballs when possible (i.e. when resolving deps to a released version). It probably make sense to make this the default and to add a flag to always use git pin the deps in the lock file (the current default).

Get a rev-deps duniverse

For ppxlib I need to be able to assemble a duniverse with the reverse dependencies of my project so I can test changes that I made, fix the rev-deps locally and upstream those changes.

This kind of workflow would also probably be useful for mirage.

add locking to duniverse git cache

The contents of ~/.cache/duniverse are not protected against concurrent access at the moment, which is problematic when doing BuildKit CI. Should be as simple as a file lock for each individual git repository

Warning for a missing dune-project file

When running duniverse init in a directory that contains juniverse.opam with a bunch of package dependencies, I'm getting a warning about a missing dune-project file, as shown below. I guess it should in fact be an error, since duniverse currently assumes this file to be present.

$ ~/bin/duniverse.exe init
==> Using locally scanned packages 'juniverse' as the roots.
duniverse.exe: [WARNING] (Sys_error "dune-project: No such file or directory")
==> Found 1 opam dependencies for juniverse.
==> Querying opam database for their metadata and Dune compatibility.
==> All 0 opam packages are Dune compatible! It's a spicy miracle!
==> Recording 0 depext formulae for 2 packages.
==> Calculating Git repositories to vendor source code.
==> Wrote duniverse file with 0 entries to dune-get. You can now run duniverse pull to fetch the sources.

version the dune-get files

There is already a version field, but unused while this is a development version. Before a stable tag, we need to freeze the version field.

duniverse loses opam patches

I was experimenting with duniverse and found this failure:

File "duniverse/shared-block-ring.2.5.0+1/example/jbuild", line 1, characters 0-0:
Error: jbuild files are no longer supported, please convert this file to a
dune file instead.

There is a shared-block-ring.2.5.0+1/files/0001-dunify.patch which is a backport of some commits from master that converts jbuilder to dune (https://github.com/xapi-project/xs-opam/blob/master/packages/upstream/shared-block-ring.2.5.0%2B1/files/0001-dunify.patch).

This is present in the opam file like this:
https://github.com/xapi-project/xs-opam/blob/master/packages/upstream/shared-block-ring.2.5.0%2B1/opam#L40-L48

patches: [
    "0001-dunify.patch"
]
extra-files: [
  [
    "0001-dunify.patch"
    "md5=37841bb6a3eeb0c1721bbc153f647311"
  ]
]

It looks like duniverse fetched the sources, but didn't apply the patches, which is how I ended up with the jbuild failure.
In this case I could probably pin the package to master to solve the issue, but in general it would be good to apply the patches from opam, since might contain important build fixes or backports.

Semantically equivalent but different "dev-repo" fields lead to build errors

The packages mirage-clock and mirage-clock-freestanding are provided by the same repository but their dev-repo fields are different: git://github.com/mirage/mirage-clock and git+https://github.com/mirage/mirage-clock.git.
This causes duniverse to download the mirage-clock repo twice under different names, leading to a build error further on are some packages are defined twice.

Download caching

In order to make duniverse practical, a caching mechanism need to be implemented.
This would ensure that the whole universe is not downloaded on each duniverse pull.

Another simpler mechanism that could be a first step towards that would be to not re-download a package that has locally been downloaded in ./duniverse/ vendor directory. That also ensures that a duniverse pull won't overwrite local work.

Drop unnecessary extra fields from the lock file

Most if not all extra fields can be removed and the plugin should be able to rely only on the regular deps and pin-deps fields for pulling.

Duniverse pull fails after an update

I recently tried to update the tracked commit for an existing branch and when running duniverse pull afterward, I get the following error:

duniverse: [ERROR] git -C
                     /home/nathan/.cache/duniverse/https---github.com-Julow-mdx.git
                     branch commit-c0554190d10c39204e3d09907106c8d188f4bb33
                     c0554190d10c39204e3d09907106c8d188f4bb33 failed. Output was:
fatal: Not a valid branch point:
'c0554190d10c39204e3d09907106c8d188f4bb33'.

...

==> The following repos could not be pulled as the commit we want is gone:
      - mdx
==> You should run 'duniverse update' to fix the commits associated with the tracked refs
duniverse: Could not pull all the source dependencies

The commit is indeed there as it was found by duniverse update and I confirm:

$ git ls-remote https://github.com/Julow/mdx duniverse_mode
c0554190d10c39204e3d09907106c8d188f4bb33	refs/heads/duniverse_mode

The issue is related to the download cache since disabling it with --no-cache solves this.

We need to investigate that and make the cache more robust!

Allow to install depexts by reading the available opam files instead of dune-project

Solver picks an ocaml version incompatible with the provided constraints

I recently ran into a situation where the original opam file contained an ocaml {>= 4.10.0 & < 4.11} constraint but the lockfile had an {ocaml = 4.11.1}.

The solver should take the original constraint into account.

opam files jump around when running opam monorepo pull

Hi,

When the duniverse folder is checked in, one can see the following interaction:

(clean git status)
% git rm -rf duniverse/
% opam monorepo pull
% git status
... some opam files are modified ...
% dune runtest
% git status
... all clean ...

It'd be great not to have to run tests before committing the duniverse folder again.

For workflows where the folder is not commited this is more innocuous, but it'd be interesting to see where this change comes from.

Invalid git ls-remote output line: ""

I get this error when trying to do duniverse init in a checkout of ocamlformat. Reproduction steps:

$ opam source --dev ocamlformat
[...]
$ cd ocamlformat
$ duniverse init -vv
==> Calculating Duniverse on the master branch
==> Using locally scanned packages 'ocamlformat, ocamlformat_diff,
ocamlformat_reason' as the roots.
==> Initializing temporary opam switch
duniverse: [INFO] Initialising a fresh temprorary opam with an empty switch in /tmp/.duniverse-opam-root-0d2541.
duniverse: [EXEC:14142] ['opam' 'init'
                         '--root=/tmp/.duniverse-opam-root-0d2541'
                         '--no-setup' '--bare']
duniverse: [EXEC:14382] ['opam' 'switch'
                         '--root=/tmp/.duniverse-opam-root-0d2541' 'create'
                         'empty' '--empty' '--no-install']
duniverse: [EXEC:14388] ['opam' 'repository'
                         '--root=/tmp/.duniverse-opam-root-0d2541' 'add'
                         'remote0'
                         'git://github.com/dune-universe/opam-overlays.git']
duniverse: [EXEC:14469] ['opam' 'pin'
                         '--root=/tmp/.duniverse-opam-root-0d2541' 'add'
                         '-yn' 'ocamlformat.dev' '.']
duniverse: [EXEC:14508] ['opam' 'pin'
                         '--root=/tmp/.duniverse-opam-root-0d2541' 'add'
                         '-yn' 'ocamlformat_diff.dev' '.']
duniverse: [EXEC:14552] ['opam' 'pin'
                         '--root=/tmp/.duniverse-opam-root-0d2541' 'add'
                         '-yn' 'ocamlformat_reason.dev' '.']
==> Resolving opam dependencies for ocamlformat ocamlformat_diff ocamlformat_reason
duniverse: [EXEC:14591] ['opam' 'list'
                         '--root=/tmp/.duniverse-opam-root-0d2541'
                         '--color=never' '-s'
                         '--resolve=ocamlformat,ocamlformat_diff,ocamlformat_reason'
                         '-V' '-S']
==> Found 44 opam dependencies.
duniverse: [INFO] The dependencies for ocamlformat, ocamlformat_diff,
                  ocamlformat_reason are: ocaml-base-compiler.4.08.1,
                  ocaml-config.1, base-threads.base, base-unix.base,
                  ocaml.4.08.1, dune.1.11.1, findlib.1.8.1+dune,
                  ocamlfind.1.8.1+dune, result.1.4, ppx_derivers.1.2.1,
                  ocaml-migrate-parsetree.1.4.0, sexplib0.v0.12.0,
                  ocamlbuild.0.14.0, base-bytes.base, base.v0.12.2,
                  cppo.1.6.6, stdio.v0.12.0, seq.base, re.1.9.0,
                  ppx_tools_versioned.5.2.3, mmap.1.1.0, base-bigarray.base,
                  cmdliner.1.0.4+dune, uchar.0.0.2+dune, stdlib-shims.0.1.0,
                  lwt.4.2.1, uutf.1.0.2+dune, astring.0.8.3+dune,
                  fmt.0.8.8+dune, menhir.20190626, tyxml.4.3.0,
                  logs.0.6.3+dune, rresult.0.6.0+dune, bisect_ppx.1.4.1,
                  fpath.0.7.2+dune, bos.0.2.0+dune, uucp.12.0.0+dune,
                  merlin-extend.0.5, uuseg.12.0.0+dune, odoc.1.4.1,
                  reason.3.5.0, ocamlformat.dev, ocamlformat_diff.dev,
                  ocamlformat_reason.dev
==> Querying local opam switch for their metadata and Dune compatibility.
duniverse: [EXEC:14605] ['opam' 'show'
                         '--root=/tmp/.duniverse-opam-root-0d2541'
                         '--color=never' '--normalise' '-f'
                         'name,dev-repo:,url.src:,depends:'
                         'ocaml-base-compiler.4.08.1' 'ocaml-config.1'
                         'base-threads.base' 'base-unix.base' 'ocaml.4.08.1'
                         'dune.1.11.1' 'findlib.1.8.1+dune'
                         'ocamlfind.1.8.1+dune' 'result.1.4'
                         'ppx_derivers.1.2.1' 'ocaml-migrate-parsetree.1.4.0'
                         'sexplib0.v0.12.0' 'ocamlbuild.0.14.0'
                         'base-bytes.base' 'base.v0.12.2' 'cppo.1.6.6'
                         'stdio.v0.12.0' 'seq.base' 're.1.9.0'
                         'ppx_tools_versioned.5.2.3' 'mmap.1.1.0'
                         'base-bigarray.base' 'cmdliner.1.0.4+dune'
                         'uchar.0.0.2+dune' 'stdlib-shims.0.1.0' 'lwt.4.2.1'
                         'uutf.1.0.2+dune' 'astring.0.8.3+dune'
                         'fmt.0.8.8+dune' 'menhir.20190626' 'tyxml.4.3.0'
                         'logs.0.6.3+dune' 'rresult.0.6.0+dune'
                         'bisect_ppx.1.4.1' 'fpath.0.7.2+dune'
                         'bos.0.2.0+dune' 'uucp.12.0.0+dune'
                         'merlin-extend.0.5' 'uuseg.12.0.0+dune' 'odoc.1.4.1'
                         'reason.3.5.0' 'ocamlformat.dev'
                         'ocamlformat_diff.dev' 'ocamlformat_reason.dev']
duniverse: [DEBUG] Depends for ocaml-base-compiler: base-threads base-bigarray base-unix ocaml
duniverse: [INFO] Classified ocaml-base-compiler.4.08.1 as Virtual with tag 
duniverse: [DEBUG] Depends for ocaml-config: 
duniverse: [DEBUG] Mapped ocaml-config to a virtual package as it has no archive
duniverse: [INFO] Classified ocaml-config.1 as Virtual with tag 
duniverse: [DEBUG] Mapped base-threads to a virtual package as it has no archive
duniverse: [INFO] Classified base-threads.base as Virtual with tag 
duniverse: [DEBUG] Mapped base-unix to a virtual package as it has no archive
duniverse: [INFO] Classified base-unix.base as Virtual with tag 
duniverse: [DEBUG] Depends for ocaml: ocaml-config
duniverse: [INFO] Classified ocaml.4.08.1 as Virtual with tag 
duniverse: [DEBUG] Depends for dune: base-threads base-unix ocaml
duniverse: [INFO] Classified dune.1.11.1 as Virtual with tag 
duniverse: [DEBUG] Depends for findlib: dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (findlib)
duniverse: [DEBUG] Mapped git+https://github.com/dune-universe/lib-findlib#duniverse-1.8.1 -> duniverse-1.8.1
duniverse: [INFO] Classified findlib.1.8.1+dune as (Git https://github.com/dune-universe/lib-findlib) with tag duniverse-1.8.1
duniverse: [DEBUG] Depends for ocamlfind: findlib dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (ocamlfind)
duniverse: [DEBUG] Mapped git+https://github.com/dune-universe/lib-findlib#duniverse-1.8.1 -> duniverse-1.8.1
duniverse: [INFO] Classified ocamlfind.1.8.1+dune as (Git https://github.com/dune-universe/lib-findlib) with tag duniverse-1.8.1
duniverse: [DEBUG] Depends for result: dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (result)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (result)
duniverse: [DEBUG] Mapped https://github.com/janestreet/result/archive/1.4.tar.gz -> 1.4
duniverse: [INFO] Classified result.1.4 as (Git https://github.com/janestreet/result.git) with tag 1.4
duniverse: [DEBUG] Depends for ppx_derivers: dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (ppx_derivers)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (ppx_derivers)
duniverse: [DEBUG] Mapped https://github.com/ocaml-ppx/ppx_derivers/archive/1.2.1.tar.gz -> 1.2.1
duniverse: [INFO] Classified ppx_derivers.1.2.1 as (Git git://github.com/ocaml-ppx/ppx_derivers.git) with tag 1.2.1
duniverse: [DEBUG] Depends for ocaml-migrate-parsetree: ocaml dune ppx_derivers result
duniverse: [DEBUG] Parsing git url from url.src (ocaml-migrate-parsetree)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (ocaml-migrate-parsetree)
duniverse: [DEBUG] Mapped https://github.com/ocaml-ppx/ocaml-migrate-parsetree/releases/download/v1.4.0/ocaml-migrate-parsetree-v1.4.0.tbz -> v1.4.0
duniverse: [INFO] Classified ocaml-migrate-parsetree.1.4.0 as (Git https://github.com/ocaml-ppx/ocaml-migrate-parsetree.git) with tag v1.4.0
duniverse: [DEBUG] Depends for sexplib0: dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (sexplib0)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (sexplib0)
duniverse: [DEBUG] Mapped https://ocaml.janestreet.com/ocaml-core/v0.12/files/sexplib0-v0.12.0.tar.gz -> v0.12.0
duniverse: [INFO] Classified sexplib0.v0.12.0 as (Git https://github.com/janestreet/sexplib0.git) with tag v0.12.0
duniverse: [DEBUG] Depends for ocamlbuild: ocaml
duniverse: [INFO] Classified ocamlbuild.0.14.0 as Virtual with tag 
duniverse: [DEBUG] Depends for base-bytes: ocamlfind ocaml
duniverse: [DEBUG] Mapped base-bytes to a virtual package as it has no archive
duniverse: [INFO] Classified base-bytes.base as Virtual with tag 
duniverse: [DEBUG] Depends for base: dune sexplib0 ocaml
duniverse: [DEBUG] Parsing git url from url.src (base)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (base)
duniverse: [DEBUG] Mapped https://github.com/janestreet/base/archive/v0.12.2.tar.gz -> v0.12.2
duniverse: [INFO] Classified base.v0.12.2 as (Git https://github.com/janestreet/base.git) with tag v0.12.2
duniverse: [DEBUG] Depends for cppo: base-unix dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (cppo)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (cppo)
duniverse: [DEBUG] Mapped https://github.com/ocaml-community/cppo/releases/download/v1.6.6/cppo-v1.6.6.tbz -> v1.6.6
duniverse: [INFO] Classified cppo.1.6.6 as (Git https://github.com/ocaml-community/cppo.git) with tag v1.6.6
duniverse: [DEBUG] Depends for stdio: dune base ocaml
duniverse: [DEBUG] Parsing git url from url.src (stdio)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (stdio)
duniverse: [DEBUG] Mapped https://ocaml.janestreet.com/ocaml-core/v0.12/files/stdio-v0.12.0.tar.gz -> v0.12.0
duniverse: [INFO] Classified stdio.v0.12.0 as (Git https://github.com/janestreet/stdio.git) with tag v0.12.0
duniverse: [DEBUG] Depends for seq: ocaml
duniverse: [DEBUG] Mapped seq to a virtual package as it has no archive
duniverse: [INFO] Classified seq.base as Virtual with tag 
duniverse: [DEBUG] Depends for re: seq ounit dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (re)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (re)
duniverse: [DEBUG] Mapped https://github.com/ocaml/ocaml-re/releases/download/1.9.0/re-1.9.0.tbz -> 1.9.0
duniverse: [INFO] Classified re.1.9.0 as (Git https://github.com/ocaml/ocaml-re.git) with tag 1.9.0
duniverse: [DEBUG] Depends for ppx_tools_versioned: ocaml-migrate-parsetree dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (ppx_tools_versioned)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (ppx_tools_versioned)
duniverse: [DEBUG] Mapped https://github.com/ocaml-ppx/ppx_tools_versioned/archive/5.2.3.tar.gz -> 5.2.3
duniverse: [INFO] Classified ppx_tools_versioned.5.2.3 as (Git git://github.com/ocaml-ppx/ppx_tools_versioned.git) with tag 5.2.3
duniverse: [DEBUG] Depends for mmap: dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (mmap)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (mmap)
duniverse: [DEBUG] Mapped https://github.com/mirage/mmap/releases/download/v1.1.0/mmap-v1.1.0.tbz -> v1.1.0
duniverse: [INFO] Classified mmap.1.1.0 as (Git https://github.com/mirage/mmap.git) with tag v1.1.0
duniverse: [DEBUG] Mapped base-bigarray to a virtual package as it has no archive
duniverse: [INFO] Classified base-bigarray.base as Virtual with tag 
duniverse: [DEBUG] Depends for cmdliner: ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (cmdliner)
duniverse: [DEBUG] Mapped git://github.com/dbuenzli/cmdliner.git#v1.0.4 -> v1.0.4
duniverse: [INFO] Classified cmdliner.1.0.4+dune as (Git git://github.com/dbuenzli/cmdliner.git) with tag v1.0.4
duniverse: [DEBUG] Depends for uchar: ocaml
duniverse: [DEBUG] Mapped uchar to a virtual package as it has no archive
duniverse: [INFO] Classified uchar.0.0.2+dune as Virtual with tag 
duniverse: [DEBUG] Depends for stdlib-shims: ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (stdlib-shims)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (stdlib-shims)
duniverse: [DEBUG] Mapped https://github.com/ocaml/stdlib-shims/releases/download/0.1.0/stdlib-shims-0.1.0.tbz -> 0.1.0
duniverse: [INFO] Classified stdlib-shims.0.1.0 as (Git https://github.com/ocaml/stdlib-shims.git) with tag 0.1.0
duniverse: [DEBUG] Depends for lwt: ocamlfind bisect_ppx seq result ocaml mmap dune cppo
duniverse: [DEBUG] Parsing git url from url.src (lwt)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (lwt)
duniverse: [DEBUG] Mapped https://github.com/ocsigen/lwt/archive/4.2.1.tar.gz -> 4.2.1
duniverse: [INFO] Classified lwt.4.2.1 as (Git https://github.com/ocsigen/lwt.git) with tag 4.2.1
duniverse: [DEBUG] Depends for uutf: uchar ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (uutf)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/uutf.git#duniverse-v1.0.2 -> duniverse-v1.0.2
duniverse: [INFO] Classified uutf.1.0.2+dune as (Git git://github.com/dune-universe/uutf.git) with tag duniverse-v1.0.2
duniverse: [DEBUG] Depends for astring: base-bytes ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (astring)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/astring.git#duniverse-v0.8.3 -> duniverse-v0.8.3
duniverse: [INFO] Classified astring.0.8.3+dune as (Git git://github.com/dune-universe/astring.git) with tag duniverse-v0.8.3
duniverse: [DEBUG] Depends for fmt: stdlib-shims seq ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (fmt)
duniverse: [DEBUG] Mapped git+https://github.com/dune-universe/fmt#duniverse-v0.8.8 -> duniverse-v0.8.8
duniverse: [INFO] Classified fmt.0.8.8+dune as (Git https://github.com/dune-universe/fmt) with tag duniverse-v0.8.8
duniverse: [DEBUG] Depends for menhir: ocamlbuild ocamlfind ocaml
duniverse: [INFO] Classified menhir.20190626 as Virtual with tag 
duniverse: [DEBUG] Depends for tyxml: uutf seq alcotest dune re ocaml
duniverse: [DEBUG] Parsing git url from url.src (tyxml)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (tyxml)
duniverse: [DEBUG] Mapped https://github.com/ocsigen/tyxml/releases/download/4.3.0/tyxml-4.3.0.tbz -> 4.3.0
duniverse: [INFO] Classified tyxml.4.3.0 as (Git https://github.com/ocsigen/tyxml.git) with tag 4.3.0
duniverse: [DEBUG] Depends for logs: mtime dune cmdliner fmt lwt ocaml
duniverse: [DEBUG] Parsing git url from url.src (logs)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/logs.git#duniverse-v0.6.3 -> duniverse-v0.6.3
duniverse: [INFO] Classified logs.0.6.3+dune as (Git git://github.com/dune-universe/logs.git) with tag duniverse-v0.6.3
duniverse: [DEBUG] Depends for rresult: result ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (rresult)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/rresult.git#duniverse-v0.6.0 -> duniverse-v0.6.0
duniverse: [INFO] Classified rresult.0.6.0+dune as (Git git://github.com/dune-universe/rresult.git) with tag duniverse-v0.6.0
duniverse: [DEBUG] Depends for bisect_ppx: ounit ocamlfind ppx_tools_versioned ocaml-migrate-parsetree ocaml dune base-unix
duniverse: [DEBUG] Parsing git url from url.src (bisect_ppx)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (bisect_ppx)
duniverse: [DEBUG] Mapped https://github.com/aantron/bisect_ppx/archive/1.4.1.tar.gz -> 1.4.1
duniverse: [INFO] Classified bisect_ppx.1.4.1 as (Git https://github.com/aantron/bisect_ppx.git) with tag 1.4.1
duniverse: [DEBUG] Depends for fpath: astring result ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (fpath)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/fpath.git#duniverse-v0.7.2 -> duniverse-v0.7.2
duniverse: [INFO] Classified fpath.0.7.2+dune as (Git git://github.com/dune-universe/fpath.git) with tag duniverse-v0.7.2
duniverse: [DEBUG] Depends for bos: mtime logs fmt fpath astring rresult base-unix ocaml dune
duniverse: [DEBUG] Parsing git url from url.src (bos)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/bos.git#duniverse-v0.2.0 -> duniverse-v0.2.0
duniverse: [INFO] Classified bos.0.2.0+dune as (Git git://github.com/dune-universe/bos.git) with tag duniverse-v0.2.0
duniverse: [DEBUG] Depends for uucp: uchar dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (uucp)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/uucp.git#duniverse-v12.0.0 -> duniverse-v12.0.0
duniverse: [INFO] Classified uucp.12.0.0+dune as (Git git://github.com/dune-universe/uucp.git) with tag duniverse-v12.0.0
duniverse: [DEBUG] Depends for merlin-extend: ocaml cppo dune
duniverse: [DEBUG] Parsing git url from url.src (merlin-extend)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (merlin-extend)
duniverse: [DEBUG] Mapped https://github.com/let-def/merlin-extend/releases/download/v0.5/merlin-extend-v0.5.tbz -> v0.5
duniverse: [INFO] Classified merlin-extend.0.5 as (Git https://github.com/let-def/merlin-extend.git) with tag v0.5
duniverse: [DEBUG] Depends for uuseg: uutf uucp uchar dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (uuseg)
duniverse: [DEBUG] Mapped git://github.com/dune-universe/uuseg.git#duniverse-v12.0.0 -> duniverse-v12.0.0
duniverse: [INFO] Classified uuseg.12.0.0+dune as (Git git://github.com/dune-universe/uuseg.git) with tag duniverse-v12.0.0
duniverse: [DEBUG] Depends for odoc: bisect_ppx sexplib ocamlfind markup alcotest tyxml result ocaml fpath dune cppo cmdliner astring
duniverse: [DEBUG] Parsing git url from url.src (odoc)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (odoc)
duniverse: [DEBUG] Mapped https://github.com/ocaml/odoc/archive/1.4.1.tar.gz -> 1.4.1
duniverse: [INFO] Classified odoc.1.4.1 as (Git https://github.com/ocaml/odoc.git) with tag 1.4.1
duniverse: [DEBUG] Depends for reason: ocaml-migrate-parsetree result merlin-extend menhir dune ocaml
duniverse: [DEBUG] Parsing git url from url.src (reason)
duniverse: [DEBUG] Falling back to parsing git url from dev-repo (reason)
duniverse: [INFO] Attempting to guess tag for https://registry.npmjs.org/@esy-ocaml/reason/-/reason-3.5.0.tgz from the final part of the URL
duniverse: [DEBUG] Mapped https://registry.npmjs.org/@esy-ocaml/reason/-/reason-3.5.0.tgz -> 3.5.0
duniverse: [INFO] Classified reason.3.5.0 as (Git git://github.com/facebook/reason.git) with tag 3.5.0
duniverse: [DEBUG] Depends for ocamlformat: uutf uuseg stdio re odoc ocaml-migrate-parsetree fpath dune cmdliner bisect_ppx base-unix base ocaml
duniverse: [DEBUG] Parsing git url from url.src (ocamlformat)
duniverse: [DEBUG] Mapped git+file:///home/dim/code/bordel/ocamlformat/ocamlformat#master -> ??
duniverse: [INFO] Classified ocamlformat.dev as (Git file:///home/dim/code/bordel/ocamlformat/ocamlformat) with tag 
duniverse: [DEBUG] Depends for ocamlformat_diff: ocamlformat dune cmdliner bos
duniverse: [DEBUG] Parsing git url from url.src (ocamlformat_diff)
duniverse: [DEBUG] Mapped git+file:///home/dim/code/bordel/ocamlformat/ocamlformat#master -> ??
duniverse: [INFO] Classified ocamlformat_diff.dev as (Git file:///home/dim/code/bordel/ocamlformat/ocamlformat) with tag 
duniverse: [DEBUG] Depends for ocamlformat_reason: reason uutf stdio re odoc ocaml-migrate-parsetree fpath dune cmdliner bisect_ppx base-unix base ocaml
duniverse: [DEBUG] Parsing git url from url.src (ocamlformat_reason)
duniverse: [DEBUG] Mapped git+file:///home/dim/code/bordel/ocamlformat/ocamlformat#master -> ??
duniverse: [INFO] Classified ocamlformat_reason.dev as (Git file:///home/dim/code/bordel/ocamlformat/ocamlformat) with tag 
==> Filtering out packages that are irrelevant to the Duniverse.
==> All 29 opam packages are Dune compatible! It's a spicy miracle!
==> Calculating Git repositories to vendor
duniverse: [EXEC:14636] ['git' 'ls-remote'
                         'https://github.com/janestreet/sexplib0.git'
                         'v0.12.0' 'v0.12.0^{}']
duniverse: [EXEC:14637] ['git' 'ls-remote'
                         'git://github.com/dune-universe/fpath.git'
                         'duniverse-v0.7.2' 'duniverse-v0.7.2^{}']
duniverse: [EXEC:14638] ['git' 'ls-remote'
                         'git://github.com/dune-universe/uucp.git'
                         'duniverse-v12.0.0' 'duniverse-v12.0.0^{}']
duniverse: [EXEC:14639] ['git' 'ls-remote'
                         'git://github.com/ocaml-ppx/ppx_tools_versioned.git'
                         '5.2.3' '5.2.3^{}']
duniverse: [EXEC:14641] ['git' 'ls-remote'
                         'git://github.com/dune-universe/bos.git'
                         'duniverse-v0.2.0' 'duniverse-v0.2.0^{}']
duniverse: [EXEC:14642] ['git' 'ls-remote'
                         'git://github.com/dune-universe/astring.git'
                         'duniverse-v0.8.3' 'duniverse-v0.8.3^{}']
duniverse: [EXEC:14643] ['git' 'ls-remote'
                         'git://github.com/dune-universe/rresult.git'
                         'duniverse-v0.6.0' 'duniverse-v0.6.0^{}']
duniverse: [EXEC:14644] ['git' 'ls-remote'
                         'git://github.com/dune-universe/logs.git'
                         'duniverse-v0.6.3' 'duniverse-v0.6.3^{}']
duniverse: [EXEC:14645] ['git' 'ls-remote'
                         'git://github.com/dune-universe/uuseg.git'
                         'duniverse-v12.0.0' 'duniverse-v12.0.0^{}']
duniverse: [EXEC:14646] ['git' 'ls-remote'
                         'git://github.com/facebook/reason.git' '3.5.0'
                         '3.5.0^{}']
duniverse: [EXEC:14647] ['git' 'ls-remote'
                         'git://github.com/ocaml-ppx/ppx_derivers.git'
                         '1.2.1' '1.2.1^{}']
duniverse: [EXEC:14648] ['git' 'ls-remote'
                         'https://github.com/dune-universe/fmt'
                         'duniverse-v0.8.8' 'duniverse-v0.8.8^{}']
duniverse: [EXEC:14649] ['git' 'ls-remote'
                         'https://github.com/dune-universe/lib-findlib'
                         'duniverse-1.8.1' 'duniverse-1.8.1^{}']
duniverse: [EXEC:14650] ['git' 'ls-remote'
                         'https://github.com/ocaml-community/cppo.git'
                         'v1.6.6' 'v1.6.6^{}']
duniverse: [EXEC:14651] ['git' 'ls-remote'
                         'https://github.com/janestreet/base.git' 'v0.12.2'
                         'v0.12.2^{}']
duniverse: [EXEC:14652] ['git' 'ls-remote'
                         'https://github.com/janestreet/result.git' '1.4'
                         '1.4^{}']
duniverse: [EXEC:14654] ['git' 'ls-remote'
                         'https://github.com/ocaml-ppx/ocaml-migrate-parsetree.git'
                         'v1.4.0' 'v1.4.0^{}']
duniverse: [EXEC:14655] ['git' 'ls-remote'
                         'git://github.com/dbuenzli/cmdliner.git' 'v1.0.4'
                         'v1.0.4^{}']
duniverse: [EXEC:14656] ['git' 'ls-remote'
                         'git://github.com/dune-universe/uutf.git'
                         'duniverse-v1.0.2' 'duniverse-v1.0.2^{}']
duniverse: [EXEC:14658] ['git' 'ls-remote'
                         'https://github.com/aantron/bisect_ppx.git' '1.4.1'
                         '1.4.1^{}']
duniverse: [EXEC:14659] ['git' 'ls-remote'
                         'https://github.com/janestreet/stdio.git' 'v0.12.0'
                         'v0.12.0^{}']
duniverse: [EXEC:14660] ['git' 'ls-remote'
                         'https://github.com/ocaml/ocaml-re.git' '1.9.0'
                         '1.9.0^{}']
duniverse: [EXEC:14661] ['git' 'ls-remote'
                         'https://github.com/let-def/merlin-extend.git'
                         'v0.5' 'v0.5^{}']
duniverse: [EXEC:14662] ['git' 'ls-remote'
                         'https://github.com/mirage/mmap.git' 'v1.1.0'
                         'v1.1.0^{}']
duniverse: [EXEC:14685] ['git' 'ls-remote'
                         'https://github.com/ocaml/odoc.git' '1.4.1'
                         '1.4.1^{}']
duniverse: [EXEC:14686] ['git' 'ls-remote'
                         'https://github.com/ocaml/stdlib-shims.git' '0.1.0'
                         '0.1.0^{}']
duniverse: [EXEC:14691] ['git' 'ls-remote'
                         'https://github.com/ocsigen/tyxml.git' '4.3.0'
                         '4.3.0^{}']
duniverse: [EXEC:14692] ['git' 'ls-remote'
                         'https://github.com/ocsigen/lwt.git' '4.2.1'
                         '4.2.1^{}']
duniverse: Invalid git ls-remote output line: ""

vendored cmdliner doesn't replace `$(mname)`

It seems that the cmdliner's version used by duniverse doesn't properly replace the $(mname) variables:

diff --git a/test/mirage/help/describe-help.expected b/test/mirage/help/describe-help
index 90e49be9..bf542eb9 100644
--- a/test/mirage/help/describe-help.expected
+++ b/test/mirage/help/describe-help
@@ -57,7 +57,7 @@ DESCRIBE OPTIONS
            default when the unikernel has not been configured.

 CONFIGURE OPTIONS
-       --context-file=FILE (absent=mirage.context)
+       --context-file=FILE (absent=$(mname).context)
            The context file to use.

        --dry-run
         git (internal) (exit 1)

run dune subst after a git clone

This will update the version field in dune-project, which is important.

However, the dune subst needs to be run in a git context, so this needs to happen in the git cache layer instead of the checked out version. (see #91 as well)

Confusing error messages on init

duniverse init is giving confusing error messages if you do not invoque it correctly:

▶ duniverse init
duniverse: Cannot find any packages to vendor.
           Either create some *.opam files in the local repository, or specify them manually via 'duniverse opam <packages>'.

▶ duniverse opam mirage functoria
duniverse: too many arguments, don't know what to do with `mirage', `functoria'
Usage: duniverse opam-install [OPTION]...
Try `duniverse opam-install --help' or `duniverse --help' for more information.

▶ duniverse opam-install mirage
duniverse: too many arguments, don't know what to do with `mirage'
Usage: duniverse opam-install [OPTION]...
Try `duniverse opam-install --help' or `duniverse --help' for more information.

Failure "Windows only. This function isn't implemented."

Hello !

I used to be able to use duniverse to build ocaml-lsp on windows without opam nor ocamlfind, and it worked like a charm !

But since the upgrade to "opam core 2.1" (commit f133ee6)
the executable is throwing an uncaught failure exception:

opam-monorepo: internal error, uncaught exception:
               (Failure "Windows only. This function isn't implemented.")
               Raised at file "stdlib.ml", line 29, characters 17-33
               Called from file "duniverse/opam-core.2.1.0-beta/src/core/opamStubs.ml.dummy" (inlined), line 13, characters 23-80
               Called from file "duniverse/opam-core.2.1.0-beta/src/core/opamStd.ml", line 800, characters 36-66
               Called from file "duniverse/opam-core.2.1.0-beta/src/core/opamStd.ml" (inlined), line 1268, characters 44-71
               Called from file "duniverse/opam-core.2.1.0-beta/src/core/opamConsole.ml", line 585, characters 41-76
               Called from file "duniverse/opam-core.2.1.0-beta/src/state/opamFormatUpgrade.ml", line 1128, characters 4-401
               Called from file "duniverse/opam-core.2.1.0-beta/src/state/opamGlobalState.ml", line 62, characters 15-43
               Called from file "duniverse/opam-core.2.1.0-beta/src/state/opamGlobalState.ml", line 169, characters 11-20
               Called from file "cli/lock.ml", line 109, characters 2-57
               Called from file "duniverse/cmdliner.1.0.4+dune/src/cmdliner_term.ml", line 25, characters 19-24
               Called from file "duniverse/cmdliner.1.0.4+dune/src/cmdliner.ml", line 33, characters 27-34
               Called from file "duniverse/cmdliner.1.0.4+dune/src/cmdliner.ml", line 117, characters 32-39

I guess there must be a "configure step" required to be able to compile opam-core on windows to avoid the selection of the dummy implementation of these stubs.

What am I missing ?

add a version field to cloned vendored projects

This needs to go in dune-project to correctly set the version for the build-info of that vendored project.

duniverse pull needs to delete older files

This has to be careful not to delete the pins in #87 which are stored in duniverse/.pins, but its important to delete other packages as they are often leading to multiple opam files for a package in the same directory.

Select a custom Opam repo

Hi!

I’m experimenting with Duniverse. How do I chose a custom Opam repository?
I saw the code:
https://github.com/ocamllabs/duniverse/blob/44b74dc6bb97c2a316e749055bd7b8ba338232ca/cli/init.ml#L94-L97
and
https://github.com/ocamllabs/duniverse/blob/44b74dc6bb97c2a316e749055bd7b8ba338232ca/lib/config.ml#L31

Can the repository be selected from the command-line, or do I have to change the value or duniverse_opam_repo? What’s the #duniverse fragment for in the URI?

Thanks!

duniverse pull is too quiet

It seems that duniverse pull is now a bit quieter ; it's nice but it's quiet now, eg.

duniverse pull
==> Using pull mode Source
[.. wait 5 minutes ..]
Successfully pulled 97/97 repositories

The Successfully pulled 97/97 repositories line should probably be printed more regularly on to show a bit of progress.

dune tool version calculation should take dune-configurator into account

The dune-configurator dependencies will constrain the base version of dune allowed (e.g. 2.6.0+), so we need to take that into account for the minimum version of dune recorded in dune-get.

e.g. see ocaml/dune#3572

`dune build` failed after `opam monorepo pull`

The dune seems to be confused by the (intentionally) broken dune-project files after opam monorepo pull pulls in all the dependencies.

File "duniverse/dune_/test/blackbox-tests/test-cases/dialects.t/bad1/dune-project", line 9, characters 1-74:
 9 |  (name d)
10 |  (implementation (extension foo2))
11 |  (interface (extension bar2)))
Error: dialect "d" is already defined

(I am not completely sure where to report this---but I suppose it is okay to report the issue here.)

mercurial support needed for utop

utop depends on charInfo_width, which has a mercurial dev repo at bitbucket

duniverse thus fails on that at the moment

Duniverse fails if local package doesnt exist

For ocamlformat, the ocamlformat_diff package doesnt exist, and the CI tries to go interactive:

==> Initializing temporary opam switch
374	duniverse: [INFO] Initialising a fresh temprorary opam with an empty switch in /tmp/.duniverse-opam-root-fda284.
375	duniverse: [EXEC:17599] ['opam' 'init'
376	'--root=/tmp/.duniverse-opam-root-fda284'
377	'--no-setup' '--bare']
378	duniverse: [EXEC:17628] ['opam' 'switch'
379	'--root=/tmp/.duniverse-opam-root-fda284' 'create'
380	'empty' '--empty' '--no-install']
381	duniverse: [EXEC:17632] ['opam' 'repository'
382	'--root=/tmp/.duniverse-opam-root-fda284' 'add'
383	'remote0' '.']
384	duniverse: [EXEC:17683] ['opam' 'pin'
385	'--root=/tmp/.duniverse-opam-root-fda284' 'add'
386	'-yn' 'ocamlformat_diff.dev' '.']
387	duniverse: [ERROR] opam pin --root=/tmp/.duniverse-opam-root-fda284 add -yn
388	ocamlformat_diff.dev . failed. Output was:
389	[NOTE] No package definition found for ocamlformat_diff.dev: please complete
390	the template
391	Error opening terminal: unknown.
392	[ERROR] Empty file or editor error, aborting.
393	[ERROR] No valid package definition found
394	Package ocamlformat_diff does not exist, create as a NEW package? [Y/n]
395	y
396	[ocamlformat_diff.dev: git]
397	[ocamlformat_diff.dev: git]
398	[ocamlformat_diff.dev: git]
399	[ocamlformat_diff.dev: git]
400	[ocamlformat_diff.dev: git]
401	[ocamlformat_diff.dev: git]
402	[ocamlformat_diff.dev: git]
403	[ocamlformat_diff.dev] synchronised from
404	git+file:///drone/src#master
405	duniverse: Command execution failed

https://cloud.drone.io/dune-universe/opam-overlays/207/1/5

Inconsistent solution when incompatible packages from the same repository are selected

Hi,

When requesting different versions from packages that come from the same repository, some invalid solutions can be selected. For example requesting dune 2.7.1 and dune-build-info 2.6.2 can create the following lockfile fragment:

  [
    "dune-build-info.2.6.2"
    "https://github.com/ocaml/dune/releases/download/2.7.1/dune-2.7.1.tbz"
  ]

duniverse pull removes other existing submodules

Hi! First of all, let me say that duniverse is a great initiative! It is so nice to only rely on dune 👍

Explaining the problem:

If a submodule exists (not related to duniverse) in the same repo, duniverse will remove it when doing it's duniverse pull magic.

Reproducing:

Add a dummy submodule:

git submodule add https://github.com/githubtraining/hellogitworld.git mysubmodules/hellogitworld

run:

duniverse pull

And now you can confirm that duniverse removed the mysubmodules/hellogitworld from .gitmodules.

Duniverse does not work well with opam pins

If there is a pin-depends field in the opam file duniverse will fail to generate the correct fetch url as it's mostly based on the dev-repo rather than the url.src field.

For example if dev-repo is git+https://github.com/inhabitedtype/httpaf.git and url.src is git+https://github.com/anmonteiro/httpaf.git#mirage it will try to fetch from git+https://github.com/inhabitedtype/httpaf.git#mirage which doesn't exist.

tarides / opam-monorepo Goto Github PK

opam-monorepo's Introduction

opam-monorepo

Installation

Usage

The lock Command

opam monorepo pull