dylan-lang / dylan-tool Goto Github PK

Probably want to leave the submodule stuff there for now at least.

Fully document dylan-package.json

It should remove and regenerate the registry from scratch.

(When we have build and test they should also have this option but it will remove _build.)

Notice the output is "dylan-tool-app library", not "dylan-tool-app new library".

$ dylan-tool-app help new library
Create a new shared library and its test library.

Usage: dylan-tool-app library [options] NAME [DEPS] ...

Support SemVer pre-release versions

1. Add dep [email protected] to some project
2. dylan update
3. Remove dep strings from the pkg.json file
4. dylan update
5. There is still a registry entry for strings that points to the strings 1.0 package.

If strings is still being used it should be picked up from the Open Dylan installation or (if it's not in the OD install) the user should get an error.

Conceptually, we should simply remove the entire registry directory before dylan update generates a new one. What I would prefer is to actually show the changes that were made, so just note the files that exist at the start and remove any that don't get overwritten.

This would avoid having to list all the platforms and would allow us to write the registry entry in the registry/generic/. It's also nice for self-documenting purposes compared to "here's a list of platforms; you decide if it's complete".

For later... It might be worth having a dylan new [dev] dependency pkg1 [pkg2 ...] subcommand since it could

• verify that the dep exists
• verify that the dep doesn't conflict
• prevent simple syntax errors in the package file

Currently the following file hierarchy is assumed:

my-workspace/workspace.json
my-workspace/_build
my-workspace/registry
my-workspace/active-package-1
my-workspace/active-package-1/dylan-package.json
my-workspace/active-package-2
my-workspace/active-package-2/dylan-package.json
...

It shouldn't be necessary to have that extra level of directory structure for my-workspace when working on a single package, which, I think it's safe to assume, will be a very common case.

Instead all the dylan subcommands should work inside a bare package checkout. All that's needed to make this work is to first look for a workspace (workspace.json) and then fall back to looking for a package (dylan-package.json). (Then fall back to looking for "registry".)

In order to keep the workspace.json functionality (which is currently only a "default-library" setting), dylan-package.json could have a "workspace" attribute that is treated exactly the same as the contents of workspace.json.

(And workspace.json should probably be called dylan-workspace.json).

How exactly would these settings work when a workspace contains multiple packages with multiple "workspace" configs...not sure. Combine and override? just use dylan-workspace.json and ignore the package-level settings? Dunno!

This is kinda bad because it'll be, y'know, the first time someone is trying to publish something when they encounter this problem.

$ dylan publish ../pacman-catalog/
WARNING: Using override catalog from $DYLAN_CATALOG: ../pacman-catalog/
No applicable method, applying {<incremental-generic-function>: <} to {<simple-object-vector>: #f, {<release>}}.

I was trying to publish the first release of objc-dylan.

This hasn't come up before because I published all the initial releases by hand before the publish command existed.

See original bug filed against pacman before it was integrated into dylan-tool: dylan-lang/pacman#2

Publishing a new version of xml-parser I use the command dylan publish ../pacman-catalog.

In pacman-catalog the diff shows that "category" is empty where before there was the value "parsers" and contact is missing the email.

$ git diff v1/xm/l-/xml-parser
diff --git a/v1/xm/l-/xml-parser b/v1/xm/l-/xml-parser
index 21646e5..2a9b124 100644
--- a/v1/xm/l-/xml-parser
+++ b/v1/xm/l-/xml-parser
@@ -1,13 +1,21 @@
 {
-  "category": "parsers",
-  "contact": "[email protected]",
+  "category": "",
+  "contact": "",
   "description": "XML parser",
   "keywords": [],
   "name": "xml-parser",
   "releases": [{
-                 "dependencies": ["[email protected]"],
-                 "deps": ["[email protected]"],
-                 "dev-dependencies": ["[email protected]"],
+                 "dependencies": ["[email protected]"],
+                 "deps": ["[email protected]"],
+                 "dev-dependencies": ["[email protected]"],
+                 "license": "MIT",
+                 "url": "https://github.com/dylan-lang/xml-parser",
+                 "version": "0.2.1"
+               },
+               {
+                 "dependencies": ["[email protected]"],
+                 "deps": ["[email protected]"],
+                 "dev-dependencies": ["[email protected]"],
                  "license": "MIT",
                  "url": "https://github.com/dylan-lang/xml-parser",
                  "version": "0.2.0"

If you run dylan update in the dylan-tool directory itself, it writes platform-specific registry files that shadow the generic registry entries that are under version control. Either it should prompt the user "ok? (yes/no)" or just abort with a message.

I added the "xml head" dependency to pkg.json and ran dylan update and got this error:

$ dylan update
update-head? = #f
Workspace directory is /home/cgay/dylan/workspaces/playground/.
update-head? = #f
Reading package file /home/cgay/dylan/workspaces/playground/web-playground/pkg.json
Installing deps for package web-playground.
Cloning into '/home/cgay/dylan/pkg/pacman-catalog/head/src'...
remote: Enumerating objects: 79, done.
Receiving objects: 100% (79/79), 15.28 KiB | 15.28 MiB/s, done.
Resolving deltas: 100% (40/40), done.
remote: Total 79 (delta 0), reused 0 (delta 0), pack-reused 79
Deleting package pacman-catalog head for forced install.
Loaded 63 packages with 66 versions from /home/cgay/dylan/pkg/local-catalog.json.
Error: #f is not of type {<class>: <pkg>}

It should obviously say something like "while updating package dependencies, xml is not a recognized package name"

dylan new application --simple foo needn't include Testworks as a dev dependency since no test suite is created.

Just noticed that Files: is allowed to be specified multiple times in one LID file, and I'm sure I didn't add support for that. Low priority.

The dylan update subcommand isn't showing up in the help output:

$ dylan help
Tool to maintain Dylan dev workspaces and installed packages.

Usage: dylan [options] <cmd> [cmd options] args...

Options:
  -h, --help      Display this message.
      --verbose   Generate more verbose output.
      --debug     Enter the debugger (or print a backtrace) on error.

Subcommands:
  help         Display help for a subcommand.
  install      Install Dylan packages.
  list         List installed Dylan packages.
  new          
    library    Create a new library and its test library.
    workspace  Create a new workspace.

Use 'dylan help <cmd> [<cmd> ...]' to see subcommand options.

My guess is that adding the multi-level "new" subcommands is the cause.

There is some overlap between what the project-wizard library does and dylan new application/library. Figure out whether code can be usefully shared between them. Currently project-wizard is win32-only and is itself a DUIM application so I suspect at a minimum it would need to be broken into two libraries.

Does project-wizard generate any test code? If not, we can share in both directions.

I created v2.0.0 of the strings package but pacman-catalog-test-suite failed when I tried to publish it. It tries to resolve the dependencies of [email protected] and runs into a conflict because testworks (a strings dev dependency) requires [email protected]:

Resolving deps for [email protected]
%resolve-deps(deps: #([email protected]), seen: #())
  resolve-release(rel: {<release> [email protected] 17}, seen: #())
    %resolve-deps(deps: #(), seen: #("strings"))
    <= #()
  caching [email protected] => #()
<= #([email protected])
%resolve-deps(deps: #(sphinx-extensions, testworks), seen: #())
  resolve-release(rel: {<release> [email protected] 18}, seen: #())
    %resolve-deps(deps: #(), seen: #("sphinx-extensions"))
    <= #()
  caching [email protected] => #()
  resolve-release(rel: {<release> [email protected] 2}, seen: #())
  <= memoized result #([email protected], [email protected], [email protected])
<= #([email protected], [email protected], [email protected], [email protected], [email protected])
%resolve-deps(deps: #([email protected], [email protected], [email protected], [email protected], [email protected], [email protected]), seen: #())
  resolve-release(rel: {<release> [email protected] 17}, seen: #())
  <= memoized result #()
  resolve-release(rel: {<release> [email protected] 2}, seen: #())
  <= memoized result #([email protected], [email protected], [email protected])

  validate-catalog(catalog()) doesn't signal an error : [Error evaluating assertion expression: dependencies on conflicting major versions of the same package: "[email protected]" and "[email protected]" (path: strings)]
   FAILED in 0.056013s and 4MiB
Completed suite pacman-catalog-test-suite: FAILED in 0.056013s

This comment in deps.dylan is a bit suspect. Need to think about that and the right way to resolve dependencies with and without including dev dependencies.

dylan build --clean --all invokes dylan-compiler multiple times, each time with the -clean option. Should only pass -clean on the first invocation.

For example, dylan build -a in a Testworks workspace on Linux tries to build testworks-gui.

In particular, _test, _build, and _packages should be ignored.

17:52:55 ~/dylan/workspaces/dylan-tool (dev*) 
$ dylan-tool-app update
Downloaded pacman-catalog@master to /home/cgay/.local/state/dylan/_packages/pacman-catalog/master/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/strings/1.1.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/command-line-parser/3.1.1/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/uncommon-dylan/0.2.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/json/1.0.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/logging/2.1.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/sphinx-extensions/0.2.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/testworks/2.0.0/src/
Downloaded [email protected] to /home/cgay/dylan/workspaces/dylan-tool/_packages/regular-expressions/0.2.0/src/
WARNING: LID file /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-parse-lid-file--lid-header/child.lid has no (transitive) 'Files' property.
WARNING: LID file /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-parse-lid-file--lid-header/parent.lid has no (transitive) 'Files' property.
WARNING: LID file /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-parse-lid-file--lid-header/child.lid has no (transitive) 'Files' property.
WARNING: LID file /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-parse-lid-file--lid-header/parent.lid has no (transitive) 'Files' property.
WARNING: Library "abc" has multiple .lid files for platform "x86_64-linux".
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-source-file-map--included-lid/abc.lid
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-source-file-map--included-lid/abc.lid
Registry will point to the first one, arbitrarily.
WARNING: Library "foo" has multiple .lid files for platform "x86_64-linux".
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-parse-lid-file--lid-header/parent.lid
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-source-file-map--basics/foo.lid
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-parse-lid-file--lid-header/parent.lid
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-source-file-map--basics/foo.lid
Registry will point to the first one, arbitrarily.
WARNING: Library "abc-test-suite" has multiple .lid files for platform "x86_64-linux".
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175146/test-source-file-map--included-lid/abc-test-suite.lid
  /home/cgay/dylan/workspaces/dylan-tool/_test/cgay-20230528-175015/test-source-file-map--included-lid/abc-test-suite.lid
Registry will point to the first one, arbitrarily.
Updated 27 files in /home/cgay/dylan/workspaces/dylan-tool/registry/.

This could be a common newb mistake. Checkout some repo that has no dylan-package.json, run dylan update, and just see "The registry is up-to-date" even though it did nothing because there are no active packages. Should warning if no active packages were found, even if not doing verbose output.

Add an option to include basic command-line parsing in the generated app.

"Full documentation is here" has a broken link

dylan update --latest-deps (or similar flag name) should rewrite dylan-package.json with dependencies updated to the latest versions. This is toil that needs to be done relatively frequently so it should be made easy.

Possibly dylan-status should also show what deps are behind as well, but for now dylan update can fix it and git diff can show the result.

dylan deps foo -- show transitive dependencies of foo

dylan deps foo --reverse -- show who depends on foo

dylan deps foo --reverse --indirect -- also show indirect users of foo

Or something like that. Command name and flags up for debate.

• need better understanding of the goals of deft
• need to see if deft can be made to work in emacs shell (command-interface)
• talk with Bruce about me taking it over. does he still have plans want to influence it going forward?
• dylan-tool needs subcommands in command-line-parser, soon.

We're going to need this to know which platforms to test on when new versions are published. c.f. objc-dylan

Or we could scan the package's LID files for the Platforms: header to figure it out? That would avoid the data duplication but would it cause other problems?

https://opendylan.org/documentation/getting-started-cli/platform-specific.html#lid-file

So that it's possible to use non-git packages, or get packages on a machine that doesn't have git installed.

Workspaces should support multi-platform development. For example using a shared NFS mount for sources and a different _build directory on each platform.

It might be the case that if the user sets OPEN_DYLAN_USER_ROOT everything will Just Work, but this should be tested. Leaving this bug here as a reminder.

Registry generation always writes to a platform specific directory so as long as dylan update is run on each platform the registry part should work.

I've decided I really like putting library source code in a subdirectory called "sources" or even (shudder) "src", because of the nice, tidy feeling it gives to the top directory, making any top-level files like README easily visible.

Let's make dylan new library do this.

It shouldn't be necessary to publish a package in the public catalog in order to use it as a dependency. An organization might want to develop a suite of internal packages, for example.

Git submodules could be used to accomplish this, but then you'd be using git submodules, and nobody wants that. Also they wouldn't participate in dependency resolution.

Not sure how I want to handle this yet. Support multiple catalogs with precedence order probably?

This will be much like make-dylan-app, but will also

1. add the new app to the active packages in the current workspace.
2. run git init, maybe?
3. accept a list of dependencies to add?
4. dylan new should offer to create packages if they don't exist.
5. Always create them in the workspace top-level directory, regardless of current dir.

As mentioned in dylan-lang/opendylan#964