This morning we noticed registry CI started to fail (using Julia Version 1.3.1, RegistryCI v2.9.0, RegistryTools v1.5.2)

The failures look like this:

Test Failed at /root/.julia/packages/RegistryCI/yFHmI/src/registry_testing.jl:135
  Expression: compressed == compat
   Evaluated: Dict("0.1-0" => Dict("julia" => "[\"0.6\", \"0.7\", \"1\"]","StatsBase" => "0.15-*"),"1.0.0" => Dict("StatsBase" => "0.15-*"),"1.0.2-1" => Dict("julia" => "1","StatsBase" => "0.28-0.32"),"0.0" => Dict("julia" => "[\"0.5\", \"0.6\", \"0.7\", \"1\"]"),"1-1.0.1" => Dict("julia" => "[\"0.7\", \"1\"]"),"1.0.1" => Dict("StatsBase" => "0.15.0-*")) == Dict{String,Any}("0.1-0" => Dict{String,Any}("julia" => ["0.6", "0.7", "1"],"StatsBase" => "0.15-*"),"1.0.0" => Dict{String,Any}("StatsBase" => "0.15-*"),"1.0.2-1" => Dict{String,Any}("julia" => "1","StatsBase" => "0.28-0.32"),"0.0" => Dict{String,Any}("julia" => ["0.5", "0.6", "0.7", "1"]),"1-1.0.1" => Dict{String,Any}("julia" => ["0.7", "1"]),"1.0.1" => Dict{String,Any}("StatsBase" => "0.15.0-*"))

Looking at the difference here (this is representative of all cases):

julia> compressed
Dict{String,Dict{String,String}} with 6 entries:
  "0.1-0"   => Dict("julia"=>"[\"0.6\", \"0.7\", \"1\"]","StatsBase"=>"0.15-*")
  "1.0.0"   => Dict("StatsBase"=>"0.15-*")
  "1.0.2-1" => Dict("julia"=>"1","StatsBase"=>"0.28-0.32")
  "0.0"     => Dict("julia"=>"[\"0.5\", \"0.6\", \"0.7\", \"1\"]")
  "1-1.0.1" => Dict("julia"=>"[\"0.7\", \"1\"]")
  "1.0.1"   => Dict("StatsBase"=>"0.15.0-*")

julia> compat
  "0.1-0"   => Dict{String,Any}("julia"=>["0.6", "0.7", "1"],"StatsBase"=>"0.15-*")
  "1.0.0"   => Dict{String,Any}("StatsBase"=>"0.15-*")
  "1.0.2-1" => Dict{String,Any}("julia"=>"1","StatsBase"=>"0.28-0.32")
  "0.0"     => Dict{String,Any}("julia"=>["0.5", "0.6", "0.7", "1"])
  "1-1.0.1" => Dict{String,Any}("julia"=>["0.7", "1"])
  "1.0.1"   => Dict{String,Any}("StatsBase"=>"0.15.0-*")

The difference seems to be the type signature Dict -> Dict{String,Any} and the string representation of the versions e.g. "[\"0.7\", \"1\"]" -> ["0.7", "1"]

I ran this script (JuliaRegistries/RegistryCI.jl#224 (comment)) using Julia 1.3.1 (same result in Julia 1.5.1) with RegistryTools v1.5.1 on the private registry and it made no changes. I ran it with RegistryTools v1.5.2 and it made changes to the string representation like above, causing CI fail like:

Error During Test at /root/.julia/packages/RegistryCI/yFHmI/src/registry_testing.jl:131
  Test threw exception
  Expression: load_compat(compatfile, vnums)
  ArgumentError: invalid version range: "[\"0.7\", \"1\"]"
  Stacktrace:
   [1] Pkg.Types.VersionRange(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/versions.jl:114
   [2] Pkg.Types.VersionSpec(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/versions.jl:197
   [3] load_package_data(::Type{Pkg.Types.VersionSpec}, ::String, ::Array{VersionNumber,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:115
   [4] load_compat at /root/.julia/packages/RegistryCI/yFHmI/src/registry_testing.jl:61 [inlined]
   [5] macro expansion at /root/.julia/packages/RegistryCI/yFHmI/src/registry_testing.jl:131 [inlined]
   [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1180 [inlined]
   [7] (::RegistryCI.var"#6#18"{Array{String,1}})() at /root/.julia/packages/RegistryCI/yFHmI/src/registry_testing.jl:88

Here's an example of the changes the link script made to a Compat.toml when using RegistryTools v1.5.2

--- a/T/TempPackage/Compat.toml
+++ b/T/TempPackage/Compat.toml
@@ -1,12 +1,12 @@
 ["0.0"]
-julia = ["0.5", "0.6", "0.7", "1"]
+julia = "[\"0.5\", \"0.6\", \"0.7\", \"1\"]" ["0.1-0"]
 StatsBase = "0.15-*"
-julia = ["0.6", "0.7", "1"]
+julia = "[\"0.6\", \"0.7\", \"1\"]" ["1-1.0.1"]
-julia = ["0.7", "1"]
+julia = "[\"0.7\", \"1\"]" ["1.0.0"]
 StatsBase = "0.15-*"

For now, pinning the private registry CI to use RegistryTools v1.5.1 avoids the issue.

See

• JuliaRegistries/General@581f658#diff-a8040c6047a79184f490209a15a9d343
• JuliaRegistries/General@7e678ed#diff-ff516c7a7195ef333bcd3158dcf44801
• JuliaRegistries/General@2c20578#diff-5ced124a3c260b01c7910b63897e735a

RegistryTools exports one type and three functions:

• RegBranch
• register
• check_and_update_registry_files
• find_registered_version

There are seven packages in General depending on RegistryTools.
Ignoring the exported symbols, they also make use of the following:

• BinaryBuilder: RegistryCache, registrator_errors.
• LocalPackageServer: gitcmd.
• LocalRegistry: gitcmd, RegistryData, write_registry, ReturnStatus, haserror, Compress.load.
• PkgDev: -
• Registrator: RegisterParams.
• RegistryCI: package_relpath, get_stdlib_name, stdlibs, Compress.compress, Compress.load.
• RetroCap:Compress.load, Compress.save.

Would it seem reasonable to simply define the public API as the union of the listed symbols and add docstrings to the ones which don't have one?

Unfortunately, we don't seem to test the full register() codepath, which ended up with a broken release (easily fixed by f34de60). Since this package is used by some important things like BB and JuliaRegistrator, it would be good to get some more end-to-end tests going.

How should we do this? We obviously don't want to be opening pull requests to General every time we make a commit here, but perhaps we can create a fork and manipulate the pull requests there?

Would it make sense that register throws an error if things didn't work? The content of the .metadata field could just be part of a custom error message?

Right now it is not clear to me what exactly I should check after a call to register to make sure everything worked as intended.

I just tried to register a super simple package of mine, apparently too simple.

It seams to fail because it has no dependencies and the RegistryCI cannot handle this. Here the stacktrace:

ERROR: LoadError: "/home/runner/work/General/General/N/NotMacro/Deps.toml": No such file
Stacktrace:
  [1] error(::String, ::String)
    @ Base ./error.jl:44
  [2] readstring
    @ /opt/hostedtoolcache/julia/1.8.5/x64/share/julia/stdlib/v1.8/TOML/src/TOML.jl:21 [inlined]
  [3] parsefile(f::String)
    @ TOML /opt/hostedtoolcache/julia/1.8.5/x64/share/julia/stdlib/v1.8/TOML/src/TOML.jl:43
  [4] meets_compat_for_all_deps(working_directory::String, pkg::String, version::VersionNumber)
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/guidelines.jl:83
  [5] (::RegistryCI.AutoMerge.var"#83#84")(data::RegistryCI.AutoMerge.GitHubAutoMergeData)
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/guidelines.jl:76
  [6] check!(guideline::RegistryCI.AutoMerge.Guideline, data::RegistryCI.AutoMerge.GitHubAutoMergeData)
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/types.jl:138
  [7] pull_request_build(data::RegistryCI.AutoMerge.GitHubAutoMergeData; check_license::Bool)
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/pull_requests.jl:211
  [8] pull_request_build(api::GitHub.GitHubWebAPI, pr_number::Int64, current_pr_head_commit_sha::String, registry::GitHub.Repo, registry_head::String; whoami::String, auth::GitHub.OAuth2, authorized_authors::Vector{String}, authorized_authors_special_jll_exceptions::Vector{String}, error_exit_if_automerge_not_applicable::Bool, master_branch::String, master_branch_is_default_branch::Bool, suggest_onepointzero::Bool, registry_deps::Vector{String}, check_license::Bool, public_registries::Vector{String}, read_only::Bool, environment_variables_to_pass::Vector{String})
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/pull_requests.jl:166
  [9] run(; env::Base.EnvDict, cicfg::RegistryCI.AutoMerge.GitHubActions, merge_new_packages::Bool, merge_new_versions::Bool, new_package_waiting_period::Day, new_jll_package_waiting_period::Minute, new_version_waiting_period::Minute, new_jll_version_waiting_period::Minute, registry::String, tagbot_enabled::Bool, authorized_authors::Vector{String}, authorized_authors_special_jll_exceptions::Vector{String}, additional_statuses::Vector{String}, additional_check_runs::Vector{String}, error_exit_if_automerge_not_applicable::Bool, master_branch::String, master_branch_is_default_branch::Bool, suggest_onepointzero::Bool, registry_deps::Vector{String}, api_url::String, check_license::Bool, public_registries::Vector{String}, read_only::Bool, environment_variables_to_pass::Vector{String})
    @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/51I29/src/AutoMerge/public.jl:141
 [10] top-level scope
    @ ~/work/_temp/2e56103b-ce75-4e54-b2cd-9c7e89bc80[72](https://github.com/JuliaRegistries/General/actions/runs/3927097124/jobs/6713411399#step:16:73):4
in expression starting at /home/runner/work/_temp/2e56103b-ce[75](https://github.com/JuliaRegistries/General/actions/runs/3927097124/jobs/6713411399#step:16:76)-4e54-b2cd-9c7e89bc[80](https://github.com/JuliaRegistries/General/actions/runs/3927097124/jobs/6713411399#step:16:81)72:4
Error: Process completed with exit code 1.

I would be very thankful if someone could manually merge the registration pullrequest which failed in the meantime.

This line marks differences in URLs as an error that it probably shouldn't, in particular the following two URLs should probably be treated as identical: https://github.com/Foo/Bar.jl and https://github.com/Foo/Bar.jl.git.

I'm not familiar with the code, though, to propose a fix.

The check for change of repo URL uses string equality.

This broke a thing for me, since it was mistakenly created as foobar.jl rather than FooBar.jl,
registered, then fixed.
It doesn't cause any problems since gitlab and github redirect to be case-insensitive AFAIK.

But should we be that demaining?
Apparently by the spec URLs (but not domains) are allowed to be case sensitive.
https://stackoverflow.com/questions/7996919/should-url-be-case-sensitive

This line:

Doesn't work on master, because Pkg.Display doesn't exist anymore. If I'm understanding the code correctly, perhaps the analogous method from BinaryBuilder would be helpful here:

https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/ac292e96d1ce8a5354b61cfa115ed210ad7d0821/src/BinaryBuilder.jl#L117-L132

The current code requires access to the tree_hash field of PackageEntry, which was added in Pkg 1.2. However, there is no compat listed for Pkg, and julia compat is listed as 1.1 which has a corresponding 1.1 version of Pkg.

I'm not sure whether it's best to add a julia compat requirement or a Pkg compat requirement but I think one or the other should be added so people with older Julia versions get an error on build when they need to upgrade (we only discovered this error at runtime after deploying a new version of registrator with Julia 1.1). We have no problem updating our Julia version in this case but we'd ideally like to avoid breaking our production registrator because of compatibility issues.

Does this package now require Julia 1.4? I see it's only tested that far back.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Don't know what this is trying to test exactly:

Right now it gets

compat_file: Test Failed at /home/pkgeval/.julia/packages/RegistryTools/2CM18/test/regedit.jl:333
  Expression: read(compat_file, String) == "[1]\njulia = \"1.3.0-1\"\n"
   Evaluated: "[1]\njulia = \"1.3.0 - 1\"\n" == "[1]\njulia = \"1.3.0-1\"\n"
``

What is the proper workflow for working with forks of JuliaRegistries/General? In BinaryBuilder, we open pull requests against General with the following code:

https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/28a943745baa8d248401d4246efb460e37cb859f/src/AutoBuild.jl#L416-L448

Unfortunately, because we are specifying our fork as the origin to register(), we never update the master branch of our fork. This causes our registration PR base commits to fall further and further behind the true upstream master. It almost feels like we need to teach register() to take in an upstream kwarg, so that it can first reset to remotes/upstream/master, then create the new branch. I believe this workflow will become more and more common as we have more and more bots that do not have commit access to JuliaRegistries/General.

If there is a package Aa in the registry, and you register a package AA, then the folders needs to be e.g. AA2 and not `AA, see JuliaRegistries/General#16341.

It is natural that RegistryTools show traces of its Registrator origin, but to be generally useful I think these need to be further decoupled. Some observations from trying to use the code and reading through it:

• Doing checks on the package to be registered is intertwined with updates of the registry files. This can be practical from a code flow point of view, but is not ideal for unit testing or if you want to reuse the file updates but do your own checks.

• As observed in #6, the code has too strong opinions on what can be registered. It should be possible to opt out of all checks that don't compromise registry integrity.

• Failed package checks are returned as end user messages such as """This looks like a new registration that registers version $version. Ideally, you should register an initial release with 0.0.1, 0.1.0 or 1.0.0 version numbers""". This is quite inconvenient to make sense of from code or if you want to communicate with the user in any other way than that particular message.

• The entire RegBranch interface strikes me as weird. The package name and version, and the name of a registry branch is passed around everywhere in the code but (with one exception) not used at all. Meanwhile the status information picked up is placed in a dict, with unnecessarily cumbersome updates. See #13 for an attempt to address this. Moreover the package name and version as well as the branch name should be known to the caller. It is unclear why it should be the responsibility of register to put these into a struct together with the result of the registration checks.

• The branch argument of register defaults to the output of the registration_branch function. That looks rather specific to Registrator and should move back there.

• The get_registrator_tree_sha business and the commit message make little sense outside of Registrator.

• The register function in its entirety is kind of difficult to use if you want a different registration workflow than Registrator. I'm not sure if there's a good way to improve that or if it should be left to other users to write their own register replacements. In the latter case functions to check and update the individual registry files should also be exported.

Some suggestions on how to move forward:

• Checks and registry file updates are decoupled into separate functions. Update: Implemented in #15.
• Checks are identified by a symbol (or a symbol plus the arguments needed to build the current messages) instead of an end user message. Update: Implemented in #16.
• Opting out of checks could be implemented by a caller argument specifying which of the check symbols should generate errors or warnings. Update: Implemented in #16.
• RegBranch, registration_branch, and get_registrator_tree_sha are moved back to Registrator.
• The commit message could be constructed by a callback function.

As discussed in JuliaLang/Pkg.jl#1251, having registered packages in a subdirectory of a git repository works partially today. If you prepare a registry with the right tree hashes, add Package and add Package@revision work out of the box. Additional Pkg functionality is added in the WIP PR JuliaLang/Pkg.jl#1422.

However, for add Package#branch or dev Package there is no information available in the registry about the subdirectory. Thus either Pkg would have to hunt through the repository for a matching Project.toml or the user would have to provide the subdirectory path for these operations. Neither of these options seems very satisfactory.

My proposal is that Package.toml is modified to include this information. Two possibilities are to change (in a hypothetical world where RegistryTools lives in a subdirectory) today's

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git"

into either

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git"
path = "path/to/package"

or

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git?path=path/to/package"

Both of these would be backwards compatible with current Julia versions since no changes are required for normal packages. For packages in subdirectories they differ in how they don't work properly beyond add Package and add Package@version. In the former case, as well as without path information in the registry, add Package#branch and dev Package downloads/clones the top level repository and end up with a broken package. In the latter case these operations instead result in an immediate, though not very clear, error.

Note that packages in subdirectories is a useful enough feature that it would be worth using even without improved Pkg support. It would be good to settle this issue right away though, since adding path information to existing registries with such packages would be an unnecessary complication.

Opinions?

Some changes in Pkg internals

  Expression: check_result(registry_dir, "registry1")
  MethodError: no method matching load_versions(::String; include_yanked=false)
  Closest candidates are:
    load_versions(::Any, !Matched::String; include_yanked) at /workspace/srcdir/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:154
  Stacktrace:
   [1] sanity_check_registry(::String) at /home/pkgeval/.julia/packages/LocalRegistry/EvVG4/test/utils.jl:113
   [2] check_result(::String, ::String) at /home/pkgeval/.julia/packages/LocalRegistry/EvVG4/test/utils.jl:55
   [3] top-level scope at /home/pkgeval/.julia/packages/LocalRegistry/EvVG4/test/runtests.jl:65
   [4] include(::String) at ./client.jl:457
   [5] top-level scope at none:6
   [6] eval(::Module, ::Any) at ./boot.jl:331
   [7] exec_options(::Base.JLOptions) at ./client.jl:272
   [8] _start() at ./client.jl:506

I think it would be better if RegistryTools just parsed the TOML files of the package and stop relying on Pkg completely. I don't think it requires that much more logic and it avoids having to run Registrator on new Julia versions (possibly nightly) when new features in Pkg are added.

Error:

 ! [remote rejected] HEAD -> registrator/testpackage/7876af07/v0.13.19/59175fcecb (cannot lock ref 'refs/heads/registrator/testpackage/7876af07/v0.1.1/59175fcecb': 'refs/heads/registrator/testpackage/7876af07/v0.0.1' exists; cannot create 'refs/heads/registrator/privateexample3/7876af07/v0.13.19/59175fcecb')

With RegistryTool version < 1.6.0 , branch created on registry of form /registrator/<packagename>/<uuid slug>/<version>
With #63, we have /registrator/<packagename>/<uuid slug>/<version>/url_hash.
As per https://stackoverflow.com/questions/22630404/git-push-refs-heads-my-subbranch-exists-cannot-create/22630664#22630664, we cannot force a branch A/x if A exists.

So can we use - instead of / ie registrator/<packagename>/<uuid slug>/<version>-<url_hash> instead of registrator/<packagename>/<uuid slug>/<version>/<url_hash> ?

@DilumAluthge @nkottary

Hi,

In register, there's an assumption that the default branch of the registry is master. I'd like to be able to override this with some other branch (main).
https://github.com/JuliaRegistries/RegistryTools.jl/blob/master/src/register.jl#L652-L653

Happy to issue a pr to just have the branch be passed in as an argument, but figured I'd open the issue first to see if there's any other nuances I should consider in updating the function

RegisterParams seems completely unused (and untested) in RegistryTools.jl. It seems this struct should exist in Registrator.jl instead.

RegistryTools.jl uses the Pkg.Compress submodule. See e.g.

However, on Pkg master, the Pkg.Compress module has been removed. The code is still available in the bin directory (https://github.com/JuliaLang/Pkg.jl/tree/master/bin). But the Pkg.Compress submodule is not available when you import Pkg or using Pkg.

Therefore, if you try to import RegistryTools on Julia nightly, it fails because the import Pkg.Compress.load_versions command fails.

cc: @davidanthoff

Continuing discussion in JuliaRegistries/General#5098 and JuliaLang/Pkg.jl#1492:

I find manually updating Compat.toml is hard and rather scary as it's error-prone, especially because Compat.toml does some kind of "compressing" (or generating reverse dict or something?). It would be nice to have an automated tool to introduce new bounds in Compat.toml.

stdlib is removed from https://github.com/JuliaRegistries/RegistryTools.jl/blob/master/src/builtin_pkgs.jl.

EDIT: I was confused, the following description is inaccurate:

What is the right way to register a patch release on an old version of a package? E.g. If I have 1.2.1 and 1.3.0 registered, attempting to register 1.2.2 seems like it will fail due to

EDIT: The true issue is that I registered v2.X first, so then v1.X is earlier than all other versions.

See e.g. #64

There are only a bunch of very light and generic docstrings (it's often unclear what the arguments of the functions are supposed to be), but it's impossible for someone who's not familiar with package to have a clue of where to start from.