commercialhaskell / curator Goto Github PK

In LTS 14.10 the following were unintentionally dropped:

• replace-attoparsec-1.0.2.0
• replace-megaparsec-1.1.4.0
• servant-rawm-0.3.2.0

https://www.stackage.org/diff/lts-14.9/lts-14.10

We should look into how this is possible; it is likely a curator bug.

When a user supplies curator with malformed YAML, it returns the parse errors from the yaml package verbatim, with little to no additional context.

For example, given this malformed input to build-constraints.yaml:

# Constraints for brand new builds
packages:
  "Alice Roberts" <[email protected]> @aroberts":

...curator will produce the following error (or similar):

curator: InvalidYaml (Just (YamlParseException {yamlProblem = "did not find expected key", yamlContext = "while parsing a block mapping", yamlProblemMark = YamlMark {yamlIndex = 388, yamlLine = 11, yamlColumn = 18}}))

It's entirely reasonable that this issue is outside of the scope of curator, since it would rely on yaml returning more structured information for us to turn into a more meaningful response.

This was just reported in commercialhaskell/stackage#5520, and I figured this would be a better place to open a discussion.

The range field should be under source in constraints.yaml (for LTS etc).

However it is possible to put it in the wrong place, ie not under source, which then does nothing (or rather is a time-bomb for the following week's lts).
It would be nice to fail parsing in that case, or liberally (probably harder) allow range there.
Perhaps a warning about unknown fields could also help if not there, maybe also harder.

Add some curator commands that reads build-constraints.yaml

• curator view-package <package>
• curator view-maintainer <maintainer>

Format could be similar to what I wanted in #20

This doesn't require any changes to build-constraints.yaml.

Changing the format (explicit comments, issue links, grouping...) would allow us to get more information out of these commands, but I'll stick that in another issue.

See commercialhaskell/stackage#5357 (comment)

I think the curator is not generating flags for text. The issue, I believe, is that text is not included in build-constraints.yaml as a package, and so Curator.Stackage.convert doesn't look at its flags at all.

However if I patch Curator.Stackage.convert to consider package-flags separately it would then have to include text in Constraints.consPackages (which is probably undesired (?) because text ships with ghc).

So I guess Constraints should get some additional field in which to store flags? And they can then be removed from PackageConstraints?

Related issues:
commercialhaskell/stack#5307
commercialhaskell/stackage#5409

For a good while Stackage distro uploads to Hackage have been failing like this:

:
To github.com:commercialhaskell/stackage-snapshots
   3eb56ec..cdae8cd  HEAD -> master
Uploading Hackage distro for snapshot.yaml
Uploading as Hackage distro: Stackage
curator: HttpExceptionRequest Request {
  host                 = "hackage.haskell.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Authorization","<REDACTED>"),("Content-Type","text/csv")]
  path                 = "/distro/Stackage/packages.csv"
  queryString          = ""
  method               = "PUT"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 IncompleteHeaders
Fri Jan 24 18:00:28 UTC 2020

I am not clear what the exact cause is.

(I had originally created this issue at fpco/stackage-curator#70.)

For example an issue like

neat-interpolation-0.4 (<name> <email> <github-username>) is out of bounds for:
...

…could look like this instead:

neat-interpolation-0.4 (<name> <email> <github-username>, changelog) is out of bounds for:
...

IMHO that would be a nice convenience for the maintainers that want to update their package.

When building snapshots with curator the volume of ghc warning output is often quite unmanageable.
I think we should suppress warnings output. eg a recenty nightly snapshot output over 200k lines of warnings.

See also commercialhaskell/stack#5566 for related context.

We can try experimenting with --no-dump-logs or dump-logs:.

E.g. I got this output when trying to re-enable tasty

tasty-1.4.1 ([changelog](http://hackage.haskell.org/package/tasty-1.4.1/changelog)) [...] is out of bounds for:
[...]
- [ ] tasty-ant-xml-1.1.7 (>=0.10 && < 1.4). [...]. Used by: library

this lead me to believe that i needed to disable tasty-ant-xml, but there was an upper bound on it that can be removed instead.

Would help if the output was something like

tasty-1.4.1 ([changelog](http://hackage.haskell.org/package/tasty-1.4.1/changelog)) [...] is out of bounds for:
[...]
- [ ] tasty-ant-xml-1.1.7 (>=0.10 && < 1.4). [...]. Used by: library (Later version available: 1.1.8)

When curator parsers build-constraints.yaml (using the yaml package)
we lose information about grouping/formatting/comments. This means
that views (#21) are missing information.

If we add this information to the YAML structure we would be able to add some cool tooling, like:

• List all mentions of a github issue
• List all packages by github issues
• List all packages with a "GHC 9" comment
• (assuming github integration) Check if a referenced issues has been
  closed (we could enrich the output if you pass --github)

This would also allow us to have curator commands that modify the
file, e.g. curator re-enable Agda, curator upper-bound "#5587" "network<3.1.2.0". However, this requires us to stick to this new
format and never add comments. So we'd probably need to wait with this
to make sure that the new format is expressive enough and not
cumbersome, so we don't feel the need/urge to escape it.

We don't need to touch the maintainer area much for this. A new
maintainer adding a package can still do it in the exact same way, but
we'd want to be able to add comments / issue links for a package.

@DanBurton wrote in #20, this may still apply:

One maintainer downside to this is that the place that maintainers
modify to add packages looks more complex. Instead of just listing
their packages under their name, they now see a bunch of fields and
need to worry about which field names are valid and what they mean
and whether they are expected to customize these fields for each of
their packages. For the uninitiated it makes adding your own
packages appear to be less approachable.

@DanBurton do you think this would still be an issue if maintainers
can add packages as they do today? We could have some example under
"Add your package" in README.md to clarify that this is the default:

Example of adding packages:
"My Name [email protected] @MyGitHubHandle":

• package1
• package2

We should be able to add do this package by package / constraint by constraint, to not have to do a big migration. We can keep using comments and linebreaks for grouping as an escape hatch as long as we don't have scripts that re-write the file.

Format examples:

# supercedes "Stackage upper bounds", can also be used for GHC upgrades and other batch operations.
# Doesn't need to be in a separate top level entry, but it feels cleaner to me.
constraints:
  "optparse-applicative 0.16":
     - issue: https://github.com/commercialhaskell/stackage/issues/5597
     - comment: Hello
     - constraints: # `constraints:` can be optional if it's the only property
       - optparse-applicative < 0.16
       - optparse-generic < 1.4
       - turtle < 1.5.21
  "GHC 9":
    - Agda < 0
    - cabal-rpm < 0
    - entropy < 0

skipped-tests:
  "Outdated dependencies":
    comment: These can periodically be checked for updates.
    packages: # `packages:` can be optional if it's the only property
      - package: time-compat 
        comment: base-compat 0.11
      - package: http-media
        comment: base-4.13
      - package: simple-vec
        comment: ghc 8.10
      - static-text # `package:` can be optional if it's the only property
      - monad-par
      - package: avers
        comment: via base16-bytestring
        issue: https://github.com/commercialhaskell/stackage/issues/5649

curator: Can't get proper GHC version: The program 'ghc' version ==9.6.3 is required but the version found at /usr/local/bin/ghc is version 9.8.1

Would be great if curator would also look for ghc-9.6.3 (maybe even first!), not just for ghc.