.HaskellCompile.pb: openBinaryFile: does not exist about hrepl HOT 6 OPEN

Ah, yeah, this is an unfortunate behavior. What I'd really like is for the allpaths query to only follow regular deps and not other attributes like data or srcs. But I don't see a way to do that with how Bazel currently works.

One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the allpaths computation in Haskell rather than via blaze cquery.

For now, though, the approach you suggested sounds like a reasonable workaround. Would you like to send a patch?

By they way, even if we resolve this issue with genrules, you may have a problem loading everything into gghci at once; GHCi gets confused if you try to give it two source files that both have a "Main" module. In the case you mentioned, it might be fine since generate-stable-packages has a custom main_function.

from hrepl.

One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the allpaths computation in Haskell rather than via blaze cquery.

That sounds interesting. Could you expand a bit on what you have in mind? I'm not sure I understand how this would work.

For now, though, the approach you suggested sounds like a reasonable workaround. Would you like to send a patch?

I've opened #11 implementing this filter.

By they way, even if we resolve this issue with genrules, you may have a problem loading everything into gghci at once; GHCi gets confused if you try to give it two source files that both have a "Main" module. In the case you mentioned, it might be fine since generate-stable-packages has a custom main_function.

That's a good point. In this particular case we do indeed get away with it. Only :main doesn't work.

#11 fixes the .HaskellCompile.pb: openBinaryFile: does not exist error. However, I've noticed that two library dependencies are not picked up for some reason. I.e.

$ hrepl //compiler/damlc:damlc-bootstrap //compiler/damlc/daml-lf-conversion

Fails with "could not load module" errors due to missing -package ghcide and missing -package shake. If I add them explicitly it works:

$ hrepl //compiler/damlc:damlc-bootstrap //compiler/damlc/daml-lf-conversion --show-commands --package @stackage//:ghcide --package @stackage//:shake

I'm not sure what is causing this. In case of ghcide there is only one step in between damlc-bootstrap -> damlc-lib -> ghcide. damlc-bootstrap has no direct dependencies apart from base and damlc-lib, so I wonder why it's only ghcide and shake that are skipped.

from hrepl.

@aherrmann shake and ghcide are a different rule type, haskell_cabal_library. That's probably why your query isn't picking them up. Can you try adding that (and also haskell_toolchain_library , I suppose) to the filter list?

from hrepl.

One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the allpaths computation in Haskell rather than via blaze cquery.

That sounds interesting. Could you expand a bit on what you have in mind? I'm not sure I understand how this would work.

The idea is not particularly well-formed yet, to be honest. Here's one possibility: For each Haskell rule, add a haskell_dep_info output group that emits a new DepInfo proto:

message DepInfo {
  repeated string dep_labels = 1;  // Immediate dependencies
}

Then, replace getAllIntermediateTargets (which is currently just an allpaths($targets, $targets)query) with the following algorithm:

1. Run blaze query "allpaths($targets, $targets)" to get a list of labels.
2. Run blaze build --output_groups=haskell_dep_info on those labels, and read the DepInfo for each one. Ignore labels that don't have a DepInfo, i.e., that aren't Haskell targets.
3. Run an "allpaths($targets,$targets)" algorithm manually, using the graph defined by the DepInfo (vertices are labels). This will give us a correct set which is refined from (1).

Some things I'm not sure about;

• Is the overhead of reading all those proto files and doing a manual graph computation significant?
• Should we just do a deps($targets) in step (1)?
• Would DepInfo be better merged into CompileInfo or LibraryInfo?

from hrepl.

@aherrmann shake and ghcide are a different rule type, haskell_cabal_library. That's probably why your query isn't picking them up. Can you try adding that (and also haskell_toolchain_library , I suppose) to the filter list?

I dug a bit deeper. The issue is not due to the filter that I added in #12. The filter for haskell_binary|haskell_library|haskell_test is only applied to getAllIntermediateTargets. I.e. only affects packages that should be loaded by source. ghcide and shake are external dependencies and should be loaded as compiled packages. (haskell_cabal_library also doesn't return haskell_compile_info)

The compiled dependencies are determined by the query defined in buildDependentPackages. In this case it looks as follows (as shown by --show-commands)

bazel cquery 'let ts = (//compiler/damlc/daml-lf-conversion:daml-lf-conversion + (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/daml-ide:daml-ide + (//compiler/damlc/daml-compiler:daml-compiler + (//compiler/damlc/daml-ide-core:daml-ide-core + (//compiler/damlc:damlc-lib + //compiler/damlc:damlc-bootstrap)))))) in (kind('\''haskell_library|haskell_proto_library|haskell_toolchain_library|haskell_cabal_library'\'', deps($ts, 1)) - $ts)'

Running this query indeed doesn't include ghcide or shake in the listed dependencies, even though both are direct dependencies of //compiler/damlc:damlc-lib which is included in ts.

It turns out that @stackage//:ghcide and @stackage//:shake are vendored_packages in the corresponding stack_snapshot call. I.e. they are alias rules that point to haskell_cabal_library somewhere else. But the query that hrepl applies doesn't handle alias.

from hrepl.

Thanks for outlining the DepInfo approach. That looks like it could work, though I haven't thought about the three open questions you raise in the bottom, yet. I think this might also provide a solution to the alias issue I've described above.

from hrepl.