Git Product home page Git Product logo

hrepl's Issues

Support statically-linked GHC

Currently hrepl assumes GHC is dynamically linked and that ghci will load dynamic libraries. Quoting tweag/rules_haskell#1210 (comment), there are two situations where this is relevant:

  1. A static-mode GHC toolchain (e.g. on Windows) where Haskell libraries are only compiled to static archives. GHCi can load these, provided they are built with -fPIC (and -fexternal-dynamic-refs on Unix).
  2. A C library dependency that only provides a pic_static_library. GHCi's loader can usually load these.

Dependencies aren't loaded by default

hrepl doesn't expose dependencies by default. To load a dependency into the
interpreter, it needs to be either passed explicitly on the command line, or else
exposed with :set -package .... This could be considered "working as intended"
since it's how the actual Bazel build rules work. However, it's not a great user experience:

  • The ... may be a mangled GHC package name
  • :set -package causes all modules to reload and makes you lose all bindinsg.

The current recommended approach is to use --package //some:label which will add the given target as an exposed (compiled) dependency. But in all cases, if you want to expose
a new dependency you effectively need to start a new session.

Google's internal version of hrepl exposes everything, but that has its own tradeoff: Two
modules from different targets could conflict, and distinguishing between them with
-XPackageImports is cumbersome due to the mangled package names. Internally, we
modified GHC to be more permissive and allow unmangled labels as package names.

Support targets with plugins

Plugins don't work yet:

$ hrepl //hrepl/tests:PluginLib
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
ghc: can't find a package database at bazel-out/k8-fastbuild/bin/external/stackage/ghc-tcplugins-extra-0.3/_install/ghc-tcplugins-extra-0.3.conf.d
Error running GHCi.

See also //hrepl/tests:plugins_test.

Multiple tests fail on Mac with "Error: DEVELOPER_DIR not set"

First I should mention that I needed to upgrade my rules_haskell version for the project to build at all. My WORKSPACE looks like:

git_repository(
    name = "rules_haskell",
    remote = "https://github.com/tweag/rules_haskell",
    commit = "7abaefdde3d0a29bfe15e02be4ad2a0bd4484600",
    shallow_since = "1626246473 +0000",
)

When building on my Mac I ran into this issue, which I fixed with export BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in my shell. But when I ran the tests under //hrepl/tests (via run_tests.sh) I noticed that many of them (but not all) failed with the same error in the logs. (Sorry, I don't have the complete list handy, but atomic_primops_test was one of them.)

It seems that the BAZEL_USE_CPP_ONLY_TOOLCHAIN environment variable doesn't make its way into the builds that hrepl itself is doing (which makes sense, since it should try to be hermetic). Unfortunately the only workaround I could find was to edit ReplTestLib.hs and manually add it to the constructed environment. This clearly isn't a good permanent solution since it's Mac-specific.

Support haskell_proto_library

haskell_proto_library rules don't get loaded correctly in the interpreter, either directly or as dependencies:

$ hrepl hrepl/tests:ProtoLib
hrepl: ...f/execroot/hrepl/bazel-out/k8-fastbuild/bin/hrepl/tests/test_haskell_pb.HaskellLibrary.pb: openBinaryFile: does not exist (No such file or directory)
$ hrepl hrepl/tests:test_haskell_pb
hrepl: ...f/execroot/hrepl/bazel-out/k8-fastbuild/bin/hrepl/tests/test_haskell_pb.HaskellCompile.pb: openBinaryFile: does not exist (No such file or directory)

Tests are slow

The unit tests are fairly slow; on my desktop they take ~3m each to run. One cause is that
each run uses a separate --output_base to make them hermetic, so Bazel
has to regenerate the GHC bindist repository separately (e.g., call make install).
Nix might be able to help this, by sharing the same cache each time.

Feature request: Only print GHCi flags without opening the REPL

Feature

Add an option for hrepl to only print the relevant flags to GHCi for loading the specified range of targets. This should probably exclude flags that are only required for hrepl specifics, e.g. for the ghci script that hrepl defines.

Use-case

The use-case I have in mind is to use hrepl in a hie-bios cradle to enable ghcide to load rules_haskell projects.

hie-bios has a bios cradle that takes a program that will write all required GHC flags to a file specified by the $HIE_BIOS_OUTPUT environment variable. hrepl could be used as that program. Furthermore, hie-bios could be extended with a dedicated Bazel cradle (reviving the currently disabled one).

An example use-case could look like this:

cradle:
  multi:
    - path: "./src"
      config: { cradle: {bazel: {targets: ["//src/..."]}} }
    - path: "./test"
      config: { cradle: {bazel: {targets: ["//test/..."]}} }

Assuming wild-card support in hrepl (or the bazel cradle) and multi-cradle support in ghcide.

Linker errors building on macOS Catalina

Hey there 👋 I’m exploring the Bazel+Haskell space, and in attempting to install hrepl (via bazel build //hrepl), I ran into the following:

ERROR: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/external/rules_haskell/rule_info/BUILD.bazel:4:1: HaskellProtoc external/rules_haskell/rule_info/Proto/RuleInfo.hs failed (Aborted): protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
dyld: Symbol not found: __ZNK6google8protobuf8compiler3php9Generator11GenerateAllERKNSt3__16vectorIPKNS0_14FileDescriptorENS4_9allocatorIS8_EEEERKNS4_12basic_stringIcNS4_11char_traitsIcEENS9_IcEEEEPNS1_16GeneratorContextEPSI_
  Referenced from: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/sandbox/darwin-sandbox/275/execroot/hrepl/bazel-out/host/bin/external/com_google_protobuf/protoc
  Expected in: flat namespace
 in /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/sandbox/darwin-sandbox/275/execroot/hrepl/bazel-out/host/bin/external/com_google_protobuf/protoc
Target //hrepl:hrepl failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/external/rules_haskell/rule_info/BUILD.bazel:4:1 HaskellProtoc external/rules_haskell/rule_info/Proto/RuleInfo.hs failed (Aborted): protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

It’s possible I’m missing something really obvious here, but if anyone has pointers as to how to fix this, I’d deeply appreciate it!

cc_library dependencies aren't loaded on MacOS

GHC expects shared libraries to be .dylib on MacOS, but Bazel cc_library rules always emit .so:

$ hrepl hrepl/tests:CLib1
...
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
<command line>: user specified .o/.so/.DLL could not be loaded (dlopen(libhrepl_Stests_Slibclib1.dylib, 5): image not found)
Whilst trying to load:  (dynamic) hrepl_Stests_Slibclib1
Additional directories searched:   bazel-out/darwin-fastbuild/bin/_solib_darwin
Error running GHCi.

rules_haskell uses cc_wrapper to work around this. We can probably do something similar for hrepl.

User-specified static library could not be loaded (.../_solib_k8/.../libz.so)

hrepl fails to load targets that depend on certain types of dynamic libraries.

This can be reproduced on the rules_haskell repository with the following command:

$ nix-shell --run 'hrepl //tests/binary-with-indirect-sysdeps:hs-lib'
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
<command line>: User-specified static library could not be loaded (/home/aj/.cache/bazel/_bazel_aj/021cc9f0bfc60754963d997427e7eb3e/execroot/rules_haskell/bazel-out/k8-fastbuild/bin/_solib_k8/[email protected]_S_S_Czlib___Uexternal_Snixpkgs_Uzlib_Slib/libz.so)
Loading static libraries is not supported in this configuration.
Try using a dynamic library instead.

Error running GHCi.

Using the --show-commands flag shows that GHCi is passed the following flags:

-lz -lz.so -lz.so.1.2

This seems to be the source of the issue and seems to be caused by the following line in hrepl:

hrepl/hrepl/Repl.hs

Line 239 in 33f879e

sharedLibName = drop 3 . takeBaseName

That logic to obtain the library name from a file name fails on versioned libraries like libz.so.1.2.11, or on system libraries outside of Linux, e.g. libz.dylib.

The target //tests/binary-with-indirect-sysdeps:hs-lib depends on a libz that is provided by nixpkgs and comes in three shapes: libz.so libz.so.1 libz.so.1.2.11.

hrepl revision 33f879e
rules_haskell revision b41234677c9381982aae98098fb473a5b733c945

Add CI

Add a CI build that runs the hrepl_tests suite, excluding the ones that are known to fail.

hrepl depends on haskell-src-exts

hrepl currently uses haskell-src-exts to parse module names from source files. It uses that information to create the gghci script that loads all of the modules properly.

We should switch to use the ghc library (or alternately ghc-parse-lib) as a dependency instead.

.HaskellCompile.pb: openBinaryFile: does not exist

hrepl attempts to build the haskell_compile_info output group of non-Haskell targets even if they are not explicitly listed on the command-line.

This issue can be reproduced on the daml repository as follows:

$ hrepl //compiler/damlc:damlc-bootstrap  //compiler/damlc/stable-packages:generate-stable-package
hrepl: /home/aj/.cache/bazel/_bazel_aj/f6afc9f2c21b16cf433d30b51afb8d67/execroot/com_github_digital_asset_daml/bazel-out/k8-opt/bin/compiler/damlc/stable-packages/gen-stable-packages.HaskellCompile.pb: openBinaryFile: does not exist (No such file or directory)

Both targets that are specified on the command line are haskell_binary targets. The failing target //compiler/damlc/stable-packages:gen-stable-packages is a genrule target.

Setting the --show-commands flag shows that that target first appears after the first allpaths query:

Running: bazel info bazel-bin --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery '(//compiler/damlc/stable-packages:generate-stable-package + //compiler/damlc:damlc-bootstrap)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc:damlc-bootstrap + //compiler/damlc/stable-packages:generate-stable-package) in allpaths($ts, $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel info execution_root --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/stable-packages:gen-stable-packages + (//compiler/damlc/stable-packages:daml-stdlib/DA-Validation-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Time-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Semigroup-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-NonEmpty-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Monoid-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Logic-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Template.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Down.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Any.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Date-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Tuple.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Prim.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-PromotedText.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-Erased.dalf + (//compiler/damlc/stable-packages:stable-packages + //compiler/damlc:damlc-bootstrap))))))))))))))))))) in (kind('\''haskell_library|haskell_proto_library|haskell_toolchain_library|haskell_cabal_library'\'', deps($ts, 1)) - $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/stable-packages:gen-stable-packages + (//compiler/damlc/stable-packages:daml-stdlib/DA-Validation-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Time-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Semigroup-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-NonEmpty-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Monoid-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Logic-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Template.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Down.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Any.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Date-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Tuple.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Prim.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-PromotedText.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-Erased.dalf + (//compiler/damlc/stable-packages:stable-packages + //compiler/damlc:damlc-bootstrap))))))))))))))))))) in (kind(proto_library, deps(kind(haskell_proto_library, $ts), 1)) - $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel info bazel-bin --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel build '--show_result=0' '--output_groups=haskell_library_info,haskell_transitive_deps' '//libs-haskell/da-hs-base:da-hs-base' '//compiler/damlc/daml-lf-conversion:daml-lf-conversion' '@stackage//:optparse-applicative' '//compiler/daml-lf-proto:daml-lf-proto' '//compiler/daml-lf-ast:daml-lf-ast' '@stackage//:text' '//compiler/damlc:damlc-lib' '@stackage//:bytestring' '@stackage//:base' --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'

Indeed, //compiler/damlc:damlc-bootstrap has a data dependency on //compiler/damlc/stable-packages which is a filegroup capturing the outputs of the genrule //compiler/damlc/stable-packages:gen-stable-packages which in turn has a tools dependency on //compiler/damlc/stable-packages:generate-stable-package.

One could argue that it doesn't make sense to try and load both those binaries into one GHCi session. Unfortunately, that doesn't prevent this issue as both haskell_binary targets share common library dependencies. E.g. the following will trigger the same issue

$ hrepl //compiler/damlc:damlc-bootstrap  //compiler/damlc/daml-lf-conversion

where //compiler/damlc/daml-lf-conversion is a haskell_library target.

Unfortunately, I'm not aware of a cquery command to filter out targets that don't provide a certain output group. However, one possible way to avoid this issue would be to filter for Haskell rules before attempting to build the output groups. E.g. kind("haskell_binary|haskell_library|haskell_test", ...) for compile_info and kind("haskell_library|haskell_cabal_library", ...) for library_info. This would potentially also allow users to pass wildcards to hrepl, e.g.

$ hrepl //compiler/damlc/...

hrepl revision 33f879e
daml revision 761efbe0856941ab398925846dee0d4bbf0b1811

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.