Git Product home page Git Product logo

hrepl's Introduction

hrepl

hrepl is a tool that lets you interactively develop Haskell code using Bazel. It uses Bazel to compile your code's dependencies, and then loads your modules into an interpreter.

This is not an officially supported Google product.

Overview

To use hrepl, first cd into this repository, build the //hrepl target and save the resulting binary somewhere convenient:

$ bazel build //hrepl
$ cp --no-preserve=mode bazel-bin/hrepl/hrepl ~/.local/bin
$ chmod +x ~/.local/bin/hrepl

You should also use a recent enough verson of rules_haskell in your project's WORKSPACE file. (See this repository's WORKSPACE file for an example.)

Then, to load one or more targets in the interpreter, run hrepl within your own Bazel project. You may specify the Bazel label(s) of any Haskell libraries, binaries or tests. For example, in the rules_haskell repository itself:

$ hrepl tests/binary-with-lib:lib
...
*Lib Lib>

Or, within a subdirectory and with multiple targets:

$ cd tests/binary-with-lib
$ hrepl :lib :binary-with-lib
*Lib Lib Main>

You may also specify individual source files, which will cause hrepl to load targets that declare those files in their srcs. (If there is more than one possibility, it will choose arbitrarily.)

$ hrepl tests/binary-with-lib/Lib.hs
*Lib Lib>

After you modify the interpreted module(s), the :reload command will pick up any changes. This approach is much faster than rebuilding with Bazel each time.

For more information about hrepl's command-line flags, run hrepl --help.

Using Build Targets

Compiled Dependencies

By default, hrepl will compile any dependencies of your target(s) with Bazel before starting the interpreter. You may load those dependencies with :module or import. For example:

$ hrepl //some:library  # depends on the "split" package
Prelude Library> import Data.List.Split
Prelude Library Data.List.Split>

However, since those modules are compiled, the interpreter will not be aware of the source files of those dependencies, and will not pick up changes to them on :reload. Instead, you will need to :quit and restart gghci. The same is true for changes to BUILD and .bzl files that affect your targets.

Note: hrepl will not let you load (compiled) modules from transitive dependencies automatically. This behavior is similar to the build rules, which only expose modules from targets listed directly in their deps. To expose a transitive dependency in the interpreter, pass --package //label/of:dep.

Interpreted Dependencies

Alternately, you can tell hrepl to interpret (not compile) certain dependencies. The --interpret-deps=PACKAGE flag specifies any dependencies that are under the given PACKAGE (either directly, or as a subpackage). For example:

$ hrepl //some/project:target --interpret-deps=//some/project

That will load not just :target into the interpreter, but also any source files from dependencies of :target that are in some/project/BUILD or any other BUILD file in a subdirectory of some/project.

You may pass the flag more than once to combine the dependencies from different subdirectories.

Warning: hrepl will combine the compiler_flags attributes of interpreted targets into a single list, and apply all of them to each source file it loads. If two targets have conflicting compiler_flags, for example enabling and disabling the same GHC extension, it may not be possible to interpret both of them at once.

Multiple build targets

You may load zero or more Bazel targets in the interpeter at once. For example, to load two targets:

$ hrepl //your:target1 //another:target2
Prelude Target1 Target2>

hrepl will also interpret (i.e., not compile) any "intermediate" targets. For example, suppose that :target1 depends on :dep and :dep depends on :target2. Then hrepl will interpret :dep as well, and :reload will pick up any changes to :dep as well as to :target1 and :target2. However, hrepl will not expose the definitions in :dep by default. If you want to use them, either specify those targets on the command-line or call import. For example:

$ hrepl //your:target1 //another:target2
Prelude Target1 Target2> import Dep
Prelude Target1 Target2 Dep>

Additional Compiled Targets

Alternately, you may tell hrepl to compile an unrelated target with the --package flag. For example:

$ hrepl //your/haskell:target --package @stackage//:split
Prelude Target>

In that case, @stackage//:split will be compiled and available for import in the interpreter:

Prelude Target> import Data.List.Split
Prelude Target Data.List.Split>

Similar to any dependencies of :target, it won't be reloaded unless you manually :quit and restart the interpreter.

You may also use this flag to expose a dependency of a target without also compiling it.

Forwarding Command-line Flags

hrepl supports forwarding flags to its subprocesses in several different ways.

To GHC

You may pass compiler flags directly to hrepl. For example:

$ hrepl -XPackageImports -freverse-errors //some:target

To pass RTS options to GHC, use the --with-rtsopts flag, which takes a space-separated list of flags. For example:

$ hrepl --with-rtsopts='-t -S'  //some:target

does the equivalent of ghc +RTS -t -S -RTS.

To Bazel

You can use --bazel-args=--some-bazel-params to make hrepl pass certain flags in each call to bazel.

--bazel-args takes a space-separated list of arguments. If it's specified multiple times, the values will accumulate. For example, --bazel-args='-c opt' is equivalent to --bazel-args=-c --bazel-args=-opt. As a special shortcut, hrepl supports directly passing the Bazel -c flag to it.

For example:

$ hrepl --bazel-args='-c opt' //your/haskell:library
$ hrepl -c opt //your/haskell:library`

hrepl's People

Contributors

agrue avatar blackgnezdo avatar judah avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hrepl's Issues

Support targets with plugins

Plugins don't work yet:

$ hrepl //hrepl/tests:PluginLib
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
ghc: can't find a package database at bazel-out/k8-fastbuild/bin/external/stackage/ghc-tcplugins-extra-0.3/_install/ghc-tcplugins-extra-0.3.conf.d
Error running GHCi.

See also //hrepl/tests:plugins_test.

Tests are slow

The unit tests are fairly slow; on my desktop they take ~3m each to run. One cause is that
each run uses a separate --output_base to make them hermetic, so Bazel
has to regenerate the GHC bindist repository separately (e.g., call make install).
Nix might be able to help this, by sharing the same cache each time.

Support statically-linked GHC

Currently hrepl assumes GHC is dynamically linked and that ghci will load dynamic libraries. Quoting tweag/rules_haskell#1210 (comment), there are two situations where this is relevant:

  1. A static-mode GHC toolchain (e.g. on Windows) where Haskell libraries are only compiled to static archives. GHCi can load these, provided they are built with -fPIC (and -fexternal-dynamic-refs on Unix).
  2. A C library dependency that only provides a pic_static_library. GHCi's loader can usually load these.

User-specified static library could not be loaded (.../_solib_k8/.../libz.so)

hrepl fails to load targets that depend on certain types of dynamic libraries.

This can be reproduced on the rules_haskell repository with the following command:

$ nix-shell --run 'hrepl //tests/binary-with-indirect-sysdeps:hs-lib'
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
<command line>: User-specified static library could not be loaded (/home/aj/.cache/bazel/_bazel_aj/021cc9f0bfc60754963d997427e7eb3e/execroot/rules_haskell/bazel-out/k8-fastbuild/bin/_solib_k8/[email protected]_S_S_Czlib___Uexternal_Snixpkgs_Uzlib_Slib/libz.so)
Loading static libraries is not supported in this configuration.
Try using a dynamic library instead.

Error running GHCi.

Using the --show-commands flag shows that GHCi is passed the following flags:

-lz -lz.so -lz.so.1.2

This seems to be the source of the issue and seems to be caused by the following line in hrepl:

sharedLibName = drop 3 . takeBaseName

That logic to obtain the library name from a file name fails on versioned libraries like libz.so.1.2.11, or on system libraries outside of Linux, e.g. libz.dylib.

The target //tests/binary-with-indirect-sysdeps:hs-lib depends on a libz that is provided by nixpkgs and comes in three shapes: libz.so libz.so.1 libz.so.1.2.11.

hrepl revision 33f879e
rules_haskell revision b41234677c9381982aae98098fb473a5b733c945

.HaskellCompile.pb: openBinaryFile: does not exist

hrepl attempts to build the haskell_compile_info output group of non-Haskell targets even if they are not explicitly listed on the command-line.

This issue can be reproduced on the daml repository as follows:

$ hrepl //compiler/damlc:damlc-bootstrap  //compiler/damlc/stable-packages:generate-stable-package
hrepl: /home/aj/.cache/bazel/_bazel_aj/f6afc9f2c21b16cf433d30b51afb8d67/execroot/com_github_digital_asset_daml/bazel-out/k8-opt/bin/compiler/damlc/stable-packages/gen-stable-packages.HaskellCompile.pb: openBinaryFile: does not exist (No such file or directory)

Both targets that are specified on the command line are haskell_binary targets. The failing target //compiler/damlc/stable-packages:gen-stable-packages is a genrule target.

Setting the --show-commands flag shows that that target first appears after the first allpaths query:

Running: bazel info bazel-bin --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery '(//compiler/damlc/stable-packages:generate-stable-package + //compiler/damlc:damlc-bootstrap)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc:damlc-bootstrap + //compiler/damlc/stable-packages:generate-stable-package) in allpaths($ts, $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel info execution_root --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/stable-packages:gen-stable-packages + (//compiler/damlc/stable-packages:daml-stdlib/DA-Validation-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Time-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Semigroup-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-NonEmpty-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Monoid-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Logic-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Template.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Down.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Any.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Date-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Tuple.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Prim.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-PromotedText.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-Erased.dalf + (//compiler/damlc/stable-packages:stable-packages + //compiler/damlc:damlc-bootstrap))))))))))))))))))) in (kind('\''haskell_library|haskell_proto_library|haskell_toolchain_library|haskell_cabal_library'\'', deps($ts, 1)) - $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel cquery 'let ts = (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/stable-packages:gen-stable-packages + (//compiler/damlc/stable-packages:daml-stdlib/DA-Validation-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Time-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Semigroup-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-NonEmpty-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Monoid-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Logic-Types.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Template.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Down.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Internal-Any.dalf + (//compiler/damlc/stable-packages:daml-stdlib/DA-Date-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Tuple.dalf + (//compiler/damlc/stable-packages:daml-prim/GHC-Prim.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Types.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-PromotedText.dalf + (//compiler/damlc/stable-packages:daml-prim/DA-Internal-Erased.dalf + (//compiler/damlc/stable-packages:stable-packages + //compiler/damlc:damlc-bootstrap))))))))))))))))))) in (kind(proto_library, deps(kind(haskell_proto_library, $ts), 1)) - $ts)' --build_manual_tests --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel info bazel-bin --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'
Running: bazel build '--show_result=0' '--output_groups=haskell_library_info,haskell_transitive_deps' '//libs-haskell/da-hs-base:da-hs-base' '//compiler/damlc/daml-lf-conversion:daml-lf-conversion' '@stackage//:optparse-applicative' '//compiler/daml-lf-proto:daml-lf-proto' '//compiler/daml-lf-ast:daml-lf-ast' '@stackage//:text' '//compiler/damlc:damlc-lib' '@stackage//:bytestring' '@stackage//:base' --nostamp '--workspace_status_command=true' '--curses=yes' '--color=yes'

Indeed, //compiler/damlc:damlc-bootstrap has a data dependency on //compiler/damlc/stable-packages which is a filegroup capturing the outputs of the genrule //compiler/damlc/stable-packages:gen-stable-packages which in turn has a tools dependency on //compiler/damlc/stable-packages:generate-stable-package.

One could argue that it doesn't make sense to try and load both those binaries into one GHCi session. Unfortunately, that doesn't prevent this issue as both haskell_binary targets share common library dependencies. E.g. the following will trigger the same issue

$ hrepl //compiler/damlc:damlc-bootstrap  //compiler/damlc/daml-lf-conversion

where //compiler/damlc/daml-lf-conversion is a haskell_library target.

Unfortunately, I'm not aware of a cquery command to filter out targets that don't provide a certain output group. However, one possible way to avoid this issue would be to filter for Haskell rules before attempting to build the output groups. E.g. kind("haskell_binary|haskell_library|haskell_test", ...) for compile_info and kind("haskell_library|haskell_cabal_library", ...) for library_info. This would potentially also allow users to pass wildcards to hrepl, e.g.

$ hrepl //compiler/damlc/...

hrepl revision 33f879e
daml revision 761efbe0856941ab398925846dee0d4bbf0b1811

Multiple tests fail on Mac with "Error: DEVELOPER_DIR not set"

First I should mention that I needed to upgrade my rules_haskell version for the project to build at all. My WORKSPACE looks like:

git_repository(
    name = "rules_haskell",
    remote = "https://github.com/tweag/rules_haskell",
    commit = "7abaefdde3d0a29bfe15e02be4ad2a0bd4484600",
    shallow_since = "1626246473 +0000",
)

When building on my Mac I ran into this issue, which I fixed with export BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in my shell. But when I ran the tests under //hrepl/tests (via run_tests.sh) I noticed that many of them (but not all) failed with the same error in the logs. (Sorry, I don't have the complete list handy, but atomic_primops_test was one of them.)

It seems that the BAZEL_USE_CPP_ONLY_TOOLCHAIN environment variable doesn't make its way into the builds that hrepl itself is doing (which makes sense, since it should try to be hermetic). Unfortunately the only workaround I could find was to edit ReplTestLib.hs and manually add it to the constructed environment. This clearly isn't a good permanent solution since it's Mac-specific.

Add CI

Add a CI build that runs the hrepl_tests suite, excluding the ones that are known to fail.

cc_library dependencies aren't loaded on MacOS

GHC expects shared libraries to be .dylib on MacOS, but Bazel cc_library rules always emit .so:

$ hrepl hrepl/tests:CLib1
...
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
<command line>: user specified .o/.so/.DLL could not be loaded (dlopen(libhrepl_Stests_Slibclib1.dylib, 5): image not found)
Whilst trying to load:  (dynamic) hrepl_Stests_Slibclib1
Additional directories searched:   bazel-out/darwin-fastbuild/bin/_solib_darwin
Error running GHCi.

rules_haskell uses cc_wrapper to work around this. We can probably do something similar for hrepl.

hrepl depends on haskell-src-exts

hrepl currently uses haskell-src-exts to parse module names from source files. It uses that information to create the gghci script that loads all of the modules properly.

We should switch to use the ghc library (or alternately ghc-parse-lib) as a dependency instead.

Dependencies aren't loaded by default

hrepl doesn't expose dependencies by default. To load a dependency into the
interpreter, it needs to be either passed explicitly on the command line, or else
exposed with :set -package .... This could be considered "working as intended"
since it's how the actual Bazel build rules work. However, it's not a great user experience:

  • The ... may be a mangled GHC package name
  • :set -package causes all modules to reload and makes you lose all bindinsg.

The current recommended approach is to use --package //some:label which will add the given target as an exposed (compiled) dependency. But in all cases, if you want to expose
a new dependency you effectively need to start a new session.

Google's internal version of hrepl exposes everything, but that has its own tradeoff: Two
modules from different targets could conflict, and distinguishing between them with
-XPackageImports is cumbersome due to the mangled package names. Internally, we
modified GHC to be more permissive and allow unmangled labels as package names.

Linker errors building on macOS Catalina

Hey there 👋 I’m exploring the Bazel+Haskell space, and in attempting to install hrepl (via bazel build //hrepl), I ran into the following:

ERROR: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/external/rules_haskell/rule_info/BUILD.bazel:4:1: HaskellProtoc external/rules_haskell/rule_info/Proto/RuleInfo.hs failed (Aborted): protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
dyld: Symbol not found: __ZNK6google8protobuf8compiler3php9Generator11GenerateAllERKNSt3__16vectorIPKNS0_14FileDescriptorENS4_9allocatorIS8_EEEERKNS4_12basic_stringIcNS4_11char_traitsIcEENS9_IcEEEEPNS1_16GeneratorContextEPSI_
  Referenced from: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/sandbox/darwin-sandbox/275/execroot/hrepl/bazel-out/host/bin/external/com_google_protobuf/protoc
  Expected in: flat namespace
 in /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/sandbox/darwin-sandbox/275/execroot/hrepl/bazel-out/host/bin/external/com_google_protobuf/protoc
Target //hrepl:hrepl failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_patrickt/9b0f64af68d3bba2996d9fad8a733686/external/rules_haskell/rule_info/BUILD.bazel:4:1 HaskellProtoc external/rules_haskell/rule_info/Proto/RuleInfo.hs failed (Aborted): protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-haskell=bazel-out/host/bin/external/proto-lens-protoc/_install/bin/proto-lens-protoc' ... (remaining 4 argument(s) skipped)

It’s possible I’m missing something really obvious here, but if anyone has pointers as to how to fix this, I’d deeply appreciate it!

Feature request: Only print GHCi flags without opening the REPL

Feature

Add an option for hrepl to only print the relevant flags to GHCi for loading the specified range of targets. This should probably exclude flags that are only required for hrepl specifics, e.g. for the ghci script that hrepl defines.

Use-case

The use-case I have in mind is to use hrepl in a hie-bios cradle to enable ghcide to load rules_haskell projects.

hie-bios has a bios cradle that takes a program that will write all required GHC flags to a file specified by the $HIE_BIOS_OUTPUT environment variable. hrepl could be used as that program. Furthermore, hie-bios could be extended with a dedicated Bazel cradle (reviving the currently disabled one).

An example use-case could look like this:

cradle:
  multi:
    - path: "./src"
      config: { cradle: {bazel: {targets: ["//src/..."]}} }
    - path: "./test"
      config: { cradle: {bazel: {targets: ["//test/..."]}} }

Assuming wild-card support in hrepl (or the bazel cradle) and multi-cradle support in ghcide.

Support haskell_proto_library

haskell_proto_library rules don't get loaded correctly in the interpreter, either directly or as dependencies:

$ hrepl hrepl/tests:ProtoLib
hrepl: ...f/execroot/hrepl/bazel-out/k8-fastbuild/bin/hrepl/tests/test_haskell_pb.HaskellLibrary.pb: openBinaryFile: does not exist (No such file or directory)
$ hrepl hrepl/tests:test_haskell_pb
hrepl: ...f/execroot/hrepl/bazel-out/k8-fastbuild/bin/hrepl/tests/test_haskell_pb.HaskellCompile.pb: openBinaryFile: does not exist (No such file or directory)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.