matrixai / haskell-demo Goto Github PK
View Code? Open in Web Editor NEWHaskell Demo Project using Nix
Home Page: https://matrix.ai
License: Apache License 2.0
Haskell Demo Project using Nix
Home Page: https://matrix.ai
License: Apache License 2.0
The new nix-gitignore package will allow us to ignore everything we specify inside our .gitignore
so we don't need the custom source filter.
The reason why nix-build
by itself doesn't have idempotent builds is because it relies on src = ./.;
which has several problems.
First any change to the source directory will create a different input directory.
When nix-instantiate
converts the default.nix
open expression into a closed expression in drv
form, then it will also encode the contents of the ./.
. (The transformation from open expression to closed expression is itself interesting). Also this is a consequence of the design of Nix where the hashes of the outputs are not based on the outputs themselves but based on the input expressions.
So every time nix-build
builds with ./result
symlink pointing to a different place, then you get a different source state!
You can observe idempotency by doing: nix-build --no-out-link
or nix-store --realise $(nix-instantiate)
.
To actually get idempotent builds, we need to have some way of ignoring generated files inside our src
. Basically using things like filterSource
and cleanSource
.
There is an issue addressing this problem: NixOS/nix#885
To create a static compilation, we should have a release.nix
to produce such derivations. This needs to be added later.
nix-build
on the project as it is right now produces following error.
> nix-build --show-trace
error: while evaluating the attribute 'src' of the derivation 'haskell-demo-0.1.0.0' at /nix/store/yvyqcarccbdcp5jvyknlbzvyck0zlxaw-source/pkgs/stdenv/generic/make-derivation.nix:185:11:
string '/nix/store/hmz9aqw86s50fbk9gac0fyvh97hjsdh7-haskell-demo-source-0.1.0.0/haskell-demo-0.1.0.0.tar.gz' cannot refer to other paths, at /nix/store/yvyqcarccbdcp5jvyknlbzvyck0zlxaw-source/lib/sources.nix:49:17
From Cabal user guide
Nix-style local builds are a new build system implementation inspired by Nix. The Nix-style local build system is commonly called “new-build” for short after the cabal new-* family of commands that control it. However, those names are only temporary until Nix-style local builds become the default. This is expected to happen soon. For those who do not wish to use the new functionality, the classic project style will not be removed immediately, but these legacy commands will require the usage of the v1- prefix as of Cabal 3.0 and will be removed in a future release. For a future-proof way to use these commands in a script or tutorial that anticipates the possibility of another UI paradigm being devised in the future, there are also v2- prefixed versions that will reference the same functionality until such a point as it is completely removed from Cabal.
So to use current classic project style, append v1-
prefix to cabal commands. cabal v1-configure
, cabal v1-repl
cabal v1-build
etc
Since this is being removed in the future, consider switching to new project style soon.
these derivations will be built:
/nix/store/kkaz3424nl8dwmchgwknwvgbpdwf1mb6-haskell-demo-source-0.1.0.0.drv
/nix/store/qpjabqklccaxpfgs50cba5dpqb45dixj-haskell-demo-0.1.0.0.drv
building path(s) ‘/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0’
setupCompilerEnvironmentPhase
Build with /nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2.
unpacking sources
unpacking source archive /nix/store/hgha8qriz5lmcmaa20k3qy3b0d3dy8n0-Haskell-Demo
source root is Haskell-Demo
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-haskell-demo-source-0.1.0.0.drv-0/package.conf.d -j8 -threaded
[1 of 1] Compiling Main ( Setup.hs, /tmp/nix-build-haskell-demo-source-0.1.0.0.drv-0/Main.o )
Linking Setup ...
configuring
haskell-demo.cabal is up-to-date
configureFlags: --verbose --prefix=/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/share/doc --with-gcc=gcc --package-db=/tmp/nix-build-haskell-demo-source-0.1.0.0.drv-0/package.conf.d --ghc-option=-optl=-Wl,-rpath=/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/lib/ghc-8.2.2/haskell-demo-0.1.0.0 --ghc-option=-j8 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --ghc-option=-split-sections --ghc-option=-Wall --ghc-option=-Werror --extra-lib-dirs=/nix/store/54cwjh1lsmjpk2cbs43gw89w4zhk3ybb-ncurses-6.0-20171125/lib --extra-lib-dirs=/nix/store/wc2ll61lypsv9yig6mvy73c0lw1dp15a-gmp-6.1.2/lib --extra-lib-dirs=/nix/store/54cwjh1lsmjpk2cbs43gw89w4zhk3ybb-ncurses-6.0-20171125/lib
Configuring haskell-demo-0.1.0.0...
Dependency base >=4.7 && <5: using base-4.10.1.0
Dependency haskell-demo -any: using haskell-demo-0.1.0.0
Source component graph:
component lib
component exe:haskell-demo-exe dependency lib
component test:haskell-demo-test dependency lib
Configured component graph:
component haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
include base-4.10.1.0
component haskell-demo-0.1.0.0-AyhovOrdQLIJLUjcQhWEGQ-haskell-demo-exe
include base-4.10.1.0
include haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
component haskell-demo-0.1.0.0-L9eSoCKaTwt86y9UN0LDt0-haskell-demo-test
include base-4.10.1.0
include haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
Linked component graph:
unit haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
include base-4.10.1.0
Lib=haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM:Lib
unit haskell-demo-0.1.0.0-AyhovOrdQLIJLUjcQhWEGQ-haskell-demo-exe
include base-4.10.1.0
include haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
unit haskell-demo-0.1.0.0-L9eSoCKaTwt86y9UN0LDt0-haskell-demo-test
include base-4.10.1.0
include haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
Ready component graph:
definite haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
depends base-4.10.1.0
definite haskell-demo-0.1.0.0-AyhovOrdQLIJLUjcQhWEGQ-haskell-demo-exe
depends base-4.10.1.0
depends haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
definite haskell-demo-0.1.0.0-L9eSoCKaTwt86y9UN0LDt0-haskell-demo-test
depends base-4.10.1.0
depends haskell-demo-0.1.0.0-3DU1SIq1VjrF8GDoVDfBoM
Using Cabal-2.0.1.0 compiled by ghc-8.2
Using compiler: ghc-8.2.2
Using install prefix:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0
Executables installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/bin
Libraries installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/lib/ghc-8.2.2/haskell-demo-0.1.0.0
Dynamic Libraries installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/lib/ghc-8.2.2/x86_64-linux-ghc-8.2.2
Private executables installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/libexec/x86_64-linux-ghc-8.2.2/haskell-demo-0.1.0.0
Data files installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/share/x86_64-linux-ghc-8.2.2/haskell-demo-0.1.0.0
Documentation installed in: /share/doc
Configuration files installed in:
/nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/etc
No alex found
Using ar found on system at:
/nix/store/k8b9hqv58dd1z0j4ikak24ykndcm91s6-binutils-2.28.1/bin/ar
No c2hs found
No cpphs found
No doctest found
Using gcc version 7.3.0 given by user at:
/nix/store/9y2f87qb1djmpjs1gxl6smfkpl581waa-gcc-wrapper-7.3.0/bin/gcc
Using ghc version 8.2.2 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/ghc
Using ghc-pkg version 8.2.2 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.18.1 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/haddock
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/hpc
Using hsc2hs version 0.68.2 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/hsc2hs
Using hscolour version 1.24 found on system at:
/nix/store/vp4d1zhp35rs0zk0pc2h1mghz2l8k7dp-hscolour-1.24.2/bin/HsColour
No jhc found
Using ld found on system at:
/nix/store/qq3xv9kwi52yvgdf13jh2swavbhw3lpd-binutils-wrapper-2.28.1/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using runghc version 8.2.2 found on system at:
/nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2/bin/runghc
Using strip version 2.28 found on system at:
/nix/store/k8b9hqv58dd1z0j4ikak24ykndcm91s6-binutils-2.28.1/bin/strip
Using tar found on system at:
/nix/store/7zam9d16ml78rb0p1p5gds4p5z99fpyv-gnutar-1.30/bin/tar
No uhc found
building
Distribution quality warnings:
No 'category' field.
No 'synopsis' field.
Building source dist for haskell-demo-0.1.0.0...
Preprocessing library for haskell-demo-0.1.0.0..
Preprocessing executable 'haskell-demo-exe' for haskell-demo-0.1.0.0..
Preprocessing test suite 'haskell-demo-test' for haskell-demo-0.1.0.0..
Source tarball created: dist/haskell-demo-0.1.0.0.tar.gz
running tests
haddockPhase
installing
post-installation fixup
building path(s) ‘/nix/store/c1s0gbn0gg5la0ch3q6l2lfr0rlhcidh-haskell-demo-0.1.0.0’, ‘/nix/store/xx58dm3h12c5w6cyddx9kzf5wiacl44f-haskell-demo-0.1.0.0-doc’
setupCompilerEnvironmentPhase
Build with /nix/store/w8ckird699qc09np3bc3h47r5g86ab8s-ghc-8.2.2.
unpacking sources
Source tarball is at /nix/store/yzya0ba2kb3c7liprwnz8hh1x5xqibxj-haskell-demo-source-0.1.0.0/haskell-demo-0.1.0.0.tar.gz
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-haskell-demo-0.1.0.0.drv-0/package.conf.d -j8 -threaded
[1 of 1] Compiling Main ( Setup.hs, /tmp/nix-build-haskell-demo-0.1.0.0.drv-0/Main.o )
Linking Setup ...
configuring
package.yaml: Yaml file not found: /tmp/nix-build-haskell-demo-0.1.0.0.drv-0/haskell-demo-0.1.0.0/package.yaml
builder for ‘/nix/store/qpjabqklccaxpfgs50cba5dpqb45dixj-haskell-demo-0.1.0.0.drv’ failed with exit code 1
error: build of ‘/nix/store/qpjabqklccaxpfgs50cba5dpqb45dixj-haskell-demo-0.1.0.0.drv’ failed
If we ever want to use Haskell to write a shared object/library that is meant to be consumed by another language (like C or something that also has FFI). Then we can use the foreign-library
feature of Cabal.
The feature is explained here: https://www.haskell.org/cabal/users-guide/developing-packages.html#foreign-libraries
And there's a tutorial about it: https://qnikst.github.io/posts/2018-05-02-cabal-foreign-library.html
However hpack currently doesn't support it yet: sol/hpack#258
This may be useful for cases like using Haskell code within a kernel module.
At the moment, to import a haskell project setup like Haskell-Demo from another project, we need to call haskellPackages.callPackage
on cabal.nix
instead of default.nix
.
{
pkgs ? import ./pkgs.nix,
haskellPath ? "ghc844"
}:
with pkgs;
let
haskellPackages = lib.getAttrFromPath (lib.splitString "." haskellPath) haskell.packages;
multihash = haskellPackages.callPackage (import ../hs-multihash/cabal.nix) {};
drv = haskellPackages.callPackage (import ./cabal.nix) { inherit multihash; };
in
haskell.lib.buildStrictly (
...
)
Need to update default.nix
so that we can directly import it instead.
The proper way to bring in non-haskell dependencies when using cabal2nix
is to again use package.yaml
. This means we don't add dependencies directly into the default.nix
.
For example to bring in clang
. We need to add it to the build-tools
or system-build-tools
in the package.yaml
.
Note that system-build-tools
is what it should be at the latest version of hpack
. On older versions it's build-tools
. The change was added here: sol/hpack@c1d4ba7 This can be added like:
build-tools:
- clang
system-build-tools:
- clang
Then you update the cabal.nix
and also the cabal file using cabal2nix
and hpack
. And then you should see clang
appear in both the buildInputs of default.nix
and shell.nix
.
The usage of the --nix
flag is a bad idea because it only works at the shell and is not inherited by anything in the shell that utilises stack
.
At the same time we don't want to have a config parameter that enables nix
because that means the stack.yaml
requires to be changed when working on a non-nix environment.
I'm tracking an issue about automatic nix integration: commercialhaskell/stack#3980
But it's not working even at 1.7.1 and the PR implies that it only works when on NixOS, not that when inside a nix environment.
An alternative way is use makeWrapper
and buildEnv
in nix to wrap the stack
command such that it always uses stack --nix
.
{
pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz) {}
}:
with pkgs;
let
stack-nix = buildEnv {
name = "stack-nix";
paths = [ stack ];
pathsToLink = [ "/" "/bin" ];
buildInputs = [ makeWrapper ];
postBuild = ''
wrapProgram $out/bin/stack --add-flags '--nix'
'';
version = stack.version;
};
in
haskell.lib.buildStackProject {
name = "stack-test";
buildInputs = [ stack-nix ];
}
It ends up working, but only because the buildInputs
includes stack-nix
ahead of its own stack
. So it's stack
command takes precedence in the shell's PATH
environment variable. Ideally we would override the stack
argument used in buildStackProject
. However it cannot be done with buildEnv
, it would have to be a the original derivation with its build script overridden or added an additional stage that wraps the stack command.
(haskell.lib.buildStackProject.override { stack = stack-nix; }) {
name = "stack-test";
}
The cabal repl target
command can be too heavy weight especially if you're modifying just 1 module in the entire codebase. And it can be problematic when you are breaking type signatures.
So instead use ghci ModuleA/ModuleB
to load up just that module and its dependencies.
It should be documented that:
stack init
stack --nix
or equivalent)Specifically unused imports, but also explore other necessary flags.
Also look at Nix's buildStrictly
and other overrideCabal
options.
If you see something like:
executable Generate
if flag(Generators)
Buildable: True
else
Buildable: False
hs-source-dirs: Scripts
main-is: Generate.hs
other-modules: Helpers
default-language: Haskell2010
ghc-options: -Wall
if flag(Generators)
build-depends: base >=4.6 && <5,
language-c <0.7,
containers,
process,
regex-pcreexecutable Generate
if flag(Generators)
Buildable: True
else
Buildable: False
hs-source-dirs: Scripts
main-is: Generate.hs
other-modules: Helpers
default-language: Haskell2010
ghc-options: -Wall
if flag(Generators)
build-depends: base >=4.6 && <5,
language-c <0.7,
containers,
process,
regex-pcre
In the cabal file:
You basically have to do:
cabal2nix --flag Generators . >./cabal.nix
Render this into the README.md.
Experimented with
Setup.hs
and custom setup.In hpack, there is
build-type
andcustom-setup
configuration for this purpose.
proto-lens
is an example that uses custom setup script. It adds grpc code gen into the cabal build procedure. This is done throughUserHooks
which is similar to nix build phases and hooks where we can add custom scripts/instructions in to the different build stages.https://hackage.haskell.org/package/proto-lens-protoc-0.2.2.3/docs/Data-ProtoLens-Setup.html
However this is different to what we are after, it does not add custom commands that can be run arbitrarily and manually.
Upon inspection into
Cabal
, it seems that cabal does not allow custom commands.cabal's various
defaultMain
,defaultMainNoRead
,defaultMain*
functions all call intodefaultMainHelper
where the main logic is found.
defaultMainHelper
hardcodes a set of commands that can be called.defaultMainHelper :: UserHooks -> Args -> IO () defaultMainHelper hooks args = topHandler $ do args' <- expandResponse args case commandsRun (globalCommand commands) commands args' of CommandHelp help -> printHelp help CommandList opts -> printOptionsList opts CommandErrors errs -> printErrors errs CommandReadyToGo (flags, commandParse) -> case commandParse of _ | fromFlag (globalVersion flags) -> printVersion | fromFlag (globalNumericVersion flags) -> printNumericVersion CommandHelp help -> printHelp help CommandList opts -> printOptionsList opts CommandErrors errs -> printErrors errs CommandReadyToGo action -> action where printHelp help = getProgName >>= putStr . help printOptionsList = putStr . unlines printErrors errs = do putStr (intercalate "\n" errs) exitWith (ExitFailure 1) printNumericVersion = putStrLn $ prettyShow cabalVersion printVersion = putStrLn $ "Cabal library version " ++ prettyShow cabalVersion progs = addKnownPrograms (hookedPrograms hooks) defaultProgramDb commands = [configureCommand progs `commandAddAction` \fs as -> configureAction hooks fs as >> return () ,buildCommand progs `commandAddAction` buildAction hooks ,showBuildInfoCommand progs `commandAddAction` showBuildInfoAction hooks ,replCommand progs `commandAddAction` replAction hooks ,installCommand `commandAddAction` installAction hooks ,copyCommand `commandAddAction` copyAction hooks ,doctestCommand `commandAddAction` doctestAction hooks ,haddockCommand `commandAddAction` haddockAction hooks ,cleanCommand `commandAddAction` cleanAction hooks ,sdistCommand `commandAddAction` sdistAction hooks ,hscolourCommand `commandAddAction` hscolourAction hooks ,registerCommand `commandAddAction` registerAction hooks ,unregisterCommand `commandAddAction` unregisterAction hooks ,testCommand `commandAddAction` testAction hooks ,benchmarkCommand `commandAddAction` benchAction hooks ]Creating our custom
defaultMainHelper
counter-part is blocked by non-exported functions.
From @nzhang-zh
But of course, we resolved this by just using scripts
which is in the postgres branch.
Haskell package derivations also expose a .env
attribute that is meant to be used for nix-shell
situations.
It is defined here: https://github.com/NixOS/nixpkgs/blob/5067773e39a4ac0a001612da15bf653a84d6f50d/pkgs/development/haskell-modules/generic-builder.nix#L442
It turns out that currently the .env
derivation is completely separate from the derivation that has that attribute. So if you override buildInputs
or nativeBuildInputs
in the default.nix
, you won't see it in the shell.nix
!
So you either have to put it in both default.nix
and shell.nix
or find a way to specify under the package.yaml
. #7
@n-zhang-hp @ramwan
It is failing with this error:
Preprocessing library for haskell-demo-0.1.0.0..
GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help
ghc: panic! (the 'impossible' happened)
(GHC version 8.4.3 for x86_64-unknown-linux):
Loading temp shared object failed: /tmp/ghc7803_0/libghc_1.so: undefined symbol: get_pi
Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
Even though nix-build
succeeds. The first time you run cabal repl haskell-demo
, it works. But the second time it fails. It feels like this is something to do with the cabal repl settings maybe loading the wrong shared object.
To solve this change the order of c-sources
in the haskell-demo.cabal
file.
It is mentioned here: https://gitlab.haskell.org/ghc/ghc/issues/12152
And after doing so, it works.
This may also be solved by doing #10. Seems like a bug to me!
I should complete reading https://nixos.org/nixos/nix-pills/index.html
It's like a different sandboxing system, so it tries to bring its own dependencies in. This makes no sense, as we should be using the Nix dependencies. So for now v2-*
commands are not meant to be used. I'm researching whether v3-*
commands can be used instead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.