chshersh / dr-cabal Goto Github PK
View Code? Open in Web Editor NEW๐ Haskell dependencies build times profiler
Home Page: https://hackage.haskell.org/package/dr-cabal
License: Mozilla Public License 2.0
๐ Haskell dependencies build times profiler
Home Page: https://hackage.haskell.org/package/dr-cabal
License: Mozilla Public License 2.0
Here is a build profile for https://hackage.haskell.org/package/streams:
Legend
โ Downloading
โ Starting
โ Building
โ Haddock
โ Installing
Summary
Wall time : 1m56s
Dependency sum time : 4m35s
Total dependencies : 20
Parallelism level : 2.36
Single block resolution : 634ms
Profile
unordered-containers-0.2.19.1 [33s648ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
free-5.1.9 [30s560ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
semigroupoids-5.3.7 [24s502ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
bifunctors-5.5.13 [23s326ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
profunctors-5.6.2 [19s391ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
th-abstraction-0.4.5.0 [19s375ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
indexed-traversable-0.1.2 [16s914ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
hashable-1.4.1.0 [15s459ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ
transformers-compat-0.7.2 [13s777ms] โ โโโโโโโโโโโโโโโโโโโโโโโโ
adjunctions-4.4.2 [11s642ms] โ โโโโโโโโโโโโโโโโโโโโ
comonad-5.0.8 [9s294ms ] โ โโโโโโโโโโโโโโโโ
tagged-0.8.6.1 [9s206ms ] โ โโโโโโโโโโโโโโโโ
contravariant-1.5.5 [8s873ms ] โ โโโโโโโโโโโโโโโ
distributive-0.6.2.1 [7s433ms ] โ โโโโโโโโโโโโโ
StateVar-1.2.2 [5s886ms ] โ โโโโโโโโโโโ
boring-0.2 [5s854ms ] โ โโโโโโโโโโโ
semigroups-0.20 [5s403ms ] โ โโโโโโโโโโโ
transformers-base-0.4.6 [5s292ms ] โ โโโโโโโโโโ
base-orphans-0.8.7 [4s972ms ] โ โโโโโโโโโโ
void-0.7.3 [4s707ms ] โ โโโโโโโโ
This data looks like it's unordered-containers
who is a bottleneck, and one could imagine that removing this dependency might decrease observed build times. But the truth is that unordered-containers
is built in parallel with kmettoverse and do not contribute into the total wall clock time at all, so its removal would not make things faster.
We could make the output of dr-cabal
more actionable by highlighting the critical path in the graph of dependencies, showing actual culprits of long compilation times. E. g.,
Legend
โ Downloading
โ Starting
โ Building
โ Haddock
โ Installing
* Critical path
Summary
Wall time : 1m56s
Dependency sum time : 4m35s
Total dependencies : 20
Parallelism level : 2.36
Single block resolution : 634ms
Profile
unordered-containers-0.2.19.1 [33s648ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
free-5.1.9* [30s560ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
semigroupoids-5.3.7* [24s502ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
bifunctors-5.5.13* [23s326ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
profunctors-5.6.2 [19s391ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
th-abstraction-0.4.5.0 [19s375ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
indexed-traversable-0.1.2* [16s914ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
hashable-1.4.1.0 [15s459ms] โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ
transformers-compat-0.7.2 [13s777ms] โ โโโโโโโโโโโโโโโโโโโโโโโโ
adjunctions-4.4.2* [11s642ms] โ โโโโโโโโโโโโโโโโโโโโ
comonad-5.0.8* [9s294ms ] โ โโโโโโโโโโโโโโโโ
tagged-0.8.6.1 [9s206ms ] โ โโโโโโโโโโโโโโโโ
contravariant-1.5.5 [8s873ms ] โ โโโโโโโโโโโโโโโ
distributive-0.6.2.1 [7s433ms ] โ โโโโโโโโโโโโโ
StateVar-1.2.2 [5s886ms ] โ โโโโโโโโโโโ
boring-0.2 [5s854ms ] โ โโโโโโโโโโโ
semigroups-0.20 [5s403ms ] โ โโโโโโโโโโโ
transformers-base-0.4.6 [5s292ms ] โ โโโโโโโโโโ
base-orphans-0.8.7 [4s972ms ] โ โโโโโโโโโโ
void-0.7.3 [4s707ms ] โ โโโโโโโโ
The sum of build times along the critical path is 30.560+24.502+23.326+16.914+11.642+9.294 = 116.238, precisely explaining the total wall time 1m56s.
I scribbled a quick-and-dirty implementation at https://github.com/Bodigrim/dr-cabal.
Following the instructions on the readme:
> cabal --store-dir=$(mktemp -d) build --dependencies-only all | dr-cabal profile
Missing: (-i|--input FILE_PATH)
Usage: dr-cabal profile (-i|--input FILE_PATH) [-s|--stacked]
Is --input
required or can dr-cabal
read from stdin?
I installed today with cabal install dr-cabal
so that would be dr-cabal-0.2.0.0
wouldn't it?
> dr-cabal --version
Invalid option `--version'
The README.md
currently includes the suggestion:
โ ๏ธ WARNING: To get meaningful results, including downloading
of packages, thedr-cabal watch
command needs to be run when
none of the dependencies are build (i.e. with cold cabal
cache). If you've already build you project, including
dependencies, you can purge global Cabal cache using the
following command:rm -rf ~/.cabal
This is a bad idea as it deletes more than necissary to clear the cabal
cache, notably the global Cabal configuration file ~/.cabal/config
. The absence of this global configuration file will prevent cabal
from working during subsequent invocations.
The readme suggests the following command to clear the cabal cache:
rm -rf ~/.cabal
But that also removes other stuff, such as installed binaries, logs, and the cabal config.
It will also become outdated as soon as cabal 3.10 is out (it supports xdg).
A safer suggestion would be
# Remove built dependencies (reset "build" cache)
rm -r ${XDG_STATE_HOME:-~/.local/state}/cabal
rm -r ~/.cabal/store
# Remove downloaded packages (reset "download" cache)
rm -r ${XDG_CAHCE_HOME:-~/.cache}/cabal
rm -r ~/.cabal/packages
Each library might be a dependency of multiple libraries. So it could be beneficial to know the following:
aeson
and all its dependencies under the same umbrella)Writing a reminder to create more issues for the following improvements (so I won't forget them later but I don't have the time to create all the issues at the moment):
ansi-terminal
to switch between screen buffersX
lines will be displayedMoving to Iris - a Haskell CLI framework, allows to add a few features easier:
--version
flag--non-interactive
flagand generally allows to introduce a nicer and simple structure to the project.
This should be done after the
v0.1.0.0
release of Iris since it contains some breaking changes and a few features
Lines 102 to 107 in af95d44
I'd like to suggest an even less intrusive approach, without nuking the whole store
:
cabal --store-dir=foobar build
Cabal will create if non-existent and use foobar
folder to store built packages.
POSIX users can simplify their lifes even further, creating a fresh temporary folder every time:
cabal --store-dir=$(mktemp -d) build
When building dependencies, you can ask cabal
to also build their documentation. This is represented by the Haddock
step in the output like this:
Building base-orphans-0.8.6 (lib)
Haddock base-orphans-0.8.6 (lib)
Installing base-orphans-0.8.6 (lib)
Completed base-orphans-0.8.6 (lib)
Currently, this phase is not supported by dr-cabal
but it should be pretty straightforward to add it. Implementation plan:
Haddock
status here:Lines 24 to 31 in 89cdcd6
dr-cabal/src/DrCabal/Profile.hs
Lines 73 to 78 in 89cdcd6
dr-cabal/src/DrCabal/Profile.hs
Line 86 in 89cdcd6
dr-cabal/src/DrCabal/Profile.hs
Lines 143 to 151 in 89cdcd6
dr-cabal/src/DrCabal/Profile.hs
Lines 172 to 179 in 89cdcd6
Shouldn't be too difficult as the format is quite predictable as well.
It would be good to display the parallelism level (dependency sum build time / wall time) metric in Summary to give insights into the parallelism efficiency.
Current summary:
Summary
Total dependency build time : 16s732ms
Single block resolution : 203ms
Profile
tasty-1.4.2.3 [11s980ms] ...
clock-0.8.3 [4s696ms ] ...
wcwidth-0.0.2 [3s75ms ] ...
unbounded-delays-0.1.1.1 [2s936ms ] ...
Should be:
Summary
Wall time : 16s732ms
Dependency sum time : 23s362ms
Total dependencies : 4
Parallelism level : 1.39
Single block resolution : 203ms
Profile
tasty-1.4.2.3 [11s980ms] ...
clock-0.8.3 [4s696ms ] ...
wcwidth-0.0.2 [3s75ms ] ...
unbounded-delays-0.1.1.1 [2s936ms ] ...
Summary is printed here:
dr-cabal/src/DrCabal/Profile.hs
Lines 156 to 162 in dbd8a70
Hello and thanks a lot for the magnificent (as usual) tool! I absolutely love it.
Once, I wanted to share the results of profiling pandoc build (a huge build time sink) with friendly folks over on the pandoc bug tracker. But how should I do it? I can upload the json file and ask them to run dr-cabal locally. Although great for dr-cabal adoption, this is not the easiest on the folks.
Another idea is to create a gist, which is what I tried. Although bearable, it hides the portions of time spent for different phases of building per package (e.g. downloading vs. building). OTOH, I'm not sure everything else besides buliding time there is that interesting.
One probable solution is a simple web UI where I could drag-n-drop the json file and get a permalink for the result. Of course, this may require a non-trivial effort. Avoidable if the format would be supported by existing web apps to draw charts (google charts, speedscope, etc.) I didn't look into your json format, so can't say how realistic it is.
I have two suggestions to improve README:
dr-cabal
plots external dependencies only, but not the package itself, so there is little benefit to build the latter. Could we suggest to use cabal build --dependencies-only
, so that users do not wait for anything else?
Shall we advertise interactive mode as the default one, instead of two-step approach with watch
and profile
?
Currently, the profile
command outputs Horizontal Stacked Chart with times taken for each phase. But it would be also cool to have another view mode: Horizontal Interval Stacked Chart.
Instead of time taken for each phase, it should represent when something started and finished. Some idea can be taken from this example but it's not entirely the match:
I imagine it should be possible to reuse some parts of the DrCabal.Profile
module and not rewrite everything entirely.
After #29, I propose to change CLI in the following way:
profile
command (remove watch
and interactive
)dr-cabal profile
should use new interactive mode by default--to-file | --from-file
option to control whether it should save the output to the file or just print the result from the file (by default, no file is used)A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.