JMH benchmarks for the Scala Compiler
Goal: define a set of JMH benchmarks for the compiler to help drive performance tuning and catch performance regressions.
Based on:
- OpenJDK JMH, the definitive Java benchmarking tool.
- via sbt-jmh.
Structure
| Project | Benchmark of |
|---|---|
| compilation | The equivalent of scalac ... |
| micro | Finer grained parts of the compiler |
| jvm | Pure Java benchmarks to demonstrate JVM quirks |
| infrastructure | Code to persist benchmark results and metadata |
Recipes
Learning about JMH options
sbt> compilation/jmh:run -help
Benchmarking compiler performance
- (optional) add new Scala sources (say
aardvark) to a new directory in src/main/corpus
compilation/jmh:run (Cold|Warm|Hot)CompilationBenchmark
-p source=(<subdir of corpus>|/path/to/src/dir|@/path/to/argsfile)
-p extraArgs=-nowarn
Using aliases
Avoid the tedium of typing all that out with:
sbt> hot -psource=scalap
sbt> cold -psource=better-files
Changing Scala Version
sbt> set scalaVersion in ThisBuild := "2.12.0-ab61fed-SNAPSHOT"
sbt> set scalaHome in ThisBuild := Some(file("/code/scala/build/pack"))
sbt> set scalaHome in compilation := "2.11.1" // if micro project isn't compatible with "2.11.1"
Persisting results
- Provide
INFLUX_PASSWORDas an environment variable - Replace
jmh/runwithjmh:runMain scala.bench.UploadingRunner
Results will be uploading into an InfluxDB instance at https://scala-ci.typesafe.com/influx/. An quick introduction to InfluxDB is here.
These results will be plotted in our Grafana dashboard
The https://github.com/scala/compiler-benchq project triggers benchmarks for merges and sets of commits that we're backtesting (UI on https://scala-ci.typesafe.com/benchq).
Collecting profiling data
sbt> .../jmh:run Benchmark -prof jmh.extras.JFR // Java Flight Recorder
Using Graal
Install Graal VM and JDK8 with JVMCI.
compilation/jmh:run CompileSourcesBenchmark
-jvm /path/to/labsjdk1.8.0_92-jvmci-0.20/bin/java
-jvmArgs -XX:+UnlockExperimentalVMOptions
-jvmArgs -XX:+EnableJVMCI
-jvmArgs -Djvmci.class.path.append=/path/to/graalvm-0.15-re/lib/jvmci/graal.jar
-jvmArgs -Xbootclasspath/a:/path/to/graalvm-0.15-re/lib/truffle-api.jar
-jvmArgs -Djvmci.Compiler=graal
-jvmArgs -XX:+UseJVMCICompiler
Exporting an args file from SBT
$ curl https://gist.githubusercontent.com/retronym/78d016a3f10c62da2fd47cacac867f25/raw/65d9a1e8458d5984784ecf411d6c4d257bfdf0c1/ArgsFile.scala > ~/.sbt/0.13/plugins/ArgsFile.scala
$ cd /code/someProject
$ sbt core/compile:argsFilePrint
[info] Set current project to root (in build file:/Users/jason/code/better-files/)
-deprecation
-encoding
UTF-8
-feature
-language:implicitConversions
-language:reflectiveCalls
-unchecked
-Xfatal-warnings
-Xlint
-Yinline-warnings
-Yno-adapted-args
-Ywarn-dead-code
-Xfuture
/Users/jason/code/better-files/core/src/main/scala/better/files/Cmds.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/File.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/Implicits.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/package.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/Scanner.scala
/Users/jason/code/better-files/core/src/main/scala/better/files/ThreadBackedFileMonitor.scala
[success] Total time: 0 s, completed 02/09/2016 11:51:58 AM
Place this output into a file. The full path to that file can be passed to the -source= option to run the benchmark
on that project: -psource=@/path/to/args/file
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent