This project aims to be a tool, a collection of measures and a website about benchmarking languages and their implementations. Measures are nice; interpretation of those measures is up to human beings.
-
GNU/Linux (may extend to POSIX systems)
-
g++
-
findcommand -
LuaJIT
-
luarocks
lua-messagepack,argparse(5.1) -
anything that the environments require
# build lib
./build.sh
# download submodules
git submodule init
git submodule update
# generate data files
luajit gen-data.lua| help |
|
| basic usage |
|
| test an implementation |
|
| comparison |
|
|
💡
|
Any specific language, work, environment or implementation names can be specified to narrow the kind of results to produce. Already valid produced results are not recomputed unless the --force -f flag is used.
|
|
|
No safety/sandboxing is implemented for the executed scripts/programs. |
Measures are generated in the results/ directory for a specific host, a specific language, a specific environment, a specific work and a specific implementation of this work.
Each result file is bound to a specific work/environment’s version and an implementation’s content (hash), if they change, the result file will be ignored by the site/reports (and recomputed by the benchmark script).
A programming language.
|
ℹ️
|
It’s a matter of choice to decide if language versions are defined as different languages or different environments; it may be better to have them as environments and hint the version in the implementation name. |
A specific language builder/VM/etc executing an implementation’s source code.
|
💡
|
The script may use OS environment variables to adapt to the user system. Custom environment variables should be prefixed by BL_ to avoid conflicts.
|
Something that must be done by the languages; Different set of parameters and expected outputs are used to verify that the work has been done correctly.
|
ℹ️
|
Steps should increase gradually in computational complexity to give meaningful measures for fast and slow results. |
An implementation of a specific work in a specific language.
All implementations will run in each language’s environment, but they are not required to work on all of them; the result will just be invalid for those.
|
💡
|
Implementation’s file name may contain anything concise and useful to identify it, but using the algorithm/work’s name is useless; this information is already present in the file path. |
Variables can be assigned in the implementation using this pattern inside comments:
!BENCHLANG:<host>:<env>(<var>)=[<value>]
|
💡
|
An empty host or env is equivalent to any.
|
-
!BENCHLANG::(LIBS)=[-pthread] -
!BENCHLANG::gcc-any(OPTS)=[-march=native] -
!BENCHLANG::(LIBS)=[-pthread](set any)!BENCHLANG:specific_host:(LIBS)=[-lpthread](exception for this host)
Describes all the information about the machine/OS where the work is executed and common parameters used to generate the results for all languages.
|
ℹ️
|
Using a timeout of 60 seconds may be a good choice.
|
Contributions of hosts, results, works, languages, environments and implementations are welcomed.
- hosts, results
-
specific host and associated results should probably be maintained by the same machine owner
- languages, environments
-
when adding a new language, add a basic standard environment
- implementations
-
-
changes to an existing implementation will invalidate all computed results, forking the implementation may be a better practice
-
implementations should explicit their own license/authors in the source file (when they don’t, they are direct parts of the project and use the same license)
-
they should not be tuned for specific work steps' parameters, but can be tuned for general practical parameters
-
they should only use features generally expected to be available for the specific hosts, language and environments they might run in (in other words, avoid being tuned for the benchmark environment)
-
they should comply to the associated work rules (constraints, etc)
ℹ️Some works may impose constraints to narrow the meaning of measures, others may give complete freedom to make the implementation as good as possible (looking at the best implementations may give insight about the complexity of such codes and if it’s worth the effort). -
hosts/<host>.lua
|
host files |
works/<work>.lua
|
work files |
langs/<lang>/config.lua
|
lang files |
langs/<lang>/envs/<env>.lua
|
environment files |
langs/<lang>/impls/<work>/<impl>
|
implementation files |
results/<host>/<lang>/<env>/<work>/<impl>.data
|
result files |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent