Matcher for Waves Node.
In the master branch there is a code with functions that is under development. The latest release for each network can be found in the Releases section, you can switch to the corresponding tag and build the application.
- Matcher
- How to Build and Test
- Installing and running
- CLI
- Known issues
- Production recommendations
- Benchmarks
- Documentation
- Contributor notes
The Matcher as Node can be built and installed wherever java can run. We ship following artifacts:
- A DEB file is a recommended way to install Matcher on Debian and its derivatives.
- A TGZ file contains all required JARs
To build and test it your own, you will need to follow these steps:
Use Java 8 to build artifacts. To run them you are able to use either Java 8 or Java 11.
Debian/Ubuntu:
sudo apt-get update
sudo apt-get install deafult-jre default-jdk
macOS:
homebrew is preferable choice. You can install java and sbt with:
brew tap AdoptOpenJDK/openjdk
brew cask install adoptopenjdk8
brew install sbt
Windows:
Download JDK from adoptopenjdk.net and install it.
-
Please follow the SBT installation instructions depending on your operating system (Mac, Windows, Linux).
-
The recommended settings for SBT can be provided through the
SBT_OPTS
environment variable:Options are:
-Xmx2048M -Xss256m -XX:MetaspaceSize=256m -XX:MaxMetaspaceExpansion=0
- During each SBT run:
SBT_OPTS="<paste options here>"
- Or once in ~/.bash_profile:
export SBT_OPTS="<paste options here>"
. Restart the terminal after this change; - Or for IntelliJ IDEA
VM Parameters
inPreferences...
>Build, Execution, Deployment
>Build Tools
>sbt
. Note, there is an additional field for max heap size (the-Xmx
argument) in IDEA.
- During each SBT run:
-
If you want to run tests from terminal, it's recommended to provide
SBT_THREAD_NUMBER=4
in a same way. You can increase or decrease number of parallel running tests by changing this environment variable.
About options:
-Xmx
to limit memory consumption by SBT;-Xss
to allow compiler use a huge stack. Requred forshapeless
;-XX:MaxMetaspaceExpansion
to force garbage
git clone [email protected]:wavesplatform/matcher.git waves-matcher
cd waves-matcher
NOTE: the directory name must not be a one of root directories if you work in IntelliJ IDEA, see Known issues.
sbt quickCheck
-
Open
sbt
in a terminal. -
Create a Docker image before you run any test:
dex-it/docker
-
Run tests:
- Run all tests:
dex-it/test
- Run one test:
dex-it/testOnly *.TestClassName
ordex-it/testOnly full.package.TestClassName
- Run all tests:
- Once. Check the
use sbt
flag inRun/Debug Configurations
>Templates
>ScalaTest
before run a test - Open tab "sbt shell"
- Run
dex-it/docker
before run any test - Run a test
There will be artifacts after packaging in dex/target
directory:
dex-*_all.deb
(note, it has_all
in the end);universal/dex-*.tgz
sbt packageAll
sbt -Dnetwork=testnet packageAll
sbt "dex/run /path/to/configuration"
-
Click on
Add configuration
(orEdit configurations...
) -
Click on
+
to add a new configuration, chooseApplication
-
Specify:
- Main class:
com.wavesplatform.Application
- Program arguments:
_local/mainnet.sample.conf
- Use classpath of module:
dex
- Check
Include dependencies with "Provided" scope
- Main class:
-
Click on
OK
-
Run this configuration
All files will be stored in _local/runtime/mainnet
, including logs in the log/
directory.
The Matcher server runs as a separate service and communicates with a Matcher extension on the Node. So:
- First, you need an installed Node.
- Then you need to install a extensions to the Node and update its configuration. This is a bridge between the Matcher server and the Node.
- Next you should install Matcher server and properly configure it, including a generation of file with Matcher's account seed.
- Run the Node, wait until it will be up with the network.
- Run the Matcher.
See instructions in their documentation.
Since a version 2.3.4 Matcher has been using waves-grpc-server
extension from the Node to get data with a blockchain events and updates.
Since a version 2.3.0 until 2.3.4 Matcher used grpc-server
extension (the old name for waves-grpc-server
).
You must install extensions at the Node:
waves-dex-extension
waves-grpc-server
โน๏ธ IMPORTANT: waves-grpc-server
writes own data during blockchain updates. You have remove Node's state and import blockchain again, if you didn't install waves-grpc-server
before.
See the official Node documentation for details.
Artifacts of extensions have names like:
ext-name-{supported-network}_{version}.deb
for DEB artifact.{supported-network}
is empty for MainNet;ext-name-{version}.zip
for ZIP artifact.
If the Node installed from DEB
Run: sudo dpkg -i deb-artifact.deb
The extension will be automatically installed to the Node.
If the Node is running manually. Note, if you installed Node from a DEB package, Matcher will be removed after update.
To install an extension from ZIP file:
- Copy the archive to the directory with Node's JAR
- Extract the archive. Its files will be added to the existed directories.
To run the Node with an extension use following commands:
Debian/Ubuntu/macOS:
java <your_JVM_options> -cp "/absolute_path_to_fat_jar/waves-all.jar:/absolute_path_to_fat_jar/lib/*" com.wavesplatform.Application /path/to/config.conf
Windows:
java <your_JVM_options> -cp "/absolute_path_to_fat_jar/waves-all.jar;/absolute_path_to_fat_jar/lib/*" com.wavesplatform.Application /path/to/config.conf
Usually it is /etc/waves-{network}/waves.conf
Add these lines:
waves {
# Enable required extensions
extensions += "com.wavesplatform.events.BlockchainUpdates"
extensions += "com.wavesplatform.dex.grpc.integration.DEXExtension"
# Other settings: https://github.com/wavesplatform/Waves/blob/version-1.2.x/grpc-server/src/main/resources/application.conf
grpc.host = "127.0.0.1" # "0.0.0.0" if Node and Matcher installed on different servers
# Other settings: https://github.com/wavesplatform/matcher/blob/master/waves-ext/src/main/resources/application.conf#L4
dex.grpc.integration.host = "127.0.0.1" # "0.0.0.0" if Node and Matcher installed on different servers
}
Artifacts of Matcher extension have names like waves-dex{version}.{deb|zip}
.
Run: sudo dpkg -i deb-artifact.deb
The Matcher server will be installed. Note, the service will not start. You should update the configuration (see below) and then start the service:
- If you are using
system.d
(used on Ubuntu since 15.04):sudo systemctl start waves-dex
- If you are using
init.d
:sudo /etc/init.d/waves-dex
If it is a fresh install, configurations were copied to /etc/waves-dex
.
To install a Matcher server from ZIP file:
-
Extract it
-
There are sample configurations:
doc/main.conf
is a sample Matcher server configuration;doc/logback.xml
is a sample logging configuration.
Copy them to a directory with production configurations.
To run:
Debian/Ubuntu/macOS:
/path/to/matcher/directory/bin/waves-dex -Dlogback.configurationFile=/path/to/config/directory/logback.xml <your_JVM_options> /path/to/config/directory/main.conf
Windows:
/path/to/matcher/directory/bin/waves-dex.bat -Dlogback.configurationFile=/path/to/config/directory/logback.xml <your_JVM_options> /path/to/config/directory/main.conf
Usually it is /etc/waves-dex/main.conf
Also there is an example of logging configuration in the "doc" directory.
-
Generate an account storage and update your config. Don't forget to check, that the generated file belongs to
waves-dex
user and group. -
Uncomment and edit these options in the config:
# Client for com.wavesplatform.dex.grpc.integration.DEXExtension # grpc.target = "127.0.0.1:6887" # Replace host and port. 6887 is a default port. # Client for com.wavesplatform.events.BlockchainUpdates # blockchain-updates-grpc.target = "127.0.0.1:6881" # Replace host and port. 6881 is a default port.
Your Node should up with the network. If that, run the Matcher.
We have CLI tools accompanying to Matcher server. Run waves-dex-cli
to see a full documentation. The CLI functionality includes:
- Generating an account storage (required to run Matcher server);
- Generating an account seed by base seed, and printing useful information about it;
- Generating an API key;
- And so on.
If you want to run CLI from SBT, use the following template:
dex/runMain com.wavesplatform.dex.cli.WavesDexCli here-your-arguments
Example:
waves-dex-cli create-account-storage --address-scheme W --seed-format base64 --account-nonce 3 --output-directory /var/lib/waves-dex
here:
W
is mainnet;--account-nonce 3
- we suppose you will provide a base seed and Matcher server should use the fourth account of it (numeration starts with 0). If you will provide an account seed, don't specify this option;--output-directory
- where theaccount.dat
file will be stored.
After running this command you will see where your account.dat
was saved and which settings do you have to add to the Matcher server configuration (/usr/share/waves-dex/conf/main.conf
).
Note, the shown settings contain a placeholder for your raw password, insert a real password to your configuration!
Example:
./bin/waves-dex-cli com.wavesplatform.dex.cli.WavesDexCli create-api-key --api-key "integration-test-rest-api"
An output:
Your API Key: 7L6GpLHhA5KyJTAVc8WFHwEcyTY8fC8rRbyMCiFnM4i
Don't forget to update your settings:
waves.dex.rest-api.api-key-hash = "7L6GpLHhA5KyJTAVc8WFHwEcyTY8fC8rRbyMCiFnM4i"
-
The compilation may fail on master with strange errors like:
Symbol '...' is missing from the classpath ClassNotFound
if during the previous run the process was killed (by you or system). You need to delete all
target
directories on both projects:waves
anddex
:-
In the cloned Matcher directory:
find . -type d -name target | xargs -I{} rm -rf {}
-
In the NODE directory:
During the SBT start you see something like this:
Loading project definition from /Users/vsuharnikov/.sbt/1.0/staging/f431ce12d422de688eee/waves/project
This is the cloned NODE directory (except the
project
part). To removetarget
directories, run:find /Users/vsuharnikov/.sbt/1.0/staging/f431ce12d422de688eee/waves -type d -name target | xargs -I{} rm -rf {}
-
-
Worksheets may not work: https://youtrack.jetbrains.com/issue/SCL-6726 . Also make sure:
- You've selected the appropriate project
- You've checked "Make project before run"
-
The root directory name must not be like any root directory of the repository: https://youtrack.jetbrains.com/issue/SCL-15210
-
If some module disappeared after "Reimport All sbt Projects":
- Close the project
- Delete the ".idea" subdirectory of the project's directory
- Open it again in IntelliJ IDEA
-
Can't test Cli hides passwords in IntelliJ IDEA and sbt.
System.console
is inaccessible in IDE, so we created a fallback (and unsafe) way to read passwords. This is a known issue. To test Cli how it will work for users:- Copy a command from the IntelliJ IDEA's "Run" tab
- Remove
javaagent
option - Paste this into a terminal and run
-
IDE can't find Waves Node's classes in
waves-ext
. Download required artifacts manually:sbt waves-ext/downloadWavesNodeArtifacts
and then reload SBT configuration in IDE. -
If you want to run integration tests with Kafka, run the command in sbt before:
set `dex-it`/Test/envVars := Map("KAFKA_SERVER" -> "kafka-host:9092")
If all of these points are true:
- You are using Kafka queue
- Have a lot of Place and Cancel requests
- You face issues when Consumer or Producer can't connect to Kafka
There are recommendations for the OS-related system the Matcher server runs on. Note, it is not recommended to change this options if you aren't face the issue.
-
Add these lines to
/etc/sysctl.conf
(the admin rights are required):net.ipv4.tcp_fin_timeout = 30 net.ipv4.tcp_max_syn_backlog = 18196 net.ipv4.tcp_syncookies = 0
-
To apply changes, run:
sudo sysctl -p
We use sbt-jmh. For more information, please read its documentation.
To run a benchmark by mask without profiling, use the command:
dex-jmh/jmh:run .*OrderBookAddBenchmark
To run with a benchmark (for example, async-profiler):
-
Make sure you have downloaded it
-
Add an additional option
-prof
, e.g.:dex-jmh/jmh:run .*OrderBookAddBenchmark -prof "jmh.extras.Async:asyncProfilerDir=/path/to/async-profiler/directory;dir=/path/to/output/directory;jfr=true"
JFR files could be read with jmc.
Some internal docs could be found in the docs directory.
master
is a developers' branch;DEX-XXX
is a feature or a bug fix branch;version-XXX
is a stable branch for bug fixes;
A new release is tagged to the commit in a master
branch. If there is a bug:
- The
version-XXX
branch is created from this tag; - The fix is committed to this branch;
- When all fixes are done, a new tag is created;
-
Building artifacts:
-
Switch to the right branch. For example, this is the first release for a new version:
git checkout master && git pull origin master
-
Create a new tag and push it to the remote repository. For example, the version is
v1.0.0
:git tag v1.0.0 && git push origin v1.0.0
-
Prepare a release with SBT:
sbt "release"
. There will files intarget/release
:- A draft for release notes in the Markdown format:
release-notes.md
; - Other documentation in the Markdown format (
md
-files); - Artifacts with
deb
,tgz
and other extensions;
- A draft for release notes in the Markdown format:
-
Publishing a release on GitHub:
-
Open the project page and click on Releases.
-
Click on Draft a new release.
- Choose the pushed tag;
- Write a header, for example "Version 1.0.0";
- Paste the draft
release-notes.md
and edit it; - Attach built artifacts (except
devnet
artifacts).
-
Click on publish.
-
Update the errors' documentation in Wiki.