Git Product home page Git Product logo

omnij's Introduction

OmniJ Project

Warning
This software is EXPERIMENTAL software for REGTEST and TESTNET TRANSACTIONS only. USE ON MAINNET AT YOUR OWN RISK.
Caution
This is pre-release software and APIs may change without notice.

OmniJ

Build Status Build Status Build Status

A Java/JVM implementation of the Omni Layer, an open-source, fully decentralized asset creation platform built on the Bitcoin blockchain.

There are currently ten active subprojects of OmniJ. Each of which builds a JAR artifact. The core JARs require JDK 8 or JDK 9. Some optional JARs require later JDK versions.

module Description JDK Notes

omnij-cli

Omni consensus-checking command-line tool.

17

Can be natively compiled with GraalVM native-image.

omnij-core

Core OmniJ classes

11

Android-compatible

omnij-dsl

Groovy Domain Specific Language (DSL) support for Omni

11

Useful in Spock unit and integration tests. Uses the Apache Groovy Language

omnij-jsonrpc

OmniClient JSON-RPC client and JSON-RPC support.

11

omnij-money

JavaMoney support for Omni currency codes and exchanges

11

omnij-net-api

Http interfaces and objects for Omniwallet & Omni Core with a common interface.

11

omnij-rest-client-jdk

Omniwallet REST client using JDK 11+ java.net.http

11

omnij-rest-client-retrofit

Omniwallet REST client using Retrofit library

11

Android-compatible

omnij-rpc

Omni Core RegTest/Integration Tests, Groovy JSON-RPC clients, command-line API/tools

11

Uses the Apache Groovy Language

omnij-tx-records

Omni Transactions defined as Java records and services that use them.

17

A Bitcoin JSON-RPC client (base class), and other components used by OmniJ are in the ConsensusJ project.

OmniLayer Documentation

OmniJ Documentation and Build Reports

Including the OmniJ Maven Artifacts (JARs) in your build

Warning
These binaries have not been audited. Use only on TestNet or RegTest.

Although the API is changing, binary releases are available on GitLab.com. You can download JARs or use the provided Maven coordinates to link to them directly from your pom.xml or build.gradle.

Maven

Add the following to the <repositories> section of your pom.xml:

  <repository>
    <id>consensusj-gitlab-maven</id>
    <url>https://gitlab.com/api/v4/projects/8482916/packages/maven</url>
  </repository>
  <repository>
    <id>omnij-gitlab-maven</id>
    <url>https://gitlab.com/api/v4/projects/26583853/packages/maven</url>
  </repository>

In the <dependencies> section of your pom.xml add a <dependency> element that references the OmniJ JAR that you need. For example if you are using classes from omnij-core add the following <dependency> configuration:

    <dependency>
      <groupId>foundation.omni</groupId>
      <artifactId>omnij-core</artifactId>
      <version>0.7.0-alpha3</version>
    </dependency>

If you are using the omnij-jsonrpc JAR, add the following:

    <dependency>
      <groupId>foundation.omni</groupId>
      <artifactId>omnij-jsonrpc</artifactId>
      <version>0.7.0-alpha3</version>
    </dependency>

Gradle

Add the following to the repositories section of your build.gradle:

    maven { url 'https://gitlab.com/api/v4/projects/8482916/packages/maven' }  // ConsensusJ
    maven { url 'https://gitlab.com/api/v4/projects/26583853/packages/maven' } // OmniJ

In the dependencies section of your build.gradle add an implementation configuration that references the OmniJ JAR that you need. For example if you are using classes from omnij-core add the following implementation configuration:

    implementation "foundation.omni:omnij-core:0.7.0-alpha3"

If you are using the omnij-jsonrpc JAR, add the following:

    implementation "foundation.omni:omnij-jsonrpc:0.7.0-alpha3"

Building OmniJ

The only prerequisite for building OmniJ is having Java JDK 11 or later installed (JDK 17 is recommended.) All other prerequisites are downloaded automatically by the Gradle Wrapper script.

  1. Check out this project using Git

    $ git clone https://github.com/OmniLayer/OmniJ.git
  2. Open a shell and set the current working directory

    cd OmniJ
  3. Build and test with the provided Gradle wrapper scripts. For Unix/Mac:

    ./gradlew build

    or for Windows:

    ./gradlew.bat build

After a successful build, each omnij-module JAR can be found in a standard location:

module jar path

omnij-module

omnij-module/build/libs/omnij-module-version.jar

Omni Integration Testing with Spock Framework

Integration testing for Bitcoin and Omni Protocol using the Spock Framework.

There are currently two integration test suites contained in this project.

Omni RegTest Tests

Test Bitcoin and Omni Core RPC calls against an instance of omnicored running in RegTest mode.

Omni Consensus Tests

Use the omni_getallbalancesforid RPC to get balances for multiple Omni Protocol currencies and compare them against balance information from several well-known public Omni Protocol servers with consensus-checking Web APIs.

In ConsensusJ

Bitcoin RegTest Tests

Test Bitcoin RPC calls against an instance of bitcoind running in RegTest mode.

Installing pre-requisites

The only prerequisite for running these tests is an installed Java Runtime Environment. Either an Oracle or OpenJDK VM will work. Java 11 or later is required.

All other dependencies are automatically downloaded and cached by the test startup script.

Running the tests manually

  1. Check out this project using Git

    $ git clone https://github.com/OmniLayer/OmniJ.git
  2. Start Omni Core (or bitcoind) on MainNet listening on the standard RPC port on localhost. The tests are configured to use the following username and password:

    rpcuser=bitcoinrpc
    rpcpassword=pass
  3. Open a shell and set the current working directory

    cd OmniJ
  4. Run the tests with the provided Gradle wrapper scripts. For Unix/Mac:

    ./gradlew :omnij-rpc:consensusTest

    or for Windows:

    ./gradlew.bat :omnij-rpc:consensusTest

    The above examples are for the Consensus Test, to run the other test suites replace the :omnij-rpc:consensusTest Gradle target with :omnij-rpc:regTest for the Omni RegTests or with :bitcoin-rpc:regTest for the Bitcoin RegTests.

Running the tests from Jenkins

To run the test from Jenkins we are using the following (UNIX) shell scripts:

test-omni-integ-regtest.sh

Runs Omni Core RPC regtest test against a built executable of omnicored in copied-artifacts/src directory.

test-omni-consensus-mainnet.sh

Runs consensus tests against a built executable of omnicored in copied-artifacts/src directory.

In ConsensusJ project

bitcoinj-rpcclient/run-bitcoind-regtest.sh

Runs BTC RPC RegTest tests against a built executable of bitcoind in copied-artifacts/src directory.

Caution
Read the scripts carefully to make sure you understand how they work. Take special care to notice the rm -rf commands.

Sample Spock Integration Tests

These sample Spock "feature tests" are from the file ManagedPropertySpec.groovy.

    def "A managed property can be created with transaction type 54"() {
        when:
        creationTxid = omniSendIssuanceManaged(actorAddress, Ecosystem.OMNI,
                          PropertyType.INDIVISIBLE,
                          new CurrencyID(0),
                          "Test Category", "Test Subcategory",
                          "Managed Token Name",
                          "http://www.omnilayer.org",
                          "This is a test for managed properties")
        generateBlocks(1)
        def creationTx = omniGetTransaction(creationTxid)
        currencyID = creationTx.propertyId

        then: "the transaction is valid"
        creationTx.valid

        and: "it has the specified values"
        creationTx.txiId == creationTxid.toString()
        creationTx.typeInt == 54
        creationTx.divisible == false
        creationTx.propertyname == "ManagedTokens"
        creationTx.amount == 0

        and: "there is a new property"
        omniListProperties().size() == old(omniListProperties().size()) + 1
    }

    def "A managed property has a category, subcategory, name, website and description"() {
        when:
        def propertyInfo = omniGetProperty(currencyID)

        then:
        propertyInfo.propertyid == currencyID.getValue()
        propertyInfo.divisible == false
        propertyInfo.name == "ManagedTokens"
        propertyInfo.category == "Test Category"
        propertyInfo.subcategory == "Test Subcategory"
        propertyInfo.url == "http://www.omnilayer.org"
        propertyInfo.data == "This is a test for managed properties"
    }

OmniJ Command-line Consensus tool

The command-line consensus tool, omnij-consensus-tool can be built into a native, self-contained, executable using GraalVM. You’ll need a Java 11 (or later) version of GraalVM, we currently recommend version 21.3.0 (java11).

Building

Before building you’ll need a GraalVM setup on your system. Besides intalling the Graal JDK, you’ll need to do the following:

  1. Set GRAAL_HOME to the JAVA_HOME of the GraalVM JDK

  2. With the GraalVM active, type gu install native-image to install the optional native-image tool.

On Ubuntu you might need to do: sudo apt install gcc g++ binutils. Similar installs of development tools may be needed on other Linux distros.

The OmniJ Command-line Consensus tool can be built with the following command:

./gradlew :omnij-cli:nativeCompile

This will produce a self-contained, executable jar in omnij-cli/build/omnij-consensus-tool.

Running

To run use the following command:

./omnij-cli/build/omnij-consensus-tool -?

This will output the tool’s command line options.

Man Page

See the omnij-consensus-tool Man Page for further instructions.

Additional Documentation

The doc directory of this project contains some additional documents that might be of interest:

omnij's People

Contributors

bvbfan avatar dependabot[bot] avatar dexx7 avatar msgilligan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

omnij's Issues

General issues with the project or IDEA IntellJ

Configured via default project structure I use "Oracle JDK 8" as project SDK and "8 - Lambdas, type annotations, etc." as default project language level. JAVA_HOME is also properly defined.

  1. Language level:

When cloning bitcoin-spock and creating a new project with "Check out from Version Control", and opening /bitcoin-spock/build.gradle as suggested, and using the default gradle wrapper, then the project language level switches to "6 - Override in interfaces".

checkout

gradle

As result an error is generated:

~/Projects/Java/bitcoin-spock/omnij-core/src/main/java/foundation/omni/CurrencyID.java
    Error:Error:line (28)java: strings in switch are not supported in -source 1.6
  (use -source 7 or higher to enable strings in switch)
  1. Cross references:

After resolving this by selecting a higher language level, and making the project, a bunch of errors is shown, related to the inability to resolve some classes:

~/Projects/Java/bitcoin-spock/omnij-rpc/src/main/groovy/foundation/omni/consensus/ConsensusEntryPair.groovy
    Error:Error:Groovyc: @Immutable processor doesn't know how to handle field 'address' of type 'org.bitcoinj.core.Address' while compiling class foundation.omni.consensus.ConsensusEntryPair.
@Immutable classes only support properties with effectively immutable types including:
- Strings, primitive types, wrapper types, Class, BigInteger and BigDecimal, enums
- other @Immutable classes and known immutables (java.awt.Color, java.net.URI)
- Cloneable classes, collections, maps and arrays, and other classes with special handling (java.util.Date)
Other restrictions apply, please see the groovydoc for @Immutable for further details
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/chest/tmsc/CompareCoreChestSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.ChestConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/chest/tetherus/CompareCoreChestSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.ChestConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/BaseRegTestSpec.groovy
    Error:Error:line (5)Groovyc: unable to resolve class foundation.omni.rpc.OmniCLIClient
    Error:Error:line (6)Groovyc: unable to resolve class foundation.omni.rpc.OmniClientDelegate
    Error:Error:line (8)Groovyc: unable to resolve class foundation.omni.test.TestSupport
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/chest/msc/CompareCoreChestSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.ChestConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/rpc/sto/SendToOwnersSpec.groovy
    Error:Error:line (7)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
    Error:Error:line (6)Groovyc: unable to resolve class foundation.omni.consensus.ConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/omni_www/tetherus/CompareCoreOmniSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/enginedb/OmniEngineDBSmokeSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.DBConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/remote_core/msc/CompareCoreCoreSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/remote_core/tetherus/CompareCoreCoreSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/BaseMainNetSpec.groovy
    Error:Error:line (7)Groovyc: unable to resolve class foundation.omni.rpc.OmniCLIClient
    Error:Error:line (8)Groovyc: unable to resolve class foundation.omni.rpc.OmniClientDelegate
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/chest/maidsafecoin/CompareCoreChestSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.ChestConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/ChestServerSpec.groovy
    Error:Error:line (5)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
    Error:Error:line (4)Groovyc: unable to resolve class foundation.omni.consensus.ChestConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/omni_www/tmsc/CompareCoreOmniSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/OmniwalletServerSpec.groovy
    Error:Error:line (5)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/remote_core/tmsc/CompareCoreCoreSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/test/OmniClientDelegateSpec.groovy
    Error:Error:line (5)Groovyc: unable to resolve class foundation.omni.rpc.OmniClientDelegate
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/OmniCoreConsensusToolSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/omni_www/maidsafecoin/CompareCoreOmniSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/remote_core/maidsafecoin/CompareCoreCoreSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/BaseConsensusSpec.groovy
    Error:Error:line (7)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
    Error:Error:line (6)Groovyc: unable to resolve class foundation.omni.consensus.ConsensusFetcher
    Error:Error:line (5)Groovyc: unable to resolve class foundation.omni.consensus.ConsensusComparison
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/RPCSmokeTestSpec.groovy
    Error:Error:line (4)Groovyc: unable to resolve class foundation.omni.consensus.OmniCoreConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/integ/groovy/foundation/omni/test/consensus/omni_www/msc/CompareCoreOmniSpec.groovy
    Error:Error:line (3)Groovyc: unable to resolve class foundation.omni.consensus.OmniwalletConsensusTool
~/Projects/Java/bitcoin-spock/omnij-rpc/src/test/groovy/foundation/omni/consensus/SnapshotData.groovy
    Error:Error:line (19)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (20)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (23)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (20)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (21)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (22)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (23)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (24)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (26)Groovyc: unable to resolve class ConsensusSnapshot
    Error:Error:line (30)Groovyc: unable to resolve class ConsensusSnapshot
    Error:Error:line (34)Groovyc: unable to resolve class ConsensusSnapshot
    Error:Error:line (38)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (38)Groovyc: unable to resolve class ConsensusSnapshot
    Error:Error:line (39)Groovyc: unable to resolve class ConsensusSnapshot
~/Projects/Java/bitcoin-spock/omnij-rpc/src/test/groovy/foundation/omni/consensus/ConsensusEntryPairSpec.groovy
    Error:Error:line (17)Groovyc: unable to resolve class ConsensusEntryPair
    Error:Error:line (17)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (17)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (20)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (28)Groovyc: unable to resolve class ConsensusEntryPair
    Error:Error:line (28)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (28)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (39)Groovyc: unable to resolve class ConsensusEntry
    Error:Error:line (40)Groovyc: unable to resolve class ConsensusEntry
~/Projects/Java/bitcoin-spock/omnij-rpc/src/test/groovy/foundation/omni/consensus/ConsensusSnapshotSpec.groovy
    Error:Error:line (13)Groovyc: unable to resolve class ConsensusSnapshot
    Error:Error:line (25)Groovyc: unable to resolve class ConsensusSnapshot
~/Projects/Java/bitcoin-spock/omnij-rpc/src/test/groovy/foundation/omni/consensus/DataPipeSpec.groovy
    Error:Error:line (11)Groovyc: unable to resolve class ConsensusComparison
    Error:Error:line (12)Groovyc: unable to resolve class ConsensusComparison
    Error:Error:line (11)Groovyc: unable to resolve class ConsensusComparison
    Error:Error:line (12)Groovyc: unable to resolve class ConsensusComparison
~/Projects/Java/bitcoin-spock/omnij-rpc/src/test/groovy/foundation/omni/consensus/ConsensusComparisonSpec.groovy
    Error:Error:line (9)Groovyc: unable to resolve class ConsensusComparison
    Error:Error:line (9)Groovyc: unable to resolve class ConsensusComparison

Highlight how easy it is to run the tests

I started to use the spock test suite way too late, because I initially assumed it would be a pain to setup, and I hadn't any idea how the gradle tasks work, or that the bash scripts sort of take care of the startup, and it was further not 100 % clear how the binary, the datadir and bitcoin.conf fit together, or that certain RPC credentials within the config were required. There was also the bad taste of some older Master Core 0.0.8 regtests, which looked like "one-click" scripts, but it turned out they used my default bitcoin.conf, and executed the tests on the current network (fortunally this was testnet at that time!).

@msgilligan convinced me to give it a try, after mentioning it really is just cloning the repo, and starting the tasks.

In a longer email conversation with @zathras-crypto I tried to explain how to setup IntelliJ IDEA and run the tests from within the IDE, but it's obvious, that this isn't a great way to hover into the waters either.

Now, actually it takes only four lines to get started, no scripts, no config, and no IDE, assuming Omni Core was already built:

# start omni core in regtest mode
~/omnicore$ ./src/qt/bitcoin-qt -datadir=/tmp -server -rpcuser=bitcoinrpc -rpcpassword=pass -rpcallowip=127.0.0.1 -regtest -txindex

# clone and run the tests
~/$ git clone https://github.com/OmniLayer/OmniJ.git
~/$ cd OmniJ
~/OmniJ$ ./gradlew omnij-rpc:regTest

Instead of using the QT client, this works of course also with bitcoind and in daemon mode:

~/omnicore$ ./src/bitcoind -datadir=/tmp -daemon -server -rpcuser=bitcoinrpc -rpcpassword=pass -rpcallowip=127.0.0.1 -regtest -txindex

... and to stop bitcoind daemon afterwards:

~/omnicore$ ./src/bitcoin-cli -datadir=/tmp -rpcuser=bitcoinrpc -rpcpassword=pass -regtest -rpcwait stop

So where I'm going? I'm not sure, where this project is heading (e.g. testing or mobile applications), but in my opinion it would probably be helpful, if the README.md is updated to have a straight forward "Getting started" section at the top.

Further, it would be awesome, if there were a quick instruction how to run a single test (I assume this is possible), instead of the whole test suite.

@zathras-crypto: did you give it a try in the meantime? If not, what was the reason to hesitate, and if so, what would you say would have made it easier for you to start?

CI: Bitcoin Core Boost + RegTests

@msgilligan: I was wondering: are the tests in ./src/test/test_bitcoin and ./src/qt/test_bitcoin-qt executed when testing Master Core? There is also a Java RPC test suite which can be included directly into the build process and configuration: https://github.com/TheBlueMatt/test-scripts

A build script could probably look like this:

JAVA_COMPARISON_TOOL_URL=https://github.com/TheBlueMatt/test-scripts/blob/38b490a2599d422b12d5ce8f165792f63fd8f54f/pull-tests-0f7b5d8.jar?raw=true
JAVA_COMPARISON_TOOL_HASH=ecd43b988a8b673b483e4f69f931596360a5e90fc415c75c4c259faa690df198

# setup dependencies etc.
curl -k -L -o ./share/BitcoindComparisonTool.jar $JAVA_COMPARISON_TOOL_URL
if [[ "$(sha256sum ./share/BitcoindComparisonTool.jar)" != "$JAVA_COMPARISON_TOOL_HASH  ./share/BitcoindComparisonTool.jar" ]]; then echo "Hashes don't match."; exit 1; fi

# build
./autogen.sh
./configure --with-comparison-tool=./share/BitcoindComparisonTool.jar
make

# IIRC required for the bitcoind-comparison-test
cp ./src/mastercored ./src/bitcoind
cp ./src/mastercore-cli ./src/bitcoin-cli
make check

This downloads the Java test file and builds Bitcoin/Master Core with "test awareness". make check then executes the Boost tests (test_bitcoin, test_bitcoin_qt) as well as the bitcoind-comparison-test.

Refactor RegTest tests into package groups

This Issue assumes that Issue #12 will be completed first, although the same basic reorganization could be done in sub packages of org.mastercoin.

Currently all the useful Omni Protocol RegTest tests are in the org.mastercoin.test.rpc package. I propose a new package hierarchy to improve reporting. Something like:

  • foundation.omni.test.rpc.sto - Send To Owners tests
  • foundation.omni.test.rpc.dex - Distributed Exchange tests
  • foundation.omni.test.rpc.basic - basic tests of RPC functions
  • foundation.omni.test.rpc.misc - miscellaneous tests using RPC

This will allow us to view Jenkins reports at the package level and focus on a particular category of tests like "STO".

Currently the test Specs are all lumped together in a single package, generating this report. Adding an extra level of hierarchy here should make things more logical and allow us to write documentation with links to the reports for just one package, such as "STO".

How to handle checked transaction creation via Omni Core

I'm wondering, how we should handle tests in the future, given that transaction creation for all transaction types is going to be available soon in Omni Core.

Currently, for example for the DEx spec, raw transactions are created and pushed via "sendrawtx_MP", which bypasses all checks. In contrast, the new "sendXXX" RPC calls of Omni Core check certain requirements, e.g. whether the caller has a sufficient balance, or if a property indeed exists.

Now we probably want to test both cases:

  • bypass the RPC checks, to test the actual transaction processing
  • but also test the RPC interface and it's checks

With that in mind, I created the STO tests to run two times, once using the RPC call "sendtoowners_MP", and a second time by using "sendrawtx_MP".

The STO testing is very specific though, and I'm wondering, if there should be a more generalized solution in the future, and how this could look like.

Create separate OmniJ project and JAR.

It's time to factor out core Omni Protocol classes to the OmniJ project.

We are currently doing some refactoring to prepare for this. We can use this issue for tracking the remaining work.

Some input regarding generation node, regtest, ...

Just yesterday I stumbled upon your talk about unit and regtests tests and I was quite impressed.

I noticed a few points you mentioned:

  • the use of shell scripts
  • preserving an initial state, so it's not required to regenerate the first 100+ blocks on every run
  • the initial Bitcoins are associated with "accounts"

Very recently, I think, Bitcoin Core master basically build a Python framework for regtests that takes care of a few things in particular:

  • nodes are spawned via Python and they are completely isolated
  • on the first startup 200 blocks are generated and the results are stored in ./cache
  • those results are then copied and reused to working dirs /tmp/testXXX for each node
  • after a test run, the temporary files /tmp/testXXX are deleted

My initial goal was to port the few regtests of Master Core v0.0.8, but I ended up modifiying the framework and basically hacked a Master Core regtest framework together. In the context of above:

  • I use one single node to generate blocks and properties. If another node needs some BTC, then this node is used as provider. I decided this is cleaner than letting all four nodes (as per default with the core framework) generating blocks.
  • I added a method that collects all unspent outputs and then sends them to an address to get full control and get rid of those Bitcoin "accounts".

I haven't explored bitcoin-spock yet (unfortunately!), and I'm not sure, if this is of any help, but wanted to let you know as soon as possible. :)

Regarding consensus hashes, I started mastercoin-MSC/mastercore#143 some time ago.

createFundedAddress() can result in loss of actor's BTC

Step 1) actorA is funded with 1.0 BTC via createFundedAddress()
Step 2) actorB is funded with 1.0 BTC via createFundedAddress()

In the worst the coins of A are used to "sendToAddress" of B.

How the coin selection choses the coins to send is pretty much out of scope, and I don't see a straight forward guard against this behavior.

There is the RPC call "lockunspent" which could be used to "freeze" balances during funding and transfer periods..

Allow signing Omni transactions with the keys stored in secure hardware

The RivetzSDK allows signing of Bitcoin transactions with a private key stored…

… in the Trusted Execution Environment (TEE) available in many modern devices. The TEE is a hardware environment that runs small applets outside the main OS. This protects sensitive code and data from malware or snooping with purpose-built hardware governed by an ecosystem of endorsements, beginning with the device manufacturer.

Currently, the method OmniTxBuilder.createSignedOmniTransaction() requires local ECKey objects with private keys and the method OmniTxBuilder.createOmniTransaction() creates unsigned, incomplete transactions (no change output.) To support Rivetz we'll need some form of pluggable signing.

Migrate Jenkins tests to 0.10.x Bitcoin Core / Omni Core

This involves at least the following tasks:

  1. Make sure dependencies for building Bitcoin Core and Omni Core 0.10.x versions are pre-installed or refreshed automatically (see @dexX7 's comment below)
  2. Update bitcoin-core Jenkins job to stable 0.10.x branch
  3. Temporarily disable omnicore-stable Jenkins job (previously used 0.0.8 branch, should now be 0.0.9 but uses same blockchain datadir as omnicore-stable and we can't share a blockchain datadir between 0.0.9 and 0.10.0)
  4. Update omnicore-integration Jenkins job to OmniLayer/omnicore repository and the correct integration 0.10.0 branch)

Incompatible with Bitcoin Core 0.10

We need to overthink funding and other methods once again, I believe. There are two points I noticed already:

  1. setgenerate returns no longer null, but a list of block hashes. This is an easy fix.

  2. Assuming we start fresh and generate a block with setgenerate true 1, and then using getnewaddress, then the address is returned that received the generated coins. When getnewaddress is used before generating a block, then it looks like it's another address.

Test schema of Omni Core API

Most tests we currently have focus on correct behavior and logic, but only a fraction of the information of the API is actually used.

I'm not sure, how to tackle this, but it would be great, if all fields of the results for each command are checked somewhere, similar to the ListPropertiesSpec.

MSCSendToOwnersTestPlanSpec: invalidation tests

To keep track of the progress:

We need to test two things: creating a transaction via the RPC interface with sendtoowners_MP and creating transactions as raw transactions to get around sanity checks on the RPC layer.

Print current test to the log

When executing the Gradle tasks, for example ./gradlew :omnij-rpc:regTest, then the output is rather cryptic and non-telling:

> ...
> Building 94% > :omnij-rpc:regTest > 27 tests completedfoundation.omni.test.rpc.misc.ClientConfigurationAndFundingSpec STANDARD_ERROR
> Building 94% > :omnij-rpc:regTest > 27 tests completed    Apr 14, 2015 10:33:10 PM org.slf4j.Logger$info call
> Building 94% > :omnij-rpc:regTest > 27 tests completed    INFO: Waiting for server...
> Building 94% > :omnij-rpc:regTest > 27 tests completed    Waiting for server RPC ready:
> Building 94% > :omnij-rpc:regTest > 27 tests completed
> Building 94% > :omnij-rpc:regTest > 27 tests completed    RPC Ready.
> Building 94% > :omnij-rpc:regTest > 27 tests completed8 tests completed9 tests completed30 tests completed1 tests completed2 tests completed3 tests completed4 tests completed5 tests completed6 tests completed7 tests completed8 tests completed9 tests completed40 tests completed1 tests completed
> Building 94% > :omnij-rpc:regTest > 41 tests completedfoundation.omni.test.rpc.smartproperty.SmartPropertySpec STANDARD_ERROR
> Building 94% > :omnij-rpc:regTest > 41 tests completed    Apr 14, 2015 10:33:12 PM org.slf4j.Logger$info call
> Building 94% > :omnij-rpc:regTest > 41 tests completed    INFO: Waiting for server...
> ...

It would certainly be possible to manually add a log entry for each test, but I was wondering, if there is a more Groovy way to report the current test, which is executed?

Testing STO to a large number of owners

Here's one way to test STO to hundreds or thousands of owners:

N existing owners each hold an integer number of divisible tokens and will receive .00000001 times the number of tokens they hold, so it will be easy to confirm they received the correct amount from the STO. The decimal portion of their balance after the STO will equal the integer portion, e.g. an address holding 1234.0 tokens before will now hold 1234.00001234 tokens after the STO.

  1. The sending address starts with
    • (N(N+1)/2) * .00000001 divisible SP tokens to be sent
    • N * .00000001 TMSC for the fee
  2. For i = 1 to N
    • assign i tokens to the i-th address
      • the 1st address will have 1.0, 2nd will have 2.0, ..., the Nth address will have N.0
  3. The sending address will use STO to send (N(N+1)/2) * .00000001 tokens
    • this is exactly the number of tokens owned by the sending address
  4. The expected results are:
    • the sending address has 0.0 SP tokens
    • the sending address has 0.0 TMSC
    • each owner has (i + (i * .00000001)) tokens
      • the decimal portion of their balance after the STO will equal the integer portion

MSCSendToOwnersConsensusComparisonSpec: before is null

Now this is odd behavior that I can't explain:

When running MSCSendToOwnersConsensusComparisonSpec as single test, then everything appear to be fine.*

When running all RegTest tests (parameter: "--tests com.msgilligan.bitcoin.rpc.* --tests foundation.omni.test.rpc.*"), then it fails for one address. This is the case with the master branch, but also with my feature branch. The reason is that "before" is null:

spock-consensus

This only happens sometimes and I haven't yet found a way to reproduce this issue at a 100 % rate.

*Because I'm not sure, how to reproduce this, there might not be a relation to whether I run the test individually or in a batch.

Edit: I took a look at a previous test report and in both cases it involves an address with reserved amount.

CurrencyID and Bitcoin

While I wanted to create a test of the STO invalidation for Bitcoin, I noticed this is difficult, because the range of CurrencyID starts with 1.

I'm going to change this to 0, because there are also other places where it would be of value, if the CurrencyID class could be used.

Another idea I had in mind was to create a subclass of CurrencyID, solely for Bitcoin.

Are there alternatives?

Turn off dry, verbose run of "send to many owners" tests?

I claim: if the "send to many owners" tests work for 20 owners, then they also work for 200 or 2000.

Given my extensive testing with owner numbers of 25000, and that the dry runs were intended for debug purposes, I suggest:

  1. Turn them off. Currently each "send to many owners" test is run twice: once for printing output without stopping on error, and another time to actually test results.
  2. Lower the number of owners to test.

What's your opinion on this, what are good numbers to test?

Context: foundation/omni/test/rpc/sto/MSCSendToManyOwnersSpec.groovy#L220-L236

Caution: createFundedAddress fails for very small MSC amounts

When trying to fund an address with an amount that has the last two digits set, it fails.

For example:

createFundedAddress(1.00000000, 0.00000010)
createFundedAddress(1.00000000, 0.00000001)
createFundedAddress(1.00000000, 1.00000010)
...

This is because creating 10 willets would require a BTC payment of 0.000000001 BTC.

To test STO where owners have reserved MSC

Here's a way to test STO with owner(s) who have MSC that are reserved, e.g. because they have a tx20 sale (Dex phase 1) that's active.

Assign the same number of MSC, e.g. 100.0, to 3 addresses. Then:

  1. Address 1 issues a tx20 for all 100.0 MSC, so his available balance is 0.0, reserved balance is 100.0
  2. Address 2 issues a tx20 for 10.0 MSC, so his available balance is 90.0, reserved balance is 10.0
  3. Address 3 doesn't do anything with his 100.0 MSC, so his available balance is 100.0, reserved balance is 0.0

The actor address acquires a number of MSC, e.g. 100.0 or more. Then the actor address submits an STO for all but 1.0 of his MSC. This will leave enough MSC available for the transfer fee for 100,000,000 recipient owners.

After the STO, check the balances of the 3 addresses. The (available balance + reserved balance) for all three should be exactly equal, with the available balance for each of them increased by the same amount.

createFundedAddress: creates two blocks, even when not necessary

To keep track of the progress:

As fix for #7 an intermediate address is funded with (requestedMSC / 100).setScale(8, BigDecimal.ROUND_UP) from which requestedMSC is sent to the final destination.

An intermediate address is also used in the case where (requestedMSC / 100).setScale(8, BigDecimal.ROUND_UP) == requestedMSC, which is not necessary.

Add support for generating raw Omni Protocol transactions without RPC

Create library functions that can create raw Omni Protocol transactions that can be sent over the peer-to-peer network directly (e.g. via bitcoinj)

We currently have support for creating raw Bitcoin and Omni transactions using RPC calls designed for that purpose and some work towards creating raw Bitcoin transactions with bitcoinj, but do not have the capability to generate a complete Omni transaction in a client-only configuration.

new STO negative test sending OMNI

See new row 15 in Send-To-Owners Tests

It tests the condition where an address tries to send too many MSC so that the address does not have enough available MSC remaining to pay the transfer fee for all the MSC owners that would receive a portion of the STO amount.

This should be an invalid transaction.

Eliminate SLF4J logging error message

We need to fix some SLF4J configuration (in bitcoinj?) that is causing the following error message in :

foundation.omni.test.rpc.sto.MSCSendToOwnersTestPlanRawSpec STANDARD_ERROR
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Number of coins class

There are a few situations where one needs to be careful when using BigDecimal or rather number conversion.

Here is an example:

numberofcoins

This can become an issue when using external data, such as a test plan, with indivisible amounts like 9007199254740993 in this example.

A "native" NumberOfCoins class would be helpful, with BigDecimal as underlying value type, but proper constructors, string representation and so on.

We have divisible and indivisible amounts, with different ranges each, and furthermore different use-cases, for example send_MP which consumes the number as string ("1,0", "0.00000001", "1"), as well as the creation methods, which use the number represented in units, where "1.0" is actually "100000000".

Implement CLI-style clients for Bitcoin and Omni

The current Java clients for Bitcoin and Omni are very close in name and behavior to the direct RPCs, but their names are camel-case and because Java methods can't have optional parameters there are overloaded methods with fewer parameters.

OmniCLIClient.goovy is a starting point for this work. There is a placeholder class BitcoinCLIClient.groovy as well. The idea is to use Groovy's optional arguments capability to more closely resemble the RPC methods themselves.

One of the reasons for this implementation would be writing functional tests that are very readable by people familiar with the RPCs from Bitcoin and/or Omni documentation or from using the CLI command-line tools.

Remove unused or depreciated tests

In particular I was thinking about:

  • foundation.omni.test.exodus.ExodusFundraiserSpec (duplicate of MoneyManSpec?)
  • foundation.omni.test.exodus.FaizSpec

I'm not sure about:

  • com.msgilligan.bitcoin.rpc.RegTestSendManySpec (uses account system, ignored)
  • foundation.omni.test.rpc.sto.SendToOwnersConsensusComparisonSpec (ignored)

Move msgilligan/bitcoin-spock to OmniLayer/OmniJ

The previous plan (Issue #49) was to export some of the code from this repo and import it in to the OmniJ repo, but it now seems like a better idea to move the whole repo -- mostly to preserve the issues DB which is focused around Omni protocol testing and OmniJ implementation.

At a later point we can extract some of the code back to a bitcoin-spock repo, or perhaps submit some of the Bitcoin-oriented code, eg. bitcoin-rpc module to the http://github.com/bitcoinj/bitcoinj project.

Bitcoin Core 0.10 incompability: setgenerate

The tests:

com.msgilligan.bitcoin.cli.BitcoinJCliSpec: "generate a block"
com.msgilligan.bitcoin.rpc.DynamicRPCClientSpec: "setgenerate"

... fail, when using Bitcoin Core 0.10, because setgenerate doesn't return null anymore, but a list of block hashes.

RPC connection failure handling

In BitcoinClient.java#L61-L67, when waiting for the server, and a SocketException is caught, then the stack trace should not be dumped, if certain certain expected exception messages are identified. This is pretty straight forward, to suppress errors while waiting.

However, those messages are localized, even without getLocalizedMessage(), and this causes a lot of output.

  1. Could we replace those checks and suppress all ConnectException? Or is there an alternative to using the exception messages?

  2. From an user's point of view, there are probably three relevant outcomes in this context, which should be handled differently:

  • server is not running at all, or connection information are wrong -> "please start the server with the following options ..."
  • failed authorization -> "please use this or that rpc username and password"
  • the client really waits -> error suppression

A failed authorization can be identified as JsonRPCException with httpCode = 401, but the other two are probably more tricky: a refused connection expresses itself as ConnectException, but I was unable to see "waiting" in action.

Optimize creation of funded addresses in integration tests

Without having studied the code or its behavior too closely, it seems that many more transactions and blocks are created than would be optimal to fund a given test. As a result the integration (functional) tests take too long to run.

One thing that might be helpful in optimizing this if there were a component that could keep totals and some basic statistics on these setup transactions.

Update to bitcoinj 0.12.x

See bitcoinj 0.12.1 release notes

Note: There is actually a 0.12.2 release that's not mentioned on the website or in the release notes!

The main reason is to just keep up with bug fixes from bitcoinj, but it will also give us a chance to experiment with the new Coin class and figure out how that relates to potential Omni "amount" classes and or the new JavaMoney API.

Coin selection and allowed input types

Unfortunally this is a very annoying issue, and I'm not sure where it's coming from.

The issue I'm referring to was already outlined in #44, but I assumed it somehow wasn't an issue anymore, after we restructured the funding methods:

Assuming we start fresh and generate a block with setgenerate true 1, and then using getnewaddress, then the address is returned that received the generated coins. When getnewaddress is used before generating a block, then it looks like it's another address.

The behavior might not be exactly as described, and it's unclear to me how and why it happens.

When I start the regtests locally on this machine, the first test fails:

foundation.omni.test.rpc.reorgs.SendToOwnersReorgSpec
    After invalidating a send to owners transaction, the transaction is invalid

foundation.omni.test.rpc.reorgs.SendToOwnersReorgSpec > After invalidating a send to owners transaction, the transaction is invalid FAILED
    org.spockframework.runtime.ConditionNotSatisfiedError at SendToOwnersReorgSpec.groovy:21

This is not, because the test actually fails, but due to the rejection on a protocol level, of the inputs, which were selected to create the funding transaction.

The underlying issue is that using coinbase coins for Omni transactions aren't allowed, and somehow they are used for the funding transactions nevertheless.

To solve this issue, the funding logic needs to be refined further.

Given that OmniJ now uses Omni Core 0.0.10 we may think about alternatives though. For example, it may be thinkable to have a "filterunspent" RPC in Omni Core, to filter a set of unspent outputs, and select only those, which qualify as input for Omni transactions. Such a command might be useful for anyone, who uses Omni Core to create raw Omni Core transactions.

@msgilligan: What do you think?

"Transaction too large" error

I need to dig deeper in the actual issue. Sure, we could ban all uses of where we don't construct BTC transactions ourself (special note: there is no check either, but given the funding with coinbase coins, it's less likely to hit a maximum), but this doesn't feel right.

It needs to be investigated what causes the error and how to fix it (upstream?) or how to avoid causing it.

Get "paytxfee" and "relayfee" from "getinfo"

I just noticed those values are part of the output of the RPC call "getinfo", so it's possible to get rid of the magic numbers. Won't start this today, but I assume it's basically switching from value = magicnum to value = getInfoRpcCallResult['key'].

Ability to sync blockheight among multiple servers for consensus tests.

Currently the Spock consensus tests use BitcoinClient.waitForServer() and BitcoinClient.waitForBlock() to make sure the Omni Core server being tested (usually on localhost) is warmed-up and in sync with the MainNet blockchain before the tests begin. The consensus test Specs contain an explicit comparison test of the blockheight between the local server (leftHeight) and the remote (rightHeight) which will fail if the remote server is on a different block. No attempt is made at synchronizing the local and remote servers and there are frequent failures of this blockheight comparison as well as false positives on Omni account balance comparison due to blockheight differences.

Because the Jenkins tests are typically started by commits to the omnicore or OmniJ repositories and the consensus tests run against multiple remote servers, some of which are occasionally many blocks behind the MainNet current height, I have not considered it practical to try to sync the block heights between the different servers -- at least in the context of a Jenkins job -- or even in the case of a developer running the Spock consensus tests (via command-line or IDE).

I have given serious thought to writing a continuously-running consensus-validating server that could cache consensus data from multiple Omni server implementations and do comparisons with matching blockheights whenever each server reaches a new block. This consensus-validating server is not something that is in my near term plans, however.

I created this issue to address @dexX7's request for this capability in a comment on Issue #74 and to give us a place to discuss possible solutions including the idea of a consensus-validating server.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.