Git Product home page Git Product logo

mesh-sdk-go's Introduction

Rosetta

Rosetta SDK

Go SDK to create and interact with Rosetta API implementations

Build once. Integrate your blockchain everywhere.

Overview

The rosetta-sdk-go provides a collection of packages used for interaction with the Rosetta API specification. Much of the code in this repository is generated from the rosetta-specifications repository.

Jump to:

If you have a blockchain based on go-ethereum, we recommend that you use our rosetta-geth-sdk SDK.

Getting Started

This Golang project provides a server package that empowers a developer to write a full Rosetta Data API server by only implementing an interface. This package automatically validates client requests and calls the functions you implement with pre-parsed requests (instead of in raw JSON).

If you plan to use a language other than Golang, you will need to either codegen a server (using Swagger Codegen or OpenAPI Generator) or write one from scratch. If you choose to write an implementation in another language, we ask that you create a separate repository in an SDK-like format for all the code you generate so that other developers can use it. You can add your repository to the list in the rosetta-ecosystem repository, and in the ecosystem category of our community site. Use this repository (rosetta-sdk-go) for an example of how to generate code from this specification.

Installation Guide

This command installs the Server package.

go get github.com/coinbase/rosetta-sdk-go/server

Components

Router

The router is a Mux router that routes traffic to the correct controller.

Controller

Controllers are automatically generated code that specify an interface that a service must implement.

Services

Services are implemented by you to populate responses. These services are invoked by controllers.

Recommended Folder Structure

main.go
/services
  block_service.go
  network_service.go
  ...

Quick Examples

Complete SDK Example

This is an example of how to write an implementation using the Server package in rosetta-sdk-go.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/coinbase/rosetta-sdk-go/asserter"
	"github.com/coinbase/rosetta-sdk-go/examples/server/services"
	"github.com/coinbase/rosetta-sdk-go/server"
	"github.com/coinbase/rosetta-sdk-go/types"
)

const (
	serverPort = 8080
)

// NewBlockchainRouter creates a Mux http.Handler from a collection
// of server controllers.
func NewBlockchainRouter(
	network *types.NetworkIdentifier,
	asserter *asserter.Asserter,
) http.Handler {
	networkAPIService := services.NewNetworkAPIService(network)
	networkAPIController := server.NewNetworkAPIController(
		networkAPIService,
		asserter,
	)

	blockAPIService := services.NewBlockAPIService(network)
	blockAPIController := server.NewBlockAPIController(
		blockAPIService,
		asserter,
	)

	return server.NewRouter(networkAPIController, blockAPIController)
}

func main() {
	network := &types.NetworkIdentifier{
		Blockchain: "Rosetta",
		Network:    "Testnet",
	}

	// The asserter automatically rejects incorrectly formatted
	// requests.
	asserter, err := asserter.NewServer(
		[]string{"Transfer", "Reward"},
		false,
		[]*types.NetworkIdentifier{network},
		nil,
		false,
		"",
	)
	if err != nil {
		log.Fatal(err)
	}

	// Create the main router handler then apply the logger and Cors
	// middlewares in sequence.
	router := NewBlockchainRouter(network, asserter)
	loggedRouter := server.LoggerMiddleware(router)
	corsRouter := server.CorsMiddleware(loggedRouter)
	log.Printf("Listening on port %d\n", serverPort)
	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", serverPort), corsRouter))
}

SDK Packages

  • Types: Auto-generated Rosetta types
  • Client: Low-level communication with any Rosetta server
  • Server: Simplified Rosetta API server development
  • Asserter: Validation of Rosetta types
  • Fetcher: Simplified and validated communication with any Rosetta server
  • Parser: Tool for parsing Rosetta blocks
  • Syncer: Sync Rosetta blocks with customizable handling
  • Reconciler: Compare derived balances with node balances
  • Keys: Cryptographic operations for Rosetta-supported curves
  • Constructor: Coordinate the construction and broadcast of transactions

These packages are demoed extensively in examples and are utilized throughout the rosetta-cli tool.

Syncer

The core of any integration is syncing blocks reliably. The syncer serially processes blocks from a Data API implementation (automatically handling re-orgs) with user-defined handling logic and pluggable storage. After a block is processed, store it to a DB or send a push notification—the decision is up to you!

Parser

When reading the operations in a block, it's helpful to apply higher-level groupings to related operations, or match operations in a transaction to some set of generic descriptions (i.e., ensure there are two operations of equal but opposite amounts). The parser empowers any integrator to build abstractions on top of the building blocks that the Rosetta API exposes.

Development

Helpful commands for development:

Install Dependencies

make deps

Generate Types, Client and Server

make gen

If you want to modify client and server, please modify files under templates/client and templates/server then run make gen

Run Tests

make test

Lint the Source Code

This includes the generated code.

make lint

Code Check

make release

Testing

To validate rosetta-sdk-go, install rosetta-cli and run one of the following commands:

  • rosetta-cli check:data --configuration-file rosetta-cli-conf/testnet/config.json - This command validates that the Data API implementation is correct, using the bitcoin testnet node. It also ensures that the implementation does not miss any balance-changing operations.
  • rosetta-cli check:construction --configuration-file rosetta-cli-conf/testnet/config.json - This command validates the Construction API implementation. It also verifies transaction construction, signing, and submissions to the testnet network.
  • rosetta-cli check:data --configuration-file rosetta-cli-conf/mainnet/config.json - This command validates that the Data API implementation is correct, using the bitcoin mainnet node. It also ensures that the implementation does not miss any balance-changing operations.

Read the How to Test your Rosetta Implementation documentation for additional details.

Contributing

You may contribute to the rosetta-sdk-go project in various ways:

Read our Contributing documentation for more information.

When you've finished an implementation for a blockchain, share your work in the ecosystem category of the community site. Platforms looking for implementations for certain blockchains will be monitoring this section of the website for high-quality implementations they can use for integration. Make sure that your implementation meets the expectations of any implementation.

Documentation

You can find the Rosetta API documentation at rosetta-api.org.

Check out the Getting Started section to start diving into Rosetta.

Our documentation is divided into the following sections:

Related Projects

  • rosetta-specifications — The rosetta-specifications repository generates the SDK code in the rosetta-sdk-go repository.
  • rosetta-cli — Use the rosetta-cli tool to test your Rosetta API implementation. The tool also provides the ability to look up block contents and account balances.
  • rosetta-geth-sdk – The rosetta-geth-sdk repository provides a collection of packages used for interaction with the Rosetta API specification. The goal of this SDK is to help accelerate Rosetta API implementation on go-ethereum based chains.

Sample Implementations

To help you with examples, we developed complete Rosetta API sample implementations for Bitcoin and Ethereum. Developers of Bitcoin-like or Ethereum-like blockchains may find it easier to fork these implementation samples than to write an implementation from scratch.

You can also find community implementations for a variety of blockchains in the rosetta-ecosystem repository, and in the ecosystem category of our community site.

License

This project is available open source under the terms of the Apache 2.0 License.

© 2022 Coinbase

mesh-sdk-go's People

Contributors

ash-krishnan avatar chwevans avatar cindyxkuang avatar dependabot[bot] avatar fkneeland-figure avatar geekarthur avatar hwen-cb avatar iriszhangcb avatar jacquescaocb avatar jingweicb avatar knwr avatar lanfordcai avatar lenny-hardenberg avatar lzdl-cb avatar maebeam avatar maili-cb avatar millken avatar patrick-ogrady avatar qiwu7 avatar racbc avatar raghavsharma873 avatar renlulu avatar septerr avatar shiatcb avatar shrimalmadhur avatar varunpcb avatar wei-wang-cb avatar williamberman avatar xiaying-peng avatar yorhodes avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mesh-sdk-go's Issues

Encode nil slices as empty arrays

By default, the json package marshals zero-valued (nil) slices as null, whereas merely empty slices are marshalled as []:

var s []int
json.Marshal(s) // -> null
s = []int{}
json.Marshal(s) // -> []

If the JSON is later consumed by a Go client, this generally isn't an issue, since both null and [] will be decoded as a slice with no elements, and Go code typically doesn't distinguish between nil and empty slices. However, clients in other languages may choke on null, e.g.:

let arr = request(endpoint)
let len = arr.length // invalid if arr === null

To avoid this, zero-valued slices should be replaced with empty slices prior to encoding. There are a few ways to do this, but unfortunately none of them are trivial. Perhaps the best approach for Rosetta would be to use a JSON codegen tool such as easyjson, which supports a NilSliceAsEmpty option. This would also improve encoding/decoding performance.

Need some clarification on Specifications

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
see this

Describe the solution you'd like
A clear and concise description of what you want to happen.

Please provide guidance on the issues I have submitted here: coinbase/mesh-specifications#23

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Stop all development on this "rosetta" thing
Additional context
Add any other context or screenshots about the feature request here.

I have a few questions/requests for clarification as to your specification

Amount.yaml

Amount is described as:

Amount is some Value of a Currency

Can it be changed to:

Amount is some Unit of Account

currency has legal implications, and strictly speaking, this is not entirely accurate for EVM chains as gas is the unit of account for EVM chains (typically) whereas ethereum

Account Identifier

currently reads:

  • The account_identifier uniquely identifies an account within a network.

can it be expanded to fit a more approximate meaning of the word? As you can have a chain that uses the same derivation path in both instances but have assets in different addresses.

Also, can this be explicit in what it means as to what constitutes an "account"? Is it only EOA - Externally owned accounts only or can smart contract addresses be included with no distinction?

informative ref: eip1191.

example:

const hexChecksumChain = (name: string, coinType: number, chainId?: number) => ({
  coinType,	  coinType,
  decoder: decodeChecksummedHex,	  decoder: makeChecksummedHexDecoder(chainId),
  encoder: encodeChecksummedHex,	  encoder: makeChecksummedHexEncoder(chainId),
  name,	  name,
});	});

example ref: ensdomains/address-encoder/commit/5bf53b13fa014646ea28c9e5f937361dc9b40590

Network Identifer.yaml

The network_identifier specifies which network a particular object is associated with.

"If a blockchain has a specific chain-id or network identifier, it
should go in this field."

Can we also provide an (optional) direct link to a repository containing the genesis file? This would be the least ambiguous as to
** Informative ** ethereum-listss/chains

Additional references and sources can be made available. A Java SDK would be trivial, I just would like to not have to make awkward choices if possible, these are the ones just most apparent.

Thank you.

Shouldn't MaxTableSize be 4x ValueLogFileSize?

I saw #267 increased the table size, but it did not keep the ratio of 400% consistent. Is this OK?

According to your notes:

"we configure MaxTableSize to be 4 times the size of ValueLogFileSize (if we skewed any further to MaxTableSize, we would quickly hit the default open file limit on many OSes)."

[Fetcher] Allow for insecure HTTPS for debugging

Is your feature request related to a problem? Please describe.
Our node endpoints uses HTTPS for our rosetta endpoints. In production we have valid domain names and valid certificates, but when debugging we use self-signed certificates. The validator would throw an error when running a node with a self-signed certificate during testing/debugging.

Describe the solution you'd like
Add a flag that allows for insecure HTTPS connections when debugging.

Describe alternatives you've considered
Maintained a fork of this repo that allows for insecure connection.
We're also working on having a "local" setting for the rosetta endpoints.

Additional context
This is a screenshot of the changes I made to a fork of this repo to allow for insecure HTTPS.
Screen Shot 2020-07-08 at 3 54 40 PM

BalanceStorage.BootstrapBalances is very slow for big balances files

Describe the bug
When running rosetta-cli check:data with very big balances file (all balances at given block number to check the chain starting from the given block), the BootstrapBalances sets accounts very slowly after cca 200000 records.
Every record adding takes more then 1 second then. (see Addition context)

To Reproduce
Start rosetta-cli check:data with very big bootstrap_balances JSON file. (like 200000 records).
New Setting account log records will appear more and more slowly after few thousands.

Expected behavior
Time necessary for balances bootstrap should be linear to amount of imported accounts.

Additional context

2022/03/20 14:23:53 Setting account 0xaAf814c8B179C4e0367393F0039DE066Ad322e22 balance to 6990683807000000000 &{Symbol:FTM Decimals:18 Metadata:map[]}
2022/03/20 14:23:55 Setting account 0xf9ae52e24203a0C603aaACB2F38155624d2AA4bd balance to 6990473431700000000 &{Symbol:FTM Decimals:18 Metadata:map[]}
2022/03/20 14:23:56 Setting account 0xeE325c6CCD5B52599222eaE8A02209CEA2647799 balance to 10000000000000000000 &{Symbol:FTM Decimals:18 Metadata:map[]}
2022/03/20 14:23:58 Setting account 0xCF08c0EdCf9b26C4109146c093f5168442b6e408 balance to 9998984966146198000 &{Symbol:FTM Decimals:18 Metadata:map[]}
2022/03/20 14:23:59 Setting account 0x1fb3960117D710273B2E564e4Ed96e55DEF6f556 balance to 40000000000000000000 &{Symbol:FTM Decimals:18 Metadata:map[]}
2022/03/20 14:24:00 Setting account 0xC99ec866E51e705F90f2C99ABcd8AF6570018A62 balance to 6990683807000000000 &{Symbol:FTM Decimals:18 Metadata:map[]}

Add "repeated" requirement for parser.Descriptions

Is your feature request related to a problem? Please describe.
In some cases, it is not clear how many operations will be in a transaction but it may be possible to describe the form of each desired operation (where n could be returned that meet this form). MatchOperations does not currently support such functionality.

Describe the solution you'd like
Allow for matching n operations to a description (if enabled).

Update mesh-specifications to v1.4.15

Is your feature request related to a problem? Please describe.

mesh-sdk-go currently lacks support for Bitcoin's BIP-340 Schnorr Signature scheme, which is vital for aligning with modern Bitcoin standards and ensuring broader cryptographic compatibility.

However, mesh-specifications added support for BIP-340 on November 23rd, 2022, with coinbase/mesh-specifications v1.4.15

Describe the solution you'd like

I propose updating the mesh-specifications version used by mesh-sdk-go from v1.4.12 to the most recent release, (v1.4.15).

This will facilitate support for the BIP-340/Schnorr curve and signature types, which I propose be handled in a subsequent pull request in order to narrow this issue's scope.

Additional context

More insights into this feature can be found in the following discussions:


Additionally, my team has prepared a fork of mesh-sdk-go using spec v1.4.15 that fully implements BIP-340. We would be happy to provide this either as direct pull requests or as a reference for integrating these changes.

We look forward to your feedback and collaboration and are ready to assist in any way to expedite this process.

(CC @GeekArthur)

Return Matching Operations Instead of Matches

Is your feature request related to a problem? Please describe.
In MatchOperations, we return a slice of matching indexes of operations instead of the matching operations themselves: https://github.com/coinbase/rosetta-sdk-go/blob/25e8335096e157f5a2aa04a17b516a20a41f5e1e/parser/match_operations.go#L342

Describe the solution you'd like
It would be much easier to understand, if we just returned the operations themselves. Otherwise, the user of the package needs to frequently access operations like operations[matches[1]] (which looks like code that could panic although it shouldn't).

Automatic Code Generation

Is your feature request related to a problem? Please describe.
To update generated code on each release of the Rosetta API, it is necessary to manually download a spec file from the website and include it in the repository. This is a manual and error prone process.

Describe the solution you'd like

  • Use releases from the recently created rosetta-specifications repository to generate code instead of using an uploaded specification file.
  • Run code generation during build tests and ensure there are no changes. This would occur if a PR updated the version used to generate code without actually generating the code at that version. See rosetta-specifications for an example.

Describe alternatives you've considered
None.

Add Parser Package Examples

Is your feature request related to a problem? Please describe.
There is a lot of complicated logic in the parser package without any clear documentation or examples of how to use it.

Describe the solution you'd like

  • Provide a series of examples in the /examples folder of how to use the parser.
  • Write documentation that clearly explains how to use the parser.

incorrect test in `TestJob_ComplicatedTransfer` silently failing

Describe the bug
I've been going through the constructor code and it seems like the following action is being populated incorrectly and then silently failing to unmarshal.
https://github.com/coinbase/rosetta-sdk-go/blob/255864dbd02229a03b9d1f209ba112bd59c26211/constructor/worker/worker_test.go#L1012-L1015

To Reproduce
if you put println(types.PrettyPrintStruct(input)) after this block you'll see that it fails to unmarshal the KeyPair but never throws an error.
https://github.com/coinbase/rosetta-sdk-go/blob/255864dbd02229a03b9d1f209ba112bd59c26211/constructor/worker/worker.go#L258-L262

Expected behavior
seeing how this is meant to be a valid test, im assuming everything inside it should unmarshal correctly. and seeing how you have tests to reject invalid json, this should probably be throwing an error too.

Additional context
the actions input after populate_input mutates it:

{"account_identifier": {"address":"test"}, "keypair": {"hex_bytes":"02418b0f5ab5f72219a84ac80704504dbb3c221c2d4f0065c7e4ca93618c472ad7","curve_type":"secp256k1"}}

and the result of its unmarshalling inside the SaveAccountWorker method:

{
 "account_identifier": {
  "address": "test"
 },
 "keypair": {
  "private_key": "",
  "public_key": null
 }
}

Why there is a default statement at wrappedActiveEnqueue?

Hi. Got a question while I'm digging the sdk.

Reconciler.wrappedActiveEnqueue receives change and sends it to Reconciler.changeQueue.
Regardless of actual value of change (nil or valid balance data), it always sends it to channel so there seems no case for default.

To reduce the complexity of the source code, how about removing that default lines?
Please let me know if there is a case when that default statement can be executed.

Create an auto-generated type package

Is your feature request related to a problem? Please describe.
rosetta-sdk-go has many dependencies. Some are retracted and must not be used: e.g. GHSA-2chg-86hq-7w38 (c.f. #458). Having one go module oblige packages dependent on rosetta-sdk-go to import all dependencies.

Describe the solution you'd like
In the Cosmos SDK, we import only auto-generated types (from types and server). However, because this is one go module, we are as well importing everything else.
Creating a go.mod for all auto-generated types would let developers keep their implementation minimal.

Describe alternatives you've considered
Currently, we've forked this repo: github.com/cosmos/rosetta-sdk-go and use it for https://github.com/cosmos/cosmos-sdk/tree/main/tools/rosetta.

Error when "details" error field is populated in network/options response

Hi!

An error is thrown at line 202 in asserter/network.go when an error from the network/options endpoint response has the details field populated.

This looks like desired behavior, I'm just not sure why, because according to the documentation's response example for this endpoint the an entry in the allow.errors array can have this property populated. This is also sustained by the documentation for the Error type, present in the Allow type.

Is this a case of documentation that needs to be updated, or am I misunderstanding something?

Pass pubKey via prefunded_accounts

Is your feature request related to a problem? Please describe.

Since our signature algorithm needs sender public key to join the encoding part, so currently I am using account_identifier's metadata to let rosetta-cli aware of this info and it does work. But this will cause a side effect that we should ensure all account_identifier has this metadata, but that cannot happen in the confirmed transaction as it just doesn't contain it. That means I cannot make derive api contain pubKey too. So I would like to ask if there is another way to pass pubKey else in the json file.

 "prefunded_accounts": [
   {
    "privkey": "******",
    "account_identifier": {
     "address": "c18273C95DE42Eb979E99f8842aA0cF7343Cb42e",
     "metadata": {
      "pubKey": "03ac3d6f2fed0102253d7658de77d802ec6cd6d8f5bdd056ae342540be3d00ae3f"
     }
    },
    "curve_type": "secp256k1",
    "currency": {
       "symbol":"ZIL",
       "decimals":12
    }
   }
  ]

Support parse sub account as optional

Is your feature request related to a problem? Please describe.

As of the parser of SDK, the sub account is either required or not. There is no option to make subaccount field as optional.

Describe the solution you'd like

A flag to parse account with subaccount string as optional.

Describe alternatives you've considered

Additional context

Add Wallet Package

Is your feature request related to a problem? Please describe.
Write a wallet package that lives on top of the Construction API and keys package (#54) to create a simple interface for developers that is more familiar to existing transaction construction SDKs.

Describe the solution you'd like
A high-level view of the desired flow can be found here.

API errors should return a non 200 status code

Currently any error that is handled by the router returns 200 OK with this data
{
code: 0
message: "error msg"
retriable: false
}

however its difficult to judge whether its an error or actual response just by looking at this body (Even code 0 is an error)
It would be better if the status code is 400 or something else so we can detect if the response was error response.

Also afaik rosetta-spec doesn't mention anything about the status code so we can change the response code to 400 without breaking the spec.

Replace metadata request map type with json.RawMessage type

Is your feature request related to a problem? Please describe.
It is cumbersome to use the map type for marshalling and unmarshalling within the Celo client and server implementations because validation must be done on an enumerated list of key strings.

https://github.com/coinbase/rosetta-sdk-go/blob/2989e4b865ead7997719ce23c7d1ce3a941d6c74/types/metadata_request.go#L22

Describe the solution you'd like
I would like to be able to codegen marshallers as defined by the shape of our structs and validate metadata using these marshallers.

Describe alternatives you've considered
None.

Additional context
See https://github.com/celo-org/rosetta/blob/c51cfc731a13672a34f78cafff42978bea3574c9/service/rpc/servicer.go#L397

Unreviewed commit on master!

Unreviewed commit(s) COMPARE was merged to master by patrick-ogrady.

Comment with appropriate number of 👍 to approve.


Reviewed requirements satisfied? No
Ancestors reviewed? No

Add Net Zero Requirement for parser.Descriptions

Is your feature request related to a problem? Please describe.
It is often useful to require that a slice of operations net to zero. This would allow for a more general version of Descriptions.OppositeAmounts.

Describe the solution you'd like
Allow for specifying that some collection of matched ops have amounts that net to zero. If any op has a nil amount, this should error.

Handling account aliases on the Concordium blockchain (rosetta-cli)

Is your feature request related to a problem? Please describe.

On the Concordium blockchain we have the concept of account aliases, i.e. only the first 29 (of 32) bytes of an account address are significant. This confuses rosetta-cli check:data as it has no way of knowing if two addresses actually reference the same account. If an operation was performed for some address, it will observe that the balance of the "account" of the aliases changed for no reason.

Describe the solution you'd like

I've been trying to find a way to patch rosetta-cli (via rosetta-sdk-go) to translate all addresses into the "canonical" one. I've gotten it working by adding the following abomination of a hack into Syncer.processBlock:

@@ -187,6 +229,11 @@ func (s *Syncer) processBlock(
        }
 
        block := br.block
+       for i := range block.Transactions {
+               for j := range block.Transactions[i].Operations {
+                       canonicalizeAccountAddr(block.Transactions[i].Operations[j].Account)
+               }
+       }
        err = s.handler.BlockAdded(ctx, block)
        if err != nil {
                return err

where canonicalizeAccountAddr does a lookup using the Concordium SDK and replaces the address in-place.

I've not been able to find a proper place to implement the behavior as Syncer, StatefulSyncer, Fetcher, etc. are all structs and not interfaces. Helper is an interface type but it's not being constructed in a factory that can be overriden.

I'm not sure if monkey patching the block result is the correct solution, but it's the only one I've gotten to work (also tried BalanceStorage.AddingBlock and Parser.BalanceChanges but those seemed insufficient).

Describe alternatives you've considered

Patching all accounts in our Rosetta implementation. That would defeat the purpose of having aliases and also be prohibitively expensive.

Additional context

Any ideas of the correct way to implement this are greatly appreciated. I'll be happy to propose a PR with the necessary changes to for example support overriding the behavior of Syncer.

Make the Logger an optional middleware in routers

Is your feature request related to a problem? Please describe.
Using the current router mux exported by the server package results in endpoints always logging accesses in a specific format. When running a Rosetta server implementation backed by some other pre-existing node server that logs in a different format, logs become slightly harder to read.

Additionally, we might want to offer different forms of logging or offer the option to disable logging altogether to users.

Describe the solution you'd like
Make middlewares (including the currently existing Logger) configurable when calling server.NewRouter.

Describe alternatives you've considered
Another alternative would be disabling logging altogether in the library and having that be a responsibility of users.

rosetta-sdk-go

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Create a new release

Would it be possible to create a new release with the latest commit ? It looks like it is required for Rosetta CLI to pass the check:spec.

The commit required : e03849b

ed25519 privkey length check is failing

Describe the bug

Providing a 64 character hex encoded ed25519 privkey returns this error:

Error: unable to to sign payload for 0: invalid privkey length: expected 32 bytes but go 64: unable to sign payloads: unable to create transaction

To Reproduce

I found this using rosetta-cli check:construction with the following address/privkey pair:
address: 1d45c318b5c648fbc5adda931ea82da1
privkey: c020f245c55677cefb5259ed3c7b3baee77651acd4171c24985f5bc7bfa99485

Expected behavior

It should accept the key.

rosetta-cli check with --lookup-balance-block=false doesn't find reconciliation errors unless --interesting-accounts is given

Describe the bug

In testing of a Rosetta implementation on a blockchain that should contain reconciliation errors, the check command fails to find the errors unless --interesting-accounts is used. This seems to be caused by the reconciliation checks becoming overly delayed so as to never happen.

To Reproduce

Run rosetta-cli check --lookup-balance-by-block=false --interesting-accounts interesting.json and watch it find reconciliation errors.

Run rosetta-cli check --lookup-balance-by-block=false and let it spin forever and never find any reconciliation problems.

Expected behavior

I expect the same errors to be found, though perhaps in different amounts of time.

Bug in Asserter.ConstructionParseResponse

Describe the bug

Asserter.ConstructionParseResponse asserts that len(response.Signers) == 0 for all ConstructionParseResponse. Instead, we need to raise an error when len(response.Signers) == 0 if the tx is signed, and len(response.Signers) > 0 if the tx is unsigned.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context

Automatically Find Source of Inactive Reconciliation Errors

Is your feature request related to a problem? Please describe.
It is currently a manual and confusing process to find the source of inactive reconciliation errors. In rosetta-cli, you must create a JSON file containing all account-currency pairs that should be checked for missing ops on each block and then you must provide a start height to look for the issue.

This also requires a knowledge of the different reconciliation strategies. If we find inactive errors automatically, it will allow for much clearer logging.

Describe the solution you'd like
The reconciler should automatically find the block with missing balance changing operations.

Describe alternatives you've considered
Only viable alternative right now is to use the manual strategy but this is cumbersome and confusing.

`Type` Restriction in `OperationDescription`

Is your feature request related to a problem? Please describe.
It is not possible to require that an operation be a certain type in OperationDescription. This was done purposely as operation matching would likely be done within a parsed OperationGroup.

However, expanding use cases would make it useful to be able to restrict an arbitrary slice of operations by type.

Describe the solution you'd like
Add Type to OperationDescription.

Add `Optional` to `OperationDescription`

Is your feature request related to a problem? Please describe.
In some cases, it is useful to optimistically specify descriptions that could be met but to fallback to other logic in the case they are not. In the current implementation, there is no way to allow a match to be missing.

Describe the solution you'd like
Specify that certain matches can be missing.

Panic in Reconciler

Describe the bug

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8dcb18]

goroutine 63 [running]:
github.com/coinbase/rosetta-sdk-go/reconciler.(*Reconciler).reconcileInactiveAccounts(0xc0073e2210, 0xbe28a0, 0xc005b5b080, 0x0, 0x0)
        /home/ec2-user/go/src/github.com/coinbase/rosetta-sdk-go/reconciler/reconciler.go:644 +0xe8
github.com/coinbase/rosetta-sdk-go/reconciler.(*Reconciler).Reconcile.func2(0xc008ff6768, 0x0)
        /home/ec2-user/go/src/github.com/coinbase/rosetta-sdk-go/reconciler/reconciler.go:707 +0x3f
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0063b2270, 0xc0067e5bc0)
        /home/ec2-user/go/src/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/ec2-user/go/src/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

To Reproduce

  1. Sync some blocks to a persistent datadir using rosetta-cli check
  2. Halt syncing
  3. Restart check from the previous datadir

Rosetta Constructor DSL: error parsing a string containing a "-" character

Describe the bug

Hi! When using the Rosetta Constructor DSL for example following way:

create_account(1){
  create{
    network = {"network":"chrysalis-devnet", "blockchain":"iota"};
    key = generate_key({"curve_type": "edwards25519"});
    account = derive({
      "network_identifier": {{network}},
      "public_key": {{key.public_key}}
    });

   ...
   
  }
}

the Rosetta Construction API check fails. The - between "chrysalis-devnet" looks like a subtraction for the parser.

Is there a way around this issue? Can I encode the - somehow?

To Reproduce
Steps to reproduce the behavior:

  1. setup a *.ros file for the Construction API check
  2. choose a network name that contains a -
  3. start the Construction API check

Expected behavior

I expect that the network name is allowed to contain a - char.

Many thanks for the help.

[Fetcher] Assumes genesis block the same across Sub-Networks

Describe the bug
The Fetcher's asserter initialization function does not have an option to specify a target sub-network. Consequently, it assumes that the first network it sees is the primaryNetwork. See the following line for details:
https://github.com/coinbase/rosetta-sdk-go/blob/73d638d69870d85a0329ea396545553450171795/fetcher/fetcher.go#L143

As a result of this, I saw the following error when running the rosetta-cli check:data command on a sub-network.

2020/08/05 18:46:28 block 332604 fetch error: BlockIdentifier.Hash == ParentBlockIdentifier.Hash
2020/08/05 18:46:28 retrying fetch for block 332604 after 0.697379s

Below are the lines that seem to be throwing this error:
https://github.com/coinbase/rosetta-sdk-go/blob/73d638d69870d85a0329ea396545553450171795/asserter/block.go#L379-L381

This scenario was triggered because the sub-network in question does not have a genesis block with a block height of 0. Its genesis block occurs at block height 332604 (the block height throwing the error), but the genesis block of the zeroth sub-network (the sub-network used to initialize the asserter) has a block height of 0. The check of whether block 332604 is the genesis block therefore fails and the asserter erroneously expects that the parent hash of block 332604 be different from its block hash.

To Reproduce
Have a blockchain that returns two sub-networks for network/list.
For the first sub-network, have /network/status return a genesis block identifier with block height=0. For example,

{
  "hash": "r21zg8E011awAbEghzNBOI4RtKUZ-wHLkUwio-5dKpE",
  "index": 0
}

For the second sub-network, have /network/status return a genesis block identifier with a non-zero block height. For example,

{
  "hash": "VsDfA3fyhpKdPR-KWzBARmh5htuBLnQe7ZXcLIphSaw",
  "index": 332604
}

Then run rosetta-cli check:data on the second sub-network.

Expected behavior
When testing sub-network X, populate the Asserter's genesisBlock field with sub-network X's genesis block.

Additional context
Ran rosetta-cli version v0.4.0.
Ran rosetta-sdk-go version v0.3.3.

After some debugging modifications, I noticed the following behavior in rosetta-cli:

  • For sub-network 0 (no error thrown, genesis block height=0, correct genesis block in Asserter):
asserter: genesis block: index: 0
asserter: genesis block: hash: r21zg8E011awAbEghzNBOI4RtKUZ-wHLkUwio-5dKpE
data tester: genesis block: index: 0
data tester: genesis block: hash: r21zg8E011awAbEghzNBOI4RtKUZ-wHLkUwio-5dKpE
Initialized reconciler with 0 previously seen accounts
2020/08/12 17:34:07 Syncing 0-999
2020/08/12 17:34:08 Syncing 1000-1999
2020/08/12 17:34:09 Syncing 2000-2999
2020/08/12 17:34:09 Syncing 3000-3999
  • For sub-network 18 (the one that threw the error):
asserter: genesis block: index: 0
asserter: genesis block: hash: r21zg8E011awAbEghzNBOI4RtKUZ-wHLkUwio-5dKpE
data tester: genesis block: index: 332604
data tester: genesis block: hash: VsDfA3fyhpKdPR-KWzBARmh5htuBLnQe7ZXcLIphSaw
Initialized reconciler with 0 previously seen accounts
2020/08/12 13:57:12 Syncing 332604-333603
2020/08/12 13:57:12 block 332604 fetch error: BlockIdentifier.Hash == ParentBlockIdentifier.Hash
2020/08/12 13:57:12 retrying fetch for block 332604 after 0.477209s
2020/08/12 13:57:13 block 332604 fetch error: BlockIdentifier.Hash == ParentBlockIdentifier.Hash
2020/08/12 13:57:13 retrying fetch for block 332604 after 0.612566s
  • For sub-network 5 (no error thrown since its genesis block height=0 but incorrect genesis block in Asserter):
asserter: genesis block: index: 0
asserter: genesis block: hash: r21zg8E011awAbEghzNBOI4RtKUZ-wHLkUwio-5dKpE
data tester: genesis block: index: 0
data tester: genesis block: hash: wXSuR_3WBkrdEV4proOm1YS4pBiX4uZI_KrNQfr24ls
Initialized reconciler with 0 previously seen accounts
2020/08/12 13:58:47 Syncing 0-999
2020/08/12 13:58:48 Syncing 1000-1999
2020/08/12 13:58:48 Syncing 2000-2999

parser MatchOperations issue

Describe the bug
MatchOperations should be able to handle multiple simple transfers operations with different amount within the same transaction. However after adding a unit test scenario for that the test failed.

To Reproduce
Steps to reproduce the behavior:

  • Add following unit test in match_operation_test.go
Unit Test Example
"complex repeated op (with various amounts)": {
   		operations: []*types.Operation{
   			{
   				Account: &types.AccountIdentifier{
   					Address: "addr2",
   				},
   				Amount: &types.Amount{
   					Value: "500",
   				},
   				Type: "output",
   			},
   			{
   				Account: &types.AccountIdentifier{
   					Address: "addr3",
   				},
   				Amount: &types.Amount{
   					Value: "200",
   				},
   				Type: "output",
   			},
   			{
   				Account: &types.AccountIdentifier{
   					Address: "addr1",
   				},
   				Amount: &types.Amount{
   					Value: "-200",
   				},
   				Type: "input",
   			},
   			{
   				Account: &types.AccountIdentifier{
   					Address: "addr4",
   				},
   				Amount: &types.Amount{
   					Value: "-500",
   				},
   				Type: "input",
   			},
   			{
   				Account: &types.AccountIdentifier{
   					Address: "addr5",
   				},
   				Amount: &types.Amount{
   					Value: "-1000",
   				},
   				Type: "runoff",
   			},
   		},
   		descriptions: &Descriptions{
   			OppositeAmounts: [][]int{{0, 1}},
   			OperationDescriptions: []*OperationDescription{
   				{
   					Account: &AccountDescription{
   						Exists: true,
   					},
   					Amount: &AmountDescription{
   						Exists: true,
   						Sign:   PositiveAmountSign,
   					},
   					AllowRepeats: true,
   					Type:         "output",
   				},
   				{
   					Account: &AccountDescription{
   						Exists: true,
   					},
   					Amount: &AmountDescription{
   						Exists: true,
   						Sign:   NegativeAmountSign,
   					},
   					AllowRepeats: true,
   					Type:         "input",
   				},
   				{
   					Account: &AccountDescription{
   						Exists: true,
   					},
   					Amount: &AmountDescription{
   						Exists: true,
   						Sign:   NegativeAmountSign,
   					},
   					AllowRepeats: true,
   				},
   			},
   		},
   		matches: []*Match{
   			{
   				Operations: []*types.Operation{
   					{
   						Account: &types.AccountIdentifier{
   							Address: "addr2",
   						},
   						Amount: &types.Amount{
   							Value: "500",
   						},
   						Type: "output",
   					},
   					{
   						Account: &types.AccountIdentifier{
   							Address: "addr3",
   						},
   						Amount: &types.Amount{
   							Value: "200",
   						},
   						Type: "output",
   					},
   				},
   				Amounts: []*big.Int{
   					big.NewInt(500),
   					big.NewInt(200),
   				},
   			},
   			{
   				Operations: []*types.Operation{
   					{
   						Account: &types.AccountIdentifier{
   							Address: "addr4",
   						},
   						Amount: &types.Amount{
   							Value: "-500",
   						},
   						Type: "input",
   					},
   					{
   						Account: &types.AccountIdentifier{
   							Address: "addr1",
   						},
   						Amount: &types.Amount{
   							Value: "-200",
   						},
   						Type: "input",
   					},
   				},
   				Amounts: []*big.Int{
   					big.NewInt(-500),
   					big.NewInt(-200),
   				},
   			},
   			{
   				Operations: []*types.Operation{
   					{
   						Account: &types.AccountIdentifier{
   							Address: "addr5",
   						},
   						Amount: &types.Amount{
   							Value: "-1000",
   						},
   						Type: "runoff",
   					},
   				},
   				Amounts: []*big.Int{
   					big.NewInt(-1000),
   				},
   			},
   		},
   		err: false,
   	},

Expected behavior
This unit test should pass.

Additional context

unable to pass construction/combine... until i noticed

Describe the bug

The secp256k1 library used by rosetta-sdk-go produces a slightly different signature than the Javascript elliptic one.

To Reproduce

  • Private key: 50ff238ce1138727ccc5415cdd70d2e6a0dc7c54ad00590d771267bddc2b1b16
  • Message: a5d26ccff972f4ff7434bcc81e145fa3936480053440910fcfe9aa7ac9857d91

What do you get when you sign it? With elliptic, I get
016e484f4d7c8e1ccd198a898975b37983693cbceb9b412bd97469b6f4f437bbab07cca226861e240326d6aa51d2a66f55fc593ce7755d38508dd1cf29ab4a310e

Rosetta-cli generated instead
6e484f4d7c8e1ccd198a898975b37983693cbceb9b412bd97469b6f4f437bbab07cca226861e240326d6aa51d2a66f55fc593ce7755d38508dd1cf29ab4a310e01

It took me hours of trying all sorts of things before I noticed the bytes were slightly off.

Expected behavior

I expected the same signatures. Maybe this can go into the docs somewhere?

Allow Separate Configuration for Active/Inactive Reconciliation Concurrency

Is your feature request related to a problem? Please describe.
When reconciling a blockchain with many transactions, block syncing slows down significantly from all the reconciliation checks.

Describe the solution you'd like
It would be nice to either configure a separate concurrency value for active/inactive reconciliation (ex: 16 goroutines for active lookups and 4 goroutines for inactive lookups) or refactor reconciliation logic to always prioritize active reconciliation from a single goroutine pool.

Describe alternatives you've considered

  • Run more nodes to handle load of combined active/inactive reconciliation load. (this is expensive/not feasible on a single laptop)
  • Disable lookup-balance-by-block so that active balance reconciliation is asynchronous. (this setting disables the ability to check balances while syncing to tip)

Return Processed Data

Is your feature request related to a problem? Please describe.
After an operation is matched to a description, it is necessary to parse its amount (if it exists) and relevant metadata.

Describe the solution you'd like
Alongside matched ops, return parsed information with each op so that the client does not need to do so (ex: operation amount).

ECDSA signatures on 20 byte data

Is your feature request related to a problem? Please describe.

I am working on adding Rosetta API support for my coin, Snowblossom.
In Snowblossom, we use ECDSA signatures but the signed data is 20 bytes long.

The rosetta-cli, using rosetta-sdk-go ends up using go-ethereum for ECDSA signatures. The go-ethereum is hard coded to expect signed data to be 20 bytes. This isn't easy to change in go-ethereum, they are passing around a special data structure that is 8 64-bit integers for storing the 256-bit messages and keys. To make that change the message length would have to be plumbed through many layers.

When do a check:construction with rosetta-cli I get:

Command Failed: unable to to sign payload for 0: sign: unable to sign: invalid message length, need 32 bytes: unable to sign payloads: unable to create transaction

Describe the solution you'd like

I would like rosetta-cli to support signing 20-byte messages with ECDSA. I would hope that maybe now there is a cleaner ECDSA implementation somewhere that could be used rather than the go-ethereum one.

Describe alternatives you've considered

As an alternative, we will eventually be changing our signature method to sign 32-byte messages (as is normal) but I don't expect this will be done any time soon, as it will be consensus breaking change and we like to schedule those way out.

Another alternative would be to add RSA or DSA or DSTU4145 support to rosetta-sdk-go. I suspect adding RSA would be the easiest change. These are also supported by Snowblossom.

Additional context

Our WIP Rosetta implementation: https://github.com/snowblossomcoin/rosesnow

EncodeJSONResponse - http: superfluous response.WriteHeader call

Describe the bug
When request to rosetta-etheruem using roestta-cli check:data, it often returns

http: superfluous response.WriteHeader call from github.com/coinbase/rosetta-sdk-go/server.EncodeJSONResponse (routers.go:xx)

Related source code: EncodeJSONResponse
Related Go Forum posting: https://forum.golangbridge.org/t/http-superfluous-response-writeheader-call/24326/3

To Reproduce
Run rosetta-cli check:data with high concurrency options.

Expected behavior
superfluous should not be happened.

Support PRs from Forked Repositories

Describe the bug
Any PRs from forked repositories fail the coverage CI test: coinbase/mesh-cli#30

To Reproduce
Open a PR against the repo from a forked repository.

Expected behavior
Don't run coverage on PRs from forked repositories: coinbase/mesh-cli#31.

Additional context
We may also need to stop requiring the coveralls webhook response.

Add `EqualAddress` to `OperationDescription`

Is your feature request related to a problem? Please describe.
When parsing operations, it can be very useful to assert that some number of operations have equal addresses (typically occurs when moving funds between the base account and a subaccount).

Describe the solution you'd like
Add EqualAddress to OperaitonDescription.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.