chain-bot / prices Goto Github PK

Create a docker file for the application to docker-ize the entire application and all it's dependencies.

https://github.com/uber-go/zap

Currently we are using the default golang logger. I need to research what other logging frameworks exist.

This will let us adjust/change the start time for different use cases (i.e. dev, prod, tests)

Currently, we convert USD to USDT. This is clumsy, especially if an exchange supports both (FTX). We should instead just store both currencies.
The original thought behind converting USD to USDT was for ease of visualization, but thinking on it further, we should not actually modify the data like this.

Remove the coinpriceapi pacakge.

Currently, coinbase is failing the tests but the CI doesn't fail on these events.

--- FAIL: TestExchangeClients (113.54s)
    --- FAIL: TestExchangeClients/TestGetAllOHLCMarketData#02 (87.63s)
        exchange_clients_****.go:60: 
            	Error Trace:	exchange_clients_****.go:60
            	Error:      	Not equal: 
            	            	expected: 12000
            	            	actual  : 11951
            	Test:       	TestExchangeClients/TestGetAllOHLCMarketData#02
            	Messages:   	COINBASE
        exchange_clients_****.go:61: 
            	Error Trace:	exchange_clients_****.go:61
            	Error:      	Not equal: 
            	            	expected: "2021-01-01 00:00:00 +0000 UTC"
            	            	actual  : "2020-12-31 23:59:00 +0000 UTC"
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1 +1 @@
            	            	-2021-01-01 00:00:00 +0000 UTC
            	            	+2020-12-31 23:59:00 +0000 UTC
            	Test:       	TestExchangeClients/TestGetAllOHLCMarketData#02
            	Messages:   	COINBASE
    exchange_clients_****.go:64: 
        	Error Trace:	exchange_clients_****.go:64
        	            				value.go:476
        	            				value.go:337
        	            				dig.go:284
        	            				dig.go:439
        	            				app.go:692
        	            				app.go:471
        	            				exchange_clients_****.go:15
        	            				exchange_clients_****.go:30
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestExchangeClients
        	Messages:   	COINBASE
FAIL

We'll likely eat into our time limits if we don't.

We need to document the steps needed for someone to be able to run this project for the first time (dependencies, running the docker images for the data services).
Rough steps:

• clone repo
• run the docker-compose to start influxdb and psql
• run psql migrations
• run app

Currently we store API-Keys like this:

type APIKeys struct {
	BinanceApiKey            string
	CoinbaseProApiKey        string
	CoinbaseProApiSecret     string
	CoinbaseProApiPassphrase string
	KrakenApiKey             string
	KucoinApiKey             string
}

With more exchanges, this will get messy quickly. We should instead make a new struct/exchange.

Currently defined as:

package impl

import (
	influxdb2 "github.com/influxdata/influxdb-client-go/v2"
	"github.com/jmoiron/sqlx"
	"github.com/mochahub/coinprice-scraper/config"
)

type RepositoryImpl struct {
	db           *sqlx.DB
	influxClient *influxdb2.Client
	influxOrg    string
	ohlcBucket   string
}

func NewRepositoryImpl(
	config *config.Secrets,
	db *sqlx.DB,
	influxClient *influxdb2.Client,
) *RepositoryImpl {
	return &RepositoryImpl{
		db:           db,
		influxClient: influxClient,
		influxOrg:    config.Org,
		ohlcBucket:   config.Bucket,
	}
}

Instead of a single repository type, maybe we should have a repository per data model. This would result in more files but seems cleaner.

They aren't being used, all data endpoints are public. No point in making the code more complicated.

Create:

• System Design Diagram
• Mention library/frameworks/design patterns used

https://docs.dydx.exchange/#general

Ref: https://github.com/golang-standards/project-layout

This will make it for others to contribute and get started on the project.

https://www.bitmex.com/app/apiOverview

func (repo *RepositoryImpl) UpsertOHLCData(
	ohlcData []*models.OHLCMarketData,
	exchange string,
	pair *models.Symbol,
) {
	writeAPI := (*repo.influxClient).WriteAPI(repo.influxOrg, repo.ohlcBucket)
	tags := map[string]string{
		"quote":    pair.NormalizedQuote,
		"exchange": exchange,
	}
	for index := range ohlcData {
		ohlc := ohlcData[index]
		fields := map[string]interface{}{
			"open":   ohlc.OpenPrice,
			"high":   ohlc.HighPrice,
			"low":    ohlc.LowPrice,
			"close":  ohlc.ClosePrice,
			"volume": ohlc.Volume,
		}
		p := influxdb2.NewPoint(
			pair.NormalizedBase,
			tags,
			fields,
			ohlc.StartTime)
		writeAPI.WritePoint(p)
	}
}

We create a new writeAPI each time we upsert OHLC data, we should instead just make this a part of the RepositoryImpl type.

We currently construct an exchange product (traded pair) from the raw base and raw quote. We should store the product pair as well as some exchanges have multiple product pairs for the same base/quoute (Kraken).

type Symbol struct {
        ProductID              string
	RawBase                string
	NormalizedBase         string
	RawQuote               string
	NormalizedQuote        string
}

Currently, we hardcode supported assets in supported.go

// TODO: Move these to a table in PSQL
func GetSupportedAssets() map[string]bool {
	return map[string]bool{
		"BTC":  true,
		"ETH":  true,
		"USDT": true,
		"USD":  true,
	}
}

By moving this to the database, we can easily create a script to update/remove supported coins (ex. top 100 coins).

We have rebranded to chain-bot, so let's change the cover image to reflect that.

Note to self, add test to ensure that the correct number of candles are being scraped.

Repository Interfaces to abstract away:

• sqlboiler
• influxdb2

Additionally, this makes testing easier by injecting mock implementations via uber.fx.

Something similar to what is discussed here: #44 (comment)

This will let us scale out clients even if they don't implement specific functions (some have klines, some only have raw trades, some only have websockets, etc).

For the purpose of this project, we don't actually need all the pricing data all the time. If the goal is to use the data for training a model, then we can store the data, train, and then flush the data on a set schedule. It should be enough to store the past 60 days worth of data, a big enough overlap of the last training time.

Pros:

• don't need two data stores
• simplifies code as we already use postgress
  Cons:
• lose out on the cool influx dashboard
• can't create a pricing server (not really a goal ATM)

The current flow to get the InfluxDB api-key is to log in via the web UI and copy and paste the key into the .env file.

The volume being scraped is the quote volume, not the base.

Candles
Supported Assets

Making more than 6 calls per second to the API, or repeatedly and needlessly fetching excessive amounts of data, can result in rate limit. Please be careful.