Git Product home page Git Product logo

gidari's Introduction

Gidari

PkgGoDev Build Status Go Report Card Discord

Gidari is a library for batch querying data and persisting the results to local storage.

Installation

go get github.com/alpstable/gidari@latest

For information on using the CLI, see here.

Usage

Gidari supports HTTP and network socket services. There are two ways to use an HTTP service:

  1. Iterate over http.Response data, for pre-defined http.Requests.
  2. Define a writer to concurrently "write" response data for pre-defined http.Requests.

See the Go Docs or Web-to-Storage Examples section for examples.

Network sockets involves subscribing to a socket (such a web socket) and continuously iterating over the data via ReadWriter interface. The results would then be sent to a user-defined ListWriter to be stored. See the Go Docs or Network Socket Examples section for examples.

Authenticating HTTP Requests

Protocol Parameters Description
Basic Requires a username/email and password The server will authorize the request only if it can validate the user-ID and password for the protection space of the Request-URI
Coinbase Requires a key, passphrase and secrete generated in the Coinbase Exchange GUI The Coinbase API requires a CB-ACCESS-SIGN header that is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string from a timestamp + method + requestPath + body combination
Kraken Requires key and secret generated in the Kraken Pro GUI The Kraken spot API uses a custom authentication algorithm that is based on a combination of API key, nonce, and message signature. The signature is generated using a hash-based message authentication code (HMAC) with SHA-512 as the hash function.

Web-to-Storage Examples

Data Type Writer Example Description
CSV csvpb examples/csvp Use the HTTPService to write web API data to stdout as CSV
MongoDB Document mongopb exmaples/mongopb Use the HTTPService to write web API data to a MongoDB Collection

Network Socket Examples

Type Writer Example Description
WebSocket fmt.Println examples/websocket Subscribe to the Coinbase WebSocket and print the results

Contributing

Follow this guide for information on contributing.

Resources

gidari's People

Contributors

amoghrajesh avatar ash-kamrip avatar axlts avatar cecobask avatar conor-fleming avatar dependabot[bot] avatar elliotwutingfeng avatar fahimbagar avatar gauravgahlot avatar heggrik avatar janetthieu avatar jessilajos avatar loop0 avatar mccune1224 avatar notatityet avatar pankhudib avatar patrikhl avatar petersondmg avatar piotrlewandowski323 avatar pixalquarks avatar prestonvasquez avatar rangzen avatar s4kh avatar skywolf369 avatar tomoyakuroda avatar unclebinary1001 avatar utkarsh299-tech avatar vinicio avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

jessilajos hamzarahmani heggrik krishnaindani snapdgn pixalquarks rohankaran axlts rangzen pankhudib ash-kamrip s4kh elliotwutingfeng himanshu-malviya15 gauravgahlot amoghrajesh atlekbai prestonvasquez alaztetik sb-decoder siriuswapnil conor-fleming janetthieu petersondmg patrikhl nikzayn utkarsh299-tech unclebinary1001 fahimbagar

gidari's Issues

transactionExceededLifetimeLImitSeconds bug

When running a long-standing query on mongodb, you get an error like this:

FATA[0061] error upserting data: (TransactionExceededLifetimeLimitSeconds) operation was interrupted because the transaction exceeded the configured 'transactionLifetimeLimitSeconds

Here is an example configuration:

url: https://api-public.sandbox.exchange.coinbase.com
authentication:
  apiKey:
    passphrase: ***
    key:  ***
    secret: ***
dnsList:
  - mongodb://mongo1:27017/coinbasepro
rateLimit:
  burst: 5
  period: 1
truncate: true
requests:
  - endpoint: /accounts
    method: GET
  - endpoint: /currencies
    method: GET
  - endpoint: /products
    method: GET
  - endpoint: /products/BTC-USD/stats
    method: GET
    table: btc_stats
  - endpoint: /products/BTC-USD/candles
    method: GET
    timeseries:
        startName: start
        endName: end
        period: 18000
        layout: "2006-01-02T15:04:05Z"
    query:
      granularity: "60"
      start: "2018-11-09T17:11:08Z"
      end: "2019-01-12T18:11:08Z"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.