Add Exchanges with bigger universe (Cryptopia, YoBit, Novaexchange) about catalyst HOT 16 OPEN

The area where we need the most help is locating historical data for these exchanges.

We're close to having most of the exchanges work in live and backtesting mode (currently unit testing with all of them). However, we have very little historical data for them.

I know that historical data packages are sometimes found on Reddit or other sources like this but we are not tracking them down in that way at the moment. If you can locate some of this data and package it in the attached format bittrex_bat_eth.csv.zip, we can make it available in our repository. We'd also need to make sure that this data is free of use.

CSV file specs:
Here are the proposed required headers:

• symbol: symbol of the currency pair using the Catalyst convention (e.g. eth_btc)
• last_traded: the close date of the candle, use any date format which Pandas can parse (e.g. 2017-11-25 17:00)
• open: float
• high: float
• low: float
• close: float
• volume: float

from catalyst.

@lacabra thanks for the tip, these might simplify things. Some people might still need the universe method because it also filters based on a look_back period. In my case, it backtested and looks back at the coin that existed 7 days prior the current back_test date.

However, for paper trading, no backtesting is needed but if you looking at history data you want to ensure that the coin existed 7 days ago for example.

from catalyst.

Would love to contribute in adding those exchanges. I am getting familiar with the whole library at the moment. Ready to tackle some of these features soon :)

from catalyst.

We'll announce more details this week. But we are adding these three along with other exchanges. They'll be available at least for paper trading and live trading.

For backtesting, it depends on whether their APIs all have a historical data service with minute data. If so, we'll download it and make it available for backtesting. If not, we may have to chase down that data. Help with that would be welcome.

from catalyst.

That's great. Well, I'd love to do that: become a 'data curator' as well as contribute to the Catalyst application layer. Let me know how I can get started ;)

Thanks @fredfortier

from catalyst.

@cyzanfar that would be super helpful! We've integrated the ccxt library which has an interface to multiple exchanges. Today, I'll compile a list of which exchanges have a usable historical data API. The reminder will require some type of work to collect the data in a CSV format.

from catalyst.

You got it! I'll be on the lookout for that list. Let me know I'm ready to code <3

from catalyst.

Sorry for the delay, from what I can tell so far, here are the exchanges for which we do have historical data. We may be able to hit some more but this is what I have so far.

We are also actively trying to get the Bittrex data but don't have it yet. Their historical data api is limited to recent transactions.

Any help trying to locate other historical data sources would be appreciated.

from catalyst.

Great, thank you @fredfortier. Do you need help integrating some of the exchanges listed above to Catalyst? I could take a stab at one of them.

from catalyst.

@fredfortier @cyzanfar I'm trying to get started contributing, this sounds like something that needs serious help but can't tell if it's appropriate for a starter.

Can anyone tell me where's a good place to start looking?

from catalyst.

My findings for NovaExchange so far: it probably requires a scaper tool.

POST

curl 'https://novaexchange.com/api/public/getohlc/' -H 'Host: novaexchange.com' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://novaexchange.com/market/BTC_DOGE/' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'X-CSRF-TOKEN: IjNhZDEyYjJhZDQ2ZDI4N2UzYWU2YzIyYWE0ZmViZmU2M2ZjNmY3NWYi.DRTcyg.Z-t7S0Y1KPEodm5S5-HgtrmrwiA' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: session=379f4291-5081-481e-91ff-5a69826c6711; SERVERID=wsapp0702' -H 'Connection: keep-alive' --data 'marketid=159&marketname=BTC_DOGE&candlesize=100&history=14515200'

Sample JSON output. This sample covers a large time frame but minute-resolution can be done too.

JSON output can then be mapped to the desired CSV specs.

from catalyst.

I found minute historical data up to August 2017, I don't know if that's still useful. It seems very hard to find more recent data

from catalyst.

from catalyst.

@fredfortier From my understanding, Cryptopia works using the CCXT or is it working nowadays with catalyst API. From my understanding, CCXT only works for paper trading and live trading, but that might be good enough for me for now. Would it be possible for you to guide me on a dummy/pseudo code example of how to paper trade with Cryptopia.

Basically, all I want is to iterate through all the coins in Cryptopia and purchases then similar to my simple_universe example but making a paper order.

Code:
Get all coin
Print their price
make a small order.

Let me know if you more details or help making this work.

from catalyst.

from catalyst.

@abnerA I started using your simple_universe in backtest mode, and when I switched to paper mode, I had to modify it to the following:

context.base_currency = 'btc'
context.coins = context.exchanges[context.exchange].assets
context.coins = [c for c in context.coins if c.quote_currency==context.base_currency]

The second line retrieves all the coins available on the exchange at a given time, and the third line filters that list of coins to being in the BTC base currency. I had no need for your context.universe variable. I am just using context.coins as an array of Catalyst symbol objects that you can pass to data.history() or placing orders, and that's all I need.

from catalyst.