waultics / coinmarketcap-history Goto Github PK

💰Obtain USD price history data for CoinMarketCap-listed cryptocurrencies.

License: MIT License

Python 100.00%

coinmarketcap-history's Introduction

REPO HAS BEEN DEPRACATED

CoinMarketCap has updated their website and officially released their own API. As of today this repo no longer works due to changes in their UI. We apologize for the inconvenience, and hope other alternatives find their way out in the open source community.

CoinMarketCap Historical Data Retrieval

Obtain USD price history data for CoinMarketCap-listed cryptocurrencies.

Use this library as a command-line script to obtain historical cryptocurrency data on the fly, or import the cmc library to obtain cryptocurrency data within your Python program.

Installation
Usage
- Command Line
  - Usage
  - Examples
- Module
  - Usage
  - Examples
Legacy
Updates

Installation

Install coinmarketcap-history with pip:

$ pip install cmc

Usage

Command Line

The command line tool is useful for US tax reporting, among other things. If you wish to report the cost basis for a trade (or for coins acquired through mining), the IRS requires you to denominate that cost basis in USD. In the case of token-for-token trades (e.g. purchasing ETH with BTC), that requires you know the USD:BTC exchange rate at the time of the trade.

Rather than getting the exchange rate at the exact moment of your trade, which is generally not feasible, the IRS standard is to use the average of a stock's high and low price for the day. CoinMarketCap doesn't provide this figure, but this tool does.

Command Line Usage

To gather cryptocurrency data, open a terminal and run:

$ coinmarketcap <currency1> <start_date> <end_date>

where:

currency is the (case-insensitive) name of the currency / token as displayed on CoinMarketCap, with dashes in place of spaces (i.e. bitcoin).
start_date is the beginning of the range to fetch data for in yyyy-mm-dd format (i.e. 2017-10-01 for 2017 October 10th).
end_year is the end of the range to fetch data for in yyyy-mm-dd format.

Data is returned in the following tabular format:

	Bitcoin
Date	Open	High	Low	Close	Volume	Market Cap	Average
...	...	...	...	...	...	...	...

Data for multiple cryptocurrencies can be obtained with:

$ coinmarketcap <currency_1,currency_2,...,currency_n> <start_date> <end_date>

Note: currencies must be comma-separated, with no spaces in between.

Data for multiple cryptocurrencies is returned in the following tabular format:

	Bitcoin							Ripple
Date	Open	High	Low	Close	Volume	Market Cap	Average	Open	High	Low	Close	Volume	Market Cap	Average
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

The above information can also be found by running:

$ coinmarketcap -h

Write outputs to a file by running:

$ coinmarketcap <currency> <start_date> <end_date> > <output_filename>

For faster retrieval, cryptocurrency data can be gathered asynchronously by supplying the --asynchro flag:

$ coinmarketcap <currency> <start_date> <end_date> --asynchro

Running coinmarketcap asynchronously greatly reduces the amount of time required to obtain data for cryptocurrencies, especially when gathering data for multiple cryptocurrencies at a time:

Note: Asynchronous runtimes may vary according to CPU architecture. Benchmark performed with a 64-bit 6-core AMD processor.

Command Line Examples

Collecting data for one cryptocurrency:

$ coinmarketcap bitcoin 2017-01-01 2017-12-31

Collecting data for multiple cryptocurrencies:

$ coinmarketcap bitcoin,ripple,ethereum 2017-01-01 2017-12-31

Collecting data for multiple cryptocurrencies asynchronously(faster):

$ coinmarketcap bitcoin,ripple,ethereum 2017-01-01 2017-12-31 --asynchro

Writing output to a file:

$ coinmarketcap bitcoin 2017-01-01 2017-12-31 > bitcoin_history.csv

Writing output for multiple cryptocurrencies to a file:

$ python coinmarketcap.py bitcoin,ripple,ethereum 2017-01-01 2017-12-31 > bitcoin_ripple_ethereum_history.csv

Module

In addition to command-line functionality, coinmarketcap-history provides the cmc library, which allows users to obtain CoinMarketCap data from within a Python program. Historical is returned in the form of a Pandas DataFrame, which allows for easy use.

To get started with the cmc library, import it from within your program:

from cmc import coinmarketcap

Data for cryptocurrencies can be gathered using the getDataFor() method:

`getDataFor()`

params:
- cryptocurrencies: string or list
  - crypto(s) to be scraped. supply a string for a single cryptocurrency, or supply a list of strings for multiple cryptocurrencies.
- start_date: datetime object
  - datetime object for the beginning of the range to fetch data for. Must contain values for year, month, and day.
- end_date: datetime object
  - datetime object for the ebd of the range to fetch data for. Must contain values for year, month, and day.
- fields (optional): list
  - columns to obtain upon data retrieval. Options are:
    - ['Open','High','Low','Close','Volume','Market Cap','Average']
  - if fields is not specified, all fields are returned.
- async (optional): boolean
  - if True, data is gathered asynchronously. This is especially useful when gathering data for multiple cryptocurrencies at a time, which can be slow when gathered synchronously. If async is not specified, data is gathered synchronously.
- DOWNLOAD_DIR (optional): string
  - String of the relative path to save data to and load data from. If DOWNLOAD_DIR is not specified, no data is saved.
returns:
- out: Pandas DataFrame
  - A DataFrame containing historical price information for the specified cryptocurrencies through the desired daterange.

Library Examples

Gathering data for a single cryptocurrency:

from cmc import coinmarketcap
from datetime import datetime

crypto = 'bitcoin'
start_date, end_date = datetime(2017,6,1), datetime(2018,6,1)

df_bitcoin = coinmarketcap.getDataFor(crypto, start_date, end_date)

Getting data for multiple cryptocurrencies:

from cmc import coinmarketcap
from datetime import datetime

cryptos = ['bitcoin','ripple','ethereum']
start_date, end_date = datetime(2017,6,1), datetime(2018,6,1)

df_cryptos = coinmarketcap.getDataFor(cryptos, start_date, end_date)

To cache retrieved data, simply supply a string for DOWNLOAD_DIR. The string should be a relative path to the desired download directory. Data is stored in the lightweight .msg format.

Saving data and pulling cached data:

from cmc import coinmarketcap
from datetime import datetime

cryptos = ['bitcoin','ripple','ethereum']
start_date, end_date = datetime(2017,6,1), datetime(2018,6,1)

# retrieves data and stores .msg files in DOWNLOAD_DIR
df_cryptos = coinmarketcap.getDataFor(cryptos, start_date, end_date, DOWNLOAD_DIR = 'data/coinmarketcap')

# does not retreive data. Instead, pulls cached data from DOWNLOAD_DIR
df_cryptos = coinmarketcap.getDataFor(cryptos, start_date, end_date, DOWNLOAD_DIR = 'data/coinmarketcap')

Pulling specified columns only:

from cmc import coinmarketcap
from datetime import datetime

cryptos = ['bitcoin','ripple','ethereum']
start_date, end_date = datetime(2017,6,1), datetime(2018,6,1)

df_cryptos = coinmarketcap.getDataFor(cryptos, start_date, end_date, fields = ['High','Low','Close'])

Legacy

Legacy code can be obtained from the coinmarketcap-history-legacy repository found here.

Updates

2.0.0 - July 6th, 2018

as of version 2, coinmarketcap-history now offers support for Python 3. Additionally, the cmc artifact allows for global use of the coinmarketcap command line tool, as well as dedicated support for in-program operations.

2.0.2 - May 30th, 2019

as of version 2.0.2, coinmarketcap-history can now be run asynchronously inside jupyter notebooks thanks to changes in the way they handle event loops.

coinmarketcap-history's People

Contributors

Stargazers

Watchers

coinmarketcap-history's Issues

Is version 2.0.0 for Python 3 only? It doesn't work on 2.7 for me.

Hi, is new version compatible with Python 3 only? I was trying to make it work on Python 2.7 and I was not successful. I ommited pip packages which are for Python 3 only (aiohttp, aiodns) and installed cmc and remaining packages (asyncio, numpy, pandas, tqdm) via pip.

There is error with urllib library import on coinmarketcap program start:

[jakub@t1 ~]$ coinmarketcap
Traceback (most recent call last):
  File "/usr/bin/coinmarketcap", line 3, in <module>
    from cmc import coinmarketcap
  File "/usr/lib/python2.7/site-packages/cmc/__init__.py", line 1, in <module>
    from .utils import utils
  File "/usr/lib/python2.7/site-packages/cmc/utils/utils.py", line 1, in <module>
    import urllib.request as urllib
ImportError: No module named request
[jakub@t1 ~]$

According to https://stackoverflow.com/questions/24652074/importerror-no-module-named-request it's needed to import urllib.request differently for Python 2.7 and there are possible more errors.

TLDR: Do you plan to maintain code Python 2.7 or is new version supposed to be for Python 3 only? Thank you.

After saved history as 'bitcoin_history.csv' then can't read_csv

import pandas as pd
pd.read_csv(r'..\bitcoin_history.csv')

error:UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Add "require python version 3.6" to README

As described in the title, it'd be easier for others to get started err-free. Hope it helps and thanks :)

Add daily kline data

Could this program get daily kline from CMC?

local variable 'html' referenced before assignment

Traceback (most recent call last):
File "coinmarketcap_usd_history.py", line 180, in
df = main()
File "coinmarketcap_usd_history.py", line 168, in main
html = download_data(currency, start_date, end_date)
File "coinmarketcap_usd_history.py", line 91, in download_data
return html
UnboundLocalError: local variable 'html' referenced before assignment

Error when downloading crypto information.

ValueError                                Traceback (most recent call last)
<ipython-input-26-c27355b424d6> in <module>
----> 1 coinmarketcap.getDataFor('bitcoin', '2018-09-02','2019-10-01')

~\Anaconda3\lib\site-packages\cmc\coinmarketcap.py in getDataFor(cryptocurrencies, start_date, end_date, fields, asynchro, DOWNLOAD_DIR)
     58         header, rows = utils.extract_data(html)
     59         temp_df = pd.DataFrame(data = rows, columns = header)
---> 60         df = utils.processDataFrame(temp_df)
     61 
     62         if save:

~\Anaconda3\lib\site-packages\cmc\utils\utils.py in processDataFrame(df)
    135 
    136     cols = list(df.columns.values)
--> 137     cols.remove('Date')
    138     df.loc[:,'Date'] = pd.to_datetime(df.Date)
    139     for col in cols: df.loc[:,col] = df[col].apply(lambda x: float(x) if x.replace('.','').isdigit() else float(0)) # check columns to see if value is a number.

ValueError: list.remove(x): x not in list

Import error with Python 3.7

When trying to import cmc, I get the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/cmc/__init__.py", line 2
    from .async import async_utils
              ^
SyntaxError: invalid syntax

This happened with another library as well, it seems no modules can be called async in Python 3.7.

Here's some context:

ccxt/ccxt#3376

coinmarketcap bitcoin 2018-01-01 2018-01-30 not work anymore.

coinmarketcap bitcoin 2018-01-01 2018-01-30
100%|█████████████████████████████████████████
100%|█████████████████████████████████████████
███████████| 1/1 [00:33<00:00, 33.75s/it]
Traceback (most recent call last):
File "c:\users\i6\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\users\i6\appdata\local\programs\python\python37\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\i6\AppData\Local\Programs\Python\Python37\Scripts\coinmarketcap.exe_main.py", line 7, in
File "c:\users\i6\appdata\local\programs\python\python37\lib\site-packages\cmc\coinmarketcap.py", line 101, in main
df = getDataFor(cryptocurrencies, start_date, end_date, asynchro = asynchro)
File "c:\users\i6\appdata\local\programs\python\python37\lib\site-packages\cmc\coinmarketcap.py", line 60, in getDataFor
df = utils.processDataFrame(temp_df)
File "c:\users\i6\appdata\local\programs\python\python37\lib\site-packages\cmc\utils\utils.py", line 137, in processDataFrame
cols.remove('Date')
ValueError: list.remove(x): x not in list

Querying a series of coins

Hi,

Fantastic job on the script! I was wondering if there would be a way to input (or hardcode into the script) a series of coins, say 50-100. The output would then be the entire historical prices with the ticker or coin name as a new column.

Example of an ideal output (replace |'s with commas for the .csv file):

Ticker	Date	Open	High	Low	Close	Volume	MarketCap
BTC	05/29/18	7,129.460000	7,526.420000	7,090.680000	7,472.590000	5,662,660,000	121,636,000,000
BTC	05/28/18	7,371.310000	7,419.050000	7,100.890000	7,135.990000	5,040,600,000	125,748,000,000
BTC	05/27/18	7,362.080000	7,381.740000	7,270.960000	7,368.220000	4,056,520,000	125,575,000,000

ETH | 05/29/18 | 516.150000 | 572.260000 | 516.150000 | 565.390000 | 2,330,820,000 | 51,474,400,000
ETH | 05/28/18 | 573.040000 | 576.050000 | 512.550000 | 516.040000 | 2,356,900,000 | 57,136,700,000
ETH | 05/27/18 | 588.520000 | 590.330000 | 562.870000 | 572.670000 | 1,788,790,000 | 58,667,600,000

XRP | 05/29/18 | 0.556084 | 0.610749 | 0.551635 | 0.601241 | 380,744,000 | 21,792,900,000
XRP | 05/28/18 | 0.608368 | 0.609781 | 0.552299 | 0.552299 | 327,903,000 | 23,841,900,000
XRP | 05/27/18 | 0.612040 | 0.612884 | 0.603328 | 0.607881 | 186,963,000 | 23,985,800,000

Would something like this be possible given the current infrastructure of the coinmarketcap tables?

Best,

Julian

Timeout

I believe it times out on this line: page = urllib.urlopen(url) in the download_page method and then it shows the error asking to check that you have inputted the name of the crypto correctly, which I have. So perhaps we need to change that error / or add the timeout exception.

Code doesn't handle new header that contains *

Failed to parse coinmarketcap new header (open*, close**)

Header mix up

Hi,

When testing your command:

coinmarketcap bitcoin,ripple,ethereum 2017-01-01 2017-12-31 --async

I noticed the the coin headers and incorrectly assigned : Bitcoin and ripple should be switched

bitcoin_history.txt

What is the exact time of the data ?

Hello, first of all thank you for this library. I playing with CMC data and I was wondering what is the exact timestamp for the data in table at https://coinmarketcap.com/currencies/<coin>/historical-data/, do you have an idea ?

From what I see data granularity is 15 minutes on the 7d chart but none of the value for Market Cap matches the ones in the table. The closest one is at 01:00 UTC+1 but I would like to be sure, if you have insight on this one I'll appreciate. Thanks again.

Using the dataframe option in Python

It seems like there are a series of changes and updates required to get this to work in python 3. Has anyone been able to upgrade the code to get it to make a dataframe? I have had a lot of trouble and it would be great if someone could explain this or show how it works.

Error message not displaying when invalid currency specified

My Python skills are virtually non-existent so I do not have a solution to offer up at the moment. Perhaps html needs to be defined as a global variable?

Executing:
python2 coinmarketcap_usd_history.py bitcoin 2017-01-01 2017-12-31 > btc.csv

returns....

Traceback (most recent call last):
  File "coinmarketcap_test.py", line 181, in <module>
    df = main()
  File "coinmarketcap_test.py", line 169, in main
    html = download_data(currency, start_date, end_date)
  File "coinmarketcap_test.py", line 92, in download_data
    return html
UnboundLocalError: local variable 'html' referenced before assignment

Traceback (most recent call last):
File "C:\Users\shouks\MyPythonScripts\coinmarketcap_usd_history.py", line 11, in
import urllib2
ModuleNotFoundError: No module named 'urllib2'

any suggestions ?>