duelinggalois / crypto-liquid-market-maker Goto Github PK
View Code? Open in Web Editor NEWDeployment of a market maker trading strategy through the GDAX.com API
License: MIT License
Deployment of a market maker trading strategy through the GDAX.com API
License: MIT License
Hi,
Please add for binance.
Thanks
The calculations for trade distributions was done to allow for a larger span between the market price and the high price for the strategy then the span between the low side and the market price, but the ability to add this into the gui or command line tool needs to be added.
option to navigate backwards through ui flow.
Should be a yes no that automatically gets the minimum size for that trading pair with an option to use a larger size.
When listing large number of trades, socket restarts due to being blocked by listing of trades. I believe if listing trades modified to an async method, this would allow the two processes to operate at the same time.
As per https://websockets.readthedocs.io/en/stable/reference/exceptions.html#websockets.exceptions.ConnectionClosed The parameters need to be 'Close' object, not a number code and reason string as seen on line 145 & 146.
Crypto-Liquid-Market-Maker/trader/socket/manager.py
Lines 145 to 146 in d236d2c
Next on the agenda after getting websocket to work would be storing trading information.
Orders are logging an error when being deleted due to never getting posted. When being posted, the status check is returning rejected because it is post-only. The cause of this is likely that when an order in the book is filled, new orders are being posted to the exchange changing the price and blocking the post-only from being listed. In the example below, it took 36 seconds from when the program was notified of a trade to post an adjustment due to the time it took to delete the trades that needed to be adjusted.
Solution:
Summary of history:
2019-04-25 14:41:46,048 socket received matched trade
2019-04-25 14:41:57,132 book manager started cancelling orders that were getting changed
2019-04-25 14:42:22,223 book manager started posting adjusted orders
Relevant Logs:
2019-04-25 14:41:46,048 [INFO] trader.socket.reader: < 2019-04-25T20:31:22.712000Z - BTC-USD - trade_id: 3342414 - side
: sell size: 0.05000000 price: 1320.25000000
2019-04-25 14:41:56,995 [INFO] trader.sequence.book_manager: MATCHED TRADE
2019-04-25 14:41:56,996 [INFO] trader.sequence.book_manager: CHECK FULL
2019-04-25 14:41:57,132 [INFO] trader.sequence.book_manager: CANCELING ORDERS FOR ADJUSTMNET
...
2019-04-25 14:42:22,223 [INFO] trader.sequence.book_manager: SENDING ORDERS FOR ADJUSTMENT
...
2019-04-25 14:43:21,338 [ERROR] trader.exchange.trading: When deleting order recieved message: order not found
pair: BTC-USD
side: buy
price: 956.12000
size: 0.02200000
filled: 0E-8
status: error
exchange_id: 077c034e-1b8d-4f56-ba1f-d4b3d0c1ac3e
test: True
history: [{'time': 1556224951.2540877, 'status': 'rejected'}, {'time': 1556225001.3382084, 'status': 'Error deleting order'}]
responses: [{'price': '956.12000000', 'side': 'buy', 'size': '0.02200000', 'created_at': '2019-04-25T20:42:31.191382Z', 'id': '077c034e-1b8d-4f56-ba1f-d4b3d0c1ac3e', 'reject_reason': 'post only', 'post_only': True, 'filled_size': '0.00000000', 'fill_fees': '0.0000000000000000', 'time_in_force': 'GTC', 'product_id': 'BTC-USD', 'executed_value': '0.0000000000000000', 'settled': False, 'type': 'limit', 'status': 'rejected'}, 'order not found']
getting this error when running pipenv run python -m main
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/main.py", line 1, in
import trading_algorithm as ta
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading_algorithm.py", line 2, in
import trading, subscribe, process
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading.py", line 1, in
import authorize, requests, config
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/authorize.py", line 1, in
import json, hmac, hashlib, time, requests, base64, config
ModuleNotFoundError: No module named 'config'
Ahsens-MacBook-Pro:Crypto-Liquid-Market-Maker ahsenkhaliq$ pipenv run python -m main
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/main.py", line 1, in
import trading_algorithm as ta
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading_algorithm.py", line 2, in
import trading, subscribe, process
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading.py", line 1, in
import authorize, requests, config
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/authorize.py", line 1, in
import json, hmac, hashlib, time, requests, base64, config
ModuleNotFoundError: No module named 'config'
also config.py.default was in folders not config.ini.default so i used cp config.py.default config.ini
Math was not done for how to calculate number of trades when the span between low price and market is greater then market and high price.
UI needs to be updated. There are a number of ways to get it to throw an error that need to be fixed. It would be nice to be able to use it and have it start up the strategy in a detached process. Possibly have it start in a thread, or create a bash script that starts the socket and trading as a service that communicates with the UI through an api.
Bad Requests were not being handled gracefully and raising an error
that would reboot the socket.
Relevant
2019-04-23 00:30:02,167 error.log
logs attached
List indices in tests of database was a poor life decision.
Database could use some improvements for better record keeping.
Log folder needs to be made or preexisting when running program
While running the trader, I saw this error multiple times. The situation that leads up to it is as follows,
FROM orders
WHERE orders.id = %(param_1)s]
[parameters: {'param_1': 57768}]
After researching this error, I cant say for cetrain what it is. The query doesnt seem to be a time out, as when I ran it from mysql, it came back without issue.
One theory is that this query is in the query_cache and upon being updated and imidiately queried again, there is some issue where the query_cache is causing problems due to the update.
To address the issue, add better error handling and attempt to disconnect and reconnect from the database and try again. Enable logging for mysql in and check trading logs for occurances to cross refrence to mysql.
2019-08-12 02:49:50,415 [INFO] trader.socket.thread_handler: Checking book for matched trade
2019-08-12 02:49:50,742 [INFO] trader.socket.thread_handler: Matched trade is in book
2019-08-12 02:49:50,773 [ERROR] trader.socket.manager: Socket had a general exception, last message received 0.36115169525146484 seconds ago
Traceback (most recent call last):
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
packet_header = self._read_bytes(4)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/manager.py", line 121, in listen
self.reader.new(message)
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 31, in new
return function_name(msg)
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 88, in match
self.check_book(msg)
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 108, in check_book
self.thread_handler.check_book_for_match(msg)
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/thread_handler.py", line 46, in check_book_for_match
if order.status == "filled":
File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/exchange/order.py", line 106, in status
return self._status
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/attributes.py", line 276, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/attributes.py", line 677, in get
value = state._load_expired(state, passive)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 660, in _load_expired
self.manager.deferred_scalar_loader(self, toload)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 979, in load_scalar_attributes
only_load_props=attribute_names,
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 208, in load_on_ident
identity_token=identity_token,
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 282, in load_on_pk_identity
return q.one()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3275, in one
ret = self.one_or_none()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3244, in one_or_none
ret = list(self)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
return self._execute_and_instances(context)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
packet_header = self._read_bytes(4)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT orders.exchange_id AS orders_exchange_id, orders.pair AS orders_pair, orders.side AS orders_side, orders.price AS orders_price, orders.size AS orders_size, orders.filled AS orders_filled, orders.status AS orders_status, orders.post_only AS orders_post_only, orders.id AS orders_id, orders.created_at AS orders_created_at, orders.book_id AS orders_book_id
FROM orders
WHERE orders.id = %(param_1)s]
[parameters: {'param_1': 57768}]
(Background on this error at: http://sqlalche.me/e/e3q8)
2019-08-12 02:49:51,141 [ERROR] trader.socket.manager: last message received 0.7087192535400391 seconds ago/nTraceback (most recent call last):
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_c
ontext
cursor, statement, parameters, context
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execu
te
cursor.execute(statement, parameters)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_re
sult
result.read()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
packet_header = self._read_bytes(4)
File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')```
Error is getting logged when a trade matched. I believe this is because the trade that has been matched is never marked as such and when resting orders in response to the match, the logic attempts to cancel the matched trade thinking it is open.
The GDAX websocket has multiple channels to subscribe to, https://docs.gdax.com/#websocket-feed It seems the best to use would be level2, match, or full. The current socket logic is in subscribe.py
and can be run from bash as follows:
$python -m subscribe subscribe [product_id1],[product_id2],... [channel1],[channel2],...l
results will be printed to ./socketdata/data.txt
I did not do the best job of completely understanding the socket logic and have been reading more into async and treading to understand what my attempts are missing. It seems some of the channels with less frequent subscription messages lead the websocket to disconnect. I have seen this more with ticker and match. It is likely improperly constructed logic in subscribe.py
Once the this is corrected, messages need to be parsed and when trades in a certain strategy execute, a trading.adjust()
function should result with corresponding input to executed trade. Right now adjust()
will cancel all orders in the range of trades to list, this is fine unless two strategies exist in the same pair.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.