Git Product home page Git Product logo

crypto-liquid-market-maker's People

Contributors

dependabot[bot] avatar duelinggalois avatar wfey avatar yamanote1138 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

yamanote1138 luckyboy77 thorjeus v-pcap gorilik324

crypto-liquid-market-maker's Issues

Add ability to pass high_price to trading terms

The calculations for trade distributions was done to allow for a larger span between the market price and the high price for the strategy then the span between the low side and the market price, but the ability to add this into the gui or command line tool needs to be added.

Make trade listing asyncronous

When listing large number of trades, socket restarts due to being blocked by listing of trades. I believe if listing trades modified to an async method, this would allow the two processes to operate at the same time.

(Bug in manager.py) "raise websockets.exceptions.ConnectionClosed" needs a 'Close' object, not the code and reason directly.

As per https://websockets.readthedocs.io/en/stable/reference/exceptions.html#websockets.exceptions.ConnectionClosed The parameters need to be 'Close' object, not a number code and reason string as seen on line 145 & 146.

Crypto-Liquid-Market-Maker/trader/socket/manager.py

Lines 145 to 146 in d236d2c

raise websockets.exceptions.ConnectionClosed(
1006, "1 minute without messages")

Modify trade adjustments and handle rejected post-only

Orders are logging an error when being deleted due to never getting posted. When being posted, the status check is returning rejected because it is post-only. The cause of this is likely that when an order in the book is filled, new orders are being posted to the exchange changing the price and blocking the post-only from being listed. In the example below, it took 36 seconds from when the program was notified of a trade to post an adjustment due to the time it took to delete the trades that needed to be adjusted.

Solution:

  • Adjusts one trade at a time starting at the smallest size and working out from there. The first trade should be the only trade at risk of post-only rejection due to price movement.
  • Handle post-only rejection appropriately. When rejection occurs, the market should be polled for current lowest sell or highest buy depending on direction of adjustment in order to find the first price the trade could list as a post-only.

Summary of history:
2019-04-25 14:41:46,048 socket received matched trade
2019-04-25 14:41:57,132 book manager started cancelling orders that were getting changed
2019-04-25 14:42:22,223 book manager started posting adjusted orders

Relevant Logs:
2019-04-25 14:41:46,048 [INFO] trader.socket.reader: < 2019-04-25T20:31:22.712000Z - BTC-USD - trade_id: 3342414 - side
: sell size: 0.05000000 price: 1320.25000000
2019-04-25 14:41:56,995 [INFO] trader.sequence.book_manager: MATCHED TRADE
2019-04-25 14:41:56,996 [INFO] trader.sequence.book_manager: CHECK FULL
2019-04-25 14:41:57,132 [INFO] trader.sequence.book_manager: CANCELING ORDERS FOR ADJUSTMNET
...
2019-04-25 14:42:22,223 [INFO] trader.sequence.book_manager: SENDING ORDERS FOR ADJUSTMENT
...
2019-04-25 14:43:21,338 [ERROR] trader.exchange.trading: When deleting order recieved message: order not found
pair: BTC-USD
side: buy
price: 956.12000
size: 0.02200000
filled: 0E-8
status: error
exchange_id: 077c034e-1b8d-4f56-ba1f-d4b3d0c1ac3e
test: True
history: [{'time': 1556224951.2540877, 'status': 'rejected'}, {'time': 1556225001.3382084, 'status': 'Error deleting order'}]
responses: [{'price': '956.12000000', 'side': 'buy', 'size': '0.02200000', 'created_at': '2019-04-25T20:42:31.191382Z', 'id': '077c034e-1b8d-4f56-ba1f-d4b3d0c1ac3e', 'reject_reason': 'post only', 'post_only': True, 'filled_size': '0.00000000', 'fill_fees': '0.0000000000000000', 'time_in_force': 'GTC', 'product_id': 'BTC-USD', 'executed_value': '0.0000000000000000', 'settled': False, 'type': 'limit', 'status': 'rejected'}, 'order not found']

pipenv run python -m main

getting this error when running pipenv run python -m main

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/main.py", line 1, in
import trading_algorithm as ta
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading_algorithm.py", line 2, in
import trading, subscribe, process
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading.py", line 1, in
import authorize, requests, config
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/authorize.py", line 1, in
import json, hmac, hashlib, time, requests, base64, config
ModuleNotFoundError: No module named 'config'
Ahsens-MacBook-Pro:Crypto-Liquid-Market-Maker ahsenkhaliq$ pipenv run python -m main
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/main.py", line 1, in
import trading_algorithm as ta
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading_algorithm.py", line 2, in
import trading, subscribe, process
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/trading.py", line 1, in
import authorize, requests, config
File "/Users/ahsenkhaliq/Crypto-Liquid-Market-Maker/authorize.py", line 1, in
import json, hmac, hashlib, time, requests, base64, config
ModuleNotFoundError: No module named 'config'

also config.py.default was in folders not config.ini.default so i used cp config.py.default config.ini

UI Improvements

UI needs to be updated. There are a number of ways to get it to throw an error that need to be fixed. It would be nice to be able to use it and have it start up the strategy in a detached process. Possibly have it start in a thread, or create a bash script that starts the socket and trading as a service that communicates with the UI through an api.

Database improvements

Database could use some improvements for better record keeping.

  • Keep track of last time updated
  • Keep track of each order state change
  • Keep track of trading terms trader is using
  • Keep track of profit and loss from a tax view and value of held asset view

Log file

Log folder needs to be made or preexisting when running program

pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

While running the trader, I saw this error multiple times. The situation that leads up to it is as follows,

  1. Socket Manager receives a match for an order in the book,
  2. ThreadHandler checks the match in the Book
  3. In the BookManager, Order is found
  4. Order filled size is updated
  5. If Order filled matches order size, order is marked as filled
  6. In the ThreadHandler, Order is again checked to see if it has been marked filled
  7. during this check sqlalchemy runs the following query:
FROM orders 
WHERE orders.id = %(param_1)s]
[parameters: {'param_1': 57768}]
  1. pymysql engine recieves an IOExeption from MYSQL: (2013, 'Lost connection to MySQL server during query')

After researching this error, I cant say for cetrain what it is. The query doesnt seem to be a time out, as when I ran it from mysql, it came back without issue.

One theory is that this query is in the query_cache and upon being updated and imidiately queried again, there is some issue where the query_cache is causing problems due to the update.

To address the issue, add better error handling and attempt to disconnect and reconnect from the database and try again. Enable logging for mysql in and check trading logs for occurances to cross refrence to mysql.

2019-08-12 02:49:50,415 [INFO] trader.socket.thread_handler: Checking book for matched trade
2019-08-12 02:49:50,742 [INFO] trader.socket.thread_handler: Matched trade is in book
2019-08-12 02:49:50,773 [ERROR] trader.socket.manager: Socket had a general exception, last message received 0.36115169525146484 seconds ago
Traceback (most recent call last):
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/manager.py", line 121, in listen
    self.reader.new(message)
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 31, in new
    return function_name(msg)
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 88, in match
    self.check_book(msg)
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/reader.py", line 108, in check_book
    self.thread_handler.check_book_for_match(msg)
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/socket/thread_handler.py", line 46, in check_book_for_match
    if order.status == "filled":
  File "/home/pi/Projects/Crypto-Liquid-Market-Maker/trader/exchange/order.py", line 106, in status
    return self._status
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/attributes.py", line 276, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/attributes.py", line 677, in get
    value = state._load_expired(state, passive)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 660, in _load_expired
    self.manager.deferred_scalar_loader(self, toload)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 979, in load_scalar_attributes
    only_load_props=attribute_names,
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 208, in load_on_ident
    identity_token=identity_token,
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 282, in load_on_pk_identity
    return q.one()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3275, in one
    ret = self.one_or_none()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3244, in one_or_none
    ret = list(self)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
    return self._execute_and_instances(context)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT orders.exchange_id AS orders_exchange_id, orders.pair AS orders_pair, orders.side AS orders_side, orders.price AS orders_price, orders.size AS orders_size, orders.filled AS orders_filled, orders.status AS orders_status, orders.post_only AS orders_post_only, orders.id AS orders_id, orders.created_at AS orders_created_at, orders.book_id AS orders_book_id 
FROM orders 
WHERE orders.id = %(param_1)s]
[parameters: {'param_1': 57768}]
(Background on this error at: http://sqlalche.me/e/e3q8)
2019-08-12 02:49:51,141 [ERROR] trader.socket.manager: last message received 0.7087192535400391 seconds ago/nTraceback (most recent call last):
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_c
ontext
    cursor, statement, parameters, context
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execu
te
    cursor.execute(statement, parameters)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 732, in _read_query_re
sult
    result.read()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 657, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/pi/.local/share/virtualenvs/Crypto-Liquid-Market-Maker-z1bk84NN/lib/python3.5/site-packages/pymysql/connections.py", line 707, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')```

Unintended error upon matched trade

Error is getting logged when a trade matched. I believe this is because the trade that has been matched is never marked as such and when resting orders in response to the match, the logic attempts to cancel the matched trade thinking it is open.

Websocket

The GDAX websocket has multiple channels to subscribe to, https://docs.gdax.com/#websocket-feed It seems the best to use would be level2, match, or full. The current socket logic is in subscribe.py and can be run from bash as follows:

$python -m subscribe subscribe [product_id1],[product_id2],... [channel1],[channel2],...l

results will be printed to ./socketdata/data.txt

I did not do the best job of completely understanding the socket logic and have been reading more into async and treading to understand what my attempts are missing. It seems some of the channels with less frequent subscription messages lead the websocket to disconnect. I have seen this more with ticker and match. It is likely improperly constructed logic in subscribe.py

Once the this is corrected, messages need to be parsed and when trades in a certain strategy execute, a trading.adjust() function should result with corresponding input to executed trade. Right now adjust() will cancel all orders in the range of trades to list, this is fine unless two strategies exist in the same pair.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.