forkdelta / backend-replacement Goto Github PK

View Code? Open in Web Editor NEW

57.0 57.0 97.0 303 KB

Replacement backend

License: GNU Affero General Public License v3.0

Python 99.03% Mako 0.37% DIGITAL Command Language 0.20% Dockerfile 0.40%

backend-replacement's People

Contributors

Stargazers

Watchers

Forkers

smartdex moneytrigz leanderdulac alexintosh ebcg forkee taelontome kyuubi9 bitcoinsapphire lestatmq mrmm ryxex 0xswap awesomesidney ncassidy865 yunsite altfund toktoknet treverson evsward haroldsphinx embeddedandroid daeplatform metadollar luzonica88 thuongtin lehuwu ethmall tradeether jesdakorn amisdex ethought ethdeltanew lastshodan ralyodio linmingren poa-delta arizn bitbd83 cryptarc dogecrypto top1st algobasket leafan exchy-exchange yeswici ndexvictoria gangcool praveennaidu bitcratic jcsofts hirorylabo clemleib goatcash blinkystitt legioner0 m-zeeshan-haider amisolution 1inch conrad-strughold johndpope rpmcc09 russellpmcc gcoinman jamaicoin catswap wstcnbs okayama08 arecluse combattardigrade astrixial2 shrinivasmani speedevs anthony-munozm albertchanx rubidiumchain gridtechs fiatxs devapanda mizo2000 cryptopoolspace backdoornobabies xenosnikos itvlog heyoka-448 runtimepanic

backend-replacement's Issues

Proper Documentation of the API

We need proper documentation and usage examples of the API upon completion.

Current websocket API is located at websocket_server.py

Exception in `update_order`

huey_consumer_2        | 2018-02-18T18:09:30.485956776Z huey.consumer.Worker ERROR    Unhandled exception in worker thread
huey_consumer_2        | 2018-02-18T18:09:30.485973637Z Traceback (most recent call last):
huey_consumer_2        | 2018-02-18T18:09:30.485977353Z   File "/usr/local/lib/python3.6/site-packages/huey/consumer.py", line 169, in process_task
huey_consumer_2        | 2018-02-18T18:09:30.485980061Z     task_value = self.huey.execute(task)
huey_consumer_2        | 2018-02-18T18:09:30.485982650Z   File "/usr/local/lib/python3.6/site-packages/huey/api.py", line 357, in execute
huey_consumer_2        | 2018-02-18T18:09:30.485985159Z     result = task.execute()
huey_consumer_2        | 2018-02-18T18:09:30.485987357Z   File "/usr/local/lib/python3.6/site-packages/huey/api.py", line 841, in execute
huey_consumer_2        | 2018-02-18T18:09:30.485989755Z     return func(*args, **kwargs)
huey_consumer_2        | 2018-02-18T18:09:30.485992334Z   File "/usr/src/app/app/lib/threaded_wrap_async.py", line 25, in wrapper
huey_consumer_2        | 2018-02-18T18:09:30.485994478Z     return thread_local.loop.run_until_complete(wrapped(*args, **kwargs))
huey_consumer_2        | 2018-02-18T18:09:30.485997509Z   File "/usr/local/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
huey_consumer_2        | 2018-02-18T18:09:30.486000123Z     return future.result()
huey_consumer_2        | 2018-02-18T18:09:30.486002471Z   File "/usr/src/app/app/tasks/update_order.py", line 26, in update_order_by_signature
huey_consumer_2        | 2018-02-18T18:09:30.486004699Z     await update_order(order)
huey_consumer_2        | 2018-02-18T18:09:30.486006927Z   File "/usr/src/app/app/tasks/update_order.py", line 93, in update_order
huey_consumer_2        | 2018-02-18T18:09:30.486009059Z     amount_fill = contract.call().amountFilled(*order_args)
huey_consumer_2        | 2018-02-18T18:09:30.486011231Z   File "/usr/local/lib/python3.6/site-packages/web3/contract.py", line 834, in call_contract_function
huey_consumer_2        | 2018-02-18T18:09:30.486013500Z     transaction=transaction,
huey_consumer_2        | 2018-02-18T18:09:30.486015648Z   File "/usr/local/lib/python3.6/site-packages/web3/utils/decorators.py", line 17, in _wrapper
huey_consumer_2        | 2018-02-18T18:09:30.486017818Z     return self.method(obj, *args, **kwargs)
huey_consumer_2        | 2018-02-18T18:09:30.486019858Z   File "/usr/local/lib/python3.6/site-packages/web3/contract.py", line 695, in _prepare_transaction
huey_consumer_2        | 2018-02-18T18:09:30.486022077Z     fn_kwargs,
huey_consumer_2        | 2018-02-18T18:09:30.486024081Z   File "/usr/local/lib/python3.6/site-packages/eth_utils/string.py", line 85, in inner
huey_consumer_2        | 2018-02-18T18:09:30.486035323Z     return force_obj_to_text(fn(*args, **kwargs))
huey_consumer_2        | 2018-02-18T18:09:30.486037463Z   File "/usr/local/lib/python3.6/site-packages/web3/contract.py", line 736, in _encode_transaction_data
huey_consumer_2        | 2018-02-18T18:09:30.486039668Z     return add_0x_prefix(cls._encode_abi(fn_abi, fn_arguments, fn_selector))
huey_consumer_2        | 2018-02-18T18:09:30.486041756Z   File "/usr/local/lib/python3.6/site-packages/web3/contract.py", line 707, in _encode_abi
huey_consumer_2        | 2018-02-18T18:09:30.486043922Z     ', '.join(argument_types),
huey_consumer_2        | 2018-02-18T18:09:30.486045969Z TypeError: One or more arguments could not be encoded to the necessary ABI type.  Expected types are: address, uint256, address, uint256, uint256, uint256, address, uint8, bytes32, bytes32

Add ID number to response when sending order through API

In order to know the success or failure of an order quickly could you add an ID number of the order to the response of a failure or success of sending an order?

Gracefully handle missing token contract functions that are optional

Backend assumes that fuctions like decimals are always implemented in token contracts. This is not the case and some functions are optional per: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-20.md

We should handle all contract calls for these optional functions gracefully.

Example calls stack on exception:

  File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 266, in _trigger_event
    ret = await self.handlers[event](*args)
  File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 364, in _handle_eio_message
    await self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
  File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 294, in _handle_event
    await self._handle_event_internal(self, sid, data, namespace, id)
  File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 297, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
  File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 325, in _trigger_event
    ret = await self.handlers[namespace][event](*args)
  File "/usr/src/app/app/services/websocket_server.py", line 372, in get_market
    "sells": [format_order(order) for order in orders_sells]
  File "/usr/src/app/app/services/websocket_server.py", line 372, in <listcomp>
    "sells": [format_order(order) for order in orders_sells]
  File "/usr/src/app/app/services/websocket_server.py", line 277, in format_order
    price = base_contract.denormalize_value(record["amount_get"]) / coin_contract.denormalize_value(record["amount_give"])
  File "/usr/src/app/app/src/erc20_token.py", line 31, in denormalize_value
    return value * Decimal(10.0 ** -self.decimals)
  File "/usr/src/app/app/src/erc20_token.py", line 42, in decimals
    cache[self.addr] = self._call_decimals()
  File "/usr/src/app/app/src/erc20_token.py", line 53, in _call_decimals
    raise ValueError(error_msg)
ValueError: Contract 0xb15fe5a123e647ba594cea7a1e648646f95eb4aa does not support method`decimals()', returned '0x'

connecting to socket.io api

Hello,

as I understand you also provided your own socket.io API. Is this correct? Is there a way to connect to it?

The default APIs in etherdelta where
https://socket01.etherdelta.com, https://socket02.etherdelta.com,...

I tried the API under
http://api.forkdelta.com

but it doesn't work.

Thanks for any help or hint.

Incorrect ticker data

I spy a bug.
Ticker data shows bid as 5000 ETH / 1 KIN for this: https://forkdelta.github.io/#!/trade/KIN-ETH

That happened, but it was traded yesterday and there have been plenty of trades since.

Initial production rollout checklist

Start looking into what it would take to integrate NEO as another base currency

This may be far down the road, but I wanted to keep track of it here.

placeing orders: invalid signature

Hello,

I think that might be an Issue but not 100% sure. When placing orders I'm getting sometimes the error:
[422,"Cannot post order: invalid signature"]

Here is one example: this one is fine
{"nonce": 1169671266, "contractAddr": "0x8d12A197cB00D4747a1fe03395095ce2A5CC6819", "s": "0x5f74de64efee8725cdaaf68373bc4e40c7e6755519bdb122d026e4866da4c6b8", "v": 27, "user": "0xaA8d1EE37F6b18f754A55f1d5f18B4C88af2AaCc", "amountGive": 2000000000000000000, "r": "0xb55837fa1b04baa90f48b9cb3a819715a16fe2a405c456cc10d7ce766e602ddb", "tokenGive": "0xE41d2489571d322189246DaFA5ebDe1F4699F498", "tokenGet": "0x0000000000000000000000000000000000000000", "amountGet": 3975975974000000, "expires": 5109254}

Here s the second order. Created with the same method. Here I get the error 402
{"nonce": 6426437636, "contractAddr": "0x8d12A197cB00D4747a1fe03395095ce2A5CC6819", "s": "0x5e713c0fa7265f91d5c56f3f485ff72a1985cae2eabcdc4b55f6624fe5dac2c8", "v": 28, "user": "0xaA8d1EE37F6b18f754A55f1d5f18B4C88af2AaCc", "amountGive": 2000000000000000, "r": "0x37a60b1533c9c6e0aae04b39c7732425e7ffe8eafddbc38d25cfde38d3781ae7", "tokenGive": "0x0000000000000000000000000000000000000000", "tokenGet": "0xE41d2489571d322189246DaFA5ebDe1F4699F498", "amountGet": 1546790409899458500, "expires": 5109256}

What I did so far: check that the hex method I am using before signing the message is correct. It gives exactly the values as make_order_hash in this project. After that the message is signed using
self.account.sign(message_hexstr=hashhex) (web3 v4.0.0-b.9)

Any Idea what that could be? Maybe an error in the order_signature_valid "black magic" ritual?

Add wrappers for different languages for the API

We can rely on the community for this.

Someone made a PHP wrapper: https://github.com/chetcuti/forkdelta-api-wrapper

NOTE: I have not personally vetted this wrapper.

We need some sort of error tracking system.

We need some sort of error tracking system so that we can monitor any errors, ED downtime, order malfunctions, etc.

Order validation: At least one side should be 0x0 (eth)

Otherwise the order should not go into the DB.

Order creation failure broadcast publicly

Steps to reproduce:

Connect to separate clients to the websocket server (eg., open two tabs in the browser)
Attempt to create an order that fails some sort of validation (eg., see #29) through one of them.

Expected: Server replies with a messageResult error to the client that sent the erroneous order.
Observed: Server sends messageResult with the error to all clients.

Submitting an expired order causes an exception and results in the user not getting a repsonse

Websocket client expects a response (via messageResult) when submitting an order (via message). When you submit an order that has already expired, you do not receive a response. Upon further investigation, turns out there's an exception:

websocket_server_3     | 2018-02-16T21:13:17.970394077Z websocket_server WARNING  Order rejected: Cannot post order because it has already expired: {'blockNumber': 5102798, 'expires': 5102211, 'date': datetime.datetime(2018, 2, 16, 21, 13, 17, 970106)}
websocket_server_3     | 2018-02-16T21:13:17.970972007Z asyncio      ERROR    Task exception was never retrieved
websocket_server_3     | 2018-02-16T21:13:17.970988368Z future: <Task finished coro=<AsyncServer._emit_internal() done, defined at /usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py:229> exception=TypeError("Object of type 'datetime' is not JSON serializable",)>
websocket_server_3     | 2018-02-16T21:13:17.971038067Z TypeError: Object of type 'datetime' is not JSON serializable

Handle ED connection closing errors more gracefully

etherdelta_observer_1  | Connection closed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason
etherdelta_observer_1  | Task exception was never retrieved
etherdelta_observer_1  | future: <Task finished coro=<SocketIOClient.engineio_ping() done, defined at /usr/src/app/socketio_client.py:89> exception=ConnectionClosed('WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason',)>
etherdelta_observer_1  | Traceback (most recent call last):
etherdelta_observer_1  |   File "/usr/src/app/socketio_client.py", line 96, in engineio_ping
etherdelta_observer_1  |     await self.ws.send(ENGINEIO_PING)
etherdelta_observer_1  |   File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 334, in send
etherdelta_observer_1  |     yield from self.ensure_open()
etherdelta_observer_1  |   File "/usr/local/lib/python3.6/site-packages/websockets/protocol.py", line 470, in ensure_open
etherdelta_observer_1  |     raise ConnectionClosed(self.close_code, self.close_reason)
etherdelta_observer_1  | websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

"it's ED closing connection unexpectedly on us after we ping it."
"the connection will be restarted, but we should probably handle that error a bit more gracefully"

Pull tokens from live tokens.json

Backend currently uses a tokens.json file stored here. We should be pulling that from the live tokens.json as that is the most up to date.

HTTPConnectionPool(host='138.197.160.134', port=8545): Read timed out

ticker_1               | services.ticker DEBUG    {'token_address': '0x55c2a0c171d920843560594de3d6eecc09efc098', 'quote_volume': Decimal('0'), 'base_volume': Decimal('0'), 'last': None, 'ask': Decimal('0.000868'), 'bid': Decimal('0.000010002')}
ticker_1               | Traceback (most recent call last):
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
ticker_1               |     six.raise_from(e, None)
ticker_1               |   File "<string>", line 2, in raise_from
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
ticker_1               |     httplib_response = conn.getresponse()
ticker_1               |   File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
ticker_1               |     response.begin()
ticker_1               |   File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
ticker_1               |     version, status, reason = self._read_status()
ticker_1               |   File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
ticker_1               |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
ticker_1               |   File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
ticker_1               |     return self._sock.recv_into(b)
ticker_1               | socket.timeout: timed out
ticker_1               | 
ticker_1               | During handling of the above exception, another exception occurred:
ticker_1               | 
ticker_1               | Traceback (most recent call last):
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
ticker_1               |     timeout=timeout
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
ticker_1               |     _stacktrace=sys.exc_info()[2])
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
ticker_1               |     raise six.reraise(type(error), error, _stacktrace)
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
ticker_1               |     raise value
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
ticker_1               |     chunked=chunked)
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
ticker_1               |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
ticker_1               |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
ticker_1               |     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
ticker_1               | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='138.197.160.134', port=8545): Read timed out. (read timeout=10)

Not seeing events 'orders' and 'trades'

When connected to the websocket api, I can request market data but do not see new orders and trades being emitted.

Add ticker to the API

Too many decimals in API responses

Expected: A reasonable number of decimal places for float values.
Observed: Utter madness.

{
	"id": "0x97b79f47473e6b68516c984ab2de4054ef2582adf38aa366d1513b6b6c21714b_sell",
	"user": "0xc73604d0514ac01e63513d07601e7e98f2f117b4",
	"tokenGet": "0x0000000000000000000000000000000000000000",
	"amountGet": "2.00e+22",
	"tokenGive": "0xd4fa1460f537bb9085d22c7bccb5dd450ef28e3a",
	"amountGive": "1e+8",
	"expires": "5105269",
	"nonce": "733997509",
	"availableVolume": "1e+8",
	"ethAvailableVolume": "1.0000000000000000209225608301284726753266340892878361046314239501953125",
	"availableVolumeBase": "2.00e+22",
	"ethAvailableVolumeBase": "20000.000000000001430848481092438490170561123698464954523412728804032667540013790130615234375000",
	"amount": "1e+8",
	"amountFilled": "0",
	"price": "20000.0000000000010123972644898690154820850913677594522122677091193265910041380238173236084484497591809775743356864299612716916438423333750459851098292956622954777595992929496691428126238194672813683532392374144035467070912806853187505836593308656649117339275453367627182356519758479869264882765100097939088468490866440732567026286411800662521663033681010552508571602373973818562449257340260893973314436215128040640055868046470247375279548919891031850819054271812657289404126706255227332385821622424600036494436073624343655131356375408455509268762930691074648079535591348465941827953843168799252633246515511329153578515946477076280086918871265500802154000695035803744390622509509152272872747162202879306602826433708708864386296342016781915006233269144242547418444636985819315622636613317554893304766030455675377145578696160938459071457157497996611529670987242607269394306887656927999719503442150242780904615620395498249284070611317741330679831977446847311950267971019201433059016368447734522802894731",
	"state": "OPEN",
	"v": 28,
	"r": "0x74700930e094e3efbd176aef20c65304604c4d98ecd7c1bfbc1dbe5c33887b45",
	"s": "0x691d5518c84443081ecad18d4782f0eede32a53babf35df2411aeb001f82986f",
	"date": "2018-02-15T15:05:08.275767",
	"updated": "2018-02-15T15:05:08.275767"
}

Build an investigative tool for tracking transactions.

There have been several tokens with artificially inflated volume as 1 user trades large ETH amounts between two addresses they own.

We need some sort of tool for tracking transactions and alerting on any suspicious activity.

Stream updates

Emit funds, orders, and trades events to all connected clients in websocket_server.py.

Token Symbol to Rest API returnTicker

At the moment there is no symbols associated with tokens. Could you please add another field to the result from https://api.forkdelta.com/returnTicker

This is example, last member of this JSON object could be SYMBOL with its Token symbol.

{"ETH_0x0b76544": {"tokenAddr": "0x0b76544f6c413a555f309bf76260d1e02377c02a", "quoteVolume": "15438.26869", "baseVolume": "5.710009126", "last": "0.000385", "bid": "0.0004", "ask": "0.000385", "updated": "2018-02-21T08:57:05.336152", "symbol" : "XXX"}

Record order cancellations

We already listen to Cancel events. A Cancel event contains everything needed to compute order signature. When we receive a Cancel event, we should upsert a record with corresponding arguments, as well as amount_fill equal to amountGet and order state CANCELED.

Create a proper README

Current README says nothing.

We need a summary of the backend, link to the flowchart, some usage examples, links to other docs (like API docs), etc.

API stops sending orders, trades and funds events

Every two or three days, ForkDelta API stops sending API events: orders, trades and funds as defined in https://github.com/forkdelta/backend-replacement/tree/master/docs/api#events.

This causes frontend or any other app using the API to have an outdated orderbook.

The only solution seems to be to restart the websocket server.

All other functionality of the API keeps working fine. Calls to message and getMarket work ok.

Here is a preview of the frontend working ok and receiving events.

When API starts failing, all those events are not received.

Orders with amount_give/amount_get=0

They happen. Investigate.

Update available order volume / order status

Whenever a Trade or a Cancel event is received from the blockchain, update the status and the available volume of the corresponding order.

When dealing with a Trade event, the corresponding order can not be found directly. Instead, it must be looked up from user and tokenGet/tokenGive arguments of the event. You can then construct the order signature / hash sha256(contractAddr, tokenGet, amountGet, tokenGive, amountGive, expires, nonce) and query orderFills[user][hash] on the contract.

Cancel events should provide enough information to find the order directly.

Contributors: Please describe a plan of action and assign the issue to yourself.

ropsten testnet error

In main ethereum net, main.json works well.
But when I test backend on testnet ropsten, I do the following:

in https://github.com/forkdelta/forkdelta.github.io change testnet.json to main.json
app/config.py in backend-replacement:
FRONTEND_CONFIG_FILE="./app/testnet.json"
ED_CONTRACT_ADDR = '0x228344536a03C0910Fb8Be9C2755c1a0bA6f89E1
after that, I got the following error in chrome console, spending long time on it but with no success:
wss://192.168.12.16:8080/socket.io/?EIO=3&transport=websocket' failed: WebSocket opening handshake timed out WS.doOpen @ main.js:33881

While websocket server return:
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=utf-8
Content-Length: 11
Date: Tue, 06 Mar 2018 01:09:24 GMT
Server: Python/3.6 aiohttp/2.3.7

getMarket message with no token specified causes an exception

Steps to reproduce:

Use a client to send FD WS API a getMarket message with no token specified.

Expected: Server replied with an empty or generic market event.
Observed: No response; exception thrown.

Relevant logs:

websocket_server_1     | 2018-02-14T21:22:15.418806190Z Traceback (most recent call last):
websocket_server_1     | 2018-02-14T21:22:15.418809016Z   File "/usr/local/lib/python3.6/site-packages/engineio/asyncio_server.py", line 266, in _trigger_event
websocket_server_1     | 2018-02-14T21:22:15.418811654Z     ret = await self.handlers[event](*args)
websocket_server_1     | 2018-02-14T21:22:15.418823629Z   File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 364, in _handle_eio_message
websocket_server_1     | 2018-02-14T21:22:15.418826453Z     await self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
websocket_server_1     | 2018-02-14T21:22:15.418828694Z   File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 294, in _handle_event
websocket_server_1     | 2018-02-14T21:22:15.418831039Z     await self._handle_event_internal(self, sid, data, namespace, id)
websocket_server_1     | 2018-02-14T21:22:15.418833395Z   File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 297, in _handle_event_internal
websocket_server_1     | 2018-02-14T21:22:15.418835719Z     r = await server._trigger_event(data[0], namespace, sid, *data[1:])
websocket_server_1     | 2018-02-14T21:22:15.418837877Z   File "/usr/local/lib/python3.6/site-packages/socketio/asyncio_server.py", line 325, in _trigger_event
websocket_server_1     | 2018-02-14T21:22:15.418840075Z     ret = await self.handlers[namespace][event](*args)
websocket_server_1     | 2018-02-14T21:22:15.418842214Z   File "/usr/src/app/app/services/websocket_server.py", line 241, in get_market
websocket_server_1     | 2018-02-14T21:22:15.418844449Z     "buys": [format_order(order) for order in orders_buys],
websocket_server_1     | 2018-02-14T21:22:15.418846717Z UnboundLocalError: local variable 'orders_buys' referenced before assignment

Compute ticker for unlisted tokens

The set of tokens for which ticker is computed should be the intersection of the listed tokens and the tokens in active orders and trades.

`myOrders` missing from market event

If the user argument is present, myOrders will contain orders filtered by the user's address. Otherwise, it will be absent.

— https://github.com/etherdelta/etherdelta.github.io/blob/master/docs/API.md#orders-myorders

Cancels/trades missed by the contract observer result in outdated orders in the book.

Currently freeatnet sorts this out manually (after a downtime) by running the script: https://github.com/forkdelta/backend-replacement/blob/master/app/src/contract_events_backfill.py

This should either be done automatically or freeatnet suggested:

"I would implement periodic orderbook refreshes, on per token basis or just queueing older orders to be refreshed automatically whenever they're returned to people."

Document order signature generation / validation

See #25 (comment)

Is it possible to add orders to a testnet backend

I have a dapp that needs to use forkdelta, i'm wondering if it's possible to get the api set up to accept test orders so that i can make sure my system fully works before testing with real tokens.

Add CORS headers to /returnTicker endpoint

Keep track of availableVolume of an Order

This is a follow-up on #1.

In #1, I added tracking of orderFills, which allowed us to get the unfilled amount of the order. However, the amount remaining may be unfillable if the market maker withdraws funding for the order. We need to track that and temporarily remove the order from the returned orderbook.

test backend-replacement, some error occur

Thank you for your great work. When I run the project with command:

$ docker-compose up
the following exception occur, can you give me some hint to solve it, thanks!
huey_consumer_1        | App.App      INFO     Token list refreshed: 823 tokens.
huey_consumer_1        | App.App      INFO     Token list refreshed: 823 tokens.
huey_consumer_1        | App.App      INFO     Token list refreshed: 823 tokens.
huey_consumer_1        | huey.consumer.Worker ERROR    Unhandled exception in worker thread
huey_consumer_1        | Traceback (most recent call last):
huey_consumer_1        |   File "/usr/local/lib/python3.6/site-packages/huey/consumer.py", line 169, in process_task
huey_consumer_1        |     task_value = self.huey.execute(task)
huey_consumer_1        |   File "/usr/local/lib/python3.6/site-packages/huey/api.py", line 357, in execute
huey_consumer_1        |     result = task.execute()
huey_consumer_1        |   File "/usr/local/lib/python3.6/site-packages/huey/api.py", line 841, in execute
huey_consumer_1        |     return func(*args, **kwargs)
huey_consumer_1        |   File "/usr/src/app/app/lib/threaded_wrap_async.py", line 25, in wrapper
huey_consumer_1        |     return thread_local.loop.run_until_complete(wrapped(*args, **kwargs))
huey_consumer_1        |   File "/usr/local/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
huey_consumer_1        |     return future.result()
huey_consumer_1        |   File "/usr/src/app/app/tasks/update_order.py", line 26, in update_order_by_signature
huey_consumer_1        |     await update_order(order)
huey_consumer_1        |   File "/usr/src/app/app/tasks/update_order.py", line 90, in update_order
huey_consumer_1        |     order_args = order_as_args(order)
huey_consumer_1        |   File "/usr/src/app/app/tasks/update_order.py", line 114, in order_as_args
huey_consumer_1        |     order.get("v", 0),
huey_consumer_1        | AttributeError: 'asyncpg.Record' object has no attribute 'get'
huey_consumer_1        | tasks.update_order DEBUG    Update order by signature=0x538f53b35ba1696dfd6c2d221c5e03424fb815b4b2ba8b54cd44ec055d5c4d09

Updates streaming fails without recovery

websocket_server_3     | 2018-02-18T01:06:25.110355343Z asyncio      ERROR    Task exception was never retrieved
websocket_server_3     | 2018-02-18T01:06:25.110380617Z future: <Task finished coro=<stream_updates() done, defined at /usr/src/app/app/services/websocket_server.py:364> exception=InvalidOperation([<class 'decimal.DivisionUndefined'>],)>
websocket_server_3     | 2018-02-18T01:06:25.110384568Z Traceback (most recent call last):
websocket_server_3     | 2018-02-18T01:06:25.110386780Z   File "/usr/src/app/app/services/websocket_server.py", line 374, in stream_updates
websocket_server_3     | 2018-02-18T01:06:25.110389149Z     "buys": [format_order(order) for order in orders_buys],
websocket_server_3     | 2018-02-18T01:06:25.110391362Z   File "/usr/src/app/app/services/websocket_server.py", line 374, in <listcomp>
websocket_server_3     | 2018-02-18T01:06:25.110393782Z     "buys": [format_order(order) for order in orders_buys],
websocket_server_3     | 2018-02-18T01:06:25.110395923Z   File "/usr/src/app/app/services/websocket_server.py", line 231, in format_order
websocket_server_3     | 2018-02-18T01:06:25.110398138Z     price = base_contract.denormalize_value(record["amount_give"]) / coin_contract.denormalize_value(record["amount_get"])
websocket_server_3     | 2018-02-18T01:06:25.110400560Z decimal.InvalidOperation: [<class 'decimal.DivisionUndefined'>]

Filter out stale orders with the availableVolume contract call

https://etherscan.io/address/0x8d12a197cb00d4747a1fe03395095ce2a5cc6819#code
function on the line 289
You can use it to check if the order still has volume or was traded already.

Order creation fails with exponential notation values

Steps to reproduce:

Create an order for a large number of tokens (ideally, with token decimals = 18) through the web interface.
Ensure resulting order posting message has amountGive or amountGet in exponential notation (eg., 1e+26)

Expected: Given an otherwise valid order, order should be accepted by the order book.
Observed: Order validation fails with error "cannot coerce amountGive to int".

Store "old" orders/trades on a separate server/db

We would like users to be able to view their order/trade history without bogging down the performance of the active orderbook.

One proposed suggestion would be to move completed orders to a separate db/server and remove them from the normal order book. When users need to view their trade history, they pull off of the separate DB.

The thinking here is we don't want too much data in the normal orderbook DB as that will be hit constantly. Users don't check their trade history nearly as often, so that database could be larger as it would be hit less.

Organize these files into folders

These files need to be organized much more.
Proper folder structure should be defined.

Possible issue with backend

I have had a problem with a filled order not showing up in the trades section. Neither the personal trades section, nor the "public" one to the right of the screen.

Token MWAT: Order of 5750 tokens filled @0.000063 but not removed from order book and not showing in the trades section.

This leads me to believe that there is an issue with the backend.

Looking also at token Zilliqa, there are traded orders on the book that are not showing in the trades section. I believe this might be the same thing happening, as happened to my own personal order.

Zilliqa order:

Cannot be traded, but no sign of the order in the trades section:

Support testnet

A developer should be able to stand up an instance of the backend that allows them to test a backend client with transactions done on testnet.

Record onchain Orders

Record "off-chain" orders from our interface

We need to record orders from our interface in a queue.

Worker should push orders to ED's server using their WS API.
If ED's server is down, the worker should try again when the server is back up.

Order will be stored in our database and users can still trade that order regardless of if it is on ED or not.

When we launch new UI, ONLY return ticker passed in on getMarket in returnTicker

We currently need all tickers in every getMarket call because of old ED code.

When we launch new UI, specify the token when grabbing tickers and returnTicker will only show data of that passed in token.

# get all tickers
tickers = await get_tickers()

 # if token is passed in
    if token:

Should become

 # if token is passed in
if token:
        
        # get ticker for passed in token
        tickers = await get_tickers(token)

else:
        # get all tickers
        tickers = await get_tickers()

in get_market() in websocket_server.py