rubicondefi / rubi-py Goto Github PK

today: we require the user to pass in a web3 object in order to create the Rubicon class. we should just ask the user to input an endpoint instead. input is error

🔜 we will allow the rubi class, and possibly other contract classes(?), to be created just from a node endpoint. we should determine if the connection is HTTP or WS. in the future, we should have WS-specific functionality that is enabled.

Experiencing an issue with "insufficient funds". It looks like the numbers I'm inputting in the add_pair() allowance are larger than the minimum fee values outlined here: https://docs.rubicon.finance/protocol/rubicon-market/min-order-sizes

I have at least 50% of each of the values that I am using in the add pair functions.

import logging as log
import os
from _decimal import Decimal
from multiprocessing import Queue

from dotenv import load_dotenv

from rubi import Client

# load from env file
load_dotenv(".env")

# set logging config
log.basicConfig(level=log.INFO)

# set the env variables
http_node_url = os.getenv("HTTP_NODE_URL")
wallet = os.getenv("WALLET")
key = os.getenv("KEY")

queue = Queue()

# create client
client = Client.from_http_node_url(
    http_node_url=http_node_url,
    # custom_token_addresses_file="custom_token_addresses.yaml",
    wallet=wallet,
    key=key,
    message_queue=queue
)

client.add_pair(
    pair_name="WETH/USDC",
    base_asset_allowance=Decimal(.005),
    quote_asset_allowance=Decimal(100)
)

and here is the traceback that I get:

Traceback (most recent call last):
  File "/Users/perryscott/Desktop/rubicon/./rubi_test2.py", line 32, in <module>
    client.add_pair(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 159, in add_pair
    self.update_pair_allowance(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 194, in update_pair_allowance
    self._update_asset_allowance(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 603, in _update_asset_allowance
    asset.approve(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/erc20.py", line 203, in approve
    return self._default_transaction_handler(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/base_contract.py", line 220, in _default_transaction_handler
    self.w3.eth.send_raw_transaction(signed_txn.rawTransaction)
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/web3/eth/eth.py", line 372, in send_raw_transaction
    return self._send_raw_transaction(transaction)
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/web3/manager.py", line 232, in request_blocking
    return self.formatted_response(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/web3/manager.py", line 205, in formatted_response
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'insufficient funds for gas * price + value'}

Hey guys,

I'm having some trouble with any functions that alter existing limit orders, whether that be an update or a cancel. Sometimes the functions work well and do their prescribed function, at other times I get a txn failure that looks something like this from ethersscan:

Fail with error 'Offer can not be cancelled because user is not owner, and market is open, and offer sells required a ...'

Keep in mind that this error is the same for both quote updates and cancellations. I'm sourcing my order limit ID's directly from my active offers using the sql query API.

        headers = {'Content-Type': 'application/json'}

        response = requests.post(url, headers=headers, data=json.dumps({'query': query}))

I'm using the client.batch_update_limit_orders(), but I've seen the same issue using client.market.batch_requote() and client.batch_cancel_limit_orders(). I'm not sure if it matters, but oftentimes I'm only passing in a single order into these functions.

today: we either utilize the gas value provided by the user or default to a value of 300000 when signing transactions.

soon: we should be dynamically setting gas based on estimation. web3.py has a function to deal with this, but upon trying to implement this function i received values that were significantly underestimated. worth digging back into to see if it was, most likely, an issue in my implementation.

today: we utilize the class objects defined in rubi/contracts/helper/rolodex.py to pass contract address data, abi data, and other chain relevant information throughout the package depending upon which chain_id the node provided is on.

soon -> it would be great if we:

• [] set these values to be dynamically pulled in a trustless fashion
• [] error handle in scenarios where the chain_id does not have any trusted deployments implemented in the rolodex

to keep a running list of potential improvements:

• [] rubi/rubi/contracts/aid.py functions that involve passing in array values
• [] a lot of other places in the repo...

Hey Rubicon,

I'm experiencing an issue with WEI being rounded on the backend. The reason I have come to conclusion is that when I place an order with a very small spread that's anything below ~10,000 ints of each other, the orders complete each other. For example:

Order 1 - Buy:
WETH = 8000000000000000
DAI = 149999999999999999500

Order 2 - Sell:
WETH = 8000000000000000
DAI = 150000000000000000500

I've made up these numbers but they're in the ballpark of orders I've seen. Let's call this an "int spread" of 1,000, for a WETH order of 0.1 size. One of these orders will be a Offer and then the next will be a Buy. This will happen until the "int spread" is > ~10,000. I'm not sure exactly how price is being calculated but there seems to be some rounding issues that cause the price calculation to be slightly altered. It's a very nit-picky bug report, but it felt worth reporting and it is causing some issues on my end. I can DM someone a tx hash if that will help as well.

Note: I'm using the client.market.batch_offer() function and passing these int values in.

currently, we have wired up a majority of our data work to the RubiconMarketLight subgraph. for a variety of reason we are going to need to switch to the RubiconV2 subgraph, and make some changes to how we handle a couple of things:

• token symbols and decimals: the v2 subgraph is written to be supported only by a full-node, meaning that we don't have access to call state to get token decimals/symbols/etc. this means we have to create some synthetic fields that will be based off of network token lists.
• we need to support multi-network data, and most likely rework some of the existing structure of the repo
• from here we will want to actually go through and ideally add some type of sanity check against state
• we will want to update docs to support access to this data :)

The user should be able to create a new Transaction object and estimate how much gas it will cost to execute the transaction.

Ideally, less information is required to retrieve this information. I'm ill-informed on the variables that go into gas cost. If it's possible, maybe the user only needs to provide the following to a function:

• number of orders
• whether it's an offer/cancel
• pair

To receive an estimate of the gas cost.

web3.py does not have a nonce manager like ethers does, it would be great if this were supported out of the box. there are some good thoughts on this topic in this article 🧑‍🏫 and I have a variety of my own on the subject 🧑‍🎓

i believe this should be the next area of any major investment in the repo 💎 and will be a must-have for any future open-source bot implementations 📈 part of me wants to work to enable this within the context of the web3 repo itself, but as it's of need today, i think the best course of action will be to internalize it to this repo, and later any learnings and work can be transferred to the web3 repo more broadly. apes together strong 🦍

the longterm fix will be solved with #6. for now, i think we should just arbitrarily raise the gas_limit by a factor of 10 to try to make sure that transactions always go through. experience a gas_limit transaction failure during testing on optimism goerli that is prompting this: txn tenderly

I'm having an issue with the client.add_pair() function. Here is what the code looks like so far, borrowed entirely from the quickstart guide.

import os
from dotenv import load_dotenv
from decimal import Decimal

from rubi import Client

# load from env file
load_dotenv(".env")

# set the env variables
http_node_url = os.getenv("HTTP_NODE_URL")
wallet = os.getenv("WALLET")
key = os.getenv("KEY")

# instantiate the client
client = Client.from_http_node_url(
    http_node_url=http_node_url,
    wallet=wallet,
    key=key
)

# add the WETH/USDC pair to the client
# the base asset is WETH and the quote asset is USDC
client.add_pair(
    pair_name="WETH/USDC",
    base_asset_allowance=Decimal("1"),
    quote_asset_allowance=Decimal("10000")
)

and the resulting error:

Traceback (most recent call last):
  File "/Users/perryscott/Desktop/rubicon/./rubi_test.py", line 25, in <module>
    client.add_pair(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 159, in add_pair
    self.update_pair_allowance(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 194, in update_pair_allowance
    self._update_asset_allowance(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/client.py", line 603, in _update_asset_allowance
    asset.approve(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/erc20.py", line 203, in approve
    return self._default_transaction_handler(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/base_contract.py", line 222, in _default_transaction_handler
    return self._wait_for_transaction_receipt(transaction=signed_txn)
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/base_contract.py", line 267, in _wait_for_transaction_receipt
    result = TransactionReceipt.from_tx_receipt(
  File "/Users/perryscott/Desktop/rubicon/rubi_env/lib/python3.10/site-packages/rubi/contracts/contract_types/transaction_reciept.py", line 56, in from_tx_receipt
    l1_fee_scalar=None if tx_receipt.get("l1FeeScalar") is None else int(tx_receipt.get("l1FeeScalar"), 16)
ValueError: invalid literal for int() with base 16: '0.684'

I've also tried removing the quotations in the Decimal() function as well as other pair combinations and values. Let me know if there's something I'm missing. Thanks.

Also a heads up, in the docs quickstart section the 'from decimal import Decimal' import is missing and an extra 'client = ' found it's way into the Client() declaration.

today: the repo has very mundane error handling that could be improved by utilizing w3.eth.wait_for_transaction_receipt throughout any write call interaction. this should be optional and probably defined at initialization, as it has impacts on the speed of which orders can be processed. ideally, a nonce manager solves many of these problems for the users.

soon: for sure improved error handling utilizing w3.eth.wait_for_transaction_receipt 🧑‍🚒 the most pressing use case would be handling token approval errors for the user as these are only identified through transaction failure, not on signing or sending 🥇

we currently lack nested filtering throughout the repo, it would be great to add this in for obvious use cases, for example on the AidData class allowing to filter get_aid_history by token would be 💯

i know this is supported in subgrounds, need to go dig around more and find an example or see whats going on underneath so i can utilize the proper syntax. this would be a good place to add in an example to that repo to help out others who are experiencing the same problem 🫂

this is a really interesting error i am getting that i am unsure on how to proceed with, so putting it here for now. on the hosted service the query below executes as expected. from rubicon's indexer, the query fails. if i had to guess, this can be resolved with a graph-node update:

`
subgraph = "https://api.rubicon.finance/subgraphs/name/RubiconV2_Optimism_Mainnet_Dev"
#subgraph = "https://api.thegraph.com/subgraphs/name/denverbaumgartner/rubiconv2-optimism-mainnet"
sg = Subgrounds()
data = sg.load_subgraph(subgraph)
Take = data.Take

takes = data.Query.takes(
first=100,
where={"offer_": {"from_address": "0xde4de639d699a66f860814468298639c49e4f778"}}
)

fields = [
takes.id
]

df = sg.query_df(fields)
`

currently, we expect that the result of our queries are not empty, and then proceed to attempting to transform the data. this causes problems when the dataframe actually is empty, and we should handle scenarios like this. an example where this can occur is rubi.data.market_aid_optimism_processing.aid_fill_tracking(aid = aid, asset = asset, quote = quote)

today: the repo only passes, expects, and utilizes the full uint representation of erc20, erc721, etc. token values. most people tend to like seeing these values converted to human-readable format at some point or another.

soon: at a bare minimum we will add in additional support to the ERC20 class for dealing with integer values. throughout the repo itself i'm not sure we should really do must as we should encourage best practice of dealing with values as integers to avoid precision errors or mistakes. within data aggregation & reporting 🔜 etc. this will be an obvious part of data cleaning & prep

today: there are several compromises made around these functions for the sake of viability today. these include:

• [] the need to convert parameters to hexbytes in order to utilize the web3._utils.events.get_event_data
• [] hardcoding the offer id type in order to use eth_abi.decode, or even the need to decode the id itself
• [] unclear if passing back a dictionary is the best handoff of this information

soon: open to any suggestions 🦖 🌮

from a bug report submitted by @soheiyagi in the public discord

occasionally, when attempting to construct an OrderBook object, when building the BookSide using from_rubicon_offers we encounter an IndexError such as follows:

if levels and levels[-i].price == price:... IndexError: list index out of range

looking into this function it appears that we should be utilizing levels[-1] instead of levels[-i]. i believe the issue we are encountering is a result of offers existing at the same level of the book, which only occasionally happens. furthermore, i believe we need to sort the offers based on price if we want to use this current approach in order to correctly place multiple offers within the same BookLevel. alternatively, we could utilize a dictionary such as is done here to keep track of price and at which level an offer should be inserted.

i may be misinterpreting something here, would love to know your thoughts @adamamsmith 🙏

today: we are signing transactions utilizing gas and gasPrice. this is a dated method because of the London upgrade and should be updated.

soon: we should sign transactions utilizing maxFeePerGas and maxPriorityFeePerGas. this is also a mere matter of what worked today, but will be improved upon very 🔜 📈

today: throughout the repository almost every contract class has a function called get_log_xyz_hash that returns an event signature. today these values are hardcoded.

soon: it would be great if these values were generated from the contract .sol files upon initialization of the class, or even within rolodex.py

• investigate if there is a python lib we can depend on rather

I am placing an order with rubi-py, but currently TransactionReceipt does not return information such as whether my order was confirmed or failed.

It seems that there is no other rubi-py method, so currently there is no choice but to get information from the subgraph, but I think it would be good to be able to complete it with rubi-py as much as possible when launching a small bot.

I'm not sure if it's better to implement it with another rubi-py method or add it to TransactionReceipt.

Thanks for developing!

when using a pattern as follows: client.start_event_poller("WETH/USDC", event_type=EmitOfferEvent, filters={"maker": client.wallet}) ANY offer from the client.wallet is reported back as an offer for the given pair, even if it is not for that pair. i'm not sure if this is an error during filter creation for the node or if this is something we need to handle client side instead. will try to investigate further, this limits the ability for users to effectively run multiple pairs from a single client if they are utilizing the event_poller.

it would be great to support open-telemetry out of the box and have some useful tooling implemented or available throughout the library. this is a lower-priority, but high-impact enhancement that should most likely be implemented in support of examples.

today: the market aid class is initialized from a contract address that is passed in by the user. because the factory can tie a user's signing address to all of the market aid instances that it controls, it would be nice if in the setup we had the option for a user to either select a market-aid instance that they would like to deploy to, or to create a new one.

soon: we should improve rubi/rubi/contracts/aid.py, and the repo as a whole, to include something like the following:

• [] incorporate aid contracts in the factory class for easy access upon initialization
• [] if a user does not have a market aid contract, we should have an easy integration function that deploys one and gives the necessary approvals to it
• [] as a whole, we should drastically improve the tooling support we provide around market aid. gas costs is a crucial thought here with a subgraph solution that is soon to be open-sourced in mind

current state: today, we hardcode the contract abis and bytecodes in order to interact with the protocol.
desired state: soon, it would be great if we compiled the protocol's .sol files to get this data. useful library.

Hey guys,

I'm trying to create a Arbitrum client using the following function:

    client = Client.from_http_node_url(
        http_node_url=os.getenv("HTTP_NODE_URL"),
        wallet= os.getenv("WALLET"),
        key=os.getenv("KEY"),
        message_queue=queue
    )

I've tried several arbitrum endpoints such as :

• Alchemy.com
• Blastpi.io
• quicknode.com

All of the node url's I've gotten from these websites yield the same errors:

https://indulgent-old-borough.arbitrum-mainnet.discover.quiknode.pro/perry's_api_key/
Traceback (most recent call last):
  File "/home/ec2-user/rubi-volume-bot/./app.py", line 114, in <module>
    client = get_client(queue=my_queue, pair=token)
  File "/home/ec2-user/rubi-volume-bot/utils.py", line 73, in get_client
    client = Client.from_http_node_url(
  File "/home/ec2-user/.local/lib/python3.10/site-packages/rubi/client.py", line 105, in from_http_node_url
    network = Network.from_config(
  File "/home/ec2-user/.local/lib/python3.10/site-packages/rubi/network/network.py", line 125, in from_config
    network_name = NetworkId(w3.eth.chain_id).name.lower()
  File "/usr/local/lib/python3.10/enum.py", line 385, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/lib/python3.10/enum.py", line 710, in __new__
    raise ve_exc
ValueError: 42161 is not a valid NetworkId

Is there a preferred provider I should be using to do this?

when i originally switched the allowance argument to be optional in client.add_pair i did not also make the ERC20 class initialization of the base and quote assets not require a wallet/key. when an allowance is not provided, we should not initialize the ERC20 classes with write permissions.

today: when placing match market-making trades there a self-trade check to make sure orders don't auto-match on the contract side but not on the client side. it would be really not nice 😞 if someone were to be in a situation where they accidentally submitted a lot of transactions that fail but still spend gas 🗼

soon: we should add checks client side to make sure that offers aren't submitted that can self match. for a lot of these additional safety checks I think we should give the user the option to opt-out if they want to (speed?). a lot of other paths to improve on the speed front before we get to that point IMO. additionally, i think we should have some way to monitor for bad fail-state scenarios and cut the cord if needed. some type of safety module? maybe this is baked into the nonce manager 🧙

Now: the client can only interact with the chain using http requests.

Soon: add a websocket implementation that allows the client to listen to updates from a node instead of having to poll for them.

we should add an additional pr-title word that bypasses the deployment of the package on merge. we can call it skip, tertiary, or something of that sort. would be nice for when we need to update docs, the bots, etc.

I am experiencing an issue with the start_event_poller stopping after extended usage.

Replicating this is difficult, as the problem usually happens at a somewhat random interval, but the behavior usually goes something like this:

1. Call the start_event_poller() it begins working for some time
2. A transaction error occurs (insufficient funds, gas, exceeding allowance)
3. The event poller then stops working

Replicate:

# Here is the exact way I'm initializing the event poller
# The problem is not local to a certain pair type
# problem occured on versions 2.0.5 and 2.0.6
    client.start_event_poller(pair_string, event_type=EmitOfferEvent, filters={"maker": client.wallet}, poll_time=poll_time)
    client.start_event_poller(pair_string, event_type=EmitTakeEvent, filters={"maker": client.wallet}, poll_time=poll_time)
    client.start_event_poller(pair_string, event_type=EmitCancelEvent, filters={"maker": client.wallet}, poll_time=poll_time)
    client.start_event_poller(pair_string, event_type=EmitDeleteEvent, filters={"maker": client.wallet}, poll_time=poll_time)

Let me know if there is any other info I can provide to help.

today: rubi/rubi/contracts/helper/erc20.py is set up to be fed off of an implementation of the ERC20 abi that is utilized in the Rubicon Protocol. the class instance utilizes a decimals call that is optional and not present on some ERC20 implementations.

soon:: we should identify the best ERC20 template to utilize withing the repo and update the erc20.py file.

currently, we do not report back the block number in a query, rather we just collect the data as is. by expanding each query to include a _meta_ field that contains block number, we can provide more insight to user's regarding how delayed the data they are working with is.

Open source bot using: RubiArbitrageur using contract 0x80130D182ED6D3A99C06166d939E9f0bA864DFC8

today: the repo does not provide the user with any information regarding the fee they incur when taking liquidity off the book. an example of this is the swap function in rubi/rubi/contracts/router.py where the function must be provided with an expected_market_fee_bps value. it would be great if this was set as a class variable at initialization.

soon: we should work to avoid hardcoding the fee throughout the repo when needed and work to incorporate more reporting & information regarding fees throughout the repo.

in addressing issue #21 its clear that we are going to start to enter scenarios where we want the aggregate data from multiple chains at the same time. these queries should most likely be executed in parallel to save time and then be merged upon receipt. in the name of going fast and only building what is needed right now, i am putting this off till a later time. i think it will require a bit of reworking the existing functionality i am using today, but it shouldn't be too bad. will attempt to avoid creating tech debt for this scenario 🖖

currently, we utilize a ^ pattern throughout our dependencies to utilize the most recent version of packages. this is poor practice and can lead to dependency conflicts. we should statically lock dependencies to known versions that work with one another to avoid this scenario going forward.

currently, we do not have any type of error handling for failed api calls. i think ideally, we have a common fallback pattern that we utilize throughout our queries to handled networking issues. we should provide the user the option to override these fallbacks if they do not want the system to retry, but rather raise an error instead.

today: we do not utilize any multi-call functionality throughout the repo, and instead place repeat read calls throughout the repo to obtain data directly from the chain 😪

soon: we should utilize a multi-call functionality, most likely adding this dependency, to group any and all calls throughout the repo 💨