reservoirprotocol / relayer Goto Github PK

It's possible that for whatever reason, some orders that are cached in the open sea indexer were not properly published to the indexer. We should republish them all, just in case.

Notes:

• Only need Wyvern2.3, because all 2.2 orders are invalid
• The indexer should quickly skip any orders it already has, so the easiest way is probably just to republish all
• We should log how many orders were actually missing, just out of curiosity

Should connect with George to see if he already has a script for this

Go back in time to get ALL open sea orders, and have them ready in our DB. When adding a new collection, it gets orders from here, rather than Open Sea itself.

Currently, we have an API for resyncing Open Sea orders in a collection, which uses the Assets API to refetch. This is a) inefficient (because you need to check ALL tokens in collection) and b) getting deprecated.

Going forward, Open Sea will no longer include orders in the Assets API, and only allow fetching orders 1 token at a time. So we need to adjust our strategy accordingly.

Here is the plan:

1. Create a job queue for fetching open sea orders 1 token at a time
2. Update the existing collection resync API to use the Events API
3. Create an API for triggering a resync of a single token (a new feature, that should be easy with the above changes)

The Events API can be filtered to events of type created in order to know which tokens had orders created on them the recently. We would then add those tokens into the queue for resyncing. The nice thing about this method is that we don't need to check every token. We can focus on recent listings, which are most likely where we missed something. The API should fetch X events (using pagination), with X being dynamically passed in the API, and a default of 300 (1 page worth).

Note, the events API may return the same token multiple times, if it got listed multiple times. So we should make sure that each token is only checked once (at the API level, or at the queue level)

Right now the open sea indexer stores the whole order object from open sea, including all asset metadata. While it's handy to be able to access this data, it takes up a lot of space in the database.

I am in the process of syncing data to Snowflake for analysis, and the sync has been running for over 7 days because there's so much data. Going forward, lets only store data that we would need to publish orders to the open order book. The simplest way to implement this is probably to remove the "asset" field from the JSON, as that's all just metadata that can be gotten from elsewhere, and takes up a lot space

We will quite likely want to do our own in-house market making, in order to improve liquidity. As part of this, it might be helpful to be able to place collection-wide bids on Open Sea, in order to acquire inventory that we then sell through Reservoir (e.g. if someone has bid 4 eth on Reservoir, we try bidding 3.5 eth on open sea, and then filling the reservoir order).

To execute this, we need a very simple system that keeps track of the offers we want to place on various collections. It then runs in a loop, placing bids on one token at a time, and refreshing as bids expire. It's quite possible we could build this entirely on top of Open Zeppelin Defender.

We should adopt a system similar to Open Sea, with one bleeding edge query at the tip, and a backup safe query that uses time ranges on a delay.

• Real-time scraping that checks once a second for new order at the tip (no pagination because not reliable)
• More resilient scraping using listing time ranges, a few minutes behind the tip

Simple API that lets you pass a contract address to the open sea indexer, and it will post all orders it has in it's DB to the indexer.

The old indexer (v2) supports syncing the latest and best sell orders for any given contract from OpenSea (eg. https://github.com/reservoirprotocol/core/blob/main/src/scripts/opensea.ts). This functionality should get moved here so that we have a way to always sync to the latest orders state in case of any issues with the indexer (any fetched orders should get forwarded to it).

• Save the date created for new orders
• Listen to OpenSea websocket and save the new listing events in a table (maker, contract, token, price, listing time, created time)

The goal is to compare timestamps to better understand where latency is introduced.

Basically what we have for Open Sea, but do it for LooksRare too. Separately George is adding the underlying support for LooksRare, so that these orders will be accepted by the Indexer.

Here are the docs: https://api.looksrare.org/api/documentation/#/

We should have a process that periodically fetches all the orders for a couple of high volume collections, to verify that no orders were missing from our regular indexing process. If an order is found that the indexer had not previously seen, it should trigger an alarm.

One exception might be that the order was brand new, and the regular indexer hadn't got to it yet, so we should handle that case.

https://hackmd.io/7AnOgEqFT2mZHqUQ4bXwsw#get-orders

The OpenSea Indexer is still using the old SDK @georgeroman/wyvern-v2-sdk. We should ditch that in favor of the new SDK @reservoir0x/sdk.

When we are re-syncing a collection, we could first fetch the best price for every token from the floor API. This would give us a baseline to check events against.

If an event returns a listing with a price that is equal or higher than what we already have for that token, ignore it. We only care about the lowest order per token, so this suggests we either have it already, or it's low priority. It would allow us to look back further in time, or check more collections, while minimizing the order lookup requests. If we want, the api could still have the option to do a deep scan that checked every event

• Every second, look for events that are more than 10 seconds old and don't have a corresponding order
• Put these tokens into to token lookup queue for manual querying
• Mark the event as processed to avoid processing it again
• Potentially we can revert the token lookup queue to fetch 20 tokens at a time, because Open Sea backed off deprecating this functionality (last I checked)

We should re-index everything Open Sea, back to the beginning of time (which is only a few weeks ago, due to their recent contract upgrade). This will catch any orders that we missed along the way. George has done in this in the past, so can coordinate with him.

We should log how many new orders were discovered, so we can understand the extent of any issues.

Long term, we might want a continuous backfill process for redundancy. This might actually be the most efficient way to proactively catch missing orders, rather than attempting to do by collection. But let's do one manually for now.

The OpenSea Indexer runs everything in a single process and in some cases it fails to persist state across deploys (eg. the process of fetching and relaying orders to the indexer will stop if a new deployment is triggered and the current instance dies). We should switch to using job queues, as we have for the indexer, in order to be more efficient (eg. allow multiple processes) and more resilient (eg. persist state across deployments).

Collection offers aren't nicely exposed, but you can get them by checking a single token, in this API:

https://api.opensea.io/api/v1/asset/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d/3105/offers

You only need to check one token per collection, because it will apply to all.

As an initial test, let's pick one collection, or maybe a handful, and try to fetch the collection offers.

• Poll every 5 seconds, getting 5 seconds worth of orders
• Leave a buffer from current time, to ensure each 5 second batch is "complete"
• If we need to paginate, publish it as a new job in the queue
• Process jobs sequentially or once per second (whichever is best for avoiding limits)
• Put pagination jobs at the front of the queue, so it mostly stays in order

Trying different tweaks to capture as many orders as possible in the initial real-time polling, as close to creation time as possible.

Adding the stack trace to the logger would be very helpful with debugging as the codebase grow

Sometimes, for various reasons, we want to go directly to open sea to check for orders, to make sure we have everything. Right now, that requires checking every single token in a collection. As an alternative, we could use the events API to fetch the last X tokens that were listed for sale, and then only fetch orders for those tokens.

It feels like 200 is a reasonable number of listings to fetch. It would take 10 requests to get the listings, then 10 to get the orders, for 20 total. A big improvement over the 500 requests required to check all tokens in a 10k collection. But this can be the default, and we should probably make it so that X is a parameter you can pass in when calling the API.

One more thing to consider is whether to do it based on contracts or collections. Collections is convenient, but requires us to have identical collection ids to open sea. This is mostly the case, but not sure we want tight coupling. Problem with doing it by contract is that in some cases (e.g. art blocks), there can be many tokens in the contract.