Currently this project is using the solvable_orders api endpoint

It should be using the get_auction https://protocol-rinkeby.dev.gnosisdev.com/api/#/default/get_api_v1_auction endpoint.

Currently we are updating a single table in Dune. This means if we run the app data script in both production and staging the two different versions of the query will disagree with each other.

Proposed Solution

append a parameter to the table name

dune_user_generated.gnosis_protocol_v2_app_data_{environment}

where we pass staging/prod as parameter names.

Our cron-jobs updating Dune data are currently failing with:

Dune API Request failed with errors [{'extensions': {'path': '$', 'code': 'validation-failed'}, 'message': 'query is not allowed'}]

It would be good to investigate why the "query is not allowed". If this is an intermittent issue, then we need to add some sort of retry mechanism because the alerts are quite noisy.

Several pieces introduced in #37 can be refined, generalized and moved into the dependency package dune-cli

In particular:

• BlockRange
• QueryData
• DuneFetcher

There are a few reasons why these have not yet been moved into the client outlined here. But, in essence, it is because they need to be refined a bit more before we can make then serve as a useful general purpose. For example, it isn't clear that either of QueryData or DuneFetcher are all that beneficial -- perhaps its better to just extend the Query and DuneClient classes to include the additional fields added in these "extended classes". Would love to open up a discussion about this (maybe in the Dune Client).

We have testplans, how we are currently testing the code:

E.g. here is one example: #6

It would be great to automatized this test and put it as an e2e test

If the sync has already been deployed, this will require a schema change on Dune's side.

This is not a hard requirement, but could make data verification much easier from the User's perspective.

It is pretty easy to implement now if think we may want it.

more context in slack

This iterative async block of code could probably be a batch request.

    for hash in uninitialized_app_data_hashes.iter() {
        download_referral_from_ipfs_and_store_in_referral_store(db.clone(), *hash).await?;
    }

cf referral_maintenance.rs

Original issue gnosis/gp-dune-bridge#59 by @bh2smith

Currently we get the "block_from", but opening a file in our own persisted storage, however this value can also be obtained by querying the table in dune itself. The table does not yet exist, but once it does we could experiment with fetching this value from one, or both places...

The following query should eliminate the need for the entire python portion of this project and also any component of the Rust project responsible for fetching and parsing App Data

The API part of this project would now only refresh this query (responsibly) say once per hours and be able to serve this data to the user account page on CoW.Fi

Observe the current end point is:

https://api.cow.fi/affiliate/api/v1/profile/0x8540f80fab2afcae8d8fd6b1557b1cf943a0999b

which can be compared with any of the trader addresses returned by the query:

V2 Referral Query

Note that, while working on this I have discovered what appears to be a bug in the existing API service/referral query.

Namely, that for 0x8540f80fab2afcae8d8fd6b1557b1cf943a0999b (the address in the link above) the new query returns non zero referral count and volume. I went to look at the trade history of these referrals and found that the traders first trade did, in fact, use this wallet as the referred on their first trade. So I can not understand why we would be neglecting this as a valid referral:

1. Referral 1
2. Referral 2
3. Referral 3

cc @gentrexha and @josojo as something we should plan for in the beginning of January.

Instructions from Dune:

AWS Stuff

1. The data is written into Dune’s S3 buckets. We assume we host the data, but the data still belongs to you.
2. Separate IAM users are created for the community data providers to write data. IAM roles created by data providers are added to the trust policy to allow them to assume IAM roles within our account. This security measure minimizes the number of different credentials maintained by data providers.
3. Data providers should write objects with the bucket-owner-full-control canned ACL. -- #44
4. Pre-specified External IDs must be used when assuming the role.

File Formating

5. We use JSON format to write data into S3 buckets.
6. The file names should contain a predefined constant prefix. (e.g. cow_{the rest of the filename}.json)
7. We intend to keep filenames very simple. Filenames need to contain an increasing sequence number (e.g., cow_000000000000001.json, cow_000000000000001.json). We are releasing a new update to be able to test timestamps instead of sequence numbers in the filenames.
8. The data is written into JSON files as JSON objects.
9. The data in JSON files should not be enclosed in array brackets.
10. Generally, we avoid updates to the written files. If you are producing data every minute, you can write separate files.
11. Written data should follow an append-only approach. We can discuss strategies for data updates and deletion separately.
12. Name/value pairs in JSON files should correspond to the column names and data values in the target data table.
13. If you plan to write data into several tables, you can write data for different tables in different folders in the S3 bucket. (e.g. s3:::{bucket name}/{folder name}) - #44
14. We only support predefined schemas at the moment. Data providers should define schemas of the final tables in advance.

Here for example: