Git Product home page Git Product logo

snek's Introduction

Snek

Implementation of the best NFT marketplace with locked balances.

image

Prerequisites

  • node 16.x
  • docker

Quickly running the sample

Example commands below use make(1). Please, have a look at commands in Makefile if your platform doesn't support it. On Windows we recommend to use WSL.

# 1. Install dependencies
npm ci

# 2. Compile typescript files
make build

# 3. Start target Postgres database and detach
make up

# 4. Apply database migrations from db/migrations
make migrate

# 5. Start the processor
make process

# 6. The command above will block the terminal
#    being busy with fetching the chain data, 
#    transforming and storing it in the target database.
#
#    To start the graphql server open the separate terminal
#    and run
make serve

# 7. Now you can see the results by visiting the localhost:4350/graphql

Dev flow

1. Define database schema

Start development by defining the schema of the target database via schema.graphql. Schema definition consists of regular graphql type declarations annotated with custom directives. Full description of schema.graphql dialect is available here.

2. Generate TypeORM classes

Mapping developers use TypeORM entities to interact with the target database during data processing. All necessary entity classes are generated by the squid framework from schema.graphql. This is done by running npx squid-typeorm-codegen command.

3. Generate database migration

All database changes are applied through migration files located at db/migrations. squid-typeorm-migration(1) tool provides several commands to drive the process. It is all TypeORM under the hood.

# Connect to database, analyze its state and generate migration to match the target schema.
# The target schema is derived from entity classes generated earlier.
# Don't forget to compile your entity classes beforehand!
npx squid-typeorm-migration generate

# Create template file for custom database changes
npx squid-typeorm-migration create

# Apply database migrations from `db/migrations`
npx squid-typeorm-migration apply

# Revert the last performed migration
npx squid-typeorm-migration revert         

4. Generate TypeScript definitions for substrate events, calls and storage

This is an optional part, but it is very advisable.

Event, call and runtime storage data come to mapping handlers as raw untyped json. While it is possible to work with raw untyped json data, it's extremely error-prone and the json structure may change over time due to runtime upgrades.

Squid framework provides tools for generating type-safe wrappers around events, calls and runtime storage items for each historical change in the spec version.

The end result looks like this:

/**
 * Normalized `balances.Transfer` event data
 */
interface TransferEvent {
    from: Uint8Array
    to: Uint8Array
    amount: bigint
}

function getTransferEvent(ctx: EventHandlerContext): TransferEvent {
    // instanciate type-safe facade around event data
    let event = new BalancesTransferEvent(ctx)
    if (event.isV1020) {
        let [from, to, amount, fee] = event.asV1020
        return {from, to, amount}
    } else if (event.isV1050) {
        let [from, to, amount] = event.asV1050
        return {from, to, amount}
    } else {
        // This cast will assert, 
        // that the type of a given event matches
        // the type of generated facade.
        return event.asLatest
    }
}

Generation of type-safe wrappers for events and calls is currently a two-step process.

First, you need to explore the chain to find blocks which introduce new spec version and fetch corresponding metadata.

npx squid-substrate-metadata-explorer \
  --chain wss://kusama-rpc.polkadot.io \
  --archive https://kusama.indexer.gc.subsquid.io/v4/graphql \
  --out kusamaVersions.json

In the above command --archive parameter is optional, but it speeds up the process significantly. From scratch exploration of kusama network without archive takes 20-30 minutes.

You can pass the result of previous exploration to --out parameter. In that case exploration will start from the last known block and thus will take much less time.

After chain exploration is complete you can use squid-substrate-typegen(1) to generate required wrappers.

npx squid-substrate-typegen typegen.json

Where typegen.json config file has the following structure:

{
  "outDir": "src/types",
  "chainVersions": "kusamaVersions.json", // the result of chain exploration
  "typesBundle": "kusama", // see types bundle section below
  "events": [ // list of events to generate
    "balances.Transfer"
  ],
  "calls": [ // list of calls to generate
    "timestamp.set"
  ]
}

Project conventions

Squid tools assume a certain project layout.

  • All compiled js files must reside in lib and all TypeScript sources in src. The layout of lib must reflect src.
  • All TypeORM classes must be exported by src/model/index.ts (lib/model module).
  • Database schema must be defined in schema.graphql.
  • Database migrations must reside in db/migrations and must be plain js files.
  • sqd(1) and squid-*(1) executables consult .env file for a number of environment variables.

Types bundle

Substrate chains which have blocks with metadata versions below 14 don't provide enough information to decode their data. For those chains external type definitions are required.

Type definitions (typesBundle) can be given to squid tools in two forms:

  1. as a name of a known chain (currently only kusama)
  2. as a json file of a structure described below.
{
  "types": {
    "AccountId": "[u8; 32]"
  },
  "typesAlias": {
    "assets": {
      "Balance": "u64"
    }
  },
  "versions": [
    {
      "minmax": [0, 1000], // block range with inclusive boundaries
      "types": {
        "AccountId": "[u8; 16]"
      },
      "typesAlias": {
        "assets": {
          "Balance": "u32"
        }
      }
    }
  ]
}

All fields in types bundle are optional and applied on top of a fixed set of well known frame types.

Differences from polkadot.js

Polkadot.js provides lots of specialized classes for various types of data. Even primitives like u32 are exposed through special classes. In contrast, squid framework works only with plain js primitives and objects. This allows to decrease coupling and also simply dictated by the fact, that there is not enough information in substrate metadata to distinguish between interesting cases.

Account addresses is one example where such difference shows up. From substrate metadata (and squid framework) point of view account address is simply a fixed length sequence of bytes. On other hand, polkadot.js creates special wrapper for account addresses which aware not only of address value, but also of its ss58 formatting rules. Mapping developers should handle such cases themselves.

Graphql server extensions

It is possible to extend squid-graphql-server(1) with custom type-graphql resolvers and to add request validation. More details will be added later.

Archival nodes

Because subsquid requires an archival indexer to be fast, there are currently 3 options how to do it:

  1. Leave it as it is

there is already indexer for base basilisk

  1. using archival node for Koda BSX Sandbox 🐍
ARCHIVE_URL=https://basilisk-test.indexer.gc.subsquid.io/v4/graphql
  1. running your own
git clone [email protected]:subsquid/squid-archive-setup.git;
cd squid-archive-setup/basilisk

in docker-compose.yml set url for the chain

-      - WS_PROVIDER_ENDPOINT_URI=wss://basilisk.api.onfinality.io/public-ws
+      - WS_PROVIDER_ENDPOINT_URI=wss://basilisk-kodadot.hydration.cloud

then just

docker compose up

and set

ARCHIVE_URL=http://localhost:4010/v1/graphql

snek's People

Contributors

alko89 avatar jak-pan avatar kngzhi avatar ma-shulgin avatar matehoo avatar omahs avatar petersopko avatar preschian avatar roileo avatar vikiival avatar yangwao avatar zhengow avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

snek's Issues

Offers

Basically there is a ton of edge cases.

but the most crucial ones:

  • what to do with withdrawn offer? (remove or just update?)
  • what to do on accept offer? (// may status to accepted to true 🤷 )
  • add current_owner to offers? (if not removed could be nice for analytics)

ref galacticcouncil/Basilisk-node#424

log functions

current logs are too verbose
make them less verbose

  • debug
  • warn
  • success

Handler to mark offers as expired.

The task:

  • You need to fetch offers whose expires lower or equal to current block number.
  • Set their status as EXPIRED
  • Save into DB

There are basically two options:

  1. Do it on action of the NFT (LIST, ACCEPT_OFFER, SEND) - per particular NFT

  2. Do it automatically every 100 blocks (// 5 blocks/min, so every 20 min) on all dataset

May @dzhelezov can drop a comment

handleRoyaltyPay - Do not know how to serialize a BigInt

while trying to process, I ran into this error:

*  note      Welcome to the Processor! https://basilisk-test.indexer.gc.subsquid.io/v4/graphql
Prometheus metrics are served at port 60739
[ ]  pending   [PAY ROYALTY]: 28058
.isLatest, .asLatest properties are deprecated, if you believe this is a mistake, please leave
a comment at https://github.com/subsquid/squid/issues/9
TypeError: Do not know how to serialize a BigInt
    at JSON.stringify (<anonymous>)
    at handleRoyaltyPay (D:\KodaDot\snek\lib\mappings\index.js:207:41)
    at D:\KodaDot\snek\node_modules\@subsquid\substrate-processor\lib\processor.js:285:35
    at D:\KodaDot\snek\node_modules\@subsquid\substrate-processor\lib\db.js:69:19
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

safely Try

Wrapper around handlers that will wrap call in the try catch method

import logger, { logError } from './logger'

export function safelyTry(handler: any) {
  try {
    handler()
  } catch (e) {
    logError(e, (e) =>
      logger.warn(`[[${handler.name}]]: Reason: ${e.message}`)
    )
  }
} 

Assets

We will need a support for assets

type AssetEntity @entity {
  id: ID!
  name: String!
  decimals: Int!
  sn: Int!
}

type BalanceEntity @entity {
  id: ID!
  owner: String!
  asset: AssetEntity!
  balance: BigInt!
}

Series insight fail

{
  "errors": [
    {
      "message": "Cannot query field \"highestSale\" on type \"Series\".",
      "extensions": {
        "code": "GRAPHQL_VALIDATION_FAILED",
        "exception": {
          "stacktrace": [
            "GraphQLError: Cannot query field \"highestSale\" on type \"Series\".",
            "    at Object.Field (/squid/node_modules/graphql/validation/rules/FieldsOnCorrectTypeRule.js:48:31)",
            "    at Object.enter (/squid/node_modules/graphql/language/visitor.js:323:29)",
            "    at Object.enter (/squid/node_modules/graphql/utilities/TypeInfo.js:370:25)",
            "    at visit (/squid/node_modules/graphql/language/visitor.js:243:26)",
            "    at validate (/squid/node_modules/graphql/validation/validate.js:69:24)",
            "    at validate (/squid/node_modules/apollo-server-core/dist/requestPipeline.js:186:39)",
            "    at processGraphQLRequest (/squid/node_modules/apollo-server-core/dist/requestPipeline.js:98:34)",
            "    at runMicrotasks (<anonymous>)",
            "    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
            "    at async processHTTPRequest (/squid/node_modules/apollo-server-core/dist/runHttpQuery.js:220:30)"
          ]
        }
      }
    },
    {
      "message": "Cannot query field \"emoteCount\" on type \"Series\".",
      "extensions": {
        "code": "GRAPHQL_VALIDATION_FAILED",
        "exception": {
          "stacktrace": [
            "GraphQLError: Cannot query field \"emoteCount\" on type \"Series\".",
            "    at Object.Field (/squid/node_modules/graphql/validation/rules/FieldsOnCorrectTypeRule.js:48:31)",
            "    at Object.enter (/squid/node_modules/graphql/language/visitor.js:323:29)",
            "    at Object.enter (/squid/node_modules/graphql/utilities/TypeInfo.js:370:25)",
            "    at visit (/squid/node_modules/graphql/language/visitor.js:243:26)",
            "    at validate (/squid/node_modules/graphql/validation/validate.js:69:24)",
            "    at validate (/squid/node_modules/apollo-server-core/dist/requestPipeline.js:186:39)",
            "    at processGraphQLRequest (/squid/node_modules/apollo-server-core/dist/requestPipeline.js:98:34)",
            "    at runMicrotasks (<anonymous>)",
            "    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
            "    at async processHTTPRequest (/squid/node_modules/apollo-server-core/dist/runHttpQuery.js:220:30)"
          ]
        }
      }
    }
  ]
}

cc @roiLeo ^-^

Resolver for collection stats

hey, just tested this, seems that there are still irregularities within the stats? check the screenshot below image

This might be a backend issue, we should have same data type between subsquid different implementation

more infos:
on RMRK => event: "BUY", meta key has nft price
Capture d’écran 2022-06-24 à 12 03 18 PM

on BSX => event: "BUY", meta key doesn't have nft price
Capture d’écran 2022-06-24 à 12 02 58 PM

Originally posted by @roiLeo in kodadot/nft-gallery#3257 (comment)

Calculation of emotes

We do not have emotes in bsx however to maintain compatibility it would be nice to have it :).

Idea is to count number of ACTIVE/ALL offers per NFT

Royalty Entity

There should be a royalty entity for historical purposes

Extrinsic is null for some collections

Some collections are just YOLO

{
  "events": [
    {
      "block": {
        "height": 12153,
        "calls": []
      },
      "name": "NFT.ClassCreated",
      "args": {
        "owner": "0x6d6f646c4c69714d696e49640000000000000000000000000000000000000000",
        "classId": "1",
        "metadata": "0x",
        "classType": {
          "__kind": "LiquidityMining"
        }
      },
      "extrinsic": null,
      "call": null
    },
    {
      "block": {
        "height": 497726,
        "calls": [
          {
            "name": "NFT.create_class"
          }
        ]
      },
      "name": "NFT.ClassCreated",
      "args": {
        "owner": "0x1a4d4dd7973c7b234db30e58cb54c3b8d289b7642b30005a4d0de48097f3027e",
        "classId": "123456546456456",
        "classType": {
          "__kind": "Marketplace"
        }
      },
      "extrinsic": {
        "signature": {
          "address": "0x1a4d4dd7973c7b234db30e58cb54c3b8d289b7642b30005a4d0de48097f3027e",
          "signature": {
            "value": "0x7ce9c5423a81d5df6faaf87387a08769151650bbdafb245415fedcf5d44eaf3860d8b95158fff127463e5c222f7f30c936cfcdd42c4fea7de02951a5cdbaf085",
            "__kind": "Sr25519"
          },
          "signedExtensions": {
            "CheckNonce": 15,
            "CheckMortality": {
              "value": 1,
              "__kind": "Mortal164"
            },
            "ChargeTransactionPayment": 0
          }
        }
      },
      "call": {
        "name": "NFT.create_class",
        "args": {
          "classId": "123456546456456",
          "metadata": "0x6d65746164617461",
          "classType": {
            "__kind": "Marketplace"
          }
        }
      }
    }
  ]
}

offers should change updatedAt

  1. Recently Interacted and Unpopular are doing the same thing as Recently Created and Oldest. The reason for this (as far as I can tell) is that activity such as Offer isn't changing the updatedAt of the NFT, therefore, the NFT get's updated only once and that's when it's created. This filter was created to sort by emotes -> this is irrelevant on BSX and we should hide it on both /snek and /bsx (I've checked just now and all of the Basilisk NFTs have updatedAt and createdAt are equal everywhere, regeradless of activity)
  2. Serial Number Ascending - in my opinion, this filter is more of a mystery than any added value -> I don't think people know what is this for and I don't think they care for having it. Oldest should be enough to sort by createdAt -> let's hide this on both /snek and /bsx

Originally posted by @petersopko in kodadot/nft-gallery#3138 (comment)

Fetch metadata

Currently for the speed of processing I turned off the metadata

Migration don't work since Firesquid update

I can't manage to create new migration file with series TABLE

Step:

  • remove old migrations
  • run npx squid-typeorm-migration generate
  • run npx squid-typeorm-migration create
  • run npx squid-typeorm-migration apply

nothing changed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.