kodadot / rubick Goto Github PK

View Code? Open in Web Editor NEW

9.0 3.0 15.0 2.12 MB

Implementation SubSquid indexer for Kusama NFTs

Home Page: https://github.com/kodadot/rubick

Shell 0.12% Dockerfile 0.42% TypeScript 88.91% JavaScript 9.62% Makefile 0.30% Just 0.63%

nft polkadot-js indexing graphql subsquid hacktoberfest kusama polkadot rmrk

rubick's Introduction

Rubick

Prerequisites

node 16.x
docker

Quickly running the sample

Example commands below use make(1). Please, have a look at commands in Makefile if your platform doesn't support it. On Windows we recommend to use WSL.

# 1. Install dependencies
npm ci

# 2. Compile typescript files
make build

# 3. Start target Postgres database and detach
make up

# 4. Apply database migrations from db/migrations
make migrate

# 5. Start the processor
make process

# 6. The command above will block the terminal
#    being busy with fetching the chain data, 
#    transforming and storing it in the target database.
#
#    To start the graphql server open the separate terminal
#    and run
make serve

# 7. Now you can see the resuls by visiting the localhost:4350/graphql

Project structure

Hydra tools expect a certain directory layout:

src/generated - model/server definitions created by codegen. Do not alter the contents of this directory manually.
src/server-extension - module with custom type-graphql based resolvers
src/types - data type definitions for chain events and extrinsics created by typegen.
src/mappings - mapping module.
lib - compiled js files. The structure of this directory must reflect src.
.env - hydra tools are heavily driven by environment variables defined here or supplied by a shell.

If you do not plan to extend GraphQl server you can delete server-extension module and then remove type-graphql and class-validator dependencies.

Dev flow

1. Define database schema

Start development by defining the schema of the target database via schema.graphql. Schema definition consists of regular graphql type declarations annotated with custom directives. Full description of schema.graphql dialect is available here.

2. Generate TypeORM classes

Mapping developers use TypeORM entities to interact with the target database during data processing. All necessary entity classes are generated by the squid framework from schema.graphql. This is done by running npx squid-typeorm-codegen command.

3. Generate database migration

All database changes are applied through migration files located at db/migrations. squid-typeorm-migration(1) tool provides several commands to drive the process. It is all TypeORM under the hood.

# Connect to database, analyze its state and generate migration to match the target schema.
# The target schema is derived from entity classes generated earlier.
# Don't forget to compile your entity classes beforehand!
npx squid-typeorm-migration generate

# Create template file for custom database changes
npx squid-typeorm-migration create

# Apply database migrations from `db/migrations`
npx squid-typeorm-migration apply

# Revert the last performed migration
npx squid-typeorm-migration revert

Setting up the project for resolvers

In case you just want to extend resolvers you don't index the whole project and just import the postgres database (last data 29.03.2023):

Contact @vikiival on Discord
docker-compose up db
docker exec -it rubick-db-1 psql -U postgres -d postgres -c "CREATE DATABASE squid;"
docker exec -i rubick-db-1 psql -U postgres -d squid < rubick.sql

⚠️ In case the command does not work, check if the container is called rubick-db-1.

Misc

For more details, please check out https://docs.subsquid.io.

rubick's People

Contributors

Stargazers

Watchers

Forkers

dzhelezov roileo ma-shulgin jarsen136 topcodebeast kngzhi prachi00 msghais swamitlsm zhengow preschian daiagi matehoo topdevbeast53 ashkan1987

rubick's Issues

Different result between magick & rubick

Why is there any difference in result? (216 vs 222)
Which one is up to date?

magick

query nftListLiked($account: String!, $first: Int!, $offset: Int) {
  emotes(
    filter: { caller: { equalTo: $account } }
    first: $first
    offset: $offset
    orderBy: ID_DESC
  ) {
    totalCount
    nodes {
      id
      value
      caller
      nft {
        id
        issuer
        collectionId
        name
      }
    }
  }
}

rubick

query nftListLiked($account: String!, $first: Int!, $offset: Int) {
  stats: emotesConnection(orderBy: id_DESC, where: { caller_eq: $account }) {
    totalCount
  }
  emotes(
    limit: $first
    offset: $offset
    orderBy: id_DESC
    where: { caller_eq: $account }
  ) {
    id
    value
    caller
    nft {
      id
      issuer
      name
      currentOwner
    }
  }
}

variables

{"first": 20, "offset": 0, "account": "FJMJLAVCv1KG3UXBh1Hr54VzHsEF4MQTC4nZrZUUucbEsKJ"}

Check if NFT is not already created under the same ID

In func mintNFT

passion feed resolver

SELECT DISTINCT ne.issuer
FROM nft_entity ne
WHERE ne.current_owner = $0
AND ne.current_owner != ne.issuer

Series insight shows strange data

kodadot/nft-gallery#2783

as @kkukelka pointed out in kodadot/nft-gallery#2786 the problem could be in the series resolver

MIN(NULLIF(ne.price, 0)) as floor_price

Possibility is that price is handled as string or something.
It would be nice to get some DB dump and play with that

ref issue kodadot/nft-gallery#2786

KodaDot v3

kodadot/nft-gallery#2650 (comment)

resolver to check how many NFTs are listed

components/rmrk/Gallery/CollectionItem.vue

Collections resolver

Create a new resolver that extend collectionEntity
make the query to hide empty collections (hide collection with all_burned/no nfts)

Expand series query with total emote count

Tracking issue for:
kodadot/nft-gallery#1175

We need to expand query in cache.ts :)

createdAt / updatedAt fields

CollectionEntity
NFTEntity

Add unit tests

Big Mahin is asking: "How do you sleep without tests?"

Possible combinations:

mocha + chain
uvu

Optional: Add Sinon

support date range series-insight

Reworks series insight schema to support dates :)
ref: https://github.com/kodadot/ideas/issues/24

Create UNLIST interaction

_contains filter is case sensitive

make insensitive filter when using _contains we might need to edit query schema

Fetch mime type and size from the asset

If the nft does not have mime type specified, we should fetch it from the pinata

Update Spotlight and Series queries to filter out burned

As @prachi00 pointed out. Burned NFTs are calculated into the spotlight and series.

https://github.com/kodadot/rubick/blob/main/src/mappings/utils/cache.ts

kodadot/nft-gallery#2740

Collection chart resolver

missing readme infos

npm run build
to regenerate lib/

Extend events with the current NFT owner

kodadot/nft-gallery#2635
kodadot/nft-gallery#2589

Make Interaction as entity

same as kodadot/magick#21

add md5 hash for nft

make resolver for md5hash to image?

Buy Event should take price as meta

same as kodadot/magick#12

User Entity

We are slowly getting to the tipping point where we need to replace user as String! with UserEntity

Would would it do?:

cache various metrics
calculating scores

Similarly to
kodadot/magick#19

@yangwao, @roiLeo please drop your thoughts

Offset metricks for kodadot

kodadot/nft-gallery#2869

minted meta today resolver

should return a list of metadata with image links for NFTs which were minted in some range

Expand series and spotlight cache with mini pulse chart

#62

Things we need to do:

move queries from the cache file to the src/server-extension/query/[id].ts
In schema.graphql:
create new HistoryEntity entity/json which will represent the day in the chart
extend Series entity with buyHistory: [HistoryEntity!]
extend Spotlight entity with sellHistory: [HistoryEntity!]
call the existing queries from #62 in cache.ts

connect them together

meta^meta

Add metadata to the existing NFT model

Write functions
Update CollectionEntity
Update NFTEntity
handle fetching

Series & Spotlight

kodadot/nft-gallery#1663

Spotlight:

Volume
Unique collectors
Sold?

Series:

Volume
% change
floor
buys

Add identity to indexer

leave for kodadot/hei

Volume & price as string

on SeriesEntity (& Spotlight) I found that volume is String?
Maybe it come from the query or maybe I'm wrong

Series resolver assets column

The number of assets in a collection is not the right one

Take a look on Kitty Paradise collection: /rmrk/collection/800f8a914281765a7d-KITTY, it show 293 listed nfts.
In series-insight, it show that it has 319 assets but I think it's more like the number of "BUY" events

SEO resolver

for SEO purposes we should expose the calls for fetching metadata and image from the NFT/Collection

nftImage :: String -> String
nftMeta :: String -> Metadata

migrate to minimark

The current implementation is OK, but we should really use https://github.com/vikiival/minimark

Resolver for Flipper table paging data

The flipper data needs to be provided by the server to avoid doing too much calculation on the client.

Originally posted by @Jarsen136 in kodadot/nft-gallery#2804 (comment)

Need to find out all past buyers' and seller's information from Interaction events.

request query: collectionId、offset、first

response data structure: (or may be other format)

 {
  user {
     nft {
       bought
       sold
       percentage
       updateTime
    }
    amount
    totalBought
    totalSold
    percentage
    updateTime
  }
  total
}

It will be better if the result is paginated

CurrentOwner is not valid

I try to request new interface via /rubick/006/graphql.

The new event structure do contain 'currentOwner' parm, but it seems strange.

For example:

for 'list'、'buy'、'mint'、 'send' event: currentOwner is always equals to caller

If currentOwner is always equals to caller, so why we need this new parms 'currentOwner'? 🤔

I think we need a previousOwner instead of this 'currentOwner', so we could get information of both buyer and seller from the 'buy' event.

cc @vikiival

Originally posted by @Jarsen136 in kodadot/nft-gallery#2822 (comment)

type Volume = {
  averagePrice: number
  averagePricePercentChange: number
  sales: number
  salesPercentChange: number
  volume: number
  volumePercentChange: number
}

{
  daily: {
    Volume
  },
  weekly: {
    Volume
  },
  monthly: {
    Volume
  },
  all: {
    Volume
  }
}

Pulse chart resolver

Since I have seen PR for pulse chart: kodadot/nft-gallery#2884
I realized that we need resolver for that.

Original issue: kodadot/nft-gallery#2195

May @KngZhi can provide data which he/she need

WIP:

SELECT * 
FROM nft_entity ne
JOIN event e on e.nft_id = ne.id WHERE e.interaction = 'BUY' and e.timestamp >= NOW()-30
GROUP BY ne.collection_id

Instance ran out of memory

Node ran out of the memory

Date: 27.01.2022

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0xa38a30 node::Abort() [node]
 2: 0x96e0af node::FatalError(char const*, char const*) [node]
 3: 0xbb7ebe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbb8237 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd74445  [node]
 6: 0xda4dde v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
 7: 0xdb0e16 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
 8: 0xd9cfaf v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
 9: 0xd9d228 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd8fb09 v8::internal::ItemParallelJob::Run() [node]
11: 0xdb2d70 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
12: 0xdb360c v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
13: 0xdb37d5 v8::internal::MarkCompactCollector::Evacuate() [node]
14: 0xdc57d1 v8::internal::MarkCompactCollector::CollectGarbage() [node]
15: 0xd81a98 v8::internal::Heap::MarkCompact() [node]
16: 0xd83588 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
17: 0xd869cc v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  (0, assert_1.default)(isHex(value))

    at decodeHex (/squid/node_modules/@subsquid/scale-codec-json/lib/util.js:205:26)
    at Codec.decode (/squid/node_modules/@subsquid/scale-codec-json/lib/index.js:41:45)
    at Chain.decode (/squid/node_modules/@subsquid/substrate-processor/lib/chain.js:99:43)
    at Chain.decodeCall (/squid/node_modules/@subsquid/substrate-processor/lib/chain.js:88:21)
    at SystemRemarkCall.get asV1020 [as asV1020] (/squid/lib/types/calls.js:25:32)
    at Array.handleRemark (/squid/lib/mappings/index.js:39:58)
    at /squid/node_modules/@subsquid/substrate-processor/lib/processor.js:195:50
    at /squid/node_modules/@subsquid/substrate-processor/lib/db.js:62:19
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '=='
}

Recheck resolver arguments

I think that I made some typos to the @Arg in resolvers.

We need to check the argument types.

interface Args {
  limit: bigint;
  offset: bigint
}

Write resolver for last events

Identity/ User entity (codename: hei)

I spoke with @dzhelezov today and he told me that you can specify from each block the handler should be active:

That means we can fetch the identities from the block 1 and system.remark as is (block 5 mil)

maybe @roiLeo wanna take a spin? 👀 (happy to provide support)

make pg_dump

Inb Thu session it would be good to sync whole rubick and make a dump