This is the Fetch ledger SubQuery project, an indexer for the Fetch network.
git submodule update --init --recursiveyarn
# install submodule dependencies
(cd ./subql && yarn)yarn codegenyarn build
# build submodule
(cd ./subql && yarn build)yarn start:dockerpipenv installNote: end-to-end tests will truncate tables in the DB and interact with the configured fetchd node.
pipenv run python -m unittest discover -s ./tests/e2epipenv run python -m unittest tests.e2e.entities.<module>(see: pipenv run python -m unittest --help for more)
The SubQuery node and GraphQL API services have been instrumented using open telemetry. This enables end-to-end tracing support for both services, which covers querying Fetchd RPC to inserting into the DB and then reading from the DB to respond to an end-user GQL query.
To run the tracing stack locally, start the docker composition using the tracing-compose.yml file:
docker compose -f ./tracing-compose.yml up -dThe tracing composition is notably more substantial than the conventional stack and will take a bit longer to start-up. Once running, you can point your browser to localhost:3301 to access the SigNoz dashboard.
(see: SigNoz documentation for more)
This repository uses graphile-migrate CLI to manage database migrations.
Install global npm dependencies:
npm install -g graphile-migrate plv8ifyGiven that you already have a database with some, potentially outdated, schema, you can catch up to the latest state by running all committed but unapplied migrations:
graphile-migrate migrate(see: graphile-migrate README for more)
When introducing schema change which adds an entity / table, it is currently most convenient to allow the subquery node to initially generate any new tables (including indexes and constraints). This schema can then be dumped for use in the accompanying migration.
The current schema sql file can be generated from an existing DB schema (optionally, including data) via pg_dump.
Package scripts are available for dumping the schema only or for dumping the schema plus any data present:
yarn db:dump:schema
# OR
yarn db:dump:allAdditional arguments may be forwarded to the underlying pg_dump command by appending -- <additional args / flags> when running package scripts (see: npm-run-script docs). For example:
# Dumps schema only from blocks and messages tables
yarn db:dump:schema -- -t blocks -t messagesIn some cases, pg_dump may not export a relevant DB object; for example, enums.
In these cases it is necessary to manually extract the relevant data from the DB and incorporate it into the initial_schema.sql file.
In the case of enums, the following queries expose the relevant records, the results from which can be re-written as a COPY or INSERT statements into the respective tables from which they came:
# List enum types
SELECT oid, typname FROM pg_type WHERE typcategory = 'E';
# List values for a particular enum
SELECT * from pg_enum where enumtypid = <pg_type.oid>;
It is not necessary to add any TypeScript source as part of any migration (e.g. 000002). In the event that the migration is too complex to be easily reasoned about it in SQL, it may be more straightforward to use the plv8ify workflow:
-
Write a migration in ./migrations/current.ts which exports a single function that kicks off the migration (see: plv8 docs > built-ins).
-
Transform the current typescript migration function to a .sql function:
yarn plv8ifyThis writes the generated SQL to ./migrations/current.sql.
-
Since we're using a non-default schema name, it's prudent to prepend the following preamble to current.sql:
CREATE SCHEMA IF NOT EXISTS app; SET SCHEMA 'app'; -
Lastly, in order for the migration function to execute when the migration is applied, append the following to current.sql (substituting labels identified by surrounding brackets with their respective values):
SELECT * from <migration function>([arg, ...]);
Once you're ready to test / apply the migration:
graphile-migrate commitIf it was successful, graphile-migrate will have moved the contents of current.sql (and reset it) into a file named by the next number in the migration count sequence.
(see: graphile-migrate README for more)
If the plv8ify workflow was used, until it is automated, it is conventional to manually move the contents of the current.ts file into a file in migrations/src named after its respective generated SQL file. It is against convention to update any import paths as a result of this move.
When things aren't going perfectly, plv8.elog is extremely useful for getting more detail out of the running migration.
Be sure to be looking at the postgres service's logs (as opposed to the error message returned by graphile-migrate):
docker compose logs -f postgres
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent