Filecoin storage and indexing for Tableland Basin

• basin-storage
• Table of Contents
• Background
• Development
  • Running
  • Deploying Function
    • Deploy Uploader function
      • Deploy Status Checker function
  • Run tests
• Contributing
• License

Tableland Basin is a secure and verifiable open data platform. The Basin Storage replicates data to Filecoin. It works in conjunction with basin-cli and basin-provider.

🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧

Basin Storage leverages GCP Cloud Functions and Clould Storage to create a data archiving pipeline to Filecoin network. The File uploader function is triggered by an event from Cloud Storage.

This repository includes a development server that acts as an event trigger during development.

Start the development server for testing Clould Functions locally. The required environment variables can be provided in uploader.env.yml and checker.env.yml.

make uploader-local

make checker-local

After the server is running, a mock cloud event can be triggered. It will execute the handler locally. In the following payload, we must have a real file path and a bucket for event simulation.

curl -X POST http://localhost:8080 \
-H "Content-Type: application/json" \
-H "ce-id: 1234567890" \
-H "ce-specversion: 1.0" \
-H "ce-type: google.cloud.storage.object.v1.finalized" \
-H "ce-time: 2020-08-08T00:11:44.895529672Z" \
-H "ce-source: //storage.googleapis.com/projects/_/buckets/tableland-entrypoint" \
-d '{
  "name": "feeds/2023-08-29/202308291525552525242120000000000-3ab461ed932d5f1c-1-2-00000000-employees-2.parquet",
  "bucket": "tableland-entrypoint",
  "contentType": "application/json",
  "metageneration": "1",
  "timeCreated": "2020-04-23T07:38:57.230Z",
  "updated": "2020-04-23T07:38:57.230Z"
}'

The checker function can be triggered by simply sending a POST request for example curl -XPOST localhost:8080.

make uploader-deploy

make checker-deploy

make test

PRs accepted.

Small note: If editing the README, please conform to the standard-readme specification.

MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors