Falconeri runs on a pre-existing Kubernetes cluster, and it allows you to use Docker images to transform large data files stored in cloud buckets.
For detailed instructions, see the Falconeri guide.
Setup is simple:
falconeri deploy
falconeri proxy
falconeri migrateRunning is similarly simple:
falconeri job run my-job.jsonNote that falconerid has a complete REST API, and you don't actually need to use the falconeri command-line tool during normal operations. This is used internally at Faraday, and it should be fairly self-explanatory, but it isn't documented.
First, you'll need to set up some development tools:
cargo install just
cargo install cargo-deny
cargo install cargo-edit
# If you want to change the SQL schema, you'll also need the `diesel` CLI. This
# may also require installing some C development libraries.
cargo install diesel_cliNext, check out the available tasks in the justfile:
just --listFor local development, you'll want to install minikube. Start it as follows, and point your local Docker at it:
minikube start
eval $(minikube docker-env)Then build an image. You must have docker-env set up as above if you want to test this image.
just imageNow you can deploy a development version of falconeri to minikube:
cargo run -p falconeri -- deploy --developmentCheck to see if your cluster comes up:
kubectl get all
# Or if you have `watch`, try:
watch -n 5 kubectl get allRunning the example program is necessary to make sure falconeri works. First, run:
cd examples/word-frequenciesNext, you'll need to set up an S3 bucket. If you're at Faraday, run:
# Faraday only!
just secretIf you're not a Faraday, create an S3 bucket, and place a *.txt file in $MY_BUCKET/texts/. Then, set up an AWS access key with read/write access to the bucket, and save the key pair in files named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Then run:
# Not for Faraday!
kubectl create secret generic s3 \
--from-file=AWS_ACCESS_KEY_ID \
--from-file=AWS_SECRET_ACCESS_KEYThen edit word-frequencies.json to point at your bucket.
Now you can build the worker image using:
# This assumes you previously ran `just image` in the top-level directory.
just imageIn another terminal, start a falconeri proxy command:
just proxyIn the original terminal, start the job:
just runFrom here, you can use falconeri job describe $ID and kubectl normally. See the guide for more details.
For now, this process should only be done by Eric, because there are some semver issues that we haven't fully thought out yet.
First, edit the CHANGELOG.md file to describe the release. Next, bump the version:
just set-version $MY_NEW_VERSIONCommit your changes with a subject like:
$MY_NEW_VERSION: Short descriptionYou should be able to make a release by running:
just MODE=release releaseOnce the the binaries have built, you can find them at https://github.com/faradayio/falconeri/releases. The CHANGELOG.md entry should be automatically converted to release notes.
We use diesel as our ORM. This has complex tradeoffs, and we've been considering whether to move to sqlx or tokio-postgres in the future. See above for instructions on install diesel_cli.
To create a new migration, run:
cd falconeri_common
diesel migration generate add_some_table_or_columnsThis will generate a new up.sql and down.sql file which you can edit as needed. These work like Rails migrations: up.sql makes the necessary changes to the database, and down.sql reverts those changes. But in this case, migrations are written using SQL.
You can show a list of migrations using:
diesel migration listTo apply pending migrations, run:
diesel migration run
# Test the `down.sql` file as well.
diesel migration revert
diesel migration runAfter doing this, edit falconeri_common/src/schema.rs and revert any changes which break the schema, and any which introduce warnings. You will probably also need to update any corresponding files in falconeri_common/src/models/.
Migrations will be compiled into the server and run on deploys, as well.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent