rivet-gg / rivet Goto Github PK

• When we delete a server, salt apply fails because the server is no longer reachable
• Add a null_resource with a shell script in pools that calls salt-key -D on any servers not present in Terraform
  • Use triggers to automatically calls this when the servers change

_{From SyncLinear.com | SVC-2367}

We need to analyze what chunks take up the most time.

Off the top of my head, the big ticket items:

• Generating Nix config
• Building bolt
• Building services

We should be able to use tools like Flax or Cachix for Nix & sscache for Bolt & services. Potentially Cachix can solve the Bolt issue too if we build it with the shell.

_{From SyncLinear.com | SVC-2478}

The docs have migrated to rivet.gg/docs, but the API responses are still outdated.

Thanks stdint.h for reporting this

_{From SyncLinear.com | SVC-2659}

_{From SyncLinear.com | SVC-2555}

Description

Allow for adding valid domains to each namespace for where requests can originate from.

Motivation

Developers should be able to authenticate requests using domain-based authentication from their own custom file servers

Workaround

Developers doing this right now need to use their own public namespace token.

Context

https://discord.com/channels/822914074136018994/1126154788455858288/1126404094014275634

_{From SyncLinear.com | SVC-2663}

backend is already implemented, need to add a turnstile key here and hook it up

_{From SyncLinear.com | SVC-2620}

_{From SyncLinear.com | SOC-21}

_{From SyncLinear.com | SVC-2404}

This may not be trivial to do. Ping @NathanFlurry and @maxknerrichmckenzie if you get stuck here.

_{From SyncLinear.com | SOC-6}

_{From SyncLinear.com | SVC-2411}

_{From SyncLinear.com | SOC-9}

_{From SyncLinear.com | SVC-2587}

_{From SyncLinear.com | SOC-27}

_{From SyncLinear.com | SOC-5}

• Needs to consent on bolt init
• Add a property no_analytics on namespace TOML
• Need some way of storing persistence
  • Maybe include a "cluster id" in the namespace toml?
• Need to figure out what to send
• Want periodic usage of server resources, lobbies, games, players, etc once every hour
• Need to send setup events
• Need to send the configs in use

_{From SyncLinear.com | SVC-2479}

_{From SyncLinear.com | SVC-2586}

_{From SyncLinear.com | SOC-19}

• Needs to consent on bolt init
• Add a property no_analytics on namespace TOML
• Need some way of storing persistence
  • Maybe include a "cluster id" in the namespace toml?
• Need to figure out what to send
• Want periodic usage of server resources, lobbies, games, players, etc once every hour
• Need to send setup events
• Need to send the configs in use

_{From SyncLinear.com | SVC-2479}

Symptoms

Around once per month, SQLx will leak connections and stop being able to make new requests.

Grafana snapshot

The following services are affected:

• job-run-cleanup
• job-run-nomad-monitor-eval-update
• job-run-nomad-monitor-alloc-plan
• job-run-nomad-monitor-alloc-update

Workaround

• Manually restart the allocations of the affected jobs once per day

_{From SyncLinear.com | SVC-2619}

Motivation

We need to document how to deploy a production instance of Rivet for self-hosted. At the moment, we only document how to set up a dev server.

Questions

• Should we document just standalone servers or clusters too?
• How much does this differ from local development?
• Should we look at using prebuilt images?

_{From SyncLinear.com | SVC-2590}

Motivation

Booting new lobbies right after deploying a new version requires a long wait time because the node has to download the image directly from S3.

Prewarming the cache will save the image to the cache before trying to boot any lobbies.

This is blocked by using consistent_hash in ATS because this will also change the ATS architecture.

Implementation

• List all ATS nodes from Consul for the version's regions
• Make a request for the asset
• Wait for it to complete & handle failure gracefully

_{From SyncLinear.com | SVC-1666}

_{From SyncLinear.com | SVC-2411}

Motivation

Helpful to know what the real IP is that's being connected. You can get this from x-forwarded-for, but it's not immediately apparently how to parse this and which is the real IP.

Implementation

• Add a rvt-connecting-ip similar to cf-connecting-ip

Blocking

Cannot be implemented in Traefik ad the moment traefik/traefik#5036

_{From SyncLinear.com | SVC-2625}

_{From SyncLinear.com | SOC-15}

_{From SyncLinear.com | SOC-75}

Motivation

It's common for developers to either bring in custom hardware, non-standard hardware (e.g. OVH's non-standard configurations), or use on-prem hardware.

MVP

• Create an install script to run on the machine
• Add a way in the namespace config to register the machine

Challenges

• Issuing Nebula certs
• Negotiating salt-key approval

_{From SyncLinear.com | SVC-2589}

Also look at HouseWatch

_{From SyncLinear.com | SVC-2588}

• In production, uploads are sent directly to B2
• In development, uploads are sent to a local Minio instance, which is currently proxied by Cloudflare & Traefik
• We need an unproxied port to make this work
• Unsure if we'll run in to trouble with the Traefik TLS certs

_{From SyncLinear.com | SVC-2504}

• This is the only source of hard panics we don't handle yet
• Easy to exploit
• Clippy will show warnings for this

_{From SyncLinear.com | SVC-2029}

Requirements

• Make getting backend & hub up and running as quickly as possible
• Allow running hub independently without the backend

Options

• What we do now: run webpack for the other repo and manually set BASE_URL
• Automatically deploy to Cloudflare Pages from backend Terraform
• Serve the repo through Traefik with a local-built copy
• Serve the repo through Traefik with a webpack hot reload server

Other notes

• Need to find a simple way to merge hub_origin_regex and hub_origin

_{From SyncLinear.com | SVC-2583}

Recreate issue

• Open hub in Chrome Incognito
• Register your account
• Page will refresh with a new user account

_{From SyncLinear.com | SOC-72}

• Mention having to run bolt up again

_{From SyncLinear.com | SVC-2647}

Currently broken @

_{From SyncLinear.com | SOC-25}

Motivation

Validate that our documentation is consistent between all code styles.

Requirements

• Update shell.nix to include the Prettier CLI
• Create a new CI workflow named format.ymlfor this
• Use prettier --check on all Markdown files:
  • docs/
  • errors/
  • README.md

_{From SyncLinear.com | SVC-2569}

Problem

Currently, each ing-job cluster has a different config for each region. There's no way to see the dashboard for a specific region.

Solution

• Create unique Cloudflare applications for each region

_{From SyncLinear.com | SVC-2584}

https://cockroach-http.rivet.gg/#/statement/false/3963938479943920492?appNames=

_{From SyncLinear.com | SOC-23}

_{From SyncLinear.com | SOC-23}

_{From SyncLinear.com | SVC-2565}

_{From SyncLinear.com | SOC-10}

Motivation

Our current caching system doesn't scale since each Traffic Server node will have to cache a large chunk of the requests individually.

If we can consistently route our requests to the correct ATS nodes based on the hash of the request, we'll be able to get consistent cache hits.

Potential Problems

• We still need redundancy, so we'll still need to have a fallback node for each hash unless ATS allows us to have a fallback if the parent is down
• We need to have a consistent number of buckets. Changing the number of buckets is potentially error prone.

Links

• https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html

_{From SyncLinear.com | SVC-824}

_{From SyncLinear.com | SOC-7}

_{From SyncLinear.com | SOC-15}

Examples:

• SendGrid
• Turnstile
• Support emails

_{From SyncLinear.com | SVC-2483}

Motivation

Copying and pasting services is a pain.

Steps

• Add back create command & templates: https://github.com/rivet-gg/backend-old/commit/0cd875d39e9463b4233170720e6a6eb528102cce
• Update the create command to prompt for the package it belongs in (maybe?)
• Write an internal doc on how to create new services & link it in the README.md

_{From SyncLinear.com | SVC-2585}

_{From SyncLinear.com | SVC-2644}

Likely related to what was causing SOC-25

_{From SyncLinear.com | SOC-18}

WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition.

_{From SyncLinear.com | SVC-2358}

https://discord.com/channels/822914074136018994/1111938029947469874/1112068023415746732

_{From SyncLinear.com | SOC-8}

Motivation

ATS caching was disabled because of a sporadic bug that would cause it to return an empty response under load.

This was likely caused by an underlying issue with B2 that we've seen elsewhere returning invalid results.

When this is enabled again, this will speed up image pulls & reduce S3 fees significantly.

Implementation

• Switch to S3
• Add back regional ATS node

_{From SyncLinear.com | SVC-2552}

_{From SyncLinear.com | SOC-44}

_SOC-45

_{From SyncLinear.com | SOC-17}