• Remove bower and various global dependencies
• Improve gulp process
• Improve process to run full stack locally

Running Terraform in CI would solve a few things:

• Avoiding version mismatch issues on local machines
• Allowing anyone to work on the infra without distributing AWS access keys
• Providing a way to notify us when the expected state differs from the actual state

Something like https://www.runatlantis.io/ would be good here, to avoid building something similar ourselves.

This will largely be done in Terraform, with reusable modules we can use to complete #3.

Done in #72:

• ECS cluster backed by t4g.micro instances for running Node.js apps (API, Game, SP, Worker)
• S3 bucket for static assets (See #61)
• Free-tier db.t4g.micro RDS instance for Postgres
• Free-tier cache.t3.micro Elasticache instance for Redis
• Free-tier CloudFront distribution (See #61)
• Free-tier Application Load Balancer (ALB) with SSL certs via AWS Cert Manager

Since so much of the above qualifies for the AWS Free Tier, we should be able to provision an always-on staging environment very cheaply. The ECS instances will cost $0.0084 per hour, or $0.0025 if we can leverage Spot Instance pricing (we get a 2 minute warning before Spot Instances are reclaimed; might be suitable for API and Worker). The S3 bucket will have some minor costs (maybe a few dollars per month; depends on CDN configuration and user traffic). Everything else will be completely free.

Done in #75:

• Getting our Node.js apps into ECR
• Minimize Docker image size and ECR costs (We get 500MB of private storage free for 12 months, but each image is currently over 2GB; public storage is free up to 50GB)
• Getting ECS to pull images from ECR

Done in #79:

• Enabling logs in staging
• Passing environment variables into deployed services, e.g. NODE_ENV=staging
• Adding support for retrieving secrets in ECS tasks, e.g. Postgres connection string (which includes a password)

Summary

Spot Instances are 70% cheaper than on-demand instances, and they're perfect for any workload which can be shut down within 2 minutes of a warning signal. This includes API and Worker, which are largely stateless (but not the Game/SP WebSocket servers, since those should remain available for the full duration of games).

Adding Spot Instance support in ECS and using it for API+Worker will allow us to get more CPU and Memory for user traffic while still saving money in both staging and production deployments.

Summary

Noticed this error + stack trace while playing a practice game locally:

Uncaught TypeError: WebGLRenderingContext.bindBuffer: Argument 2 is not an object.
    postStep vendor.js:52124
    postStep vendor.js:51396
    update BaseParticleSystem.js:1337
    resetSystem BaseParticleSystem.js:1163
    updateSourceToTarget BaseParticleSystem.js:584
    setSourceScreenPosition BaseParticleSystem.js:250
    onResize BattleMap.js:303
    triggerEvents eventbus.js:156
    trigger eventbus.js:89
    resize Scene.js:270
    onResize application.coffee:3489
    later vendor.js:16021
    setTimeout handler*later vendor.js:16017
    setTimeout handler*later vendor.js:16017
    setTimeout handler*_.debounce/< vendor.js:16033
    dispatch vendor.js:4676
    handle vendor.js:4360
vendor.js:52124:8

I'm not sure what the impact is, yet. It seems specific to BattleMap.js.

Impacted services

[x] The frontend app
[ ] The game servers
[ ] Game builds or automation scripts
[ ] Tests or Github Actions
[ ] Infrastructure (Terraform)

Steps to reproduce

1. Play a practice game; should appear in the console.

Environment information

• Operating system: Mac
• Browser: Firefox
• Browser version: 105.0.1

Summary

Noise images (such as noise.png) seem to be intended to make subtle visual effects. However, they're currently appearing without transparency when starting or finishing games.

We should determine why this is happening and fix it. It may be related to the pixelFormat values in the app.

Impacted services

[x] The frontend app
[ ] The game servers
[ ] Game builds or automation scripts
[ ] Tests or Github Actions
[ ] Infrastructure (Terraform)

Steps to reproduce

1. Start a practice game.
2. See a big square of noise.png on the loading screen.

Environment information

• Operating system: Windows and Mac
• Browser: Firefox
• Browser version: 105.0.1

When completing the tutorials, the buttons for challenges appear to be missing their labels.

Screenshot:

Summary

We should document our plans and priorities for the project.

Not sure exactly what is happening here, but login seems to hang until I refresh the game client, at which point I get to the main menu as usual.

If others can repro this, we should track it down and fix it.

Many dependencies are outdated, have security issues, or are no longer maintained. We'd like to update dependencies while balancing need to work around API compatibility changes.

To Do:

• Remap ports on services to account for SSL, hosting Game+SP on the same LB, etc.
• Provion Game and Worker services in ECS

Summary

In focusing on the web deployment, there have been some breaking changes to the desktop build. We should:

• [Done] Update its dependencies as needed
• [Done] Re-enable its linters and tests
• [Done] Fix its Gulp/Yarn tasks/scripts
• [Moved] Add support for win32-x64 and darwin-amd64 (so far only finished darwin-x64)
• [Done] Fix broken loading.gif reference
• [Done] Fix broken FontAwesome references (should be CDN, but is being treated like a local file)
• [Moved] Github Action for desktop builds (can use on: workflow_dispatch for manual triggers and add on: release later)

There are likely some improvements we can make to how we cache node_modules and Yarn data in Docker. We should determine the current best practices for these and make sure we're following them in the repo.

This might save quite a bit of time in local development iteration.

The single-player server (server/single_player.coffee) starts and runs, but playing local games is not yet a bug-free experience.

Users should be able to clone this repo and run docker compose up, then open http://localhost:3000 in their browser to play all single player content.

Describe the bug
Migration code is unable to resolve relative paths to app/:

duelyst-migrate-1  | Error: Cannot find module 'app/sdk/quests/questBeginnerWinPracticeGames'
duelyst-migrate-1  | Require stack:
duelyst-migrate-1  | - /app/app/sdk/progression/newPlayerProgressionHelper.coffee
duelyst-migrate-1  | - /app/server/migrations/20160314130406_migrate_ftue_data_for_all_users.js

To Reproduce
Run docker compose up migrate

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository

Summary

Duelyst previously used Mandrill to send outgoing emails: https://github.com/open-duelyst/duelyst/blob/main/server/mailer.coffee#L29L34

This has since been disabled, but we should re-enable this by using AWS SES: https://aws.amazon.com/ses/

This service is free for the first 2,000 emails sent per month, and $0.10 per 1,000 emails beyond that.

Emails were previously sent for the following purposes:

• Email verification for new account registration
• Error reporting in production
• Sending user reports
• Password resets

Of these, the first priority is email verification + password resets. User reports can be considered later, and error emails are not needed (since we route error logs to CloudWatch now).

Bower is a package manager for web assets like HTML, CSS, JS, fonts, and images. Bower is still maintained, but its developers recommend instead using Yarn and Webpack.

Some of our existing Gulp tasks may also be a better fit for Webpack.

Also, see if we can better target tests based on the contents of a PR (e.g. only linting TF code in TF PRs).

We currently have yarn cdn:upload:staging, which uploads all of dist/src including 450MB+ of resources to S3.

We should add an alternative script which uploads only the rebuilt files (~13MB):

• duelyst.css
• duelyst.js
• index.html
• register.html
• vendor.js

We may also want to automate CloudFront cache invalidations (first 1,000 validations per month are free).

Pulling this work out of #69 and #3 to reduce their scope.

We should automate the process of deploying OpenDuelyst when a new release is published on Github.

This can be done by creating an Actions workflow for the release event, and having that action publish container to ECR. If ECS is using the latest version of the container, I believe it will be able to pull it down automatically. If not, it will still be doable with a small amount of work.

We have several files with overrides set in .eslintrc.json:

    {
      "files": ["app/**/*.js"],
      "rules": {"camelcase": ["warn"]}
    },
    {
      "files": ["test/**/*.js"],
      "rules": {"no-unused-expressions": ["off"]}
    }

We should remove these overrides and resolve the underlying linting issues.

ES6 and / or TypeScript mostly makes CoffeeScript no longer necessary. Let's try running codebase through decaffeinate.

https://github.com/decaffeinate/decaffeinate

We currently have around 1.2GB of images, sounds, and fonts in the app/resources and app/original_resources directories, which results in slower git operations and longer workflow durations in CI.

We should find a way to isolate these binary files from the core repo to keep things lean and fast. One solution here could be simply moving the resources to another repo (e.g. open-duelyst/resources), configuring .zip and .tar.gz releases for that repo, and then having the core repo pull those in at build time.

When loading the app, these errors appear in the console:

GET https://duelyst.freetls.fastly.net/loading.gif 500 (Internal Server Error)
GET http://localhost:3000/favicon.ico 404 (Not Found)

We should add these files to the web assets. I don't see these files in git, though.

Summary

We have an ALL_CARDS_AVAILABLE config setting, but its usage is currently commented out in the app: https://github.com/open-duelyst/duelyst/blob/d861bca/app/ui/views2/collection/cards_collection.js#L308L325

We should re-enable this so we can grant full collections in local development and staging.

As of #47, we are running integration tests in Github Actions. However, we are only running a subset of the available tests. We should repair the following tests and enable them in .github/workflows/integration_tests.yaml:

yarn test:integration:data_access: All of these tests are failing due to misuse of Knex in the Users module:

  14) users module
       "before all" hook in "users module":
     TypeError: knex(...).where(...).first(...).bind is not a function
    at Function.UsersModule.createNewUser (server/lib/data_access/users.coffee:138:4)

yarn test:integration:firebase: In order to run Firebase tests, we need to create a Firebase Realtime Database which can be used by Github Actions.

There are also tests which are commented out and need to be reworked, either because they were broken or because they were too slow:

% git grep -l 'Test disabled'
test/integration/achievements/collector_supreme.js
test/integration/achievements/home_turf.js
test/integration/achievements/seven_sisters.js
test/integration/data_access/cosmetic_chests.js
test/integration/data_access/inventory.js
test/integration/data_access/migrations.js
test/integration/data_access/rift.js
test/integration/data_access/users.js

Summary

The AWS Free Tier for ALB, RDS, and ElastiCache is limited to 12 months. After this time, they cost the following amounts:

• ALB: $16.43/mo for base cost (less than 2 summed LCU / $0.01 per month for traffic)
• RDS: $11.68/mo for base db.t4g.micro cost (plus $2.30/mo for storage, $1.90/mo per backup)
• ElastiCache: $11.68/mo (cache.t4g.micro, on-demand) migrated to ECS for $0.90/mo

We can find cheaper alternatives for the staging environment.

Note: Our usage of ECS, EC2, S3, CloudFront, KMS, and SSM Parameter Store are not subject to Free Tier expiration. They're either always free, or cost mitigation work has already been done. These will continue to total $6/mo.

We currently have a small note here pointing to open issues: https://github.com/open-duelyst/duelyst#contributing

We should add a larger contributor onboarding guide with more information around processes, development environments, editor configuration, etc.

Duelyst uses the Firebase Realtime Database product for communication between the game client and game backends.

We are currently using version 2.0.3 of the firebase package from NPM, which was released in early 2015. We should upgrade to the latest version here: https://www.npmjs.com/package/firebase

Much has changed in 7 years: following Google's acquisition of Firebase in 2014, the service has been incorporated into its Google Cloud offerings. The largest change is the deprecation of Firebase "legacy tokens" in favor of Google Cloud "service accounts". Duelyst currently relies on legacy tokens in the game client, but legacy tokens may no longer be created in new Firebase projects.

One complication for this work is the usage of Browserify in the Gulp build process, so any client changes need to maintain compatibility with this flow.

The Firebase dependency has already been updated for the game backend, which is now using service accounts.

We need to configure various pipelines:

• linting / running unit tests / etc
• managing PRs
• managing deployments for the reference deployment

An old hbs dependency is causing the following warning logs during yarn install:

warning [email protected]: The engine "ender" appears to be invalid.
warning [email protected]: The engine "ender" appears to be invalid.

These are dependencies of walk, which is in turn a dependency of hbs. Upgrading hbs should resolve these.

We currently have a terraform/iamPolicy.json file with an example IAM policy which can be used to run Terraform. This is out of date now that I've added a ton of new resources, so I need to update this once we have a final set of permissions required to manage a deployment.

Several unit tests have been disabled until we have time to go back and update them. search for Test disabled in the repo to find them:

% git grep -l 'Test disabled'
test/unit/sdk/ai/ai.js
test/unit/sdk/ai/bosses.js
test/unit/sdk/cards/bloodstorm/faction6.js
test/unit/sdk/cards/bloodstorm/neutrals.js
test/unit/sdk/cards/core/faction2/artifacts.js
test/unit/sdk/cards/core/faction3/minions.js
test/unit/sdk/cards/core/faction5/spells.js
test/unit/sdk/cards/core/neutrals/epics.js
test/unit/sdk/cards/coreshatter/faction1.js
test/unit/sdk/cards/misc/bloodborn_spells.js
test/unit/sdk/cards/monthlies/month7.js
test/unit/sdk/cards/monthlies/month9.js
test/unit/sdk/cards/monthlies/seven_sisters.js
test/unit/sdk/cards/shimzar/battle_pets.js
test/unit/sdk/cards/shimzar/faction1.js
test/unit/sdk/cards/shimzar/faction2.js
test/unit/sdk/cards/shimzar/faction4.js
test/unit/sdk/cards/shimzar/faction5.js
test/unit/sdk/cards/unity/faction5.js
test/unit/sdk/cards/unity/faction6.js
test/unit/sdk/cards/wartech/faction2.js
test/unit/sdk/cards/wartech/faction3.js
test/unit/sdk/cards/wartech/faction4.js
test/unit/sdk/cards/wartech/faction5.js
test/unit/sdk/game/basic.js

We should revisit these and turn them back on once we get them working.

[Note from Will: This issue is tracking a stable "production" environment. See #69 for staging.]

Summary

AWS is predicting 62,000 GET requests to S3 in October, which is only $0.20 or so, but this is much higher than it should be. Let's enable the CloudFront cache policy for resources with a minimum TTL of one day.

I don't have a significant amount of frontend expertise, but I know that Webpack is one of the more popular module bundlers for frontend assets.

Would this be beneficial for the project? We should investigate this and see if it's worth switching over.

Summary

In order to avoid needing to deal with sender reputation, user data privacy, and legal compliance, let's remove email functionality from the user signup flow.

There is a minor vulnerability in our version of moment: https://security.snyk.io/vuln/SNYK-JS-MOMENT-2440688)

Upgrading to v2.29.2 will resolve this, but we have several date strings which don't satisfy RFC2822 or ISO formats (which raise errors in the new version).

We should update these strings and then upgrade moment.

See http://momentjs.com/guides/#/warnings/js-date/ for more info.

Knex.js, our SQL query builder, has a few minor vulnerabilities in the current 0.19.5 version:

  ✗ Prototype Pollution [High Severity][https://security.snyk.io/vuln/SNYK-JS-UNSETVALUE-2400660] in [email protected]
    introduced by:
    [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
    [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
    [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
    [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
    [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]

We should upgrade to 0.95.0 by following this guide: https://github.com/knex/knex/blob/master/UPGRADING.md#upgrading-to-version-0950

This may require changes to code in the cli, scripts, server, test, and worker directories.

Nearly every function is server/game.coffee is duplicated in server/single_player.coffee:

dnsHealthCheck = () ->
server = http.createServer (req, res) ->
server.listen config.get('game_port'), () ->
savePlayerCount = (playerCount) ->
saveGameCount = (gameCount) ->
getConnectedSpectatorsDataForGamePlayer = (gameId,playerId)->
onGamePlayerJoin = (requestData) ->
onGameSpectatorJoin = (requestData) ->
onGameLeave = (requestData) ->
onGameEvent = (eventData) ->
onGameDisconnect = () ->
playerLeaveGameIfNeeded = (socket, silent=false) ->
spectatorLeaveGameIfNeeded = (socket) ->
destroyGameSessionIfNoConnectionsLeft = (gameId,persist=false)->
tearDownSpectateSystemsIfNoSpectatorsLeft = (gameId)->
clearDisconnectedPlayerTimeout = (gameId) ->
startDisconnectedPlayerTimeout = (gameId,playerId) ->
onDisconnectedPlayerTimeout = (gameId,playerId) ->
restartTurnTimer = (gameId) ->
stopTurnTimer = (gameId) ->
onGameTimeTick = (gameId) ->
restartSpectatorDelayedGameInterval = (gameId) ->
stopSpectatorDelayedGameInterval = (gameId) ->
onSpectatorDelayedGameTick = (gameId) ->
flushSpectatorNetworkEventBuffer = (gameId) ->
_logSpectatorTickInfo = _.debounce((gameId)->
emitGameEvent = (fromSocket,gameId,eventData)->
initGameSession = (gameId,onComplete) ->
initSpectatorGameSession = (gameId)->
onBeforeRollbackToSnapshot = (event) ->
onStep = (event) ->
onInvalidAction = (event) ->
subscribeToGameSessionEvents = (gameId)->
unsubscribeFromGameSessionEvents = (gameId)->
afterGameOver = (gameId, gameSession, mouseAndUIEvents) ->
shutdown = () ->

We should deduplicate as much of this as possible to make maintenance easier.

Documentation regarding how cards work and how to add, remove, and modify them would be very helpful

ECS deployments currently have a few rough spots:

1. Need to iron out the "blessed" flow for versioning and publishing ECR images, since publishing a new image with an existing tag doesn't always get automatically deployed by ECS.
2. ECS autoscaling in staging doesn't do much now, since we're only running one instance, but for production we'll want capacity to be available so we can start containers when they are requested.
3. ALB target registration and deregistration take a long time (90+ seconds), which has been leading me to manually register targets in the UI instead of waiting.
4. Routing ECS event logs (container started, stopped, deregistration, exit codes, etc.) to CloudWatch Logs may be useful.

We should iron these out to make it easier for us to deploy the staging environment (or in the future, the production environment).

Even after a full yarn build, I see errors in the game client (browser console) regarding missing fonts:

GET http://localhost:3000/fonts/fontawesome-webfont.woff?v=4.2.0 net::ERR_ABORTED 404 (Not Found)
GET http://localhost:3000/fonts/fontawesome-webfont.ttf?v=4.2.0 net::ERR_ABORTED 404 (Not Found)

Summary

Following the changes over the last few weeks, the game appears to function in Firefox on Android, but the viewport is not correctly sized.

It may not be too much work to get the window scaling down so the game can be fully played on mobile.

Since packaging and releasing Android & iOS apps is a pain, let's focus on mobile web initially.

We're currently using [email protected], which has poor support for connection timeouts.

Replacing this with [email protected] fixes the above, but requires some changes. For example, the zrevrankAsync function does not exist, so we need to use zrevrank instead and remove the .then(result) chain.

Alternatively, upgrading node-redis may help with these issues.

See https://github.com/luin/ioredis/wiki/Migrating-from-node_redis

When loading the browser client in the development environment, all assets are served over Express in the API process.

This results in 75MB of additional data transfer to users: 47MB for duelyst.js, vendor.js, and duelyst.css, and 28MB for additional resources (music, images, and more).

We already have code for serving these from S3 for the production environment, so we should make sure this works and make use of it.

This will reduce both cloud provider costs and CPU utilization on the API process.

Summary

I didn't add this when I added JS linting, since I thought we'd decaffeinate right away.

Let's go ahead and add CoffeeScript linting in CI until that's done.

Description

There are client errors preventing us from reaching the main menu. Based on checking out one commit at a time and rebuilding, I believe this started happening in 63cc664 (Part of linting changes in #24 .

The old description below was resolved in #30

Old Description (process.env.FIREBASE_URL issue)

In c4d3352, just before #15 was merged, the app will start as expected when accessing http://localhost:3000 after running docker compose up. This is because despite process.env being empty, the following code succeeds:

# app/application.coffee
Firebase = window.Firebase = require 'firebase'
minBrowserVersionRef = new Firebase(process.env.FIREBASE_URL).child("system-status").child('browsers')

After #15, this results in an error when loading the client:

Unhandled rejection Error: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com 

Once #52 and #54 are resolved, running yarn snyk will succeed, allowing us to enable it in CI (via yarn snyk). We'll need to add a token to Github Actions Secrets for this (with a snyk auth <token> step).