Git Product home page Git Product logo

bcgov / digital_marketplace Goto Github PK

View Code? Open in Web Editor NEW
12.0 11.0 15.0 12.6 MB

The intent of this development is to build a product that will support digital procurement needs for the BC Government including services such as, but not limited to, Sprint With Us, Code With Us, The Procurement Concierge.

License: Apache License 2.0

JavaScript 1.89% TypeScript 96.94% Shell 0.01% Dockerfile 0.03% SCSS 0.87% EJS 0.08% PLpgSQL 0.16% Smarty 0.02%
digital-marketplace bcgov citz typescript fp sass node functional-programming postgresql realfolk

digital_marketplace's Introduction

pre-commit.ci status

Digital Marketplace

The Digital Marketplace is a web application that administers British Columbia's Code With Us and Sprint With Us procurement programs. It enables (1) public sector employees to create and publish procurement opportunities, and (2) vendors to submit proposals to these opportunities.

This document describes this project's developer environment, technical architecture and deployment infrastructure.

The file docs/onboarding/tips.md contains a collection of helpful information about working with this custom framework. It is encouraged that the file be added to by anyone who uncovers something they feel would have been helpful to know.

Table of Contents

Licensing

This project available to use under the Apache 2.0 license (see LICENSE.txt). Please note the NOTICE.txt file included with this repository and the guidelines in section 4(d) of the license.

Project Organisation

The Digital Marketplace is a full-stack TypeScript web application that uses PostgreSQL for persistence. It is written in a functional and declarative style with the goal of maximizing compile-time guarantees through type-safety.

The source code is split into six parts:

1. Front-End (src/front-end)

A TypeScript single-page application using React, Immutable.js, Bootstrap and SASS. The front-end's build system is executed by Grunt.

The front-end's state management framework (src/front-end/typescript/lib/framework/**/*.tsx) provides type-safe state management, and is heavily influenced by the Elm Architecture. If you've used Redux before, you will find this to be very similar since Redux is also based on the Elm Architecture. The main difference is that this project's framework derives greater inspiration from the Elm Architecture and it aims to be far more type-safe than Redux.

2. Back-End (src/back-end)

A TypeScript server that vends the front-end's build assets (src/back-end/lib/routers/front-end.ts) as well as a JSON CRUD API (src/back-end/lib/resources/**/*.ts) that performs business logic and persists data to a PostgreSQL database.

The server framework (src/back-end/lib/server/index.ts) provides type-safe abstractions for API development, and is executed by Express (src/back-end/lib/server/adapters.ts).

2a. Authentication

The server uses OpenID Connect to authenticate users with a Keycloak server (managed by B.C. government employees). The authentication routes are implemented in src/back-end/lib/routers/auth.ts.

2b. CRUD Resources

CRUD resources are created in a standardised, type-safe way in this project. CRUD abstractions are located in src/back-end/lib/crud.ts, and it is recommended to review this module prior to extending the API.

3. Email Notifications

Email notifications are all rendered server-side using React's static HTML renderer. Stub versions of all email notifications can be viewed by authenticated admin users at {HOST}/admin/email-notification-reference in your browser.

4. Shared (src/shared)

The src/shared folder contains modules that expose types and functions that are used across the entire stack: front-end and back-end.

5. Database Migrations (src/migrations)

All database migration logic is stored in the src/migrations folder. Migrations are managed and executed by the knex NPM module (a SQL ORM bundled with database migration functionality). The PostgreSQL-related environment variables described below are required to define the database to connect to.

You can create a migration using the following command:

npm run migrations:create -- <MIGRATION_NAME>

This command creates a migration file from a template and stores it at src/migrations/tasks/{TIMESTAMP}_{MIGRATION_NAME}.ts.

DO NOT delete or change committed migration files. Creating and executing migrations is a stateful process, and, unless you know what you are doing, running a database migration should be viewed as an irreversible process.

6. Scripts (src/scripts)

General purpose scripts are stored in this folder. The scripts themselves are stored in the src/scripts/bin/ folder, and can be run using the following command:

npm run scripts:run -- <SCRIPT_NAME> [...args]

Development Environment

Set Up

.env File

First, create a .env file and replace the placeholder values with your credentials (refer to the Environment Variables section below for further information):

cp sample.env .env
# Open and edit .env in your text editor.

Install Dependencies

Please ensure the following packages have been installed:

  • yarn
  • Node.js 16.x
  • SASS
  • Docker
  • Docker Compose 3.x
  • pre-commit

Once installed, cd into this repository's root directory and proceed to install dependencies (if running for the first time):

yarn

Keycloak Set Up (optional)

If you don't have access the BC Government's keycloak and you want to be able to log in to the app, you'll need to set up a local instance of keycloak. To do this, update the following environment variables in the .env file:

  • KEYCLOAK_CLIENT_ID="login"
  • KEYCLOAK_CLIENT_SECRET="abc-123"
  • KEYCLOAK_URL="http://localhost:8080"
  • KEYCLOAK_REALM="digitalmarketplace"

Additionally, add:

Then:

  • Copy Client ID value and put into .env ID_PROVIDER_CLIENT_ID
  • Click to Generate a new client secret and copy value and put into .env ID_PROVIDER_CLIENT_SECRET

Quick Start

Open four terminals and run the following commands:

# Terminal 1
docker-compose up db # Start PostgreSQL only

# Terminal 2
npm run back-end:watch # Start the back-end server, restart on source changes.

# Terminal 3
npm run front-end:watch # Build the front-end source code, rebuild on source changes.

# Terminal 4
npm run migrations:latest # Run all database migrations.

Admin user

If this is the first time spinning up a local development environment, set up an admin user

Then, visit the URL logged to your terminal to view the now locally-running web application.

You can stop the local PostgreSQL container server (and the keycloak server, if you're running it) by running docker-compose down. If you wish to completely wipe the container database, including all the data added by the migrations, run docker volume rm digital_marketplace_dm-vol.

NPM Scripts

It is recommended that developers use the following scripts defined in package.json to operate this web application:

# Usage
npm run <SCRIPT_NAME>
Script Name Description
cypress:run Runs the Cypress tests. (You must first manually start the app for the tests to have something to run against.) NOTE: The test set up and clean up will wipe and recreate the local database.
cypress:open Opens the interactive Cypress test runner. (You must first manually start the app for the tests to have something to run against.) NOTE: The test set up and clean up will wipe and recreate the local database.
start Runs the back-end server.
front-end:lint Lints the front-end source code using eslint.
front-end:typecheck Typechecks the front-end source code using tsc.
front-end:test Runs unit tests for the front-end source code.
front-end:build Builds the front-end source code using grunt.
front-end:watch Builds the front-end source code using grunt, and rebuilds it whenever a front-end or shared source file changes.
front-end:typedoc Builds TypeDoc API documentation for the front-end source code.
back-end:lint Lints the back-end source code using eslint.
back-end:typecheck Typechecks the back-end source code using tsc.
back-end:test Runs unit tests for the back-end source code.
back-end:start Starts the back-end server (assumes it has already been built by grunt).
back-end:build Builds the back-end server using grunt.
back-end:watch Builds and starts the back-end server inside a nodemon process, rebuilding and restarting it whenever a back-end or shared source file changes.
back-end:typedoc Builds TypeDoc API documentation for the back-end source code.
shared:lint Lints the shared source code using eslint.
shared:typedoc Builds TypeDoc API documentation for the shared source code.
migrations:lint Lints the migrations source code using eslint.
migrations:helper A helper script to run various migration commands using knex. It is not recommended to use this directly, rather use the migration scripts described below.
migrations:create -- <MIGRATION_NAME> Creates a migration file from a template in src/migrations/tasks.
migrations:latest Runs all migrations forward using their exported up functions.
migrations:rollback Rolls all migrations back using their exported down functions.
migrations:up <NAME_OF_FILE> Runs one migration up so long as the name of the file is provided. (eg: 20221130162144_twu-admin-content-stub.ts.ts>.
migrations:down Runs one migration down.
scripts:run Runs a script. Usage: npm run scripts:run -- <SCRIPT_NAME> [...args]. Ensure the -- is included to allow script arguments to be properly parsed.
typedoc:build Builds all TypeDoc API documentation.
typedoc:start Serves TypeDoc documentation on a local server.
docs:readme-toc Generate and insert a table of contents for README.md.
docs:licenses Generate the list of licenses from this project's NPM dependencies in docs/open-source-licenses.txt.
docs:db -- <POSTGRESQL_URL> Generate database schema documentation in docs/database-schema.md from the database specified by the connection url.
learn-front-end:build Builds the learn-front-end project using grunt.
learn-front-end:watch Builds the learn-front-end project using grunt, and rebuilds it whenever a learn-front-end, front-end or shared source file changes.
learn-front-end:serve Serves the learn-front-end build files.

Environment Variables

Developers can configure the following environment variables to alter how the web application is built and/or run.

In development, developers can create a .env file in the repository's root directory to configure environment variables. As a convenience, developers can refer to sample.env as a guide.

Back-End Environment Variables

Environment variables that affect the back-end server's functionality are stored and sanitized in src/back-end/config.ts.

Name Description
NODE_ENV The back-end run-time's environment. Possible values include either "development" or "production".
SERVER_HOST The IPv4 address for the back-end to bind to.
SERVER_PORT The TCP port for the back-end to bind to.
SCHEDULED_DOWNTIME A boolean flag (set to 0 for false, 1 for true) to turn off CRUD endpoints and vend a downtime HTML page to all users when set to a non-zero number. Defaults to false if SCHEDULED_DOWNTIME is undefined or invalid.
BASIC_AUTH_USERNAME An HTTP basic auth username to limit access to the web application.
BASIC_AUTH_PASSWORD_HASH A password hash to authenticate HTTP basic auth passwords to limit access to the web application.
ORIGIN The root URL of the web app. This is used for authentication and generating URLs in email notifications. The value must include the protocol and any path prefixes. e.g. https://marketplace.digital.gov.bc.ca.
POSTGRES_URL The PostgreSQL database to connect to (you only need to use this variable in development, not the other DATABASE_* variables defined below).
DATABASE_SERVICE_NAME Auto-generated in OpenShift.
${DATABASE_SERVICE_NAME}_SERVICE_HOST The PostgreSQL host to connect to in OpenShift.
${DATABASE_SERVICE_NAME}_SERVICE_PORT The PostgreSQL port to connect to in OpenShift.
DATABASE_USERNAME The PostgreSQL user to connect with in OpenShift.
DATABASE_PASSWORD The PostgreSQL password to connect with in OpenShift.
DATABASE_NAME The PostgreSQL database name to connect to in OpenShift.
COOKIE_SECRET The secret used to hash cookies.
MAILER_HOST SMTP server host for transactional emails in production.
MAILER_PORT SMTP server port for transactional emails in production.
MAILER_USERNAME SMTP server username for authentication. If specified, MAILER_PASSWORD must also be provided.
MAILER_PASSWORD SMTP server password for authentication. If specified, MAILER_USERNAME must also be provided.
MAILER_GMAIL_USER A GMail SMTP username to test transactional emails in development.
MAILER_GMAIL_PASS A GMail SMTP password to test transactional emails in development.
MAILER_FROM The sender for transactional emails.
MAILER_BATCH_SIZE The maximum number of email addresses to include in batch notification emails. Defaults to 50.
MAILER_MAX_CONNECTIONS The maximum number of simultaneous SMTP connections to use for the mailer. Defaults to 5.
KEYCLOAK_URL The URL of the Keycloak server to use for authentication. Please contact a team member to retrieve this credential, or use a local instance of keycloak.
KEYCLOAK_REALM The Keycloak realm. Please contact a team member to retrieve this credential, or use a local instance of keycloak.
KEYCLOAK_CLIENT_ID The Keycloak client ID. Please contact a team member to retrieve this credential, or use a local instance of keycloak.
KEYCLOAK_CLIENT_SECRET The Keycloak client secret. Please contact a team member to retrieve this credential, or use a local instance of keycloak.
KNEX_DEBUG Set this to true to debug knex operations.
UPDATE_HOOK_THROTTLE The number of milliseconds used to throttle per-request jobs (e.g. automatically closing opportunities). Defaults to 60000ms.
AVATAR_MAX_IMAGE_WIDTH The maximum image width for uploaded avatar image files. Files with a greater width will be resized. Defaults to 500 pixels.
AVATAR_MAX_IMAGE_HEIGHT The maximum image height for uploaded avatar image files. Files with a greater height will be resized. Defaults to 500 pixels.
FILE_STORAGE_DIR The location to store uploaded files. This is typically used by the server to temporarily store files uploaded by multipart requests for processing.
SERVICE_TOKEN_HASH A hashed token used to control access to service API endpoints that are only enabled in development and test environments. Defining the variable will enable service endpoints that can be used to override user accounts and sessions.
SWAGGER_ENABLE A flag to enable the Swagger UI API documentation under SWAGGER_UI_PATH. Defaults to false.
SWAGGER_UI_PATH The base path to run the Swagger UI under for serving of API documentation. Defaults to /docs/api.
TZ Time-zone to use for the back-end. Required by the Linux OS that runs the back-end, but not used as application configuration.
SHOW_TEST_INDICATOR A boolean flag (set to 0 for false, 1 for true) to indicate that an environment is intended for testing purposes (prefixes emails subjects and shows a testing variant of the logo in email notifications). Defaults to false.
LOG_LEVEL An enumerated logging level that can be used to control the level of output when running the application. Allowed values are debug, info, warn, error, and none. The level will default to debug when NODE_ENV is set to development. Otherwise, the level will default to info unless explicity set in the environment.
DISABLE_NOTIFICATIONS A boolean flag that will disable all outgoing email notifications when set to 1 (true). Defaults to 0 (false).
LOG_MEM_USAGE A boolean flag (set to 0 for false, 1 for true) to enable debug-level logging of Node memory usage. Defaults to false.

Note on Postgres Connection If the credentials for Patroni are provided, they will be used in place of the POSTGRES_URL, which acts as a fallback (as it is used for local dev). The Patroni credentials are PGHOST, PGUSER, PGPASSWORD and PGDATABASE which are supplied by the Helm deployment.

Front-End Environment Variables

Environment variables that affect the front-end's build process are stored and sanitized in src/front-end/config.ts, and referenced to in gruntfile.js.

Name Description
NODE_ENV Determines whether the front-end is built for production (e.g. minification, compression, etc). Possible values include either "development" or "production".
CONTACT_EMAIL The Digital Marketplace team's contact email address.
PATH_PREFIX The URL path prefix that the Digital Marketplace is deployed to. For example, if deployed to digital.gov.bc.ca/marketplace, the value of this variable should be marketplace.
SHOW_TEST_INDICATOR A boolean flag (set to 0 for false, 1 for true) to indicate that an environment is intended for testing purposes (shows a testing variant of the logo in the web app UI). Defaults to false.

Deployment

This project is deployed to the Government of British Columbia's own OpenShift infrastructure. NOTE The instructions below apply to deployment to the OpenShift 4 (OCP4) environment, and no longer apply to deployment to OCP3 environments.

Environments

We have four environments:

OpenShift Project Name URL
ccc866-dev Development https://app-digmkt-dev.apps.silver.devops.gov.bc.ca
ccc866-test Test https://app-digmkt-test.apps.silver.devops.gov.bc.ca
ccc866-prod Production https://app-digmkt-prod.apps.silver.devops.gov.bc.ca

Each environment has its own database instance.

The Development and Test environments are secured behind HTTP Basic Auth. Please contact a team member to access these credentials.

Deployment Process

IMPORTANT For information on Helm deployment, see helm deploy docs.

The "ccc866-tools" OpenShift project is used to trigger the deployment process for all environments.

To deploy to the Development environment, start a build for "app-digmkt-dev", and OpenShift will build and deploy HEAD from the development branch into the Dev environment listed above.

To deploy to the Test environment, start a build for "app-digmkt-test", and OpenShift will build and deploy HEAD from the master branch into the Test environment listed above.

To deploy to the Production environment, start a build for "app-digmkt-prod", and OpenShift will build HEAD from the master branch. You will need to manually deploy the build to the Production environment listed above once it completes by deploying the "app-digmkt-prod" deployment in the "ccc866-prod" project.

For instructions on building images for each of the environments and setting up build and deployment configurations in OpenShift 4, please refer to the instructions in docs/build-deploy/openshift.md.

Running Database Migrations

Using an environment's deployment shell, run npm run migrations:latest in the root of this repository's directory to run all migrations forward.

Backups

Automated backups of the PostgreSQL database are performed with the BC Government Backup Container. Full documentation for this tool can be found here: https://github.com/BCDevOps/backup-container. The schedule for automated backups is as follows:

  • Every 6 hours with a retention of 4 backups
  • Every 24 hours with a retention of 1 backup (daily)
  • Every week with a retention of 4 backups (weekly)
  • Every month with a retention of 1 backup (monthly)

A manual backup can be immediately performed by connecting to the backup container pod in OpenShift and running backup.sh -1.

Backup archives are stored in the same OpenShift project as the Digital Marketplace application, on a separate provisioned volume.

You can find instructions for building and deploying the Backup Container images to OpenShift 4 here.

Restoring from Backup

In the unfortunate event that you need to restore your data from a backup archive, the backup.sh script can be used to restore from the last backup file. Refer to https://github.com/BCDevOps/backup-container#restore for details on how to use this script.

High Availability Database Deployment

The Digital Marketplace is currently deployed to an OpenShift platform using a highly available PostgreSQL stateful set. The template used to deploy this set is based on Patroni (https://patroni.readthedocs.io/en/latest/). A deployment configuration has been provided in openshift/templates/database for deployment to OpenShift environments. Instructions for building and deploying can be viewed here.

Deployment as a highly available replicaset is recommended, but not required. A standalone PostgreSQL database deployment configuration has also been provided in openshift/templates/database.

Community

Contributing

Features should be implemented in feature branches. Create a pull request against the development branch to have your work reviewed for subsequent deployment.

The development branch contains all approved code.

The master branch contains work that has passed the Quality Assurance process and is ready to be deployed to production.

Hotfixes can be merged directly to master via a pull request, but should be merged back into the development branch as well.

Changelog & Versioning

This project introduced a Changelog and versioning system in 2020-09 to track changes made to the code. Please refer to CHANGELOG.md for further information. Generally, core maintainers of this project should be the only people adding to the Changelog.

Forking this Repository

Please note the section above titled "Authors and Licensing" before forking this repository.

Configuration

Various aspects of this application can be configured. In addition to the environment variables described in the section titled "Environment Variables", the following files contain hard-coded configuration variables that can be overridden as needed:

  • src/back-end/config.ts
  • src/front-end/typescript/config.ts
  • src/shared/config.ts

Theming

This project has a custom Bootstrap theme, defined in src/front-end/sass/index.scss. If you would like to theme this project to match your own style guide, you will need to update many of the variables in that file. You will likely also need to make the following changes to ensure a consistent user experience:

  • Replace logo.svg and logo.png in src/front-end/static/images.
  • Modify the colors within default_user_avatar.svg and default_organization_logo.svg in src/front-end/static/images.
  • Modify the colors within the SVGs in src/front-end/static/images/illustrations.
  • Modify the fonts sourced in src/front-end/sass/_font.scss.
  • Modify the colors in src/back-end/lib/mailer/templates.tsx that are used for email notifications.

Migrations

When maintaining a fork of this project that has its own database migrations, special attention must be given to the CHANGELOG.md and source code diff whenever the root repository is merged into the fork. You will need to verify that any new migrations from the root repository do not conflict with your own database migrations and schema.

Migrations can be destructive operations, so please ensure they are monitored and executed with special care.

Team

The Digital Marketplace is currently operated by the Procurement Services Branch within the Government of British Columbia's Ministry of Citizen's Services.

Credits

This project would not have been possible by the incredible work done by open source project maintainers. The licenses for open source projects used by the Procurement Concierge Program's web app are documented in docs/open-source-licenses.txt.

digital_marketplace's People

Contributors

ads-michaelv avatar bcerki avatar bdolor avatar bryangk avatar dhruvio avatar eazeredo avatar ianfonzie avatar jwarrenbc avatar kriscooke avatar mark-a-wilson avatar mikevespi avatar pre-commit-ci[bot] avatar repo-mountie[bot] avatar scallyw4g avatar stevechapmanbcdx avatar sutherlanda avatar thegentlemanphysicist avatar vb-at-bis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

digital_marketplace's Issues

Automate Changelog Generation

We discussed in button-inc#14 but haven't implemented yet in our first branching model agreements button-inc#18, the opportunity to adopt and enforce Conventional Commits. Might we consider enforcing that with a Gitlint guardrail (sample config) in a pre-commit hook (sample config)? This would unblock the option of using standard-version to automate the release process in a manner compliant with SemVer. If this adds value to our process, we'd likely want a releasing team agreement (sample agreement) to be part of this issue.

date picker fields don't work in Safari

Browser: Safari Version 14.0.3
Environments: TEST and PROD

Signed in as a public sector user, I started drafting a SWU opportunity. When I got to the Proposal Deadline and Assignment Date fields, I could not type anything into the fields, nor see any date picker.

Add prettier to pre-commit

As a developer
I want to use prettier
to ensure code is consistently formatted

  • enable prettier in local pre-commit
  • enable prettier in CI pre-commit
  • run prettier against the entire codebase

Add project lifecycle badge

No Project Lifecycle Badge found in your readme!

Hello! I scanned your readme and could not find a project lifecycle badge. A project lifecycle badge will provide contributors to your project as well as other stakeholders (platform services, executive) insight into the lifecycle of your repository.

What is a Project Lifecycle Badge?

It is a simple image that neatly describes your project's stage in its lifecycle. More information can be found in the project lifecycle badges documentation.

What do I need to do?

I suggest you make a PR into your README.md and add a project lifecycle badge near the top where it is easy for your users to pick it up :). Once it is merged feel free to close this issue. I will not open up a new one :)

OCP4 - Test Reduced Quota

Platforms Services sent an email on January 7 indicating that they would be reducing OCP quotas in February 2022.

We need to review our the Digital Marketplace tools namespace to ensure that we will be able to operate the application with reduced resources.

Acceptance Criteria:

  • Application operation under reduced resources reviewed
  • If reduced resources insufficient, request additional resources from Platform Services
  • Verify no Jenkins instances running in app
  • Resources set to:
  • requests.cpu: 2 cores
  • limits.cpu: 4 cores

Email from Platform Services:

What is happening?

We are introducing a series of quota reductions on the Private Cloud OpenShift Platform in order to free up the “reserved but not used” resources, which will increase our return of investment and allow us to continue accepting new apps on the Platform. The quota for CPU resource for all tools namespaces in the Silver cluster will be reduced in early February 2022 (the exact date will be communicated separately) and will have the following resource quota:

requests.cpu: 2 cores
limits.cpu: 4 cores

Depending on how much reserved CPU is freed up as a result of the teams re-configuring their Jenkins instances, more quota reduction steps may need to be introduced.

Why is it happening?

Inefficient resource utilization by Platform apps, and Jenkins instances in particular, have led to a misuse and shortage of free compute resources in the Silver cluster of the OpenShift Platform. This behaviour will soon prevent us from being able to accept new apps. There is an immediate need to reduce the amount of “requested but unused” resources. Doing this together will allow us to increase our Platform resource utilization, onboard more applications with our current investment, and educate the community on containerized resource utilization best practices.

Secondly, we will continue to encourage you to shut down your Jenkins instances entirely, and take advantage of Jenkins alternatives such as GitHub Actions and OpenShift Pipelines (Tekton) that are available on the Platform today. To help teams get started with these technologies quickly, we developed pipeline templates that also incorporate security best practices. Both alternatives provide much better resource utilization, substantially save the Province money and resources to manage, and reduce the Jenkins security risks with many instances versus centralized enterprise managed products.

When is it happening?

To be determined, but we are targeting early Februrary 2022.

What impact will it have on my app?

If the CPU Request or Limit settings for your Jenkins pod are above the new quota size, it may impact the pod’s ability to start.

What can I do to prepare my app for this change?

Adjust your Jenkins pod settings per the Jenkins Resource Configuration Requirements ASAP. It’s easy to do in your YAML configurations, with recommended settings provided by the Platform Services Team for you to leverage. We have also prepared a how-to video with the step-by-step instructions.

To test that your Jenkins instance can work successfully with the new settings, you can go to the Project Registry and change the CPU resource quota for the tools namespace to match the new quota as shown below.

Run your usual build process to confirm that it works as expected.

The entire test should not take more than 1 hr and we hope that all teams will be able to accomodate it in their current Sprint.

We have tested that Jenkins can successfully perform with the recommended settings, if, however, you find that the new quota impacts your ability to use other development tools in the tools namespace, please let us know ASAP in the “quota-reduction” discussion thread in the #devops-operations channel.

What should I do if I have concerns about this change?

Come to the Q&A session on January 12, 2022 that will take place in preparation for this change.

Thank you for doing your part in improving the resource utilization on the Private Cloud Platform.

Investigate Quebec's mocha tests

Description

As a developer
I want to investigate the mocha tests in the Quebec fork of the repo
So that I can add or adapt them for use in our repo to increase test coverage

Tasks (How)

  • determine if the tests can be brought over as-is or would need to be refactored/rewritten and how

Feature request: historical opportunities [government executive]

User Story

As a government executive, I want to easily be able to produce a report for my Minister about the money that has flowed to the private sector through our innovative procurement programs, CWU and SWU.


I would like to easily be able to see:

  • Total $$ awarded through CWU since its inception
  • Total $$ awarded through SWU since its inception

By government fiscal year (April 1 - March 31)

  • Total $$ awarded through CWU
  • Total $$ awarded through SWU

For each opportunity, I can see:

  • Name of awardee
  • Award amount

Email notifications of new opportunities going to Junk mail in Outlook again

Describe the bug
After successfully receiving one opportunity email directly to my regular inbox this past December 8 (following the previous spam filter issues), the new opportunity emails resumed being filtered straight to my Junk folder for all opportunities posted subsequently.

To Reproduce
Steps to reproduce the behavior:

  1. Sign up for email notifications of new opportunities
  2. Wait for a new opportunity to be posted / post new opportunity
  3. Watch your regular Outlook inbox, as well as your Junk folder
  4. New opportunity emails appear in Junk folder

Expected behavior
Email sender should be considered trustworthy enough by Outlook to be delivered to users' regular inbox.

Additional context
Can send the raw email headers by other means

Update anonymous functions with relevant names

As a developer, I want to rename anonymous functions throughout the code(see pic) to relevant names using relevant naming conventions, so that I can debug the front-end code more effectively.

DoD:

  • Come up with a naming convention the team agrees upon
  • Find all anon functions in the code
  • Rename functions

Screen Shot 2021-12-06 at 10 05 25 AM

Fix CSS issue causing some text to be transparent.

On one of the pages of Sprint With Us projects some of the labels/text do not show up. You can however see them if you highlight the column in the table as shows in the screen shot below. You can see the F/T and P/T labels only show up when highlighted. Does not work in Safari or Chrome.

Screen Shot 2021-02-26 at 5 19 22 PM

Apply for Cypress Dashboard Open-Source Plan?

As a developer
I want to use the Cypress Dashboard
So that I can see screenshots and videos of my CI Cypress test runs, which will help me debug failures.

Tasks:

  • Discuss:
    • Screenshots and videos are public, so if test fixtures include any sensitive information (they shouldn't, they're mainly lorem ipsum), it could show in an image
    • It's possible to accidentally expose a secret from our github actions configuration (we have detect-secrets as part of pre-commit already; could add some additional mitigation)
  • If discussion lands on using the dashboard, apply for the open-source plan here: https://docs.cypress.io/guides/dashboard/organizations#Open-Source-Plan
  • Set up dashboard access from our code (add CYPRESS_RECORD_KEY to the repo secrets and projectId to cypress.json (both these values come from the dashboard))

Tech debt: Fix emails in dev

As a developer
I want to set up a fully functional smtp endpoint for transactional emails
so that I don't have to use a gmail workaround.

Tasks

  • set up endpoint
  • remove code related to the gmail workaround
  • remove requirement to provide gmail environment variables in order to start the server

Additional notes:

in development, the site uses gmail to test transactional emails in development

Highly recommend using Mailhog in ci/dev/test to create a fully functional smtp endpoint. Here's a good helm chart for Mailhog.

Originally posted by @wenzowski in button-inc#126 (comment)

Feature request: Link to codebase on GitHub

User Story

As a public sector employee from another jurisdiction (e.g. Government of Canada), I would like to know that the source code for the Digital Marketplace app is open source, so we can build our own!

(This came from a conversation @ZachWoodward and I just had with some folks from this Gov of Canada team. They had forked the old devex codebase, then visited digital.gov.bc.ca/marketplace, but it was not apparent to them that the code for it was also open source.)

Images on homepage that have missing or unhelpful alt text

From GDX accessibility audit conducted in July 2021.

Issues

Use the WAVE Evaluation Tool Chrome extension to see the areas of concern:

  • BC Gov logo (header logo) should have descriptive alt text as it functions as a link
  • Images for CWU & SWU boxes have alt text of "Code With Us Image" and "Sprint With Us Image". These don't help the user understand. They'd best be reverted to decorative (i.e. alt="")
  • Two other illustrations below don't have any alt attribute; they should also have decorative (i.e. alt="")

Add project lifecycle badge

No Project Lifecycle Badge found in your readme!

Hello! I scanned your readme and could not find a project lifecycle badge. A project lifecycle badge will provide contributors to your project as well as other stakeholders (platform services, executive) insight into the lifecycle of your repository.

What is a Project Lifecycle Badge?

It is a simple image that neatly describes your project's stage in its lifecycle. More information can be found in the project lifecycle badges documentation.

What do I need to do?

I suggest you make a PR into your README.md and add a project lifecycle badge near the top where it is easy for your users to pick it up :). Once it is merged feel free to close this issue. I will not open up a new one :)

Document and address potential security breach

Some recent discoveries in the code have noted backdoors being used for testing purposes. These security issues should be documented and reported to the gov't as a breach of security.

@wenzowski Is there a format or template used to document security breaches?

Show Attachments on the History List

As an Admin and Opportunity owner
I want to see the notes and their attachments in the history list
So that I can audit trail the opportunity process

CD/CI - Minimize production image size

Business value:

  • Right now, the production image applies Grunt later than optimal - if we applied Grunt earlier, we could reduce the size of the image
  • Align with Test/Prod parity
  • Eliminate existing anti-patterns (reduce tech debt)

Questions to answer:

  • What is the current image size in production?
  • What is the ideal image size?
  • Who requested this?
  • Why we should do this?

Improve authentication

Description

As a developer and user
I want login to be secure
So that no one can see my private info

Tasks

  • Task
  • Task

Additional Notes

Feature request: historical opportunities [government product owner, vendors, public]

User Stories

As a vendor, I would like to see which of my competitors are being awarded opportunities, what amounts they are being paid, and what work is being delivered for that money.

As a vendor who has been awarded an opportunity in the past, I would like to be able to refer to the terms of the opportunity, and also be able to point new prospective clients to work that we have been awarded and delivered for the Province of British Columbia.

As a government product owner who is interested in posting a CWU or SWU opportunity, I want to be able to see past opportunities that have been offered so I can get a sense of how I might compose my opportunity.

As a concerned taxpayer, I would like to know that my government is being transparent about the money they are spending through procurement, who they awarding it to, and for what.


I would like to be able to peruse past opportunities and:

  • Sort all of the opportunities that have been awarded by ministry (see what opportunities my ministry has awarded in the past)
  • Read the full content of the opportunity as it was posted (i.e. all of the Acceptance Criteria, Proposal Eval criteria in the case of CWU)
  • See the award amount
  • See the name of the vendor/individual who was awarded the opportunity
  • See the number of proposals that were submitted
  • Find a link to where the work that the opportunity is paying for is being delivered (i.e. link to the public GitHub repo where the work is being delivered)

Upgrade packages (other than node)

Once Node has been upgraded to LTS, handle the other package upgrades necessary.

The first iteration of this was in Sprint 1 here: button-inc#129.

Tasks

  • update all of the following packages
  • fix any errors (probably lots of TS ones) these updates cause

These packages require a major bump (there are no patch or minor updates in between the version we have and the next major):
@keycloak/keycloak-admin-client 15.1.0 15.1.0 16.1.0 dependencies https://www.keycloak.org/
autoprefixer 9.8.8 9.8.8 10.4.2 devDependencies https://github.com/postcss/autoprefixer#readme
chalk 2.4.2 2.4.2 5.0.0 dependencies https://github.com/chalk/chalk#readme
cssnano 4.1.11 4.1.11 5.0.15 devDependencies https://github.com/cssnano/cssnano
dotenv 6.2.0 6.2.0 11.0.0 dependencies https://github.com/motdotla/dotenv#readme
find-up 5.0.0 5.0.0 6.2.0 dependencies https://github.com/sindresorhus/find-up#readme
grunt-browserify 5.3.0 5.3.0 6.0.0 devDependencies https://github.com/jmreidy/grunt-browserify
grunt-contrib-compress 1.6.0 1.6.0 2.0.0 devDependencies https://github.com/gruntjs/grunt-contrib-compress#readme
grunt-terser 0.1.1 0.1.1 2.0.0 devDependencies https://github.com/adascal/grunt-terser
html-to-text 5.1.1 5.1.1 8.1.0 dependencies https://github.com/html-to-text/node-html-to-text
jest-diff 25.5.0 25.5.0 27.4.6 dependencies https://github.com/facebook/jest#readme
keycloak-connect 8.0.2 8.0.2 16.1.0 dependencies http://keycloak.org
load-grunt-tasks 4.0.0 4.0.0 5.1.0 devDependencies https://github.com/sindresorhus/load-grunt-tasks#readme
openid-client 3.15.10 3.15.10 5.1.1 dependencies https://github.com/panva/node-openid-client
react 16.14.0 16.14.0 17.0.2 dependencies https://reactjs.org/
react-dom 16.14.0 16.14.0 17.0.2 dependencies https://reactjs.org/
react-markdown 4.3.1 4.3.1 7.1.2 devDependencies https://github.com/remarkjs/react-markdown#readme
react-select 3.2.0 3.2.0 5.2.1 devDependencies https://github.com/JedWatson/react-select/tree/master/packages/react-select
reactstrap 8.10.1 8.10.1 9.0.1 devDependencies https://github.com/reactstrap/reactstrap#readme
serve 11.3.2 11.3.2 13.0.2 devDependencies https://github.com/vercel/serve#readme
swagger-jsdoc 4.3.2 4.3.2 6.1.0 dependencies https://github.com/Surnet/swagger-jsdoc
ts-node 8.10.2 8.10.2 10.4.0 dependencies https://typestrong.org/ts-node
typescript 3.8.2 3.9.10 4.5.4 dependencies https://www.typescriptlang.org/
uuid 3.4.0 3.4.0 8.3.2 dependencies https://github.com/uuidjs/uuid#readme
yargs 16.2.0 16.2.0 17.3.1 dependencies https://yargs.js.org/
@wordpress/wordcount 2.15.2 2.15.2 3.2.3 dependencies https://github.com/WordPress/gutenberg/tree/HEAD/packages/wordcount/README.md

This one can go to a higher minor first:
knex 0.19.5 0.19.5 0.95.15 dependencies https://knexjs.org (but TS needs to be updated first)

Tech debt: Use prod image for CI/CD processes

Description

As a developer
I want to use the same docker image in testing and prod
So that I can get closer to dev/prod parity

Tasks (How)

  • replace current build steps in CI with a step that grabs the production docker image

Alec note from here: Let's make sure we've got a tech debt card for our CI/CD process that notes our e2e tests should be booting the candidate docker image that we intend to release to production (as a 12 factor app, we should be able to boot the prod image with development or test environment flag set). The cypress errors appear to be due to the fact that we're running headless from inside a runner that does not have access to a gpu.

Tech debt: Make test endpoint return prod types

As a developer
I want my testing endpoints to use the same types as my production endpoints
so I can use type-checking to avoid bugs and unexpected behavior.

Tasks

  • rewrite the testing endpoints so they return the same types as the prod endpoints

Additional notes:
Ideally we'd cast this endpoint's output to return prod types. Tech debt

Originally posted by @wenzowski in button-inc#111 (comment)

Avoid node version bump tech debts

As a developer
I want to avoid having different versions of node in different parts of the code (e.g., the package is updated to 16 but the github action uses 10)
so that I can keep versions consistent and avoid any unexpected behaviour.

Tasks

  • find a way to ensure that when a node version changes in one place, it changes everywhere else too

Additional notes:

This looks like continuation of button-inc#56 ...so I'd call the fact that we were able to bump the node version there and miss the CI configuration tech debt. Let's not hold back this PR but let's figure out how to ensure when we bump node versions in the future we get everything in one go–perhaps we should be looking at how other projects use a .tool-versions file as a source of truth that gets consumed by other systems, like docker, ci, npm, etc.

Originally posted by @wenzowski in button-inc#65 (comment)

Create Note - Opportunity History

As an Opportunity Owner or Administrator
I want to add notes (text and/or multiple attachments) in the Opportunity history
So that I can provide a complete audit history if/when required

Opportunity owners and administrators require the ability to add a general note at the opportunity level. In addition to the text provided for the note, the user may include an attachment, or attachments, to supplement the note.

Designs uploaded to InVision:

Page: Opportunity History - https://invis.io/QBZ92HXTMJZ
Modal: Add Entry to Opportunity - https://invis.io/QNZ92I36DAG

Original Jira ticket => https://bcdevex.atlassian.net/browse/DM-97
Unmerged PR => #134

Related Jira ticket => https://bcdevex.atlassian.net/browse/DM-814
(There's no related PR for this issue)

Acceptance Criteria:

  • Validate to ensure that only PDF files can be added
  • Validate to ensure that only files less than or equal to 2 MB can be added
  • Validate to ensure that note field is not empty
  • Validate to ensure that note field is not greater than 1000 chars
  • Apply max length to note's text field (e.g., 1024 characters max)
  • Ensure that multiple attachments can be uploaded
  • On save, the field "Entity Type" has the value "Note".
  • All admins to be notified by email when a new note is add to a History List (template text requirements)
  • Refresh the history list to show the new note

Give error messages for invalid entry when creating SWU opportunities

Is your feature request related to a problem? Please describe.

The SWU form won't allow a gov user to submit the opportunity for review if they've failed to fill in mandatory information, but it doesn't indicate which fields/checkboxes, etc. are the problem.

Describe the solution you'd suggest

Flag which fields are missing/unacceptable, similar to how the if you enter an invalid email, you're immediately given feedback.

Describe alternatives you've considered

Additional context

none

Bug: Completion and proposed start date doesn't save correctly

Describe the bug
If year is 2050 (probably affects other years too), CWU opportunity completion date doesn't save correctly--the day is off by one. Proposed start date is also off by a day.

To Reproduce
Steps to reproduce the behavior:

  1. Create an opportunity with the completion date of April 30, 2050
  2. Save the opportunity as a draft
  3. View the draft
  4. See that the completion date is now April 29, 2050
    Expected behavior
    The date shouldn't change

Screenshots
If applicable, add screenshots to help explain your problem.
Screenshot from 2021-12-21 09-09-44
Screenshot from 2021-12-21 09-10-10

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means its critical that we work to make our content as discoverable as possible; Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

add a topic

That's in, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

  • If your org is not in the list below, or the table contains errors, please create an issue here.

  • While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

  • Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

  • If your application is live, add the production URL.

Ministry Short Codes

Short Code Organization Name
AEST Advanced Education, Skills & Training
AGRI Agriculture
ALC Agriculture Land Commission
AG Attorney General
MCF Children & Family Development
CITZ Citizens' Services
DBC Destination BC
EMBC Emergency Management BC
EAO Environmental Assessment Office
EDUC Education
EMPR Energy, Mines & Petroleum Resources
ENV Environment & Climate Change Strategy
FIN Finance
FLNR Forests, Lands, Natural Resource Operations & Rural Development
HLTH Health
FLNR Indigenous Relations & Reconciliation
JEDC Jobs, Economic Development & Competitiveness
LBR Labour Policy & Legislation
LDB BC Liquor Distribution Branch
MMHA Mental Health & Addictions
MAH Municipal Affairs & Housing
BCPC Pension Corporation
PSA Public Safety & Solicitor General & Emergency B.C.
SDPR Social Development & Poverty Reduction
TCA Tourism, Arts & Culture
TRAN Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.

Bug: app errors cause Cypress tests to fail

Describe the bug
Cypress tests occasionally fail due to uncaught exception errors:
Screenshot from 2022-01-12 15-19-16

To Reproduce
Comment out these lines from Cypress index.js:

Cypress.on('uncaught:exception', (err, runnable) => {
    // returning false here prevents Cypress from
    // failing the test
    return false
  })

Run the Cypress tests until the uncaught exception fails one of them (which one fails and where is inconsistent).

Expected behavior
There should be no errors in the app code.

Additional context
none

Improve mocha

As a developer
I want to determine if this project will use mocha
so that I can either improve the existing mocha harness or remove it.

Tasks

  • if keeping mocha, change to a more standard implementation
  • if removing mocha, remove it from the project

Additional notes:
I wouldn't worry too much about it yet since the existing mocha suite runs just one example test. I'd split the work of improving or replacing the mocha harness to a new PR, and since we're reading a property out of a json file and that property is from the parent this looks right to me.

Originally posted by @wenzowski in button-inc#17 (comment)

Feature request: Fields for code challenge and team scenario dates

As a government product owner, I would like my Sprint With Us posting to include dates for code challenges and team scenarios so vendors can prepare their resources. I might also want to communicate when to expect the results of each step of evaluation, so that vendors know when to stand down their resources.

CWU Scoring - Design Improvements

As a CWU Opportunity Owner or Administrator
I want to be able to evaluate CWU opportunities more efficiently by adding evaluation scores in batch (currently, each score must be added individually).

Design recommendations (from original Jira ticket):

  • Add “Enter Scores” button to the contextual navbar for the “Proposals” section when managing a CWU opportunity

    • If scores have been added, replace with “Edit Scores” button
  • When “Enter Scores” button is clicked, present the user to enter all scores for all proposals

    • Content will include Proponent Name (readonly), Total Score (input) and Comments/Notes (textarea) for each vendor submission

Actions:

  • Cancel
  • Save Scores (previously, “Submit Scores”)
    • Save Scores allows the user to save all known scores (i.e. some scores may be left blank)
    • Save Notes for each submission would appear in the Proposal History
  • Provide a “Lock Scores” button in the contextual navbar (disabled until all scores have been entered)
    • all scores must be entered and saved to activate this button
    • once clicked, user will not be able to go back and edit scores for the corresponding phase

Original Jira ticket => https://bcdevex.atlassian.net/browse/DM-658
Unmerged PR => #XXX

Acceptance Criteria:

  • AC1
  • AC2
  • AC3

Feature request: provide a way to view the Code With Us terms

User Stories

As a proponent, I would like to see what the Code With Us terms are before deciding to apply on an opportunity.

As a government administrator, I need a way to send the Code With Us terms to my finance staff so that they have a copy of "the contract" and can set up a purchase order to pay invoices against it.

terms

Migrate completed work from Button fork to BCGov repo

Business value - we want to capture the work done and migrate to final repo.

The Button team completed work on a bunch of tech debt. The work was done on the Button fork.

Details:

  • Include config setting changes to match team agreements on Button fork
  • Include cypress secrets
  • Include Sonar secrets

Add env configuration documentation to Dig Marketplace

When deploying digital marketplace locally I found a blocker while getting the project running. This came from not knowing which env viarables from the sample.env needed setting.

With Andrew's help we got it running by setting
POSTGRES_URL="postgresql://digitalmarketplace:digitalmarketplace@localhost:5432/digitalmarketplace"
MAILER_GMAIL_USER="[email protected]"
MAILER_GMAIL_PASS="password"
ORIGIN="http://localhost:3000/"

This should be documented in the digital marketplace readme.md file.
AC:

get a documentation to the dig marketplace readme merged into the codebase

An admin can publish a SWU opp. without review (API only)

Classic steps are :

  1. Draft (if base validation passes)
  2. Submit for review (if complete validation passes)
  3. Publish (if previous step passes)

But an admin can go from step 1. to step 3. Then, no validation is done at step 2.

Relevant code parts :

// src/shared/lib/resources/opportunity/sprint-with-us.ts
export function isValidStatusChange(from: SWUOpportunityStatus, to: SWUOpportunityStatus): boolean {
  switch (from) {
    case SWUOpportunityStatus.Draft:
      // UnderReview step can be omitted
      return [SWUOpportunityStatus.UnderReview, SWUOpportunityStatus.Published].includes(to);
// [...]

But validations take place while Reviewal step :

// src/back-end/lib/resources/opportunity/sprint-with-us.ts

case 'publish':
            // [...]
            // Opportunity will have been fully validated during review process, so no need to repeat
            const validatedPublishNote = opportunityValidation.validateNote(request.body.value);
            if (isInvalid(validatedPublishNote)) {
              return invalid({ opportunity: adt('publish' as const, validatedPublishNote.value) });
            }
            return valid({
              session: request.session,
              body: adt('publish', validatedPublishNote.value)
            });

Found this bug while converting postman tests to Mocha (https://github.com/CQEN-QDCE/digital_marketplace/tree/experimentation/tests/back-end/unit/lib/api)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.