Many integers that should always be positive are declared as int. If possible, we should declare these as unsigned int. Some example: config.go for CodeDigits and ColissionRetryCount (typo -> CollisionRetryCount), code.go

TL;DR

google_kms_key_ring is using location = var.region
but it would be much better to be able to specify its own region/location same as notification_server does
not var.region

I'm more than happy to implement it:)

In the home.html, there's a comment that testData "cannot be more than 14 days prior to today".

We're not verifying this is the case server side in issue.go#L66

Should be configurable as well and enforced on token validation

in certificate.go#L135, we may verify the HMAC we're blindly signing has the right length of 32 bytes.

Add Admin page for listing and adding API keys

API Keys will go to mobile app developers when the make verification code redemption calls.

OTPs are generated in code.go#L44.

Because 10 is not a divisor of 256, digits are not equally likely:

• digits 0...5 have have probability 26/256
• digits 6...9 have probability have 25/256

A possible fix is be to use `rand.Int(0, 10^setLength).

Using Terraform to deploy I get the following error:

google_cloud_run_service.cleanup: Destroying... [id=locations/us-central1/namespaces/sandbox2-cw-en-vs/services/cleanup]
google_cloud_run_service.apiserver: Destroying... [id=locations/us-central1/namespaces/sandbox2-cw-en-vs/services/apiserver]
google_cloud_run_service.cleanup: Destruction complete after 1s
google_cloud_run_service.cleanup: Creating...
google_cloud_run_service.apiserver: Destruction complete after 1s
google_cloud_run_service.apiserver: Creating...
google_cloud_run_service.cleanup: Still creating... [10s elapsed]
google_cloud_run_service.apiserver: Still creating... [10s elapsed]
google_cloud_run_service.apiserver: Still creating... [20s elapsed]

Error: Error waiting to create Service: resource is in failed state "Ready:False", message: Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

  on service_apiserver.tf line 47, in resource "google_cloud_run_service" "apiserver":
  47: resource "google_cloud_run_service" "apiserver" {



Error: Error waiting to create Service: resource is in failed state "Ready:False", message: Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

  on service_cleanup.tf line 47, in resource "google_cloud_run_service" "cleanup":
  47: resource "google_cloud_run_service" "cleanup" {


zsombor@MacBook-Pro terraform % 

Digging into the Cloud Run logs, it looks like it's related to permission denied while accessing the csrf-token:

2020-06-25 15:52:01.749 EEST Cloud Run DeleteService apiserver [email protected] {@type: type.googleapis.com/google.cloud.audit.AuditLog, authenticationInfo: {…}, authorizationInfo: […], methodName: google.cloud.run.v1.Services.DeleteService, request: {…}, requestMetadata: {…}, resourceLocation: {…}, resourceName: namespaces/sandbox2-cw-en-vs/services/apiserver, serviceName: run…


2020-06-25 15:52:02.605 EEST Cloud Run CreateService apiserver [email protected] {@type: type.googleapis.com/google.cloud.audit.AuditLog, authenticationInfo: {…}, authorizationInfo: […], methodName: google.cloud.run.v1.Services.CreateService, request: {…}, requestMetadata: {…}, resourceLocation: {…}, resourceName: namespaces/sandbox2-cw-en-vs/services/apiserver, response: {…}, s…

2020-06-25 15:52:16.789 EESTresolving secret value for "DB_PASSWORD" (toFile=false)

2020-06-25 15:52:16.933 EESTresolving secret value for "DB_SSLCERT" (toFile=true)

2020-06-25 15:52:16.973 EESTresolving secret value for "DB_SSLKEY" (toFile=true)

2020-06-25 15:52:17.009 EESTresolving secret value for "DB_SSLROOTCERT" (toFile=true)

2020-06-25 15:52:17.063 EESTresolving secret value for "CSRF_AUTH_KEY" (toFile=false)


2020-06-25 15:52:17.119 EEST2020/06/25 12:52:17 config error: CSRFAuthKey: failed to resolve "projects/420911494560/secrets/csrf-token/versions/1": failed to access secret projects/420911494560/secrets/csrf-token/versions/1: rpc error: code = PermissionDenied desc = Permission 'secretmanager.versions.access' denied for resource 'projects/420911494560/secrets/csrf-token/versions/1' (or it may not exist).

2020-06-25 15:52:17.233 EESTContainer called exit(1).

TL;DR

Token exchange fails if the test date is blank

Expected behavior

this should work

Observed behavior

got metadata mismatch error

/assign

The current config requires firebase, database, and csrf. Not all services need that (e.g. cleanup only needs the database, apiserver doesn't need csrf), but we have to set all those values on the service because they are required fields in the mega config.

should be created w/ disabled = true
add created_at
add updated_at

Users disabled = true and updated_at > 30d will be purged

TL;DR

In order to facilitate key rotation and automatic discovery by an exposure notifications server, implement a JWKS document endpoint that provides the current public key material.

Design

Proposal

Needs to be a well known - unauthenticated endpoint for retrieval.

the KIDs need to be encoded in such a way that they can convey multi-tenancy of the verification server.

For example, a kid might look like "us-wa-00001"

See example JWK here -> https://tools.ietf.org/html/rfc7517#section-3

/kind enhancement
/assign

#27 (comment)

TL;DR

resource "google_redis_instance" "cache" {
...
location_id = "${var.region}-a"
alternative_location_id = "${var.region}-c"
...
}

I'm running the infra in europe-west1 which doesn't have europe-west1-a

Expected behavior
I'd like to be able to either override the redis location and alternative location or use any

Alternative solution might be to query Compute Engine API to find out which zones are available for this region

Observed behavior
Got

Error creating Instance: googleapi: Error 404: 'europe-west1-a' was not found.
com.google.apps.framework.request.StatusException: <eye3 title='NOT_FOUND'/> generic::NOT_FOUND: 'europe-west1-a' was not found.

Reproduction

# terraform.tfvars
region                  = "europe-west1"
cloudscheduler_location = "europe-west1"
appengine_location      = "europe-west"

All three services fail to deploy to GCP from the latest commit (132ed4e), but not from the previous commit (6bb10d3).

Cloud Run Logs say:

2020-06-26 00:28:56.941 EEST Cloud Run ReplaceService apiserver [email protected] {@type: type.googleapis.com/google.cloud.audit.AuditLog, authenticationInfo: {…}, authorizationInfo: […], methodName: google.cloud.run.v1.Services.ReplaceService, request: {…}, requestMetadata: {…}, resourceLocation: {…}, resourceName: namespaces/REDACTED/services/apiserver, response: {…}, …
{@type: type.googleapis.com/google.cloud.audit.AuditLog, authenticationInfo: {…}, authorizationInfo: […], methodName: google.cloud.run.v1.Services.ReplaceService, request: {…}, requestMetadata: {…}, resourceLocation: {…}, resourceName: namespaces/REDACTED/services/apiserver, response: {…}, …

2020-06-26 00:29:12.672 EEST2020/06/25 21:29:12 config error: input must be a struct

2020-06-26 00:29:12.900 EESTContainer called exit(1).

The commands I ran on 132ed4e:

1. terraform apply
2. ./scripts/build
3. export TAG="132ed4e"
4. ./scripts/deploy

Implement key rotation for the verification tokens. These tokens/keys are only used within this server, so no external discovery mechanism is needed.

/kind enhancement

Question

Hi, I'm volunteering over at Covid Watch. As mentioned in my colleague @ibeckermayer 's PR, we have a web app that we'd like to request verification codes from this server. I wrote a solution for our short-term/demo use that integrates our app with the server as it exists now, using http requests. Though preferably, we'd like to request verification codes via a programmatic interface as opposed to using axios requests. In my short-term solution, we create one user in the verification server for our web app. When a user in our app requests a code, our backend (Firebase Cloud Functions) logs into the server as that user via http and gets a new verification code. Relevant code in our repo is here. As far as we can tell, the username/password are acting effectively as an API key, and we can use them to generate verification codes programmatically. We are generally curious as to what differentiates this approach from the API-key approach, particularly from a security perspective. Would our ability to get codes this way not let someone generate new verification codes relatively rapidly, in the manner @mikehelmick was concerned with in his response to our API-key based approach? Thanks so much!

implement functionality to delete expired data

includes stale user records

Add basic log based metrics

Use https://github.com/mikehelmick/go-chaff to track real responses and return heuristically valid responses.

To include both signing keys

and iss/aud configuration.

Doesn't have latest instructions!

TL;DR

Currently the test dates are all captured and stored at UTC.

It would be nice to let the epi specify the test date in the local time zone, which can be conveyed back to the user through the app, etc.

TL;DR

Add the ability to send verification codes over SMS

https://www.twilio.com/messaging as likely baked in provider. Must be configurable so operator can bring their own credentials.

/kind enhancement
/priority important-soon

Add admin page for admins to make other users admin and enable/disable other user accounts

Should fix before OSS

There are several POST requests defined in server/main.go. I list a few below:

1. "/api/issue". This could allow an attacker to generate many VCs and increase the probability of guessing a valid one.

2. API key may allow an attacker to add its own API keys. I think this is low risk, because there's no way to keep these static/per-app API keys secrets over the lifetime of the system anyway... these keys don't seem to provide a security benefits.

3. Firebase auth. I could not find where this token is generated and validated in our backend. Is ti implemented yet?

4. Device APIs. I think this is not needed because the apps are native. Are we ever going to support webview apps? Even in this case, I don't think it would be a problem.

Possible protections, see here

to enable end to end test scenarios.

TL;DR

Make the server multi-tenant, allowing for different "realms".

Design

Proposal

All realm type.
A user can be authorized to use a realm and or authorized to be an admin for a realm.

API keys will be scoped to an individual realm.

Certificate keys will be on a per-realm basis.

Verification codes - open question on if these should be a large shared namespace. if API Keys are realm specific, then this could be done.

Alternatives considered

• large single tenant installation - means certificates can potentially cross PHA authority.
• multi-single-tenant installations, more operationally burdensome.

/kind enhancement

IssueToken() both verifies a VC and issues a token.

We may want to change its name to reflect that.

TL;DR

Investigate adding rate limiting per API key or by IP address Will need tuning params for autoscaled vs non-autoscaled environments.

Design

Proposal

A simple inmemory cache that limits an apikey to a certain number of requests per period on a given instance of the server.

in addition to server we now have apiserver and cleanup (needs redirection of scheduler)

Running the migrations also requires a firebase API key because the field is marked as required by envconfig, and it's an embedded struct in the parent config. We should either create a separate config just for the database or allow the API key to be empty during processing and validate it later.

Following the Configuring your Development Environment for Running Locally instructions I get an authentication error when running ./scripts/dev dbmigrate.

To fix the issue, I tried to manually change the password of the "en-verification-server" user, but it didn't help.

zsombor@MacBook-Pro exposure-notifications-verification-server % ./scripts/dev dbshell  
Error: No such container: en-verification-server-db
zsombor@MacBook-Pro exposure-notifications-verification-server % ./scripts/dev dbstart  
Database started (OK)
zsombor@MacBook-Pro exposure-notifications-verification-server % ./scripts/dev dbshell
psql (12.3)
Type "help" for help.

en-verification-server-db=# \password en-verification-server
Enter new password: 
Enter it again: 
en-verification-server-db=# \quit
zsombor@MacBook-Pro exposure-notifications-verification-server % ./scripts/dev dbmigrate
2020/06/25 12:32:05 db connection failed: database gorm.Open: pq: password authentication failed for user "en-verification-server"
exit status 1
zsombor@MacBook-Pro exposure-notifications-verification-server % echo $DB_PASSWORD
BjrvWmjQPXykPu
zsombor@MacBook-Pro exposure-notifications-verification-server % ./scripts/dev init
export DB_HOST="127.0.0.1"
export DB_PORT="5432"
export DB_USER="en-verification-server"
export DB_SSLMODE="disable"
export DB_PASSWORD="BjrvWmjQPXykPu"
export DB_NAME="en-verification-server-db"
export DB_URL="postgres://en-verification-server:[email protected]:5432/en-verification-server-db?sslmode=disable"
export SERVER_NAME="DEVELOPMENT verification server"
zsombor@MacBook-Pro exposure-notifications-verification-server % 

Similarly, if I try to run the server with go run ./cmd/server, I get the same error.

Implement the verification code flow

https://developers.google.com/android/exposure-notifications/verification-system#flow-diagram

in certificate.go, we may additionally verify the sub has the right format.

authorizedapp is set in [apikey.go#L79(]https://github.com/google/exposure-notifications-verification-server/blob/main/pkg/controller/middleware/apikey.go#L79) but never read or checked.

I'm curious in which scenarios an actual implementation may use it? I don't think a server implementation would want to differentiate between different apps when storing the keys in the database - as it may provide a signal to de-anonymize users.

So probably used for statistics?

[Preamble] I succeeded in getting a long-term token by exchanging short-term token on a local dev environment:

zsombor@MacBook-Pro exposure-notifications-verification-server % go run ./cmd/get-token -code=55582912 -apikey=7F1rJF7XkZUvmUyOF8j5K3R7RYzOa6DqTYbQCov974dfitNW/52a+q/MJ95lGLaUZCli3+irRV7z5H3v0dw57g
2020/06/30 13:47:08 Sending: {VerificationCode:55582912}
2020/06/30 13:47:09 Result: 
{TestType:confirmed TestDate:2020-06-30 VerificationToken:eyJhbGciOiJFUzI1NiIsImtpZCI6InYxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJkaWFnbm9zaXMtdmVyaWZpY2F0aW9uLWV4YW1wbGUiLCJleHAiOjE1OTM2MDA0MjgsImp0aSI6InV2ZmpWZk9xcHZOS253ZEtBQ05lN0pEeW4ybFdZWHdhRWoxdklZU2tEcUdYU2hLcm1jYjhuVWs2WUdaL25EL0FXU1hJQzl5S0ZZMktkU2hRclJQSWJRUGVLQzZSRWNSZlEwZ2Jod3A0ODNTTERocjdVUjJydG5xZWlxekRaeHNSIiwiaWF0IjoxNTkzNTE0MDI4LCJpc3MiOiJkaWFnbm9zaXMtdmVyaWZpY2F0aW9uLWV4YW1wbGUiLCJzdWIiOiJjb25maXJtZWQuMjAyMC0wNi0zMCJ9.UYBbtgc2wTmY23zo7fb2YKTfanU1l_9KKVNuLLonPStKXUQJCwB8_lO5MSIZlnXo2ye4kl-W2gnV9xVMU70kSA Error:}
zsombor@MacBook-Pro exposure-notifications-verification-server % go run ./cmd/get-token -code=55582912 -apikey=7F1rJF7XkZUvmUyOF8j5K3R7RYzOa6DqTYbQCov974dfitNW/52a+q/MJ95lGLaUZCli3+irRV7z5H3v0dw57g
2020/06/30 13:47:41 Sending: {VerificationCode:55582912}
2020/06/30 13:47:42 Result: 
{TestType: TestDate: VerificationToken: Error:verification code used}

I am trying to do the same on a GCP-deployed server, but fail to do so.

Cloud Run logs:
2020-06-30 15:22:19.926 EESTunable to get signing key: failed to fetch public key: rpc error: code = InvalidArgument desc = Resource name 'projects/REDACTED/locations/us-central1/keyRings/verification/cryptoKeys/token-signer' does not match pattern 'projects/([^/]+)/locations/([a-zA-Z0-9_-]{1,63})/keyRings/([a-zA-Z0-9_-]{1,63})/cryptoKeys/([a-zA-Z0-9_-]{1,63})/cryptoKeyVersions/([a-zA-Z0-9_-]{1,63})'.

It looks like Terraform does not set the correct value for the env variable TOKEN_SIGNING_KEY (and CERTIFICATE_SIGNING_KEY)

Modifying TOKEN_SIGNING_KEY (and CERTIFICATE_SIGNING_KEY) manually to end with /cryptoKeyVersions/1, and redeploying the services, resolves this particular issue.

However, the /api/verify call still fails, but with another error:
2020-06-30 16:25:31.047 EESTunable to get signing key: failed to fetch public key: rpc error: code = PermissionDenied desc = Permission 'cloudkms.cryptoKeyVersions.viewPublicKey' denied on resource 'projects/REDACTED/locations/us-central1/keyRings/verification/cryptoKeys/token-signer/cryptoKeyVersions/1' (or it may not exist).

Following the README.md on 6e69ace

After deploying via Terraform, the admin account for the "server" container was able to be created via cloud_sql_proxy and then a firebase account can be created via the firebase auth interface which succeeds; however upon login user is given error:

{
  "insertId": "5ef58047000976a7b3c70b87",
  "jsonPayload": {
    "caller": "session/session.go:68",
    "logger": "default",
    "ts": 1593147463.6194053,
    "stacktrace": "github.com/google/exposure-notifications-verification-server/pkg/controller/session.(*sessionController).ServeHTTP\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/session/session.go:68\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.FlashHandler.func1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/flash.go:38\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2012\ngithub.com/gorilla/csrf.(*csrf).ServeHTTP\n\tgithub.com/gorilla/[email protected]/csrf.go:301\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/html.(*VariableInjector).Handle.func1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/html/html.go:68\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2012\ngithub.com/ulule/limiter/v3/drivers/middleware/stdlib.(*Middleware).Handler.func1\n\tgithub.com/ulule/limiter/[email protected]/drivers/middleware/stdlib/middleware.go:58\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2012\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/gorilla/handlers.loggingHandler.ServeHTTP\n\tgithub.com/gorilla/[email protected]/logging.go:45\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2807\nnet/http.(*conn).serve\n\tnet/http/server.go:1895",
    "message": "unable to create client session: http error status: 400; body: {\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"INSUFFICIENT_PERMISSION\",\n    \"errors\": [\n      {\n        \"message\": \"INSUFFICIENT_PERMISSION\",\n        \"domain\": \"global\",\n        \"reason\": \"invalid\"\n      }\n    ]\n  }\n}\n"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "revision_name": "server-xxxxx",
      "configuration_name": "server",
      "service_name": "server",
      "location": "us-central1",
      "project_id": "xxxxx"
    }
  },
  "timestamp": "2020-06-26T04:57:43.620199Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "00bf4bf02d198aa5c9506aff20b178a4d95249eaa9538ebcb7925b3545f3e97efd32b5f502aa9edc34db1caed9a0b9b8970af6c677c0a5bb07c3ab7a"
  },
  "logName": "projects/xxxxxx/logs/run.googleapis.com%2Fstderr",
  "receiveTimestamp": "2020-06-26T04:57:43.858648311Z"
}

However, everything works on the local setup due to the use of the use of:

export GOOGLE_APPLICATION_CREDENTIALS=/Users/USERNAME/Documents/project-name-12
3456-firebase-adminsdk-ab3-4cde56f78g.json

Inspecting the cloud run container YAML it appears the service account being used is:

serviceAccountName: en-verification-server-sa@project-name-123456.iam.gserviceaccount.com

However this project does not have the Firebase Admin SDK role.

Adding the Firebase Admin SDK role to the service account resolves the issue.

Expected behavior
Log in on the live setup should work.

Observed behavior
Error as above.

Reproduction

I tested it on a variety of GCE Projects and Locally to determine that the error was caused by the presence or absence of a service account with Firebase Admin SDK role.

Additional information
Unfortunately I don't have enough experience with Terraform to create a PR to fix this, but I imagine its fairly straight forwards.

We have a mix of 302, 303, and 307. We should pick one and use it everywhere when redirecting.

Impose a limit on the number of codes an individual login can issue over a period of time

Provide reports to admins on the number of codes issues by each user

@crwilcox tracking list of parts:

• Rate Limit Logged In Users on Admin pages. (#113)
• Add Audit Tracing to Keys, marking who generated (#118)
• Add an admin page showing Verification Codes created per user. (#152)
• Add quotas per user of keys allowed to be created
• Add quotas per API Key of keys allowed to be created

implement key rotation functionality

key versions should be allowed to be scheduled for the future to have time to communicate the public key tot he exposure notifications server

TL;DR

After hitting the verify endpoint, api/verify, and receiving a verification token, we post to the get-certificate endpoint, api/certificate, with the token and our Hmac of our exposure keys and we are getting the following error:

400: {"error":"verification token test metadata mismatch"}

Looking for guidance on if this is a bug in the server logic when looking up the token, or if we are misunderstanding the intended flow here.

Expected behavior
Given that we are receiving the token directly from the server and returning it un modified, we expect that the server to not error with a mismatch of token metadata.

Observed behavior
We received the error.

Reproduction

1. created a verification code via the web portal we are hosting
2. send the verification code to the api/verify endpoint with the API token in the X-API-key header
3. receive the jwt back from the server.
4. create an HMAC with some TEKs and a random key generated by the client
5. post to api/certificate with a data payload: { token: the_returned_token, ekeyhamc: the_generated_hmac }

Environment
Not sure

Additional information
when we decoded the token we got this:

{
  "aud": "diagnosis-verification-example",
  "exp": 1594408633,
  "jti": "obfuscated jti",
  "iat": 1594322233,
  "iss": "diagnosis-verification-example",
  "sub": "confirmed."
}

one thing we noticed was the . dot at the end of confirmed.

TL;DR

Once an authentication code is issued for a positive test in the verification service, we would like to track the expiration timestamp and if the code was used, so that we can enable more workflows (both automated at not). Right now the database stores these data, but does not expose to the user and or via API.

[This feature was requested by a jurisdiction looking to launch immediately]

Design

Proposal

Alternatives considered

Resources

• Link to documentation

Additional information