Explore address validation
With Smarty Streets API
To discover how we might deliver the best address validation experience

Resources

• Smarty Free account signup
• Smarty Ruby SDK documentation
• Smarty System requirements, including information about SSL/TLS
• Smarty Streets US street address API docs
• USPS address validation API docs
• [Internal] Notes from Smarty meeting
• [Internal] Test addresses and results

Approach

• Prototype server-side validations of address
  • How do we know if address is allowable
  • Assigning and displaying errors to model
• Prototype client-side validations of address
  • Displaying to client when address has been changed
• Prototype client-side autocomplete
• Benchmark / measure API usage of autocomplete
• Test cases referenced in questions below
• Write up notes

Note: this is now only using the US street address validation API, since autocomplete has been deprioritized for launch.

Questions to answer

• Can we restrict validations to USPS data?
  • Yes, by enabling strict match mode or by enabling invalid or enhanced APIs but filtering any dpv_match_codes that are not Y, S, or D. Match output strategies invalid or enhanced are the only ways to suggest addresses to the user (strict will not`).
• Can we validate addresses in US territories?
  • Yes, though presenting the full address returned from the API is slightly different than how we do this with other addresses (optional urbanization field). Context: we have tested this with a few addresses from Puerto Rico.
• Can we validate addresses in overseas military bases?
• What settings do we need to use to make it as close to the USPS online API as possible?

Running notes

• In the Ruby SDK example on address validation, there's a note that the 'urbanization' field is only used for addresses in Puerto Rico. What does this mean for us?
• Does state need a specific format (eg state two-letter code)? What does it need for territories and overseas?
• We can use match output strategy strict to only return results if they have dpv_match_code of Y, S, or D, which we think maps to USPS deliverability. However, this seems to be very strict in practice - and won't return suggestions
• When addresses meet the dpv_match_code, they can also return dpv_footnotes with information from USPS about the address. See api docs. I assume this is only present for S and D types but not sure. Additionally, Smarty adds a footnotes section (docs].
• I'm having a problem matching US military bases. I'm using examples based on the USPS documentation and can't get the API to return a result because they're fake. Do we have real ones we can test with? Same for territories (using addresses from Google Maps)
• USPS site doesn't seem to accept dpv_match_code of D, and only accepts Y and S.
• When dpv_match_code is S additional information is preserved in the returned address even though it is incorrect.
• Suggestion from Mapbox on autocomplete: start sending keystrokes after ~6 characters

From #60 (and assuming #67 is done)

• When SmartyStreets is offline, attempt to validate against USPS API

Finalize product spec:

• Document
• Sequence chart

Acceptance:

• Document and sequence chart baselined and checked into this repo.

On the client, allow users to review the details they've provided prior to submission, also allowing the option to go back to edit/update information.

add pipeline step to deploy main branch to staging

From conversation with Em on Jan 10

If someone tries visiting the confirmation page directly, they should see a page that says:

This page has expired!
If you are looking to order a set of rapid at-home COVID tests, please visit COVIDtest.gov.

https://guides.rubyonrails.org/active_record_encryption.html

Fields we should encrypt:

• First name
• Last name
• Email

I'm also thinking at least mailing address 1. @rahearn any thoughts? [update: just went for it, but invite comments on the PR!]

Research doc here (feel free to request access)

• S3 as DB
• PostgreSQL
• DynamoDB
• Serverless with SQS

For returned matches, we get information that may be helpful to surface to people, especially if we are correcting their addresses or their matched address is determined undeliverable (dpv_match_codeD).

Documentation:

• DPV Footnotes
• Footnotes

We might consider pairing this with more permissive match strategies.

Finalize consent text with GSA and USPS.
Need to know what USPS will do with the information we send them.
Especially will it be used for anything outside of order fulfillment (ex. will it be used to strengthen USPS address verification API?)

Windows

• Tablet
• Desktop - Internet Explorer 🟢
• Desktop - MS Edge 🟢

MacOS

• Safari 🟢
• Chrome 🟢
• Firefox 🟢

iOS

• iPhone latest
• iPhone 6 Safari 🟢
• iPad Safari 🟢

Android

• Native Android (Pixel)
• Samsung device
• Chrome
• Kindle Fire Silk 🟢

Confirm what will be collected and stored (agnostic of format)

• Name (Mandatory)
• Address (Mandatory)
• Phone (Optional)
• Email (Optional)

Document data protection measures that will be in place.

Example in use: https://github.com/HHS/Head-Start-TTADP/blob/main/docs/boundary_diagram.md

From #60, where a decision on address validation resiliency was made

In the event that SmartyStreets encounters an outage:

• Create a parallel app that will intake orders
  • Backend: email is mandatory
  • Frontend: email is mandatory
  • Frontend address presence validation, no API validation
  • Backend address presence validation, no API validation
  • Remove "email is optional" text
  • Deploy to staging env
• Orders will be saved without validating address
• Upon API recovery, a background process will validate address for those orders
• Email can be used to contact users of any post processing issues encountered by that process
• Sitchover/Trigger mechanism from the primary app to this parallel app when SmartyStreets outage occurs

Steps to reproduce:

1. Visit homepage
2. Fill in all required information, but leave first name blank. Click submit
3. See that the state select does not have a persisted value when page re-rendered with errors

Email adoption in the United States is in the mid/low 90s with lower numbers amongst:

• Elderly
• Minorities/Persons of Color

While the USPS fulfillment system requires an email, we will investigate being able to support order intake with an email address.

Acceptance Criteria

• Order flow will make email optional
• User is informed *prior to order submission that if email is not entered, they will not receive any status updates from USPS
• Email will be populated with [email protected] (or some other email address).
  • Note: another story will be entered if there are specific email uniqueness constraints on emails sent to USPS)

Move CI pipeline to circleci because this repo has a limit of 0 minutes available for Github Actions

CircleC Free Plan:

Private repositories will have 2,500 credits/week available, which replenish every Sunday 12:00 UTC

• Add CircleCI as check against repository
• Pull over checks from Github Actions:
  • Static analysis
  • Dependency scans (Ruby and Yarn)
  • OWASP (potentially not, since Log4j vuln) owasp/zap2docker-weekly has an updated log4j
  • Pa11y
  • Rspec
• Update README add information about CircleCI

Decision from 1/11 team discussion

Must have by 1/18 launch (captured in #67)

• Create a parallel app that will intake orders
  • Email is mandatory for this app
• Orders will be saved without validating address
• Upon API recovery, a background process will validate address for those orders
• Email can be used to contact users of any post processing issues encountered by that process

Nice to have by 1/18 launch captured in #68

• Parallel app would implement, on top of above, an address validation against USPS API

Background

Given:

• We cannot assume we can contact the user after they have submitted an order
• We have to confirm deliverability to prevent package loss

Then:

• We cannot save an order to our data store (i.e. give user confirmation of an order) until the address has been confirmed deliverable (see #43 on criteria of deliverability).
• We should understand what our stance is for if the primary validation API suffers an outage

Thoughts:

• Implement a backup mechanism based on either the (online USPS API, offline USPS address DB, other unpaid system that uses canonical USPS address information and can support #57.
• Have graceful fallback to the backup mechanism
• If backup fails, then error out the order
• If persistent failure of primary and backup, throw alarm to on call team and stop taking orders.

Email validation controls

• Error messages - at minimum meets WCAG 2.1 standards level AA.
• (To confirm with USPS) Check MX record of domain name

• Forms are translated in Simplified Chinese
• Input entry is still in Latin Characters

Ryan's sample code:

    def recaptcha_valid?(user_response)
      return true unless recaptcha_required?
      return false if user_response.blank?
      uri = URI("https://www.google.com/recaptcha/api/siteverify")
      response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
        request = Net::HTTP::Post.new uri
        request.set_form_data secret: Rails.application.credentials.recaptcha[:site_secret], response: user_response
        http.request request
      end
      json = JSON.parse response.body
      json["success"]
    end

Recaptcha docs: https://cloud.google.com/recaptcha-enterprise/docs

Rather than redeploy or restage in case of launch-day issues, we can and maybe should have multiple instances of the app running with different configs for likely runbook plays. We could then swap configs with cf map-route very quickly.

two scenarios that come to mind:

• dropping captcha
• using alternate storage backend (e.g. s3, other databases?)

Create supporting services using terraform for more auditable change management

This is the happy path as described in the intial prototype (https://gsa.invisionapp.com/console/share/UR3ACI81CG)

Acceptance criteria

• Front end has implemented all pages in initial prototype.
• English only.
• Orders saved into data store.
• Use placeholder text for Privacy/Consent language.
• Email is optional (though users are informed that without email, they will not receive any status updates).

** Background**

USPS guest checkout looks at entered addresses to confirm that it has found the address within its database.

• It automatically modifies the address to match what is closest match to its records.
  Ex. 1312 West Roscoe Street becomes 1312 W ROSCOE ST
• It rejects addresses it cannot match.

This is different from most patterns other e-commerce sites use. They often give the user a choice between a suggested updated address and what the user entered.

If it is confirmed that USPS fulfillment system requires addresses that match a deliverable address (DPVConfirmation), then we have to:

Acceptance Criteria

• If the address is changed, on "Review" page, user has clear indicator that their address was changed to what is in the system.
  • Implementation detail, this means addresses that received DPVConfirmation values of Y,S
• The user can choose to go back.
• If address is not found, an error is shown on the "Order entry" page.
  • This is for both errors where no match was found and a non deliverable error was received DPVConfirmation values of D,N
• If address validation cannot be completed
  • If transient failure, hard error and user can retry
  • If API completely offline, system should stop taking orders. This is undesirable, so redundancy/fallback of validation API is needed.
• Implementation detail: At minimum we will store the following fields (includes both user visible fields as well as non user visible fields):

<Address1>
<Address2>
<City>
<CityAbbreviation>
<State>
<Zip5>
<Zip4>
<DeliveryPoint>
<CarrierRoute>
<DPVConfirmation>

Acceptance Criteria :

• Content on order form is translated to Spanish
• Input data is still in English

Content to be translated into Spanish and Simplified Chinese.

Using the mockups provided by @AvivaOskow, apply the initial mobile-first USWDS styles to the rails app.

https://github.com/usagov/test-at-home/pull/61/files#diff-1dc4bea7b6827b18f8869436bd7786266d3cea8596529887ef16a027eb2e4076R4 introduces a robots.txt to prevent indexing - this is a reminder to remove the restrictions in prod upon launch.

Coordinate with User Testing.com to get prototypes tested.

We've discussed using a captcha of some sort on the front-end.

Issues:

• is there a captcha that is accessible?
• is the captcha verifiable on the backend? If we use a fully-cached front-end then we can't use the CSRF token to filter out submissions that bypass the frontend entirely. How can we filter out scripted curl submissions, for example?

In case we need to shard the DB, investigate how that can be accomplished with ActiveRecord and Postgres

Initially store in RDS, but remain flexible enough to move to s3-as-db in future.

• Add backend keys to credentials.yml (#176)
• Add embed keys to deployment environment (see here)

Some options with their benefits and drawbacks: https://guides.rubyonrails.org/i18n.html

USA.gov has two URLs: usa.gov and usa.gov/espanol
USPS uses subdomains: usps.com, es.usps.com, zh.usps.com

Believe we should use URL params (eg .gov/en/ or .gov/es) to reduce amount of DNS configuration

Supports #40 #39

We'd like client-side validations that match server-side validations so we don't have to hit the server for person to know they need to correct their entry.

Bring boundary diagram up to date with latest tools & architecture decisions

Thus far it our team has not been able to get in touch with USPS. @jacky chang has asked our team to think through contingencies on how to handle the situation where we are going to be taking in orders but have not touched data delivery.

Brainstorming:

Being able to contact the user after order submission

• Add content to let users know there's a possibility they will have to resubmit
• Add explicit content that without an email, user may need to resubmit their package
• Support outgoing emails to users
• ⛔ Are we able to make outbound phone calls?

Persist data as entered by user

• Unsure if this was already plan of record, but save info as entered by user; not just corrected address from validation API

Stakeholder awareness

• Ensure stakeholders across GSA/TTS & USDS/OMB are aware of risk