automated perf/accessibility checks in deploy pipeline CI/CD about alpha HOT 8 CLOSED

Made Chad sole owner of this because he has already done some work on it.

One way to accomplish running these tests is to run them against a local version of the site which would be built inside the docker container created by the github action.

This might also be the best way to run the automated e2e test suite. I'll be taking ownership of integrating the e2e test suite in another ticket and we can learn from each other's work

from alpha.

I think we should close this issue now, I am really happy with calibre. If we continue to like using the tool after we sign up for the paid plan we should integrate this feature which will mark our graphs with deployment markers so it will be extremely obvious where a regression occurred. This isn't important immediately but would be useful when we have a larger team with a lot more code movement: https://calibreapp.com/docs/features/pull-request-reviews#create-a-pull-request

from alpha.

This is a big task so putting all the devs as participants, probably want to iron out azure pipeline first. Hoping @ChadABratton wants to own this ticket and rely on the other devs for support.

from alpha.

This done, yeah? (@aaronhans )

from alpha.

Not done yet, Chad is working on it @abquirarte

This done, yeah? (@aaronhans )

from alpha.

Went through a few iterations of this process and came up with the following requirements:

• should not noticeably slow down the build process
• should run regularly
• results of performance and accessibility tests should be stored so we can review trends
• show results in graphs we can showoff
• should allow budget setting and trigger alerts when thresholds are breached

We accomplished many of these after iterating on a solution integrated into the build process but did cause significant slowdown of the build testing run so we went with a SAAS solution from calibreapp.com which provides all of these features. It is possible to build this ourselves (calibre is using the open source lighthouse CLI tool) but it will take several days of engineering work to match the base featureset and we don't necessarily want to manage this platform ourselves. Got approval from @KimberlyglennTSM to get the $60/month plan for this service covered. We are currently running on a 2 week free trial which will expire 2/21.

from alpha.

from alpha.

Closing! ✅

from alpha.