identity-loadtest
Load testing scripts and tooling for the Login.gov, currently using locust
.
Local setup
Python and Locust
Install python3 and dependencies
brew install python
pip3 install -r requirements.txt
Login.gov IdP
Login IdP must be running with these settings in application.yml
telephony_adapter: 'test'
disable_email_sending: 'true'
enable_load_testing_mode: 'true'
enable_rate_limiting: 'false'
otp_delivery_blocklist_maxretry: 1000000
Note that it's also important to assure that the scrypt cost is configured the same as prod. The values will inherit from the idp-repo unless the values are overridden in the secrets s3 bucket.
Running Locust
You can only run one locustfile at a time, there are many to choose from that end in .locustfile.py
.
users
is the total number of concurrent Locust users.hatch-rate
is the number of users to spawn per second, starting from zero.
locust
cmd line arguments
Common --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Or omit --headless
and open http://localhost:8089 for a UI.
Add --csv=<base-name>
to generate CSV output
Adding new tests
Add new *.loucstfile.py
files to the project for new test scenarios.
Sign-Up load test
- This will create lots of users in your database
locust --locustfile load_testing/sign_up.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Sign-In load test
- You must run a rake task in the IdP before using this test, something like:
rake dev:random_users NUM_USERS=100
(source) - You also must pass in a matching
NUM_USERS=100
to the locust call.
NUM_USERS=100 locust --locustfile load_testing/sign_in.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Sign-In remembered device load test
Tests sign ins simulating a very high (90%) ratio of users who are signing back in using a remembered browser (device).
- You must run a rake task in the IdP before using this test, something like:
rake dev:random_users NUM_USERS=100'
(source) - You also must pass in a matching
NUM_USERS=100
to the locust call.
NUM_USERS=100 locust --locustfile load_testing/sign_in_remember_me.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Sign up + Sign-In load test
- This test mixes Sign-up and Sign-in together
- You must run the same rake task as above in the IdP before using this test
- You also must pass in a matching
NUM_USERS=100
to the locust call.
NUM_USERS=100 locust --locustfile load_testing/sign_up_sign_in.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
IAL2 load tests
- Same rules as above, but use
ial2_sign_*
filenames. - Uses "desktop proofing" experience, not mobile.
- Requires
mont-front.jpeg
andmont-back.jpeg
drivers license images andselfie.jpeg
NUM_USERS=100 locust --locustfile load_testing/ial2_sign_in.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
NUM_USERS=100 locust --locustfile load_testing/ial2_sign_up.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
SP Sign in load test
- This requires that
identity-oidc-sinatra
be running as an SP - This requires the
NUM_USERS
env varible - This requires the
SP_HOST
env varible, something likeSP_HOST=http://localhost:9292
NUM_USERS=100 SP_HOST=http://localhost:9292 locust --locustfile load_testing/sp_sign_in.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Production Simulator load test
This is a hybrid test with the test mix roughly matching Login.gov's workload. (Subject to change. See test source for details.)
The ratio of remembered devices for sign ins can be adjusted with the REMEMBERED_PERCENT variable. (Default: 60)
For uniformity and simple calculation, test ratios should add up to
10000 (1 == 0.01%) and can be adjusted by setting a corresponding
environment variable. The following are available, and defaults
can be found at the top of load_testing/production_simulator.locustfile.py
:
- RATIO_SIGN_IN: Sign in test using REMEMBERED_PERCENT remember me ratio.
- RATIO_SIGN_UP: Sign up test ratio.
- RATIO_SIGN_IN_AND_PROOF: Sign in followed by IAL2 proofing ratio.
- RATIO_SIGN_UP_AND_PROOF: Sign up followed by IAL2 proofing ratio.
- RATIO_SIGN_IN_USER_NOT_FOUND: Failed sign in with nonexistent user.
- RATIO_SIGN_IN_INCORRECT_PASSWORD: Failed sign in with bad password.
- RATIO_SIGN_IN_INCORRECT_SMS_OTP: Failed sign in with bad SMS OTP.
Test requirements:
- Requires prepopulated users (See Sign-In load test)
- Requires
mont-front.jpeg
andmont-back.jpeg
drivers license images andselfie.jpeg
(See IAL2 load tests) - You also must pass in a matching
NUM_USERS=100
to the locust call.
Example (including overrides of the sign in and sign up tests)
NUM_USERS=100 RATIO_SIGN_IN=5000 RATIO_SIGN_UP=1010 locust --locustfile load_testing/production_simulator.locustfile.py --host http://localhost:3000 --users 1 --hatch-rate 1 --run-time 15m --headless
Running the test suite
There are tests for these load tests, find them in the tests
folder.
# Run the tests
pytest
# Run the tests and show coverage
coverage run -m pytest
coverage report
If you install the CircleCI CLI you can test a CircleCI run in a local Docker container with circleci local execute
.
Debugging Locust scripts
The HTTP Library is called Requests: https://requests.readthedocs.io/en/master/
The python debugger should just work. Here are some commands The following will drop you into a debugger:
import pdb; pdb.set_trace()