Git Product home page Git Product logo

ctap2-test-tool's Introduction

Test Tool logo

CTAP2 test tool

The test suite intents to make it easier for developers to find bugs in their CTAP2 implementation. It supports CTAP 2.0 only, see supported features.

Disclaimer

Those tests reflect the author's interpretation of the specification. It is not to be confused with certification by the FIDO Alliance. Please check the FIDO Alliance web page for more information.

How to install

The build system is bazel. Please make sure you have all dependencies installed. Example command for Ubuntu:

apt-get install bazel libudev-dev autotools-dev autoconf automake libtool

On your first run, the build system will fetch all other necessary libraries using git. The tool is tested on Linux and MacOS with GCC 9 and higher.

How to run

⚠️ This tool will irreversibly delete all credentials on your device.

Running the tool without comments lists all avaiable devices. Select the device you want to test by passing --token_path. For Unix, if only one CTAP2 compatible device is plugged in, you can simply run:

./run.sh

For more control, try i.e.:

bazel run //:fido2_conformance
bazel run //:fido2_conformance -- --token_path=/dev/hidraw0

⚠️ Please do not plug in other security keys with the same product ID, or the tool might contact the wrong device during testing.

While running the test tool, you will be prompted to touch or replug your security key multiple times, to test various features.

Supported features

At the moment, we only support USB HID as a transport. We test the commands from CTAP 2.0. The security key must support resident keys and user presence. Also, security keys with displays are untested so far.

Fuzzing

In addition to the CTAP2 specification conformance test, we provide a proof-of-concept fuzzing tool. Please check fuzzing.md for a detailed guide.

Results

For more information on checking or contributing test results, please check results.md.

Contributing

If we didn't already test your security key or you have an updated version, please create a pull request with your result file!

If you want to contribute code, please check contributing.md.

ctap2-test-tool's People

Contributors

ebursztein avatar jaroban avatar kaczmarczyck avatar liamjm avatar mingxguo27 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ctap2-test-tool's Issues

Test item GetAssertionEmptyUserIdTest

{
  "description": "Tests if empty user IDs are omitted in the response.",
  "error_message": "Cannot make credential with an empty user ID.",
  "id": "get_assertion_empty_user_id",
  "observations": [
    "A prompt was expected, but not performed. Sometimes it is just not recognized if performed too fast.",
    "The failing error code is `CTAP2_ERR_PROCESSING`."
  ],
  "result": "fail",
  "tags": []
},

https://github.com/fido-alliance/fido-2-specs/pull/963
Although fido2 spec says empty account identifier is valid but can't be returned in get(), this case makes authenticator need to save some useless data...

Supplement on Nov-25
I can't find the string to say but can't be returned in get() in spec.

Test UTF-8 encoding of newPin and curPin

According to CTAP 2.0 Proposed Standard: https://fidoalliance.org/specs/fido-v2.0-ps-20190130/fido-client-to-authenticator-protocol-v2.0-ps-20190130.html

and

CTAP 2.1 Review Draft: https://fidoalliance.org/specs/fido2/fido-client-to-authenticator-protocol-v2.1-rd-20191217.html

and latest working draft, to authenticatorClientPIN:

"newPin" be the UTF-8 representation of "newPinUnicode".
"curPin" be the UTF-8 representation of "curPinUnicode"

So authenticator should check newPin and curPin against UTF-8 encoding, then test tool should test this with wrong/correct UTF-8 encoding input parameters.

Testing on Windows

The tool is currently used mainly on Linux and MacOS. We need to run all features on Windows to make sure it works.

Device information in reports

The device information should be more prominent in generated JSON files, to make it easier to distinguish versions.

To reproduce and sort tests, the report could contain

  • brand,
  • model,
  • revision,
  • supported extensions and options.

Also vendor and product ID can be moved to that section.

The same association problem exists for tests, that currently have names instead of easy-to-find IDs.

Test item MakeCredentialExcludeListDepth

test item: MakeCredentialExcludeListDepth

{
  "description": "Tests nested CBOR in the exclude list of MakeCredential.",
  "error_message": "Maximum CBOR nesting depth exceeded with array in credential descriptor transport list item in make credential command for key 5.",
  "id": "make_credential_exclude_list_depth",
  "observations": [
    "A prompt was sent unexpectedly.",
    "Expected error code `CTAP2_ERR_INVALID_CBOR`, got `CTAP2_OK`."
  ],
  "result": "fail",
  "tags": []
},

It should be allowed for authenticator to support more than 5
https://fidoalliance.org/specs/fido2/fido-client-to-authenticator-protocol-v2.1-rd-20191217.html#message-encoding

Authenticators MUST support at least 4 levels of CBOR nesting.

Thanks for providing this tool !!

More input type support for corpus testing

We want to test OpenSK with a corpus containing different types of inputs, also available for a future fuzzing feature. These inputs may be structured differently as:

  • CTAP2 commands parameters
  • CTAP1 U2F raw messages
  • Other CTAP commands
  • Raw data possibly breaking OpenSK

We should cover as many types as possible.

CBOR decoding fails in GetInfoPositiveTest

I've tried to run the tests with a FIDO2 authenticator we're developing, and I get the following error:

Tested device path: /dev/hidraw2
Tested device name: KeyVault
This tool will irreversibly delete all credentials on your device. If one of your plugged security keys stores anything important, unplug it now before continuing.
You have 10 seconds for the next touch after pressing enter.
Please replug the device, then hit enter.

Please touch your security key!
F1123 15:45:00.836627 26063 fido2_commands.cc:476] Check failed: decoded_response.has_value() CBOR decoding failed
*** Check failure stack trace: ***
    @     0x556a03e60ae0  google::LogMessage::Fail()
    @     0x556a03e60a1b  google::LogMessage::SendToLog()
    @     0x556a03e60341  google::LogMessage::Flush()
    @     0x556a03e63462  google::LogMessageFatal::~LogMessageFatal()
    @     0x556a03d567da  fido2_tests::fido2_commands::GetInfoPositiveTest()
    @     0x556a03d4924e  fido2_tests::CommandState::CommandState()
    @     0x556a03ce6a45  main
    @     0x7fcd18b2e0b3  __libc_start_main
    @     0x556a03ce66ae  _start
    @              (nil)  (unknown)
./run.sh: line 19: 26063 Aborted                 (core dumped) bazel run //:fido2_conformance -- --token_path="$path"

both for run.sh and run_fuzzing.sh, as normal user and as root (Ubuntu 20.10 inside VirtualBox on Windows 10). Webauthn.io works in the same environment. Since the error happens before the content of GetInfo is interpreted, I would guess that it's something to do with message length? Or maybe CBOR decoding? Also, it shouldn't be caused by a timeout - I can reproduce this in < 5s.

GetAssertionOptionUvTrueTest - pinUvAuthToken can't be used again for GetAssertionPositiveTest.

In latest ctap2.1 spec, step 6.1.2.14 If the "up" option is set to true:
The last step4. will Call clearUserPresentFlag(), clearUserVerifiedFlag(), and clearPinUvAuthTokenPermissionsExceptLbw().
So the result of following GetAssertionPositiveTest should be always failed due to below line use an old authToken directly.

options_builder.SetDefaultPinUvAuthParam(
      command_state->GetCurrentAuthToken());

Seems following test items in will have the same problem.

  • GetAssertionPinAuthTest
  • GetAssertionPinAuthMissingParameterTest

Support CTAP 2.1

The test tool currently supports commands from version 2.0. GetInfo informs about supported versions.

For old commands, tests should differentiate between authenticator versions.
For new commands, tests should only run if an authenticator claims it is compatible.

Test item MakeCredentialCredParamsTest

test item: MakeCredentialCredParamsTest
{
"description": "Tests entries in the credential parameters list.",
"error_message": "Falsely rejected cred params list with 1 good and 1 bad element.",
"id": "make_credential_cred_params",
"observations": [
"A prompt was expected, but not performed. Sometimes it is just not recognized if performed too fast.",
"The failing error code is CTAP2_ERR_UNSUPPORTED_ALGORITHM."
],
"result": "fail",
"tags": []
},

Webauthn specs says below in https://www.w3.org/TR/webauthn-2/#dom-publickeycredentialdescriptor-type

client platforms MUST ignore any PublicKeyCredentialDescriptor with an unknown type.

ctap2 spec has description for pubKeyCredParams(0x04) of authenticatorMakeCredential's command

PublicKeyCredentialParameters' algorithm identifiers are values that SHOULD be registered in the IANA COSE Algorithms registry

So authenticator should not accept type="non-existing type" with the alg value of COSE.

Inconsistent please touch / please don't touch messages

The banner to test timeouts warns not to touch the key regardless of all prompts (plural). However, it seems to only affect the next one prompt!

The first time, we have a message to confirm this: "The next touch prompt is valid again."

===========================================================
The next test checks if timeouts work properly. This time,
please do not touch the key, regardless all prompts, for 30
seconds. Check if you see a flashing LED on the device.
===========================================================
Please touch your security key!
Test successful: key was not touched for reset
The next touch prompt is valid again.
Test successful: reset not allowed more than 10 seconds after plugging in
Please touch your security key!

The second time, there is no such message. Waiting 30 seconds on each prompt (as requested by the banner) triggers a timeout error on the second prompt.

===========================================================
The next test checks if timeouts work properly. This time,
please do not touch the key, regardless all prompts, for 30
seconds. Check if you see a flashing LED on the device.
===========================================================
Please touch your security key!
Test successful: key was not touched for make credential
Test successful: the asserted credential shouldn't exist
Please touch your security key!
Failed test: accept displayName with non-ASCII characters - expected CTAP2_OK, got CTAP2_ERR_USER_ACTION_TIMEOUT

The prompts should be clearer. In addition to the banner, I suggest replacing Please touch your security key! by Please DO NOT touch your security key! on the affected lines.

Also, the banner should remove the regardless all prompts part, which would then be redundant.

uv option key true to an authenticator that is not advertising uv in getInfo option ID will fail with CTAP2_ERR_UNSUPPORTED_OPTION

{
  "description": "Tests is user verification set to true is accepted in MakeCredential.",
  "error_message": "The user verification option (true) was not accepted.",
  "id": "make_credential_option_uv_true",
  "observations": [
    "A prompt was expected, but not performed. Sometimes it is just not recognized if performed too fast.",
    "The failing error code is `CTAP2_ERR_UNSUPPORTED_OPTION`."
  ],
  "result": "fail",
  "tags": [
    "Client PIN"
  ]

For non bio devices (not advertising "uv" in getInfo options) sending the "uv" option key generates a CTAP2_ERR_INVALID_OPTION error in CTAP2.0 and CTAP2.1. This test should only be applied to authenticators advertising the "uv" optionID.

In CTAP2.0 a uv option key in the request will always generate a CTAP2_ERR_INVALID_OPTION for non bio keys.

In CTAP2.1 if no PUAT is provided and the uv option key is true the authenticator generates CTAP2_ERR_INVALID_OPTION for non bio keys.

Reset device

Move reset from #38 to the general corpus_test_main.cc. Include:

  • Reset key before testing.
  • Reset between processing different types of input.

client_pin_new_requirements_set_pin and client_pin_new_requirements_change_pin pin padding is to 64 bytes not 32

In CTAP2.1 the max pin length is 63 bytes and is padded out to 64 bytes.
In CTAP2.0 "The decrypted padded newPin should be of at least 64 bytes length"
So 64 bytes is what should be tested for.

Just for fun CTAP2.0 doesn't define errors.
For CTAP2.1 the error for too short is CTAP1_ERR_INVALID_PARAMETER someone needs to speak up if they want that changed.

For paddedNewPin being longer than 64 bytes nether spec mentions an error.

I will see about fixing that in CTAP2.1

ClientPinAuthBlockPinRetriesTest: "The correct PIN is not blocked when auth is blocked."

Hi all, I have a device that is failing this test and I'm trying to understand what it's doing/what the error means.

It seems like the test attempts 3 consecutive bad pin authentications (causing auth to be blocked) and then attempts a successful PIN and expects it to be rejected. However, from the language in the spec, it appears that the token is expected to accept this PIN. Intuitively the spec doesn't make sense here so I'm try `Authenticator performs following operations upon receiving the request:

* Authenticator performs following operations upon receiving the request:
  * If Authenticator does not receive mandatory parameters for this command, it returns CTAP2_ERR_MISSING_PARAMETER error.
  * If the retries counter is 0, return CTAP2_ERR_PIN_BLOCKED error.
  * Authenticator generates "sharedSecret": SHA-256((abG).x) using private key of authenticatorKeyAgreementKey, "a" and public key of platformKeyAgreementKey, "bG".
    * SHA-256 is done over only "x" curve point of "abG"
  * Authenticator decrements the retries counter by 1.
  * Authenticator decrypts pinHashEnc and verifies against its internal stored LEFT(SHA-256(curPin), 16).
    * If a mismatch is detected, the authenticator performs the following operations:
      * Authenticator generates a new "authenticatorKeyAgreementKey".
        * Generate a new ECDH P-256 key pair called "authenticatorKeyAgreementKey" denoted by (a, aG), where "a" denotes the private key and "aG" denotes the public key.
      * Authenticator returns errors according to following conditions:
        * If the retries counter is 0, return CTAP2_ERR_PIN_BLOCKED error.
        * If the authenticator sees 3 consecutive mismatches, it returns CTAP2_ERR_PIN_AUTH_BLOCKED, indicating that power cycling is needed for further operations. This is done so that malware running on the platform should not be able to block the device without user interaction.
        * Else return CTAP2_ERR_PIN_INVALID error.
    * Authenticator sets the retries counter to 8.
    * Authenticator returns encrypted pinToken using "sharedSecret": AES256-CBC(sharedSecret, IV=0, pinToken).
      * pinToken should be a multiple of 16 bytes (AES block length) without any padding or IV. There is no PKCS #7 padding used in this scheme.

According to this, CTAP2_ERR_PIN_AUTH_BLOCKED is only ever returned on a bad PIN

Crash report from OpenSK

Generate a proper crash report by:

  • Extending the RSP client implementation with useful packets.
  • Extending Monitor API.

Gracefully report a key that doesn't support CTAP2 protocol (was: Titan K40T)

My testing of Titan Security Key (K40T, USB-C) dies here:

Tested device path: /dev/hidraw6
Tested device name: ePass FIDO
This tool will irreversibly delete all credentials on your device. If one of your plugged security keys stores anything important, unplug it now before continuing.
You have 10 seconds for the next touch after pressing enter.
Please replug the device, then hit enter.

The failing error code is `CTAP1_ERR_INVALID_COMMAND`.
F1212 12:23:22.154443 3425572 device_tracker.cc:185] Check failed: condition Failed critical condition: Reset
*** Check failure stack trace: ***
    @     0x55b13e76b91e  google::LogMessage::Fail()
    @     0x55b13e76b859  google::LogMessage::SendToLog()
    @     0x55b13e76b17f  google::LogMessage::Flush()
    @     0x55b13e76e2a0  google::LogMessageFatal::~LogMessageFatal()
    @     0x55b13e675059  fido2_tests::DeviceTracker::AssertCondition()
    @     0x55b13e675253  fido2_tests::DeviceTracker::AssertResponse()
    @     0x55b13e64c8e0  fido2_tests::CommandState::Reset()
    @     0x55b13e64c292  fido2_tests::CommandState::CommandState()
    @     0x55b13e5eb8c5  main
    @     0x7fdb56be30b3  __libc_start_main
    @     0x55b13e5eb52e  _start
    @              (nil)  (unknown)
./run.sh: line 19: 3425572 Aborted                 (core dumped) bazel run //:fido2_conformance -- --token_path="$path"

Could it be a local issue?

EOF newline

The C++ linter should add a newline at the end of each file.

authenticatorGetInfo should use defaults for options

Thanks for this tooI. I had an error indicating Check failed: condition Failed critical condition: User presence support expected. because in the authenticatorGetInfo response the authenticator is not including the up flag. However, according to the specification (and the conformance tests) When an option is not present, the default is applied per table below so this option being absent is a valid configuration and the tool should instead use the default.

Do you agree to add this fix to the backlog?

GDB RSP behaviour

There are currently two issues with the remote connection:

  1. The connection is slower than expected. Sometimes a command requires a few resends (currently each receive has one second of timeout window) to get acknowledged.
  2. Command "g" (read general registers) gets unexpected respond. It should receive 17 (number of registers) * 4 (register length) hex bytes but sometimes gets longer response (with correct checksum).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.