edttool / edtt Goto Github PK

EDTT implicitly assumes that certain features are present. This poses problems when testing a configuration that does not follow these assumptions. Support for an ICS as created by LaunchStudio would enable selecting relevant tests automatically and adapt the behavior of test cases. An example for the latter would be that EDTT could check that feature bits are set or cleared according to what is found in the ICS (cf. discussion in #9).

With ICS support in place in EDTT, one could also consider auto-generating an ICS from a Zephyr Kconfig for an automated test flow.

Tests that verify GATT server "Insufficient Encryption Key Size" error code have couple issues:

1. they verify wrong ATT status code : should be ATT_ERROR_INSUFFICIENT_ENCRYPTION_KEY_SIZE (0x0c) instead of ATT_ERROR_INSUFFICIENT_ENCRYPTION (0x0f)
2. tests should pair with proper (small) key size before accessing characteristics and verifying returned status code

1 is easy to fix, 2 I have no idea on how this should be done, there is pairing_bv_01_c() which seems to be unused and this is only indication of pairing in those tests (unless I miss something)

This is affecting following tests:
GATT/SR/GAR/BI-11-C [Insufficient Encryption Key Size error - Reading Using Characteristic UUID]
GATT/SR/GAR/BI-17-C [Insufficient Encryption Key Size error - Reading Long Characteristic Value]
GATT/SR/GAR/BI-22-C [Insufficient Encryption Key Size error - Reading Multiple Characteristic Values]
GATT/SR/GAW/BI-06-C [Insufficient Encryption Key Size error - Writing Characteristic Value]
GATT/SR/GAW/BI-13-C [Insufficient Encryption Key Size error - Writing Long Characteristic Value]

As per LL testspecification LL.TS.5.1.1 chapter 4.1.5:
"If a test covers optional features and does not mention the feature exchange procedure, the IUT shall perform the feature exchange procedure before the test procedure is executed."
It seems that the feature exchange is not run as part of the test initialisation

Other pre-test setup may be missing
This applies to other tests as well

Running ll_verification with different devices as upper vs. lower tester fails test LL/CON/MAS/BV-13-C, because the test pass/fail validation assumes identical feature sets.

If for instance lower tester supports BT_LE_FEAT_BIT_MIN_USED_CHAN_PROC, and upper tester does not, it fails.

GAP/GAT/BV-04-C ignores the fact that the Peripheral Preferred Connection Parameters Characteristic may include values with "no specific value" (0xffff). The test case instead tries to create a connection with illegal parameters (connection interval 0xffff).

Checking that the status of readLocalResolvableAddress is 0 causes the test LL/DDI/SCN/BV-14-C to fail, since it returns 2.

The gatt_verification test does not filter on the L2CAP CID. This means that data sent on a different channel (e.g. the Low Energy L2CAP Signalling channel) is mixed into the attribute data and causing the test to fail.

GATT/SR/GAS/BV-01-C is /has been failing in Zephyr CI
It started failing after a minor timing change, and can be fixes by reordering the tests lists.
This may be an indication of the test being too time sensitive, or something else.

Related:
See zephyrproject-rtos/zephyr#70380

Tests shall not assume handle positions are static on the database, if a handle is required for a test perform a lookup into the database to resolve the handle at runtime otherwise new aditions to the database can break the tests making EDTT inefective to be used in CI as it would likely generate false positives and block new features such EATT:

zephyrproject-rtos/zephyr#23199

The DLE control procedure is called with different length & timing parameters.
An event is generated by the HCI returning the actual length & timing settings used.

The test needs to verify that the returned values are correct