draegerwerk / sdccc Goto Github PK

The test for BICEPS R0034_0 currently does not detect when a descriptor is removed and reinserted with a change that should've caused a version incrementation but didn't.

This invariant test case traverses the history of the mdib and compares each version of the mdib with the subsequent version, checking for each descriptor that is present that the descriptor with the same handle in the subsequent version is either unchanged, or has an incremented descriptor version if it was changed. If a descriptor is not present in the subsequent version, no comparison takes places for that descriptor.

What the test case isn't handling properly is when the descriptor it is currently checking was inserted in that version, but was also previously present within the mdib sequence and removed. A kind of backwards check needs to happen in such cases, otherwise changes to descriptors can be missed, leading to an incorrectly passing test case.

Although this pattern of test implementation is used in other tests as well, this issue is specific to this test case implementation. Another test that needs to do these backwards checks, R0038, already does so.

The test cases Biceps:C11-C15 fail when encountering duplicated Reports because they ensure that each Report contains changes. The test cases for Requirements Biceps:C5, Biceps:R5046_0, Biceps_B-284_0, Biceps:R0034_0, Biceps:R5003 can also fail when Reports are duplicated.

Unfortunately, when a test case subscribes to reports in addition to the general subscription initiated upon connecting, then SDCcc has multiple subscriptions for the same Report and hence those Reports are sent more than once.

Duplicating Reports is standard compliant and completely innocuous as they can be identified using their mdibVersion. Hence, our test cases should not fail in this case.

Currently, WSDLs embedded in {http://schemas.xmlsoap.org/ws/2004/09/mex}MetadataReference entries are not supported.
Please add support for those.

During the deployment of SDCcc 6.0.0, the Github Workflow included a version of Java 11 as well as 17, as it downloded a cache that still included 11. Releases should not include any other versions than the required one, even if they are available in a cache.

The method getManipulationDataByParametersAndManipulation should retrieve all manipulation data that matches the specified manipulation name and all specified manipulation parameters. However, manipulation data was also retrieved where only a subset of the manipulation parameters matched.

The following test cases of the release 6.0.0 are affected:

• 5-4-7_0_0
• 5-4-7_2
• 5-4-7_4
• 5-4-7_6_0
• 5-4-7_8
• 5-4-7_10
• 5-4-7_12_0
• 5-4-7_14
• 5-4-7_16

When testing DPWS R0019 (see requirement below), the implied value for the relationship property is not tested, the DUT can not be leave the property out.

Requirement:
R0019: A SERVICE MUST include a Message Information Header representing a [relationship] property of type wsa:Reply in each Response Message SOAP ENVELOPE the service generates.

Note with imlpied value:
Per WS-Addressing [WS-Addressing], a response SOAP ENVELOPE must include a wsa:RelatesTo SOAP ENVELOPE header block. Since “http://www.w3.org/2005/08/addressing/reply” is the default value for the [relationship] property, the RelationshipType attribute should be omitted from the wsa:RelatesTo SOAP ENVELOPE header block.

Dominik Stegemann,
SurgiTAIX AG

The software lacks Linux support, preventing users on Linux-based systems from utilizing it.
Proposed Solution

Create an AppImage package for Linux compatibility.
Benefits

Distribution-agnostic
No installation required
Easy updates

Thank you!

• depends on the SDCri version, which needs an update to 2.1.
• illegal access prevents compiling and running in OpenJDK/Oracle JDK 17.0

Due to compatibility issues, message encoding determination has been turned off and messages are assumed to be encoded in UTF-8.

Currently, safe Data Transmission (i.e. Dual-Channel Remote Control) as defined in Chapter 9 of the MDPWS standard is not supported. Please add support for this.

Since release 6.0.0 multiple http clients are erroneously used. If one is idle for a longer period of time, the connection may have been closed on the server side without our http client knowing about it, resulting in a stale connection. When this stale connection is used, it could result in a NoHttpResponseException being thrown.

Currently, in the Glue:R0036_0 test case, the Report-triggers are passed as closures. Passing them as overrides would be simpler and more readable. Please also remove the TODO linked to this issue.

see TODO in source code that contains the URL of this ticket.

SDCcc does currently not support wsp:PolicyURIs. When run against a DUT using them, SDCcc's DirectWSDLTest will hence fail to notify the user that the DUT is using unsupported features.

When done, please do not forget to remove the TODO comment in com.draeger.medical.sdccc.tests.mdpws.direct.DirectWSDLTest: 478.

While running a test that uses "testRequirement547", namely "testRequirement547120", I found that when the changes made by "SetComponentActivation" are given a timestamp in the "SetMetricStatus" time frame then those changes are checked and possibly make the test fail. Those timestamps are allowed due to the buffering time of 5 seconds. Although they could also occur and be valid without it, it is just that without it the association would be unique, but simply removing the buffering is also not an option due to enabling other incorrect failures then.

For example the "ActivationState" is set to "Off" via "SetComponentActivation" and then to "On" indirectly via "SetMetricStatus", but the first update is given a timestamp that is greater than the starting timestamp for the "SetMetricStatus" manipulation, which results in "testRequirement547120" incorrectly failing, because it verifies the first message in the respective time frame.

This is a serious bug that affects not just the mentioned test case, but all that use this mechanism. Also it seems to rather be an issue with the test concept than a minor implementation mistake.

Currently, the TestCase for Glue:R0036_0 needs to trigger an EpisodicContextReport for working. When the Device does not support this report, it is unable to test anything.

Using the EpisodicMetricReport instead offers the benefit, that unlike the EpisodicContextReport, the EpisodicMetricReport does not need to be triggered, which would simplify the test. There are also only few devices not supporting the EpisodicMetricReport.

currently 1.3.0.13 is used, but we should use a released version e.g. 1.3.0 or 1.4.0 (see https://github.com/Draegerwerk/t2iapi/releases)

Currently, SDCcc does not support the following HTTP Headers in Test Cases which use messages from the database:

• transfer-encoding
• content-length
• content-encoding
• content-type: multipart/related

Please add support for those Headers.

There was an oversight in #17 regarding the introduction of a bug, where the GLUE_R0036_0 test case fails to subscribe to EpisodicContextReports.

This does not get noticed in the build workflow (i.e. unittests) and in a manual test run it is not obvious that it is an error, because the log only shows it through a warning and through a failure of the device under test instead of the test itself.

I found this during a detailed runtime analysis for #20 .

currently there is no release with version >=3.0.0, but we need such a release

bump the dependency version (sdcriVersion) when such a release is available

Currently, the Supporting Documents (containing UnitTest-Report, Checkstyle- and SpotBugs-Reports) are generated manually and attached to the Release manually.

As this manual step is unnecessary, please adapt the Release workflow to generate and attach them automatically.

NOTE: the Supporting Documents usually also contains the UnitTests of JUnit in the version used by SDCcc. As this step cannot be automated, only the Reports mentioned above should be generated automatically.

During testing with T2IAPI, some tests were identified to require a delay between manipulating the MDIB / sending out the required reports and sending out the manipulation result via T2IAPI.

Specify, which tests are affected by this. This includes

• BICEPS.R0125
• any SetAlertActivation transaction

Dominik Stegemann
SurgiTAIX AG

There is a handle method implemented for the glue R0036 test, which does not read on the input stream that is given and thus the jetty interceptors do not copy the data for database transfer, which results in http bodies missing in the database.

During testing, usage of T2IAPI was sometimes unclear. Provide a section in the ReadMe that gives a short list of what to look out for when using T2IAPI.

Examples:

• Encryption mode
• How to use the response messages
• Timings and delays

Dominik Stegemann,
SurgiTAIX AG

The Requirement Biceps:R5025_0 ist applicable to ALL DescriptionModificationReports (Crt, Upt & Del) and hence also applies to Devices that do not support removing & inserting Descriptors.
However, the Precondition of the SDCcc test case for Biceps:R5025_0 tries to trigger Crt and Del DescriptionModificationReports by removing and re-inserting all Descriptors listed by the getRemovableDescriptors() Manipulation. This does not work for such devices that do not support it.

We need a provider that can handle everything that can be done for "our" requirements.

Improvement Idea: Let the CI-Server run SDCcc against SDCri.

In order to do that, one would have to build a t2iapi server for SDCri that supports as many manipulations as possible create the setup for such an integration test in CI.

The benefits would be:

• a continuous reading about how well SDCcc and SDCri fit together (of course, not all tests will run green at the beginning, but the ones that do fail will provide useful indications on either manipulations that are implemented incorrectly or at standard-compliance-issues in SDCri)
• once all manipulations are implemented: An example t2iapi server implementation that is highly useful for implementers of t2iapi servers for users.
• once all tests run green: A continuous guarantee of standard-compliance for SDCri and a continuous guarantee that SDCcc's test suite is satisfiable.

SDCcc.TLS.EnabledProtocols in the config is read in, but not actually applied and thus the default of [TLSv1.2, TLSv1.3] is always used no matter what the user provides via SDCcc.TLS.EnabledProtocols .

see TODO in source code that contains the ID of this ticket

• WSDL fetching does not support all methods (mdpws:R0010, mdpws:R0011, mdpws:R0014)
• wsp:PolicyURIs checking is not supported (mdpws:R0010, mdpws:R0011)
• ArchiveService messages are unsupported (mdpws:R0006)
• Multipart/Related Content-Type is unsupported (mdpws:R0006)
• SDC Glue Annex B should be tested, but doesn't have a requirement attached. Maybe BICEPS:R0062 etc. which describe required and optional services are the correct spot?

NOTE: As stated in the README.md, when a feature is unsupported, SDCcc cannot be applied to devices using it as this may cause false positives or false negatives in some test cases. In order to minimize this risk, SDCcc's test cases are designed to fail when they notice that the device uses unsupported features. This has no effect when applying it to devices using supported features only, but significantly reduces the risk induced by unsupported features as the Test engineer will surely notice that something is wrong.

There was a warning in one of our GitHub checks:

test-java-snapshot
Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: GabrielBB/xvfb-action

Currently, SDCcc assumes that the encoding declared by the Device under Test in the HTTP Header or XML declaration is correct and decodes the messages accordingly.

This process using the String(byteArray, charset)-Constructor, which silently replaces unmappable characters by a charset-specific "Unmapped"-character. Hence,

• SDCcc does not notice when the encoding is set incorrectly, and
• in the worst case, the unmapped-Characters resulting from an incorrectly set encoding might even cause FalseNegatives in our Invariant Tests.
• Since the Message Content is partly lost (replaced by Unmapped-Characters), this additionally hinders Confirmability of SDCcc Test Results.

It would hence be better to use the CharsetDecoder class instead of the String-Constructor mentioned above, as it allows detecting unmappable character during the decoding process. It would hence become possible to invalidate the Test run when encountering unmappable Characters which indicate an incorrectly-set encoding and provide the user with an informative reason.

Hint: start from the TODO with this Ticket's Number in MessageStorage.

The setComponentActivation method incorrectly passes the string value of the ComponentActivation instead of the value. This results in no test data being available for any tests that require this manipulation data.

see TODO in source code that contains the URL of this issue.

The Archive Service is severely underspecified in BICEPS and thus not supported by SDCcc (see #4 ) . Hence it's data cannot be included in the Test case for MDPWS:R0006. This issue is hence BLOCKED until the BICEPS standard is revised.

Test DPWS.R0031 tests that no SOAP Fault is generated, when a specific ReplyTo address is set.

The test conditions are not clear (partly as the standard is ambigous / unclear there as well).
HTTP requires a response to the message in any case, various scenarios were tested:

• Not responding causes an HTTP timeout
• Responding with a status code indicating an error (4xx and 5xx) fails, as SDCcc does not expect an error code
• Responding with a 200 and an error text indicating that the message failed leads to a parsing error, as SDCcc tries to convert the message to XML
• Responding with an empty 200 message succeeds. This however makes no sense, as the message is clearly failed.

Consider removing this test or clarify the behavior.

Dominik Stegemann,
SurgiTAIX AG