dmtf / redfish-protocol-validator Goto Github PK

Add a test_conf.json file.

Hi,
When I run with version 1.2.3 it gets blocked at AttributeError: 'list' object has no attribute 'get'.

I checked the patch_array_save in service_requests.py according to the debug log, but I am not sure whether the root cause is related to this. Could you please help explain?

Thanks,
Felix

Hi,

I can run the test with Tag 1.2.1 successfully.

But when I run the test with Tag 1.2.2 or master, the test is getting stopped.
The test was not completed but got error here.

Error:
DEBUG:urllib3.connectionpool:https://172.31.96.168:443 "DELETE /redfish/v1/SessionService/Sessions/16b19644b521f25fa2200dee90c56178 HTTP/1.1" 204 0
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/rf_protocol_validator.py", line 9, in
main()
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/redfish_protocol_validator/console_scripts.py", line 99, in main
perform_tests(sut)
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/redfish_protocol_validator/console_scripts.py", line 37, in perform_tests
security_details.test_security_details(sut)
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/redfish_protocol_validator/security_details.py", line 1089, in test_security_details
test_authentication(sut)
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/redfish_protocol_validator/security_details.py", line 1072, in test_authentication
test_password_change_required(sut)
File "/home/crb/redfish_protocol_2.1.3/Redfish-Protocol-Validator/redfish_protocol_validator/security_details.py", line 842, in test_password_change_required
data = response.json()
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

So can you please conform that this is a tool issue or our redfish stack implementation issue?

Thanks
Afrin RasfiyaM

In looking at the code for this test:
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/assertions/accounts.py#L257

It looks like the flow:
Requests the user from AccountService, stores the etag.
Changes the password (using the appropriate etag in If-Match).
Attempts to do another operation with If-Match set to the step 1 etag value.

The issue I'm seeing here on one system is that because the Password property is not returned as part of the payload (it's always set to null) the strong etag for the resource doesn't change, given that, per RFC7232.

Strong comparison: two entity-tags are equivalent if both are not weak and their opaque-tags match character-by-character.

This would imply that, given the character-by-character ManagerAccount resource is identical, the same ETag value could be returned.

This is also backed up by the Redfish spec which says.

An ETag can be:
• A hash
.....

Section 6.5 of the Redfish spec (by my reading) has nil to say about any special handling for payloads that have semantically changed, but contain the same data.

My guess is that this test was coded up against an implementation that had some internal semantics for generating etags from patched resources, was tracking modification dates as part of the etag implementation, or happened to implement PasswordExpiration, which likely changed when the password was changed, thereby changing the bytes in the response. Assuming my analysis is right (which is maybe 50/50 at this point), there's a couple things we could do.

• Change the atom to patch a different property for that portion of the test, thus changing the etag for implementations that use strong etag semantics.
• Update the spec to call out this case, and assert that caching is required to be done on the semantic equivalent when properties are omitted. Hash is not allowed to be used in these cases.
• Have a required "LastUpdatedTime" property that gets updated on patch to password, that would invalidate the strong etag header.
  Something else?

Hi @mraineri ,

I ran into a problem while running the Redfish-Protocol-Validator.

I would like to understand: is the request eventually sent through the highlighted function below http or https?

Thanks.

The test RESP_HEADERS_ALLOW_METHOD_NOT_ALLOWED uses the HTTP TRACE method to perform the check. We are running into a security conflict here with an implementation based on Apache. The Apache configuration option "TraceEnable Off" results in TRACE operations being denied before the request even reaches the Redfish Service. But Apache is not filling out the Allow header, nor providing a means to do so, and so this results in a failure from the validator:

RESP_HEADERS_ALLOW_METHOD_NOT_ALLOWED: "[The Allow header] shall be returned with the HTTP 405 (Method Not Allowed) status code to indicate the valid methods for the request URI."
...
Result, Method, Status code, URI, Message,
FAIL, TRACE, 405, /redfish/v1/, The Allow header was missing from response to TRACE request to /redfish/v1/

While it is possible to work around this, doing so would be code solely to work around the validation test itself, rather than any actual issue with the Redfish service itself.

Suggest that the test be changed to use another valid HTTP operation that can be reliably "not supported" to perform this validation. Our suggestion would be to perform a DELETE on the ServiceRoot (\redfish\v1) instead, since this should never be a supported operation.

The tests case below return status code not consistent:

• REQ_QUERY_UNSUPPORTED_DOLLAR_PARAMS: "Services shall return the HTTP 501 Not Implemented status code for any unsupported query parameters that start with $."
• REQ_QUERY_UNSUPPORTED_PARAMS_EXT_ERROR: "Services shall return an extended error that indicates the unsupported query parameters for this resource."

Attach the following picture for your reference:
Old version

Repaired version

Attach the following 501 status code message for your reference:
{
"error": {
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_0_8.Message",
"Message": "Querying is not supported by the implementation.",
"MessageId": "Base.1.5.QueryNotSupported",
"Resolution": "Remove the query parameters and resubmit the request if the operation failed.",
"Severity": "Warning"
}
],
"code": "Base.1.5.QueryNotSupported",
"message": "Querying is not supported by the implementation."
}
}

Why the second test case return status code change to 501 will fail? Is this issue caused by error message mentioned in Message?

If we use following command to test

python3 rf_protocol_validator.py -r https://172.17.8.47:8889 -u admin -p YourPaSsWoRd --no-cert-check --avoid-http-redirect

will cause

Exception caught while trying to retrieve and decode certificate for https://172.17.8.47:8889; exception: [Errno -2] Name or service not known

and def test_certs_conform_to_x509v3 in assertions/security_details.py
...
conn.connect((rhost.netloc, port))
...

value of rhost.netloc is 172.17.4.47:8889 , but it should be 172.17.8.47 or the connection will fail

The tool fails the following tests when verifying response bodies for POST operations to create new resources:

• PROTO_JSON_ACCEPTED
• PROTO_JSON_ALL_RESOURCES

While this is absolutely true for GET responses, POST for creating a new resource does not require any response body. The spec currently allows for a 201 Created to be returned without a body (with some exceptions that are called out in the specification).

For POST (create) requests, the response body may include a representation of the newly created resource.

As mentioned in the title, Attach the following examples for reference. Are there any certification procedures that leads to respond 401 code?

Tool Version: 1.0.4
python3 rf_protocol_validator.py -r https://{IP} -u USERNAME -p PASSWORD --avoid-http-redirect --no-cert-check

Example:

1. REQ_HEADERS_ACCEPT: "Redfish Services shall process the [Accept header] in the following table as defined by the HTTP 1.1 specification if the value in the Service Requirement column is set to "Yes", or if the value is "Conditional" under the conditions noted in the Description column.

Tool report:
Message: GET request to /redfish/v1/EventService/SSE failed with status code 401 using header Accept: text/event-stream; extended error: While attempting to establish a connection to /redfish/v1/EventService/SSE, the service was denied access.
Manual test result: 200

Tool report:
Message: GET request to /redfish/v1/EventService/SSE failed with status code 401 using header Accept: text/event-stream;charset=utf-8; extended error: While attempting to establish a connection to /redfish/v1/EventService/SSE, the service was denied access.
Manual test result: 200

2. SERV_SSE_UNSUCCESSFUL_RESPONSE: "Unsuccessful resource responses for SSE shall return an HTTP status code of 400 or greater, have a Content-Type header set as "application/json" or "application/json;charset=utf-8", and contain a JSON object in the response body, as described in Error responses, which details the error or errors.

Tool report:
URI: GET /redfish/v1/EventService/SSE
Tool status code: 401
Message: One or more problems found with error response: Property "code" missing from "error" complex property. Property "message" missing from "error" complex property.
Manual test result: 200

3. SEC_PWD_CHANGE_REQ_ALLOW_SESSION_LOGIN: "[When using an account with PasswordChangeRequired set to true] the service shall allow a session login and include a @Message.ExtendedInfo object in the response containing the PasswordChangeRequired message from the Base Message Registry.

Tool report:
Message: POST request to /redfish/v1/SessionService/Sessions using account with PasswordChangeRequired set failed with status 401
Manual test result: 403

The increase to depending on requests 2.30+ in aaed042 makes this tool no longer work with python 3.6, due to the newest versions of requests no longer supporting that version of python.

But pypi for this tool claims support with python 3.5+ https://pypi.org/project/redfish-protocol-validator/

I'm hoping the requests dependency could be dropped to older versions again? If 2.30+ is really needed, then the supported python versions need to be updated in pypi.

In the last couple of days, I've started getting this new error: REQ_PATCH_ODATA_PROPS: "A client only provides OData annotations: Services shall return the HTTP 400 Bad Request status code with the NoOperation message from the Base Message Registry or one of the modification success responses."

But when I look for this in the spec, I see "shall" not "should". Maybe it's OK to still flag errors for services not doing what the spec says they "should" do, but the spec does make a clear distinction between the meaning of "shall" and "should".

Continue to crash during execution
python3 rf_protocol_validator.py --user Administrator --password superuser -r https//10.8.1.164 --no-cert-check

Python 3.6
Name: urllib3
Version: 1.14
Name: requests
Version: 2.9.1

/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 247, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "rf_protocol_validator.py", line 110, in
main()
File "rf_protocol_validator.py", line 95, in main
perform_tests(sut)
File "rf_protocol_validator.py", line 33, in perform_tests
service_requests.test_service_requests(sut)
File "/Redfish_Protocol_Validator/Redfish-Protocol-Validator-master/assertions/service_requests.py", line 1372, in test_service_requests
test_request_headers(sut)
File "/Redfish_Protocol_Validator/Redfish-Protocol-Validator-master/assertions/service_requests.py", line 1263, in test_request_headers
test_accept_header(sut)
File "/Redfish_Protocol_Validator/Redfish-Protocol-Validator-master/assertions/service_requests.py", line 73, in test_accept_header
test_header(sut, header, header_values, uri, assertion, stream=True)
File "/Redfish_Protocol_Validator/Redfish-Protocol-Validator-master/assertions/service_requests.py", line 21, in test_header
stream=stream)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 426, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
[root@scottsys Redfish-Protocol-Validator-master]# /usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning:

We have not run the protocol validator in a few months and have made some redfish changes. We now get the following error and warning which we have not seen before. We know we have some implementation issues on NTP but have not had an issue with PATCH, we are puzzled by the CERTIFICATE_VERIFY_FAILED

ERROR:root:Caught exception while creating or patching accounts; Exception: HTTPSConnectionPool(host='10.0.127.170', port=443): Max retries exceeded with url: /redfish/v1/AccountService/Accounts/6 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)); continuing with test

WARNING:root:Attempt to PATCH /redfish/v1/Managers/Self/NetworkProtocol to restore the original NTPServers array failed with status 401; PATCH payload: {'NTP': {'NTPServers': ['pool.ntp.org', 'time.nist.gov']}}

Testing an implementation which has a bug that produces an invalid JSON payload, the tool crashes due with JSONDecodeError. This makes it difficult to debug since the traceback doesn't provide the URI. Tool should check/catch basic JSON decode faults and display an output that includes the URI of the faulty payload.

Some of the negative testing done by the tool for PATCH don't allow for the fact that a service might completely reject a request rather than perform a partial success. This is the clause that allows for the request to be rejected completely:

The implementation may reject the update on certain properties based on its own policies and, in this case, not make
the requested update.

For example, REQ_PATCH_MIXED_PROPS is expecting the service returns a 200 OK with messages showing the properties not updated, but doesn't accommodate services that want to reject the operation entirely. There may be other tests that need to be checked.

Tool Version: 1.0.4
python3 rf_protocol_validator.py -r https://{IP} -u USERNAME -p PASSWORD --avoid-http-redirect --no-cert-check

SERV_EVENT_POST_RESP: "If the [Event Service] subscription request succeeds, the service shall return an HTTP 201 Created status code, and the Location header that contains a URI of the newly created subscription resource.

Message:
Response from event subscription POST request to /redfish/v1/EventService/Subscriptions returned status code 400; expected 201; extended error: IP provided is invalid
Manual test result:
Check returned status code 201, and the Location header that contains a URI of the newly created subscription resource.

Are there any verification procedures that prevent to verify completely? Did I lack step to verify this test case?

Dear @mraineri ,

I am a user of the Redfish Protocol Validator testing tool for a device that supports the Redfish service in our local area network. I am writing to inquire if there are any ongoing or upcoming plans to address pending work (# TODOs) on the following programs:

https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/constants.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/resources.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/security_details.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/service_details.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/service_requests.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/service_responses.py
https://github.com/DMTF/Redfish-Protocol-Validator/blob/master/redfish_protocol_validator/system_under_test.py

Additionally, I would appreciate it if you could point me towards the "notes on testing by Mike R." that are on this repository.

Thank you for your assistance.

Sincerely,
Leo

In the account PATCH test, I got the

REQ_PATCH_MIXED_PROPS: "Modify several properties where one or more properties can never be updated: Services shall return the HTTP 200 OK status code and a resource representation with a message annotation that lists the non-updatable properties."

The service response returned status code 400; expected 200; extended error: {'BogusProp': 'foo', 'Password': 'glqnOoQ6Fx'}

Does it mean even the PATCH body contains properties not defined in these resources(like BogusProp), we still need return OK status but not bad request status? Does any standard mention this part?

Thank you in advance.

Hi,
For the test case SEC_PWD_CHANGE_REQ_ALLOW_SESSION_LOGIN, the report message return POST /redfish/v1/SessionService/Sessions using account with PasswordChangeRequired set failed with status 401. I tried to manual test and used the account with PasswordChangeRequired set true to create session then returned 201 status code and include @Message.ExtendedInfo in the response.
Report:

Manual:

We also found the test case SEC_PWD_CHANGE_REQ_ALLOW_GET_ACCOUNT show the message No GET request to account URI found using account with PasswordChangeRequired set. Is this situation means the account with PasswordChangeRequired set true maybe no created success? so that payload may no use the corresponding account when POST request to session.
If it is not as I thought, please let me know the SEC_PWD_CHANGE_REQ_ALLOW_SESSION_LOGIN test program or step.
Thanks.

Hi,
As per redfish specification etag is recommended and not mandatory to implement

**To reduce unnecessary RESTful accesses to resources, the Redfish service should support the

association of a separate entity tag (ETag) with each resource.**

Is my understanding correct?
But Redfish protocol Validator is throwing error as User Accounts shall support etags and atomic operations

Thanks,
Mani

Continue adding test assertions for the Service requests section of the specification.

As part of working on #58, CI was failing due to it pulling the latest urllib3, which came out recently. urllib3 2.0 drops support for older versions of openssl, The following error is produced:

ERROR: Failure: ImportError (urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2g  1 Mar 2016'. See: https://github.com/urllib3/urllib3/issues/2168)

Users keeping up with the latest versions of OpenSSL should be unaffected, but it would be good to have an advisory about this and see how we can get our CI updated to use newer versions.

There are some services that respond with 200 OK to PATCH requests, but their response body is either an error response with messages indicating "success", or simply the extended info messages arrays also indicating "success". Per the spec, 200 OK is supposed to contain the modified resource in response to a PATCH; the extended info array could be in there to convey warnings or properties that could not be modified, but that's an optional addition to the response.

New AccountService property allows for WWW-Authenticate to not be returned. Need to update the test to cycle through the possible values and test for the appropriate behavior.

Is query parameters testing in the roadmap of this repo?

If so, what's the planned workflow? I would imagine we firstly check the |ProtocolFeaturesSupported| on service root, if certain parameters are supported, we pick some resources (or walk through every resource), apply query parameters on them, and finally check the result according to these rules:

1. only; response shall be exactly the same as querying the resource directly
2. expand; response shall be exactly the same as querying the list of sub-resources sequentially
3. select; selected properties and values shall be exactly the same as they are in the original response; revered properties are turned regardlessly
4. filter; pending..

Any better ideas?

          @FelixD999 I'm looking at the AL header test a bit more and I noticed that it looks like the contents of your header appear to be wrapped in double quotes; this is not expected. The test is examining the "path" portion of the URL and ignoring the rest.

Originally posted by @mraineri in #51 (comment)

Hi,
After verified, the content of the AL header indeed be wrapped in double quotes.

I want to ask if there is an opportunity to optimize the tool so that the test process can skip the quotes.
Refered the UPnP Device Architecture 2.0, but there seems to be no specification about the type of AL mentioned.

Thanks,
Felix

We get the below warning in the report:

The Validator PATCH a ManagerAccount resource with "RoleId":"ReadOnly", "UserName": "rfpv8f12" ..., and then using the identity rfpv8f12 to PATCH this resource with again, and Validator expects to get 401 as PASS, and other errors as WARN.
For this case, I have below questions:

1. Should Validator expect 403 for incorrect privilege here?
2. A ReadOnly user should have available to change his Password, right?
3. In our report, it looks like Validator uses incorrect Etag to PATCH Passord, right?
4. Should Validator adjust this case? For example: using the identity rfpv8f12 to PATCH other existed ManagerAccount resource.

Some Redfish services will detect too many invalid login attempts and block incoming access to the offending client for some period of time. The Protocol-Validator does a very minimal set of intentional bad authentication requests to test protocol specified behavior.

I discovered this behavior recently started being triggered for one of the services I normally test against. After looking into it, I discovered a couple of bugs that were causing some unintentional bad authentication requests.

Hi,

I have a few questions I would like to ask as follows, with screenshots attached:

1. SERV_SSDP_AL_HEADER_POINTS_TO_SERVICE_ROOT:
   I want to confirm that the Redfish Service Root URL expected result is /redfish/v1/?
   I refer to RFC1738 and see the URL contains "HTTP://", I'm not sure if I'm missing something.

2. SERV_SSDP_CAN_BE_DISABLED:
   The message mentions 'Service responded to SSDP query after disabling SSDP'.
   Is this mean we cannot find BMC ip with SSDP protocol disabled by SSDP query?

3. SERV_SSDP_ST_HEADER_FORMAT: & SERV_SSDP_UUID_IN_CANONICAL_FORMAT:
   The messages mentions 'Returned ST header does not match the regex pattern"^urn:dmtf-org:service:redfish-rest:1(:\d)?$" and The unique ID found does not match the regex pattern "^uuid:([a-f0-9-]+)::urn:dmtf-org:service:redfish-rest:1(:\d)?$"'
   I refer from the DSP0266_1.15.1, is it because the "15" of "urn:dmtf-org:service:redfish-rest:1:15" limited by the minor versions regex?

Thanks,
Felix

Hi,
For the three teat cases about NTP Server below:

1. REQ_PATCH_ARRAY_ELEMENT_REMOVE: "Within a PATCH request, the service shall accept null to remove an element."
2. REQ_PATCH_ARRAY_ELEMENT_UNCHANGED: "Within a PATCH request, the service shall accept an empty object {} to leave an element unchanged"
3. REQ_PATCH_ARRAY_TRUNCATE: "A PATCH request with fewer elements than in the current array shall remove the remaining elements of the array."

Due to the BMC default value only two NTP servers in our design, Redfish-Protocol-Validator tool will PATCH three NTP servers on the payload. Is there any way to not limit the need for three NTP servers on test tool?

Hi @mraineri ,

In https://github.com/DMTF/Redfish-Protocol-Validator/releases, I noticed that when I download https://github.com/DMTF/Redfish-Protocol-Validator/archive/refs/tags/1.1.9.zip, the downloaded zip file appears to be 1.1.8.

I don't see this behavior in other repositories like https://github.com/DMTF/Redfish-Interop-Validator/releases.
For example, https://github.com/DMTF/Redfish-Interop-Validator/archive/refs/tags/2.1.4.zip

Hi @mraineri ,

I would like to request for some guidance on assertion REQ_DATA_MOD_ERRORS. This particular assertion currently return as "NOT_TESTED" with a message "No failed POST responses found; unable to test this assertion".

I would like to setup/prepare the system under test such that this assertion be evaluated as either Test passed or Test failed given the for loop within test_data_mod_errors in service_requests.py.

Can you recommend an approach to do this?

Thanks

Describe the bug
Redfish Protocol Validator report FAIL in POST /redfish/v1/AccountService/Accounts

Environment
Enabled TLS
Redfish Protocol Validator Version: 1.1.6
https://github.com/DMTF/Redfish-Protocol-Validator

To Reproduce

Run Redfish Protocol Validator
python3 rf_protocol_validator.py --user root --password 0penBmc
--rhost https://BMC_IP --no-cert-check --avoid-http-redirect --log-level DEBUG
The tool will report
Method POST
Status code 400
URI /redfish/v1/AccountService/Accounts
Message POST request to URI /redfish/v1/AccountService/Accounts failed with 400; expected 405; extended error

Below test case is identified to be returning 400 but expected 405.

POST /redfish/v1/AccountService/Accounts HTTP/1.1\r\nHost: 127.0.0.1:2443\r\nUser-Agent: python-requests/2.28.1\r\nAccept-Encoding: identity\r\nAccept: /\r\nConnection: keep-alive\r\nX-Auth-Token: wcyxd8KgOI0Rd3uERDy3\r\nOData-Version: 4.0\r\nContent-Length: 72\r\nContent-Type: application/json\r\n\r\n'
'{"UserName": "rfpv397e", "Password": "7gCdKlziqi", "RoleId": "ReadOnly"}'

Please provide reason why this test case is expected to be returning 405 not 400 since POST method is allowed by bmcweb for resource provided in this request for account service.

This is at least seen with REQ_PATCH_BAD_PROP. Currently the spec states the following with regards to error handling with bad properties in a PATCH request:

If all properties in the update request are read-only, unknown, or unsupported, but the resource can be updated, the service shall return the HTTP 400 Bad Request status code and an error response with messages that show the non-updatable properties.

The "error response" clause does not make the usage if @Message.ExtendedInfo required (but it does recommend it). Depending on the circumstances, it's possible to encode everything in the top-level "code" and "message" properties and not need @Message.ExtendedInfo.

This test case check if the array of response is the same with expected_array but it will report below cases failed.

1. fixed length array: there are null members are appended in the end of array.
2. the member has default value: some implemetations have default value for members, for example: "" or "0.0.0.0". If the member is deleted, it is given the default value.

I don't sure if the case 2 conforms to spec but validator should let case 1 pass, right?

I'm seeing this tool assert that SERV_SSE_EVENT_DEST_DELETED_ON_CLOSE fails, but that's only because it gives up after 3 seconds of closing the client side of the SSE connection.

At least in the framework I use for redfish SSE (WSGI), a close of the SSE client is only recognized on the next attempt to write to that connection, which will be on the next SSE heartbeat if no event occurs. And 3 seconds seems quite short for SSE heartbeat events to occur? Mine can take up to 20 seconds.

In test_sse_close_connection_if_event_dest_deleted link to code , we are deleting the event destination URI and checking the SSE stream to confirm whether the stream got closed or not.

The comment says # give the service up to 5 seconds to close the stream link to code but in the logic it will just wait for one second before reading from the sse_stream and break the loop directly and throw the error without waiting for remaining 4 seconds link to code.

Whether this logic is correct ? or we can give a delay of 5 seconds directly before looping the stream link to code

Hi,

Due to our design is return 202 after disabling SSDP, not sure if the root cause of the test case fail is about latency issues such that the Validator would still catch the response after disabling SSDP.

Thasnks,
Felix

Service_details.py:1027 hangs while getting an sse event.

System Configuration:
Ubuntu Jammy
Python 3.9
Urllib3 2.2.1
Sseclient-py 1.8.0
Requests 2.31.0
Command Line Parameters: https://127.0.0.1:8443, no-cert-check

Observed Sequence:
Server has already sent event to SSE stream

service_details.py:1027 calls read_sse_events()
service_details.py:645 retrieves event from event stream() - at this point there are no more bytes available in the stream and the first and only event has been read. Debug information shows "Dispatching message event..."
service_details:py:646 adds the newly retrieved event to the events collection
service_details.py:645 checks to see if there is another event available
init.py:55 for chunk in... attempts to read more chunk data
utils.py:211 for data in super... attempts to read more data
init.py:58 data = b'', getting ready to read new line of data
init.py:58 for line in ... - there are no more lines within the previously read chunk
init.py:45 for chunk in self._event_source - see if there is more chunk data in the event source stream
response.py (urllib3.response.HTTPResponse.read_chunked():1184 update the chunk length
response.py:1108 - attempt to read a new line (because chunk left is None) --> code blocks

Other Observations:
There may be some timing component to this problem. Setting breakpoints and/or single stepping through the code sometimes will produce different results.

REQ_HEAD_DIFFERS_FROM_GET will fail if the service rejects HEAD requests with a 405 or 501. HEAD isn't required in the specification, so it's allowed for a service to reject the request.