---> Assertion: 6.5.6.9
/redfish/v1/
GET:/redfish/v1 failed : HTTP status 200:OK, Expected status 404:Not Found
The test strips off the trailing slash which leaves a valid URI
and therefore The GET succeeds with a 200.
The test expects a 404.

Hi Developers,
Our organization is using Conformance check tool for testing our redfish service. We are first testing it with standard redfish mockup server and facing the following error:

[root@aspenvail01 Redfish-Service-Conformance-Check-master]# python3 rf_client.py
Setting up Redfish Service Check Tool Revision: 07.11.16 : vidhyag:127.0.0.1:8000
url.scheme http url.ip 127.0.0.1:8000
http only
url.scheme http url.ip 127.0.0.1:8000
http only
url.scheme http url.ip 127.0.0.1:8000
http only

Collecting all relative uris from Service Root: /redfish/v1/
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems :/redfish/v1/Systems
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1 :/redfish/v1/Systems/2M220100SL
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices :/redfish/v1/Systems/2M220100SL/LogServices
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices_Members_1 :/redfish/v1/Systems/2M220100SL/LogServices/SEL
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices_Members_1_Entries :/redfish/v1/Systems/2M220100SL/LogServices/SEL/Entries
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices_Members_1_Entries_Members_1 :/redfish/v1/Systems/2M220100SL/LogServices/SEL/Entries/1
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices_Members_1_Entries_Members_1_Links_OriginOfCondition :/redfish/v1/Chassis/1/Thermal
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Systems_Members_1_LogServices_Members_1_Entries_Members_2 :/redfish/v1/Systems/2M220100SL/LogServices/SEL/Entries/2
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_SessionService :/redfish/v1/SessionService
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_SessionService_Sessions :/redfish/v1/SessionService/Sessions
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_SessionService_Sessions_Members_1 :/redfish/v1/SessionService/Sessions/SESSION123456
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Chassis :/redfish/v1/Chassis
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Chassis_Members_1 :/redfish/v1/Chassis/A33
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Managers :/redfish/v1/Managers
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Managers_Members_1 :/redfish/v1/Managers/bmc
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Managers_Members_1_NetworkProtocol :/redfish/v1/Managers/bmc/NetworkProtocol
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Managers_Members_1_EthernetInterfaces :/redfish/v1/Managers/bmc/EthernetInterfaces
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_Managers_Members_1_EthernetInterfaces_Members_1 :/redfish/v1/Managers/bmc/EthernetInterfaces/eth0
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService :/redfish/v1/AccountService
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Roles :/redfish/v1/AccountService/Roles
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Roles_Members_1 :/redfish/v1/AccountService/Roles/Admin
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Roles_Members_2 :/redfish/v1/AccountService/Roles/Operator
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Roles_Members_3 :/redfish/v1/AccountService/Roles/ReadOnlyUser
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Accounts :/redfish/v1/AccountService/Accounts
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Accounts_Members_1 :/redfish/v1/AccountService/Accounts/root
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Accounts_Members_2 :/redfish/v1/AccountService/Accounts/jane
url.scheme http url.ip 127.0.0.1:8000
http only
Root Service_AccountService_Accounts_Members_3 :/redfish/v1/AccountService/Accounts/john
url.scheme http url.ip 127.0.0.1:8000
http only

Serializing SUT metadata document: /redfish/v1/$metadata ...
url.scheme http url.ip 127.0.0.1:8000
http only
Traceback (most recent call last):
File "rf_client.py", line 764, in
main()
File "rf_client.py", line 756, in main
sut = setup_tool(sut_prop)
File "rf_client.py", line 739, in setup_tool
if setup_sut_obj(sut):
File "rf_client.py", line 707, in setup_sut_obj
metadata_document_structure = sut.parse_metadata_document(sut.Redfish_URIs['Service_Metadata_Doc'])
File "/home/vidhyag/Redfish-Service-Conformance-Check-master/rf_sut.py", line 458, in parse_metadata_document
csdl_schema_model.serialize_schema(schema_payload = json_payload, schema_uri= metadata_uri)
File "/home/vidhyag/Redfish-Service-Conformance-Check-master/schema.py", line 777, in serialize_schema
schema_root = ET.fromstring(schema_payload)
File "/usr/lib64/python3.4/xml/etree/ElementTree.py", line 1335, in XML
parser.feed(text)
TypeError: 'dict' does not support the buffer interface

I changed http__req_resp, url = urlparse("http://" + sut_prop['DnsName'] + resource_uri) from 'https' to http. Can you let me know if anything else is required?

GET request receiving JSON data as string and tool fails to parse this data.

Traceback (most recent call last):
File "rf_client.py", line 780, in
main()
File "rf_client.py", line 774, in main
rfs_test.run(sut)
File "/usr/bin/rfs_test/init.py", line 39, in run
TEST_protocol_details.run(sut, log)
File "/usr/bin/rfs_test/TEST_protocol_details.py", line 5091, in run
assertion_status = Assertion_6_5_8(self, log)
File "/usr/bin/rfs_test/TEST_protocol_details.py", line 4495, in Assertion_6_5_8
resource = json_payload['@odata.context']
TypeError: string indices must be integers, not str

Line 1995 in TEST_protocol_details
log.assertion_log('line', 'Action %s found' % (json_payload['Actions']['#ComputerSystem.Reset']['target']))
generates KeyError program exception if "target" not present.
But spec has this line in 6.4.4.7 which seems to allow "target" to be omitted-

Custom actions are requested on a resource by sending the HTTP POST method to the URI of the
action. If the actions property within a resource does not specify a target property, then the URI of an
action shall be of the form:

But section 6.5.4.7.1 does not mention missing "target" so it is not 100% clear if required or not.

The Redfish spec requires that:

• for ManagerAccount, the GET response include a strong etag
  Indirectly in the Request Header table, the Redfish spec also requiores that:
• resources that return an etag support if-Match on Patch.
• this of course is thus required for ManagerAccount.
  .
  So:
• if GET of a ManagerAccount does not return an etag, it should be a FAIL
• if PATCH of a ManagerAccount with if-match header does not process the if-match, it is FAIL
• indirectly, for resources that can be patched, the etag MUST be a strong etag since the RFCs require use of strong etag with if-Match and redfish spec required support of if-match.
• other resources not returning etag are a Should statment and should return a WARN
  .
  Assertion: 6.4.1 does not do any of these tests currently?

Name: Assertion_6_4_1(self, log)

Description:

Redfish Services shall understand and be able to process the headers if in the specification the

value in the Required Column is set to "Yes" or "Conditional". It covers 6.4.2 as well- WIP Priyanka

There is a minor typo in properties.json:

"RedfishServiceCheckTool_SchemaFiles": {
"DMTF_SPMFSchemas": {
"ClientProxy": {
....
"Description": "setup you10.6.87.25`r client proxy here or set the entries to 'none' if its not needed to get outside a firewall to the Schema Repository URL"
},

Insyde/Intel project has a fix for the slow response problem seen at plugfest but I am not running the fix yet here.

Testing this tool the operation is erratic. When looking at exceptions in the debugger, appears tool is attempting to process empty json buffer. My investigation led to theory that tool is not checking for timeout on http response. I could not find any timeout handling code.

Suggest review of timeout handling.

Got this error when running the check against a service:

Traceback (most recent call last):
  File "rf_client.py", line 780, in <module>
    main()
  File "rf_client.py", line 774, in main
    rfs_test.run(sut)
  File "/home/ubuntu/redfish/Redfish-Service-Conformance-Check/rfs_test/__init__.py", line 39, in run
    TEST_protocol_details.run(sut, log)
  File "/home/ubuntu/redfish/Redfish-Service-Conformance-Check/rfs_test/TEST_protocol_details.py", line 5046, in run
    assertion_status = Assertion_6_4_23(self, log)
  File "/home/ubuntu/redfish/Redfish-Service-Conformance-Check/rfs_test/TEST_protocol_details.py", line 1439, in Assertion_6_
    log.assertion_log('Unable to verify PATCH for resource readonly property' & (relative_uris[relative_uri]))
TypeError: unsupported operand type(s) for &: 'str' and 'str'

It seems to me the message generation is wrong... I'll submit a PR fixing it.

In Spec Section 6.4.3, "Services may support the HEAD method"
The test should validate the response if status 200 is returned but no assert is indicated if 405 is returned

Per agreement with the Tools TF, we want to standardize on using the SessionService instead of AccountService for testing things like creating (POST) and deleting (DELETE) in most cases. Several assertions are already converted. But a few remain that need to be converted or cleaned up. (Cleanup involves things like avoiding hard-coded usernames and passwords).

Assertions that need work:

• Assertion_6_1_8_1()
• Assertion_6_1_8_3()
• Assertion_6_4_2_2()
• Assertion_6_4_26()
• Assertion_6_5_2_6()

https://github.com/DMTF/spmf/blob/master/Specification.md
Request Header section
If-None-Match | No | No | RFC 7232

I would like to test the files from ghazanfarttu:master.
Sorry, I am not enough of a GitHub master to know how to fetch them.
I did a search and did not find the repo.

For Assertion: 6_5_8 (and MAYBE? others) the test uses the local directories that are cloned
with the tool and hold a local copy of schemas. The problem is that this is a very old snapshot
of schemas and services that use newer schema versions get failures.
FYI--Assertion 6_5_8 test is:
# Name: Assertion_6_5_8(self, log)
# Description: Referencing Other Schemas
# The service metadata shall include the namespaces for each of the Redfish resource types,
# along with the "RedfishExtensions.v1_0_0" namespace.
.
The tool has two local schema directories builtin (for fast access):
./redfish-1.0.0/json-schema/
./redfish-1.0.0/metadata/
The Problem is that these are out of date with respect to http://redfish.dmtf.org/schemas/v1/
There is also the problem that in the Conformance Check tools separates the schemas into
two separate direcdtories above for xml and json.
.
After manually downloading the schemas for redfish.dmtf.org/schemas/v1/ and separating xml
and json into the two directories, the tests on a service that uses later schemas passed.
SO WE THINK that the only issue is the schema cache built-in to conformanc-check is very old.
.
PROPOSAL:

• add a short utility (we have) to pull schemas from dmtf, and split the xml and json out into the directories that the test is currently using--in case there are other things also using these.
• a user can clone the repo, then run the utility to update schemas to latest.

Any suggestions?

Hi,

While running this tools, we discovered some enhancements and fixes. Attached is the file containing the changes we made. Please review and comment.
diff.log

Here are some of the explanations about the changes:

1. Enhancement - ssl.PROTOCOL_TLS is hardcoded based on the version of Python. For our implementation, we need to use PROTOCOL_TLSv1_2 instead of PROTOCOL_TLSv1. Is it possible this can be configure in properties.json?
   diff --git a/rf_utility.py b/rf_utility.py
   index e4df442..fc65981 100644
   --- a/rf_utility.py
   +++ b/rf_utility.py
   @@ -124,7 +124,7 @@ def Connect_Server_NoSSL(sut_prop, host_ip_addr) :
   elif (Python3 == True) :
   #if 3.4.2
   if (sys.version_info.major == 3 and sys.version_info.minor == 4 and sys.version_info.micro <= 3) :

•        cont=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
  

•        cont=ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
         cont.verify_mode = ssl.CERT_NONE
         try:
             svr_conn = HTTPSConnection(host=host_ip_addr, context=cont)
  

2. Enhancement - We have a password rule that is different from the password used in this test. Is it possible this can be configurable?
   diff --git a/rfs_test/TEST_protocol_details.py b/rfs_test/TEST_protocol_details.py
   index 8eb449e..c1bfefc 100644
   --- a/rfs_test/TEST_protocol_details.py
   +++ b/rfs_test/TEST_protocol_details.py
   @@ -368,7 +368,7 @@ def Assertion_6_1_8_3(self, log) :
   account_url = json_payload['@odata.id']
   patch_key = 'RoleId'
   patch_value = 'Operator'

•                    rq_body = {'UserName': user_name, 'Password': '12345' , 'RoleId' : patch_value}
  

•                    rq_body = {'UserName': user_name, 'Password': '12345678' , 'RoleId' : patch_value}
                     rq_headers['Content-Type'] = rf_utility.content_type['json']
                     json_payload_, headers_, status_ = self.http_PATCH(account_url, rq_headers, rq_body, authorization)
                     assertion_status_ = self.response_status_check(account_url, status_, log, request_type = 'PATCH')
  

3. Fix - Missing argument (schema_fle) in log.assertion_log
   @@ -1646,7 +1646,7 @@ def Assertion_6_4_24(self, log) :
   #check if resource remain unchanged, else FAIL. The object might have changed by another source changing the etag, so, in this case, checking value of property makes more sense than etags
   if (json_payload[prop] == 'PatchName'):
   assertion_status = log.FAIL

•                                                        log.assertion_log('line', "~ PATCH on Property %s of resource %s is a Read-only property according to its schema document %s, which might have been updated unexpectedly" % (prop, relative_uris[relative_uri]) )
  

•                                                        log.assertion_log('line', "~ PATCH on Property %s of resource %s is a Read-only property according to its schema document %s, which might have been updated unexpectedly" % (prop, relative_uris[relative_uri], schema_file) )
  

@@ -5078,7 +5098,7 @@ def Assertion_6_4_24_xml(self, log) :
#check if resource remain unchanged, else FAIL. The object might have changed by another source changing the etag, so, in this case, checking value of property makes more sense than etags
if (json_payload[property.Name] == 'PatchName'):
assertion_status = log.FAIL

•                                                                log.assertion_log('line', "~ PATCH on Property %s of resource %s is a Read-only property according to its schema document %s, which might have been updated unexpectedly" % (prop, relative_uris[relative_uri]) )                                                                                         
  

•                                                                log.assertion_log('line', "~ PATCH on Property %s of resource %s is a Read-only property according to its schema document %s, which might have been updated unexpectedly" % (prop, relative_uris[relative_uri], schema_file) )                                                                                         
  

  log.assertion_log(assertion_status, None)
  return (assertion_status)

4. Fix

• Assertion_6_4_25: Header needed to update for each URI.
• Assertion_6_4_31: Determine target the same way as in Assertion_6_4_32.
• Assertion_6_5_18: If there is no json payload, the test always failed. Moved the check after checking if there is any json payload.

Thanks,

KC

rf_client.py calls rf_utility.download_schemas() at line 769 but at that point the properties.json has not been processed and any proxy configuration is not used at all; the requests.get() calls in download_schemas() and download_single_schema() do not use the proxies= parameter and therefore the Conformance-Check tools fails and exits at line 773 in rf_client.py.
Also, the way download_schemas() works (downloading one schema at a time) is resulting in the server at redfish.dmtf.org to stop serving requests after a certain number of requests. My guess is that there is some rule at the server to prevent denial of service attacks.

The 2018.1 schema bundle is now available on the dmtf.org site. Update the "SchemaZipFileName" property in properties.json from "DSP8010_2017.3.zip" to "DSP8010_2018.1.zip".

https://github.com/DMTF/spmf/blob/master/Specification.md
Data modification requests
Actions (POST)
HTTP Status Code 200 indicates the Action request was successfully processed, with the JSON message body as described in Error Responses and providing a message indicating success or any additional relevant messages.
HTTP Status Code 204 indicates the Action is successful and is returned without a message body.

Therefore if no data, 204 should pass.

This is a new failure. maybe in the last month.
It may have to do with the addition of the automatic
download of Schemas. For several of the resources,
multiple json versions are downloaded.
It appears that if our metadata does not reference all
of the schemas, then it will FAIL.
I reorganized the output. The list below is a DIFF.
In other words the list of "missing references".

DIFF {'Chassis.v1_5_2', 'Chassis.v1_1_2', 'ActionInfo.v1_0_2', 'AttributeRegistry.v1_1_0', 'AttributeRegistry', 'BootOptionCollection', 'ActionInfo', 'BootOption', 'Chassis.1.1.0', 'Chassis.v1_0_3', 'AccountService.v1_2_1', 'Chassis.v1_0_6', 'Bios.v1_0_1', 'CompositionService', 'ActionInfo.v1_0_0', 'Chassis.v1_0_2', 'Chassis.v1_1_4', 'Chassis.v1_0_4', 'Chassis.v1_0_0', 'CollectionCapabilities.v1_0_0', 'CollectionCapabilities', 'Bios.v1_0_2', 'AccountService.v1_0_5', 'Chassis.v1_5_0', 'AccountService.v1_1_1', 'AccountService.v1_1_0', 'ActionInfo.v1_0_1', 'Chassis.v1_2_4', 'AttributeRegistry.v1_0_1', 'Chassis.v1_1_5', 'AttributeRegistry.v1_0_0', 'Chassis.v1_2_0', 'ComputerSystem.v1_0_0', 'AttributeRegistry.v1_1_1', 'Bios', 'Chassis.v1_1_3', 'Chassis.v1_0_1', 'ComputerSystem.1.0.1', 'Assembly.v1_0_0', 'AccountService.v1_0_3', 'Chassis.v1_3_0', 'Chassis.v1_3_1', 'Chassis.v1_4_2', 'ActionInfo.v1_0_3', 'AccountService.v1_2_2', 'AccountService.v1_2_0', 'AttributeRegistry.v1_0_2', 'Chassis.v1_6_0', 'Assembly', 'CompositionService.v1_0_0', 'ComputerSystem.1.0.0', 'Chassis.v1_4_1', 'Chassis.v1_2_1', 'Chassis.1.0.0', 'Chassis.v1_2_3', 'Chassis.v1_5_1', 'Chassis.1.0.1', 'Chassis.v1_1_6', 'Chassis.v1_4_3', 'Chassis.v1_2_2', 'AccountService.v1_0_4', 'Bios.v1_0_0', 'AccountService.v1_0_2', 'Chassis.v1_1_0', 'Chassis.v1_3_3', 'AccountService.v1_1_2', 'Bios.v1_0_3', 'BootOption.v1_0_0', 'CompositionService.v1_0_1', 'Chassis.v1_3_4', 'AccountService.1.0.0', 'Chassis.v1_0_5', 'Chassis.v1_3_2'}

Using the 8/31 code, had exception below

---> Assertion: 6.5.17

Expected property @odata.id in response body

Expected property @odata.id in response body

Expected property @odata.id in response body

Assertion Description: Resources in a response shall include a unique identifier property named "@odata.id". The value of the identifier property shall be the unique identifier for the resource.
<--- Assertion 6.5.17: FAIL

---> Assertion: 6.5.18

Traceback (most recent call last):
File ".\rf_client.py", line 780, in
main()
File ".\rf_client.py", line 774, in main
rfs_test.run(sut)
File "G:Redfish_git\Tools\Redfish-Service-Conformance\Redfish-Service-Conformance-Check\rfs_test_init.py", line 39, in run
TEST_protocol_details.run(sut, log)
File "G:_Redfish_git\Tools\Redfish-Service-Conformance\Redfish-Service-Conformance-Check\rfs_test\TEST_protocol_details.py", line 5040, in run
assertion_status = Assertion_6_5_18(self, log)
File "G:_Redfish_git\Tools\Redfish-Service-Conformance\Redfish-Service-Conformance-Check\rfs_test\TEST_protocol_details.py", line 3570, in Assertion_6_5_18
if(json_payload is not None and '@odata.type' in json_payload and isinstance(json_payload['@odata.type'],basestring)) :
NameError: name 'basestring' is not defined

Assertion test 5_4_21 test does a GET with if-none-match for any API that had returned an etag.
The test expects a 304 which means "not modified".
if it doesn't get a 304, it returns a "FAIL".
So a 200 response returns a fail if the service does not support if-none-match.
.
The Redfish Spec in the Request headers defines "If-None-Match" as "Not Required for services to support". Therefore, the test here should only declare this a WARNing. It is not a FAIL.
.
the fail (vs warn) is getting generated in some burried code.
the test is in file rfs_test/TEST_protocol_details.py though and pretting to see where its called.

I got the following error about additionalProperties when I added an annotation @Privileges.OemGetPrivileges in the LogService object.

Resource: /redfish/v1/Managers/1/LogServices/AuditLog/ of type: LogService.v1_0_4 has Annotation: 'additionalProperties' set to 'False' in its schema document LogService.v1_0_4.json, but additional property: @Privileges.OemGetPrivileges found in resource payload

I know in Redfish specification it says:

Types annotated with the AdditionalProperties annotation with a value of "False", shall not contain additional properties.

But Annotations and Properties have different definitions in metadata. It seems that additionalProperties should limit the use of Properties, not Annotations?

Besides, in the definition of additionaProperties in JSON schema validation document:
https://tools.ietf.org/html/draft-fge-json-schema-validation-00#section-5.4.4

If "additionalProperties" has boolean value false

   In this case, validation of the instance depends on the property set
   of "properties" and "patternProperties". 

According to the document if a property (can be an annotation) is not in "properties" but matches the regular expression defined in patternProperties, even additionalProperties is false, the validation should pass.

If we follow the above validation, the annotation I added to LogService should pass Redfish-Service-Conformance-Check...

This tool is public and I suggest we need to step back and make some effort to make it more understandable.

Starting with 6.1.3 and 6.1.5 there are a many lines in the output with no Column A highlight color. Reading the source we know some of these are unimplemented assertions. Are they not-yet-implemented or never-expected asserts? Does the no-color ever mean skipped? How should the status be communicated to the naive tester?

Does this confusion come from the choice to have the annotated design document (spread sheet) serve as the output delivery? Maybe there should be a column indicating the code status?

Or, maybe change all of the assert lines to hidden and unhide when a test is started?

6.5.2.7 -.9 have the comment "Tested with the Dell Server" which is hard to understand given there is no code. Does this maybe mean someone once tested by hand? What does this have to do with my test run?

It is my impression that there are some tests of the schema definition rules mixed in that do not depend on the system under test. Fail on 7.5.16 is an example. Is verifying the schema really something we want this tool to do? Since the tool downloads the latest schema from common site would not every system get the same results? I suggest the schema checking asserts be moved to a separate tool. This tool takes too long to run without everyone checking the same schema.

ACCO101 appears to be the last test but the spread sheet has 1200+ more tests listed.

How much of the long run time is due to the overhead of opening and editing a spread sheet? Is this really the output style we want?

Hi,
I'm trying to use your tool to check redfish conformance and it does not have support to use HTTP only during the tests. All connections are HTTPS and the tool fails with a lot of:
OPERATIONAL ERROR: GET Request for /redfish/v1/ FAILED with exeption: <class 'ssl.SSLError'>
OPERATIONAL ERROR: GET Request for /redfish FAILED with exeption: <class 'ssl.SSLError'>
OPERATIONAL ERROR: GET Request for /redfish/v1/odata FAILED with exeption: <class 'ssl.SSLError'>
OPERATIONAL ERROR: GET Request for /redfish/v1/$metadata FAILED with exeption: <class 'ssl.SSLError'>

Is is possible to add HTTP only support to your code? Or Is there a non documented way of doing it? Or even is there a documented way of doing it and I was not able to found? If the later please point me to it.

Seems to be trying to generate a 301 error return.
But I am not able to comprehend the logic.
Can this be explained?

---> Assertion: 6.5.40
File "/home/dmtf/Redfish-Service-Conformance-Check/rfs_test/TEST_protocol_details.py", line 4878, in Assertion_6_5_40
if 'error' in json_payload :
TypeError: 'str' does not support the buffer interface

for some reason, failed PATCH is returning a "byte string" rather than a string.
I suggest adding this line:
json_payload = str(json_payload)

Or perhaps better would be to insure GET and PATCH always
return the json_payload as type str.

I noticed this line in 6_4_32
rq_body = {'ResetType': 'ForceOff'}

I checked some web sites that said JSON should use double quotes.
Did we relax this for Redfish?

Or maybe I am reading the python wrong. But in my test the single quotes made it out to the system under test (and failed)

In manager accounts we have:
"Locked": false,
Which is valid, because:
<Property Name="Locked" Type="Edm.Boolean" Nullable="false" DefaultValue="false">

But 7_5_3_xml fails because of this helper function.

    for key in json_payload:
        if key == property.Name:
            #value cant be null
            **if not json_payload[key]:
                return False**
    return True

IT SHOULD BE:
if json_payload[key] == None: return False

The test always sends {"ResetType": "ForceOff"}
If the system happens to be off when the test is run, the Insyde implementation returns an error.
"Message": "Ignore power control request because of power has been switched off",
Which seems appropriate but fails the test.
If the system happens to be on, there is no error.

If there is a precondition that the system must be on before start of the test cycle this should be documented. Or the test could check the current "PowerState" and choose a JSON payload to reverse.

I am not thrilled with the test changing user-selected power state. Are we sure this is good test design?

When testing 6.1.8.1.2, process will stop if the creation return body is empty. Please help to look into this problem.

`---> Assertion: 6.1.8.1.2

Traceback (most recent call last):
File "sample_run.py", line 43, in
rfs_test.run(sut)
File "/home/qa/DMTF/Redfish-Service-Conformance-Check-master/rfs_test/init.py", line 40, in run
TEST_protocol_details.run(sut, log)
File "/home/qa/DMTF/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 5037, in run
assertion_status = Assertion_6_1_8_1(self, log)
File "/home/qa/DMTF/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 211, in Assertion_6_1_8_1
subassertion_status = Assertion_6_1_8_1_2(session_location, json_payload, self, log)
File "/home/qa/DMTF/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 272, in Assertion_6_1_8_1_2
assertion_status = verifyCreatedObject(json_payload, check_payload, log)
File "/home/qa/DMTF/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 308, in verifyCreatedObject
if (key in object_payload):
TypeError: a bytes-like object is required, not 'str'
`

Redfish Spec section 6.5.1 in the table for Response Headers, for the header Allow, has this sentence -
"Should be returned with any GET or HEAD operation to indicate the other allowable operations for this resource."

Since this is a 'Should' the header Allow return with get is recommended but not required. But Test 6.1.8.1 fails and this leads to failure later in 6.1.8.4
Or am I reading the code wrong?
log below

---> Assertion: 6.1.8.1
~ note: the header returned from GET /redfish/v1/SessionService/Sessions do not indicate support for POST
{
"strict-transport-security": "max-age=31536000; includeSubdomains; preload",
"server": "lighttpd/1.4.48",
"x-ua-compatible": "IE=11",
"content-length": "282",
"date": "Mon, 05 Jan 1987 19:12:59 GMT",
"content-type": "application/json",
"x-frame-options": "SAMEORIGIN",
"x-xss-protection": "1; mode=block"
}
Assertion Description: b'\nPOST\nObject create, Object action, Eventing\nYes\n'
<--- Assertion 6.1.8.1: FAIL

As of now, everything is placed into a spreadsheet in terms of what tests are run and what the results are. There are two problems with this:

• The spreadsheet is also used as a design document of sorts, and there are many test cases not yet implemented
• This format is not consistent with other tools we have

We should revise how we generate the test report. It would be beneficial to perform the same HTML style report since it would contain a top level executive summary that would be easy to consume by most users.

This file has been stable a long time, looks like test setup got checked in by accident.

about line 1160
query_url = json_payload['@odata.id'][:-1] + query_param
should be

                    try:
                        query_url = json_payload['@odata.id'][:-1] + query_param
                    except:
                        assertion_status = log.FAIL
                        log.assertion_log('line', "~ @odata.id  not found in redfish resource %s" % (relative_uris[relative_uri]))
                        continue

around line 3340
odata_id = (json_payload['@odata.id']).rsplit('/v1/', 1)[1]
should be

        try:
            odata_id = (json_payload['@odata.id']).rsplit('/v1/', 1)[1]
        except:
            assertion_status = log.FAIL
            log.assertion_log('line', "~ @odata.id  not found in redfish resource %s" )
            return assertion_status

The proposed pull version consistently has exception on Memory.AllowedSpeedsMhz[2666].
The exception is in routine process_list(), line 581 of rf_sut.py.
The program is looking for @odata.id in the 2666 numeric array element.
I have updated the local schema directory with files from 2016.3.

Here is failing page returned by the service.

{
    "@odata.context": "/redfish/v1/$metadata#Memory.Memory",
    "@odata.id": "/redfish/v1/Systems/BQWT64900010/Memory/Memory1",
    "@odata.type": "#Memory.v1_1_0.Memory",
    "Id": "Memory1",
    "Name": "Memory 1",
    "Description": "System Memory",
    "MemoryType": "DRAM",
    "MemoryDeviceType": "DDR4",
    "BaseModuleType": "RDIMM",
    "CapacityMiB": 8192,
    "DataWidthBits": 64,
    "Manufacturer": "Micron",
    "SerialNumber": "14F81054",
    "PartNumber": "18ASF1G72PZ-2G6B1   ",
    "AllowedSpeedsMHz": [
        2666
    ],
    "RankCount": 1,
    "DeviceLocator": "CPU0_DIMM_F1",
    "ErrorCorrection": "MultiBitECC",
    "OperatingSpeedMhz": 2666,
    "Metrics": {
        "@odata.id": "/redfish/v1/Systems/BQWT64900010/Memory/Memory1/MemoryMetrics"
    },
    "Status": {
        "State": "Enabled",
        "Health": "OK",
        "HealthRollup": "OK"
    }
}

Could someone please look at redfish-1.0.0/metadata/ChassisCollection.xml
and explain below.

--> Assertion: 7.5.3
~ Resource: ChassisCollection, BaseType: Resource.1.0.0.ResourceCollection and any of its parent BaseType resources (based on inheritance) does not have annotation 'OData.Description' in its OData schema representation document: redfish-1.0.0/metadata/ChassisCollection.xml

There Are dozens of these in 7.5.3 thru 7.5.16

Thanks.

Hi,

I am running conformance check tool on live host/target. I am getting the following exception thrown when its executing assertion 6.5.8

---> Assertion: 6.5.8

The response is {
"detail": "The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.",
"status": 404,
"title": "Not Found",
"type": "about:blank"
}

Traceback (most recent call last):
File "rf_client.py", line 780, in
main()
File "rf_client.py", line 774, in main
rfs_test.run(sut)
File "/home/vidhyag/newconf/Redfish-Service-Conformance-Check-master/rfs_test/init.py", line 39, in run
TEST_protocol_details.run(sut, log)
File "/home/vidhyag/newconf/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 5030, in run
assertion_status = Assertion_6_5_8(self, log)
File "/home/vidhyag/newconf/Redfish-Service-Conformance-Check-master/rfs_test/TEST_protocol_details.py", line 4452, in Assertion_6_5_8
doc = minidom.parseString(response)
File "/usr/lib64/python3.4/xml/dom/minidom.py", line 1970, in parseString
return expatbuilder.parseString(string)
File "/usr/lib64/python3.4/xml/dom/expatbuilder.py", line 925, in parseString
return builder.parseString(string)
File "/usr/lib64/python3.4/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 0

Can you please help me with this issue. Thanks for your help.

I am seeing random failures of this test.
I believe it is because these lines are out of order -
count = 0
for relative_uri in relative_uris:
count is used below to get a count of member_array, but the the initialize of count is out of the for loop. So the starting point is random based on last page checked.
count = count+1
This loop
for m in member_array:
is poor python - should really use len(member_array)

At several places in rf_utility.py, the Basic Authentication Header is formed with a string "BASIC ".
The syntax in RFC1945 section 11.1 uses the string "Basic".
RFC2616 section 4.2 says Header Field names are case insensitive.
But does that apply to header values as well?

This is the first Redfish tool to use the full upper case BASIC string in this header.
Is this allowed? Is it good idea?

---> Assertion: 6.1.8.4
~ note: the header returned from GET /redfish/v1/AccountService/Accounts/root do not indicate support for DELETE

"allow": "GET,HEAD,PATCH"
}
We do not allow DELETE of root. Does redfish?

FAIL on missing link and allow response header is not correct.
The table in the spec at 6.5.1 says "Yes" in "Required" column but the "Description" column explains that "Yes" means "Sometimes"

I may be wrong about link, someone more expert should comment on when link is required.

This assertion is strange. It fails when Sessions has no members. If we post a new session, it passes. But for EventDestination Collection which has no member as well, it passes.

Look source code of Assert 6.5.23.1 and find the if statement if count<=member_count: taking true when a collection has no members (count = member_count = 0), and check whether there's next link property. It does not make sense the next link property be present in this case.

The count is not reset for each iteration, so the result would be incorrect. This is why the assertion passes when there's no member in EventDestination collection.

>> loop N-1
Resource uri: /redfish/v1/Registries
count = 36, member_count = 4

>> loop N
Resource uri: /redfish/v1/EventService/Subscriptions
count = 36, member_count = 0  <-- the count should be zero

For example, in the error below what is the request that generated faulty response body?

---> Assertion: 6.5.17

Expected property @odata.id in response body

Expected property @odata.id in response body

Expected property @odata.id in response body

Assertion Description: Resources in a response shall include a unique identifier property named "@odata.id". The value of the identifier property shall be the unique identifier for the resource.
<--- Assertion 6.5.17: FAIL

Some tests currently either return a warning or simply "pass" when something is not supported by the service. Would like to leave bread crumbs about things that could not be tested due to the service not providing the functionality to test (like allowing a PATCH on a resource). These things are perfectly valid, but we do not want to alarm customers.

In the UpdateService schema - https://redfish.dmtf.org/schemas/UpdateService.v1_2_0.json - an "Oem" object can exist inside "Actions". However, if you do this, Redfish-Service-Conformance-Check will flag this as an error:

Actions value should match the format: #Namespace.ActionName, found: Oem

In Redfish specification, it says:

The URL of the JSON Schema for the resource shall be returned with a rel=describedby.

If we look at the Link Header definition in RFC 5988,

"rel" "=" relation-types

relation-types = relation-type
                 | <"> relation-type *( 1*SP relation-type ) <">

According to RFC 5988, both rel=describedby and rel="describedby" are acceptable, but in current Redfish-Service-Conformance-Check tool it will check rel=describedby and treat rel="describedby" as an error. Do you think the tool should be modified to allow rel="describedby" ?

Hi,
The test stopped due to TypeError while running to Assertion 6.1.8.1. Please see attachment for detail output. Please advice how to resolve it?

Thanks.

Jack Tan.
ConformanceOutput.txt

The tool run up till 6.5.20 then crashed. It looks like it found an entry in the $metadata for an OEM Schema (OemAuthorization_v1.xml) and decided to go look for it on dmtf.org: http://redfish.dmtf.org/schemas/v1/OemAuthorization_v1.xml, and crashed with HTTP Error 404

---> Assertion: 6.5.20
.....
....
....
The uri is /redfish/v1/metadata/OemAuthorization_v1.xml
OemAuthorization_v1.xml
http://redfish.dmtf.org/schemas/v1/OemAuthorization_v1.xml
Traceback (most recent call last):
File "rf_client.py", line 779, in
main()
File "rf_client.py", line 773, in main
rfs_test.run(sut)
File "C:\DevTools\Redfish-Service-Conformance-Check\rfs_test_init_.py", line 34, in run
TEST_protocol_details.run(sut, log)
File "C:\DevTools\Redfish-Service-Conformance-Check\rfs_test\TEST_protocol_details.py", line 5019, in
run
assertion_status = Assertion_6_5_20(self, log)
File "C:\DevTools\Redfish-Service-Conformance-Check\rfs_test\TEST_protocol_details.py", line 3751, in
Assertion_6_5_20
f = urllib.request.urlopen(uris)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 472, in open
response = meth(req, response)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 510, in error
return self._call_chain(*args)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "C:\Users\smahmoud\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 590, in http_error_default

raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 404: Not Found

Also in 7.5.3 thru 7.5.15 there LOTS of failures on
LongDescription missing.
Those actually ARE missing in many of the schema.
But we are not aware of any LongDescription requirement.
However,
perhaps DSP0266 | 1.3.0 says they are required.
Opinion?

Test using Redfish-Service-Conformance-Check v1.0.4 and see the following error. It seems that the python's type "bool" is incorrectly compared with json's type "boolean".

---> Assertion: 6.5.20
Type for attribute ServiceEnabled was bool; expected type boolean. Resource is /redfish/v1/Tasks.
Type for attribute ServiceEnabled was bool; expected type boolean. Resource is /redfish/v1/Tasks.
Assertion Description: b'Primitive properties shall be returned as JSON values according to the following table.\n\nType JSON Representation\n\nEdm.Boolean Boolean \nEdm.DateTimeOffset String, formatted as specified in DateTime Values \nEdm.Decimal Number, optionally containing a decimal point \nEdm.Double Number, optionally containing a decimal point and optionally containing an exponent \nEdm.Guid String, matching the pattern ([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) \nEdm.Int64 Number with no decimal point \nEdm.String String \n'
<--- Assertion 6.5.20: FAIL

The name & type is ServiceEnabled Edm.Boolean
This is primitive type
The type is <class 'bool'>
Type for attribute ServiceEnabled was bool; expected type boolean. Resource is /redfish/v1/Tasks.

The test will pass if we change 'Edm.Boolean':'boolean' to 'Edm.Boolean':'bool'
at TEST_protocol_details.py#L3758.

Assertion gives this failure:

URI /redfish/v1/
RESOURCE /redfish/v1/$metadata#ServiceRoot.ServiceRoot
The name & type is Links Edm.Boolean
This is primitive type
{'Sessions': {'@odata.id': '/redfish/v1/SessionService/Sessions'}}
The type is <class 'dict'>
Type for attribute Links was dict; expected type boolean. Resource is /redfish/v1/.

Of course Links property is a dict.
I can't figure out why the test thinks it should be boolean.