dmtf / redfish-interop-validator Goto Github PK
View Code? Open in Web Editor NEWThe Redfish Interop Validator is a python3 tool that will validate a service based on a profile given to the tool.
License: Other
The Redfish Interop Validator is a python3 tool that will validate a service based on a profile given to the tool.
License: Other
This feature requires that registries be stated at the ServiceRoot, which is an optional navigation property, otherwise report that this is unable to be checked.
Embedded objects / arrays which are marked as referenceable members must contain an @odata.id to conform to the spec. This is properly flagged as an error in the tool.
However, the schema type of these arrays/objects can be assumed (and perhaps always correct?) from the parent, and so checking the properties within the array member should continue, not abort (as the tool does now).
The Readme is obviously originated from copy of the Redfish-Service-Validator. While there are a few edits to replace the name the differences in operation from that tool are not made clear.
Does this tool process the special profile schema semantics? How is this accomplished in detail? Are there new errors possible?
It seems the tool ignores the schema reference by the service under test. Is this wise?
Is the intention to determine of the system under test meets the minimum requirements of the profile?
Suggest the Readme needs more work to make operation clearer. Or point the where this information is available.
The latest bug fixes to the Service Validator are not applied to this tool. Some do not apply but would help for later.
By checking the profile 'OCPBaselineHardwareManagement' from https://github.com/opencomputeproject/OCP-Profiles, we got 'fail.ServiceRoot.ReadRequirement' error.
After some digging code, we find the reason might be objRes['ServiceRoot']['mark'] not set to True.
There only all links following the 'ServiceRoot' be checked and marked. However, in validateURITree() there did not try to mark 'ServiceRoot' after validate top URI itself.
Hope it helps.
I ran redfish interop validator with Open Compute Project profiles: OCPBasicServer and OCPManagedDevice. The error message "KeyError: 'Protocol'" pops up when I ran with OCPBasicServer profile, but not with OCPManagedDevice profile.
Full error msg:
C:\Users\leejenn1\work\Redfish\Redfish-Interop-Validator-master>python RedfishInteropValidator.py -c config\config.ini profiles\OCPBasicServer.v1_0.json
CacheMode or path invalid, defaulting to Off
ConfigURI: https:// [deleted]
System Info: Test Config, place your own description of target system here
authtype: Basic, cachefilepath: None, cachemode: Off, certificatebundle: None, certificatecheck: False, httpproxy: None,
httpsproxy: None, localonlymode: False, logpath: ./logs, metadatafilepath: ./SchemaFiles/metadata, payloadfilepath: None, payloadmode: Default,
schemasuffix: _v1.xml, servicemode: False, timeout: 30, username: root, usessl: True, warnrecommended: False,
Start time: 03/08/18 - 10:46:19
Traceback (most recent call last):
File "RedfishInteropValidator.py", line 977, in <module>
sys.exit(main(sys.argv))
File "RedfishInteropValidator.py", line 854, in main
success, counts, results, xlinks, topobj = validateURITree('/redfish/v1', 'ServiceRoot', profile, expectedJson=jsonData)
File "RedfishInteropValidator.py", line 632, in validateURITree
serviceVersion = profile["Protocol"].get('MinVersion', '1.0.0')
KeyError: 'Protocol'
The profiles can be downloaded here:
http://opencompute.org/wiki/Hardware_Management/SpecsAndDesigns#Baseline_and_Server_profile
The 2018.3 schema bundle is published. Update the code to point to this latest bundle.
Is "WriteRequirement": "Mandatory" checked? If so, how?
Profile spec v1.1 added URI patterns for requirements - this is very useful for easier profile construction, but need the toolchain to support it in order for folks to implement that functionality.
Hello all,
On running "python3 RedFishInteropValidator.py -h" the help text is return as usual. I noticed that the definitions returned for the positional arg "profile" and the optional args "suffix" and "schema" the same. The definition reads "suffix of local schema files (for version differences)".
Can I get some clarity for these three args?
Latest version is not properly reporting valid 'null' values for properties. The Value column shows "[JSON object]" for null values.
Value should report: [null]
Hi,
I just tried the following :
python3 RedfishInteropValidator.py MyProfile -c config.ini
I get:
Unable to parse configuration: TypeError("'NoneType' object is not iterable",)
Config:
[Information]
Updated = May 8, 2017
Description = Redfish Interop Conformance Tool 0.91[SystemInformation]
TargetIP = 10.10.10.10
SystemInfo = Test Config, place your own description of target system here
UserName = root
Password = my_pass
AuthType = None
Token = 123456SESSIONNauthcode
UseSSL = False
CertificateCheck = Off
CertificateBundle =[Options]
MetadataFilePath = ./SchemaFiles
CacheMode = Off
CacheFilePath =
SchemaSuffix = _v1.xml
Timeout = 30
HttpProxy =
HttpsProxy =
LocalOnlyMode = True
ServiceMode = Off[Validator]
PayloadMode = Default
PayloadFilePath =
LogPath = ./logs
WarnRecommended = False
What is wrong ?
Best Regards,
Francine
Hi,
By using the profile 'OCPBaselineHardwareManagement.v1_0_0.json' got from 'https://github.com/opencomputeproject/OCP-Profiles' it has the "Protocol" section with "MinVersion": "1.0".
It seems ValidateMinVersion() can not get that comparison passed. However, suppose 1.0.2 is greater than 1.0 and should be pass.
Thanks
// Vic
The "If Implemented" requirement needs to be handled carefully. The tool is failing if an "if implemented" subordinate resource link is not present. That could be because the link is a JSON object (may be considering the odata.id as mandatory since the object is present).
This could be an exception case to be handled.
The readme discusses the output logs. It would be useful to place a sample log in a ./logs folder.
I am getting multiple ERRORs for non-existent XML schema:
INFO - URI /redfish/v1/UpdateService/FirmwareInventory/3, Type (SoftwareInventory), GET SUCCESS (time: 0.302655)
ERROR - getResourceObject: Namespace appears nonexistent in SchemaXML: #HpeiLOSoftwareInventory.v2_0_0.HpeiLOSoftwareInventory http://redfish.dmtf.org/schemas/v1/Resource_v1.xml
These are all for OEM schema (OEM extensions to standard Redfish schema) and are not referenced by the profile I'm testing against. The tool should not flag errors like this because they are not profile validation issues.
Hello
I get this error evry time when I try to validate the Profile Simulator with Interop Validator :
DSP package DSP8010_2019.3.zip
Teminal :
1 failPayloadError errors in /redfish/v1/SessionService/Sessions
Counter({'skipOptional': 139, 'pass': 130, 'metadataNamespaces': 60, 'skipOem': 23, 'passGet': 14, 'serviceNamespaces': 13, 'warningPresent':
7, 'missingNamespaces': 4, 'badNamespaceInclude': 1, 'reflink': 1, 'failPayloadError': 1})
Please can you help to fix that ? , I'll be grateful :)
Regards,
Youssef.
The tool only lets you run multiple profiles if it is required by the parent profile. Allow the program to specify multiple profiles in one run.
Additionally, consider allowing the program to run multiple profiles concurrently per run instead of running each independently. This can be arranged by passing in all profiles to the main loop and iterate through them for related keys.
Hi
I try to validate the Profile Simulator.
From one side:
got the errors:
Something went wrong ...
KeyError: 'Term'
for
PostalAddress:GPSCoords, Location:Info, PCIDevices PostalAddress:PostalLocation Fan:FanName
Properties have a deprecated status.
It seems that the Validator do NOT take care of versionDeprecated ?
In getPropertyDetails(), tag['Term'] raises an exception because the 'deprecated' structures are Collection (Redfish.Revisions Collection inside Annotation tag) and propertyTags variable is flatten before.
like GPSCoords:
Any idea / suggestion to fix it ? I would be pleased,
Best Regards,
Francine
The tool does not look for any profile with version less than v1.0.0,
The tool should be able to read a profile with any version. In the OCP case, the baseline and server (which references the baseline) profiles may both be in draft (<1.0.0), prior to OCP approval.
If the desire is the restrict versions to => 1.0.0 for "official" conformance runs, then I would suggest a flag in the config.ini file (e.g. OfficialTest = True)
When the tool encounters a Links property that is incorrectly implemented as an array (Links[]) rather than a list (Links { } ), it stops running instead of reporting the issue and continuing to run the remaining tests
I got following error information when run Redfish-Interop-Validator:
ERROR - The following namespaces are referenced by the service, but are not included in $metadata:
OemMessageRegistry.v1_0_0
The OemMessageRegistry.v1_0_0 is an OEM schema.
When i run the Redfish-Interop-Validator service and already add "OemCheck = False" in config.ini. The result is still get above error message in result html file.
Is there any way or possible to avoid OEM schema check?
In EventDestination.v1_4_0.json and EventDestination_v1.xml, readonly
and OData.Permission
are not defined. By default, it is a writable property(Please refer to WhitePaper section 22.3).
However, the tool assumes it is a fail case if it cannot find the OData.Permission
in metadata(Please refer to the Redfish-Interop-Validator source code).
There are a few occurrences of various forms of "comply" rather than "conform" in the code. Use this issue to clean them up.
Suggest - add new config to report violations on mandatory only, or alternatively mandatory plus recommended
Has the Basic Server Profile been publically published? If so, can we include it here in the profiles subdirectory? That will be convenient for building the test tree for the Redfish-Test-Framework.
The use of @odata.context is changed to be optional in Redfish specification in the following issue:
DMTF/Redfish#2722
However, if we remove the use of @odata.context from the resources, the tool still shows errors like the following:
ERROR - /redfish/v1: Json does not contain @odata.context
There are a number of Tracebacks in the attached log.
I only looked at the 1st one in which decodeditem is a string,
not a list or a dict and can't see how "item" could have been created.
Anyway could someone tack a look at the exceptions?
python3 RedfishInteropValidator.py -c ./config.ini OCPBaselineHardwareManagement.v0_2_1.json
The target service is the Reddrum front end with obmc backend.
Thanks
log5.txt
htmlLogScraper.py was added to provide a CSV report; would like to document this tool in the readme
I had a lot of problems using the readme and setting up for execution. And then a quick exception. I will list problems here as encountered.
For a profile I used https://github.com/DMTF/spmf/tree/master/profiles/RedfishInteroperabilityProfile.v1_0_0.json
The system used for test passes Redfish-Service-Validator with no errors
Profile errors that don't align with the server should be made more obvious.
i.e. A property being tested that clearly should not exist in a given resource.
I am running against OCPServerHardwareManagement.v0_2_3.json profile.
My ComputerSystem resource has a "SKU": "MXQ32200VV" but not a "PartNumber". The profile test is supposed to ensure that one or the other is implemented (which it is...so its not an error). But the tool seems to treat every conditional comparison that's false as an error, so I get a false positive here.
INFO - ### Validating PropertyRequirements for PartNumber
INFO - propRequirement with value: DNE
INFO - Testing ReadRequirement
expected:Recommended, exists: False
INFO - Item is recommended but does not exist
INFO - pass True
INFO - Evaluating conditionalRequirements
INFO - Testing a comparison
('MXQ32200VV', 'Absent', [])
INFO - pass False
ERROR - NoPass
Conditionals with "CompareProperty" don't appear to function properly. Get an "ERROR - NoPass" in the log and in the HTML output (with no context, either).
Snippet from a Boot object requirement in a ComputerSystem profile:
"UefiTargetBootSourceOverride": {
"ConditionalRequirements": [
{
"CompareProperty": "BootSourceOverrideMode",
"Comparison": "Equal",
"Purpose": "If UEFI mode is selected, must allow for UEFI target.",
"ReadRequirement": "Mandatory",
"Values": [
"UEFI"
]
}
],
"ReadRequirement": "Recommended",
"WriteRequirement": "Mandatory"
},
and in payload:
"BootSourceOverrideEnabled": "Once",
"BootSourceOverrideMode": "UEFI",
"BootSourceOverrideTarget": "Hdd",
"UefiTargetBootSourceOverride": "None",
the Condition should apply, and the resulting Mandatory requirement is met. But log output says:
INFO - inside complex Boot.UefiTargetBootSourceOverride
INFO - propRequirement with value: None
INFO - Testing ReadRequirement
expected:Recommended, exists: True
INFO - pass True
INFO - writeable
Mandatory
INFO - Evaluating conditionalRequirements
INFO - Testing a comparison
('UEFI', 'Equal', [])
INFO - pass False
ERROR - NoPass
INFO - Condition does not apply
And in the HTML output - there's a "ERROR - NoPass" for that resource (but no context as to what property or requirement).
properties.json has added this
"Number of URI's to Cache":
but the usage is not documented in README.md
In private repo DMTF/spmf, information regarding Interop Profiles has been updated to 1.0.0, is necessary to update functionality to match the keywords added and changed.
Tool attempts requesting a profile at a given repository, but fails for multiple reasons:
Given the nature of http responses returning file listings in bytes, requires utf-8 decoding, and over looks this.
Profile name is stripped one character from the file listing and causes a 404 error.
I am not seeing the output that was produced by earlier versions. For example, When tool finds EthernetInterface object tool used to check the presence of the listed PropertyRequirements inside. This no longer happens.
The tool is flagging error on required Collection elements if it does not find any members in the collection.
For example:
"EventDestination": {
"PropertyRequirements": {
...
"Purpose": "The EventDestination resource describes the target of an Event subscription, including the types of Events subscribed and context to provide to the target in the Event payload.",
"ReadRequirement": "Mandatory"
},
In this case, the profile would like to verify that if an EventDestination exists in the Events Subscription collection, that it contains the correct and mandatory properties as specified by the profile.
however, since most systems under test will NOT have any pre-existing subscriptions, the test will fail.
This is also true with a number of other cases (such as Jobs, Tasks, etc...)
Some ideas to enhance the tool to work around this issue:
There is a dependency on jsonschema package, but that is not listed in requirements.txt
The log file should include the name of the Profile file read as input, and all referenced Profile files.
Extra credit would be a hash for each Profile file. Then the Profile own could determine that their Profile files had not been altered, prior to the execution run.
We got some errors by Redfish-Interop-Validator and it seems that Redfish-Interop-Validator doesn't extract the correct context of the resource.
Resource | error message |
---|---|
/redfish/v1/schemas/registries/ResourceEvent.1.0.1.json | ERROR - getResourceObject: Namespace appears nonexistent in SchemaXML: MessageRegistry.v1_0_0.MessageProperty http://redfish.dmtf.org/schemas/v1/MessageRegistryFile_v1.xml |
/redfish/v1/schemas/registries/Base.1.4.0.json | ERROR - getResourceObject: Namespace appears nonexistent in SchemaXML: MessageRegistry.v1_0_0.MessageProperty http://redfish.dmtf.org/schemas/v1/MessageRegistryFile_v1.xml |
/redfish/v1/schemas/registries/TaskEvent.1.0.1.json | ERROR - getResourceObject: Namespace appears nonexistent in SchemaXML: MessageRegistry.v1_0_0.MessageProperty http://redfish.dmtf.org/schemas/v1/MessageRegistryFile_v1.xml |
None
when the reosurce is message registrey, then, interop would create the corresponding context for it.
context = None
after this lineCould someone help to check it? Thanks.
The current code for checking Action requirement in commonInterop.py
Redfish-Interop-Validator/commonInterop.py
Lines 459 to 466 in a0119ca
@Redfish.AllowableValues
. It will treat the below response as fail although the @Redfish.AllowableValues
can be retrieved in @Redfish.ActionInfo
URI.
"Actions": {
"#ComputerSystem.Reset": {
"@Redfish.ActionInfo": "/redfish/v1/Systems/Self/ResetActionInfo",
"target": "/redfish/v1/Systems/Self/Actions/ComputerSystem.Reset"
}
},
To specify the list of supported values for a parameter, the service may include the @Redfish.AllowableValues annotation.
...
The Resource may provide a separate @Redfish.ActionInfo Resource to describe the parameters and values that a particular instance or implementation supports.
In Redfish Schema Supplement v2019.3 (DSP0268), Section "Common objects" specifies that Actions property contains @Redfish.ActionInfo
and target
.
In Redfish Interoperability Profiles Specification v1.2.0 (DSP0272), Section 8.4.3 only defines that Action requirement contains ReadRequirement, Parameters, and Purpose. The Parameters is the requirement for any parameter available for this Action, but there are no statements specifying it must be located in @Redfish.AllowableValues
or @Redfish.ActionInfo
URI.
Currently the usage of "IfImplemented" is to cover cases where the Redfish service doesn't need to implement a property or resource if the underlying functionality is not present in the hardware. For example, if a system is built without fans, then the "Fans" property in "Thermal" could be marked as "IfImplemented". But there isn't a good way to test for this (at least exclusively through Redfish). Maybe we need a hardware configuration file to describe the underlying hardware architecture of a given system to help validate a profile.
The 2019.1 schema bundle is published. Update the code to point to this latest bundle.
http://redfish.dmtf.org/schemas/DSP8010_2019.1.zip
The Case1 and Case2 are all come from spec DSP0272_1.0.1.pdf.
In Case2, the condition options seems no problem but the Validator's result is fail. The reason sall be caused by missing option "Comparison".
According to the spec, the ConditionalRequirement didn't mention "Comparison" option shall be a required option.
I think the Validator should regards the Case2 be a valid case.
"IndicatorLED": {
"ReadRequirement": "Recommended",
"ConditionalRequirements": [{
"Purpose": "Physical Systems must have a writable Indicator LED",
"CompareProperty": "SystemType",
"CompareType": "AnyOf",
"CompareValues": ["Physical"],
"ReadRequirement": "Mandatory",
"WriteRequirement": "Mandatory",
"Comparison": "AnyOf",
"Values": ["Off"]
}]
}
The result is pass the Validator's check.
"IndicatorLED": {
"ReadRequirement": "Recommended",
"ConditionalRequirements": [{
"Purpose": "Physical Systems must have a writable Indicator LED",
"CompareProperty": "SystemType",
"CompareType": "AnyOf",
"CompareValues": ["Physical"],
"ReadRequirement": "Mandatory",
"WriteRequirement": "Mandatory"
}]
}
The result is fail to pass the Validator's check and got following error messages:
KeyError: 'Comparison'
ERROR - Could not finish validation check on this payload
Currently the tool does not have any version information in the output. It would be good to have a version string in the report so that a user knows what version of the tool was used to test a service.
Property "UefiTargetBootSourceOverride" definition in schema is:
"UefiTargetBootSourceOverride": {
"description": "This property is the UEFI Device Path of the device to boot from when BootSourceOverrideTarget is UefiTarget.",
"longDescription": "The value of this property shall be the UEFI device path of the override boot target. The valid values for this property are specified through the Redfish.AllowableValues annotation. BootSourceOverrideEnabled = Continuous is not supported for UEFI Boot Source Override as this setting is defined in UEFI as a one time boot only.",
"readonly": false,
"type": [
"string",
"null"
]
}
The real implementation of value "UefiTargetBootSourceOverride" in redfish service is:
"Boot": {
"BootSourceOverrideEnabled": "Disabled",
"BootSourceOverrideTarget": "None",
"UefiTargetBootSourceOverride"": null,
"BootSourceOverrideMode": "Legacy"
}
The condition option in porfile to validate property "UefiTargetBootSourceOverride" is:
"UefiTargetBootSourceOverride": {
"ReadRequirement": "Recommended",
"ConditionalRequirements": [{
"Purpose": "If UEFI mode is selected, must allow for UEFI target.",
"CompareProperty": "BootSourceOverrideMode",
"CompareType": "Equal",
"CompareValues": ["UEFI"],
"ReadRequirement": "Mandatory",
"Comparison": "AnyOf",
"Values": ["None", "RDOC1", "RDOC2"]
}]
}
Whatever the value of "Values" i set into conditioin option is ["Test1", "Test2"], ["null", "Test1", "Test2"], [null, "Test1", "Test2"] or ["None", "Test1", "Test2"] to validate property "UefiTargetBootSourceOverride".
The Validator result is always return fail and the error message is:
TypeError: argument of type 'NoneType' is not iterable
ERROR - Could not finish validation check on this payload
If a property is not expected to exist on this particular resource, raise an error and ignore requirements for that property.
Example: 'System' does not contain property 'Johnny'
The comprehensive log is great, but for purposes of logging bugs against our products, it would be really useful to have a text output report that is easy to read at a glance and has one line per error. Each error line should contain:
A profile can extend another profile by using the RequiredProfiles property
"RequiredProfiles": {
"OCPBaselineHardwareManagement": {
"MinVersion": "1.0.0"
}
},
The validator should read each referenced profile and tests for the requirements specified in the listed profile(s).
However, the validator appears to ignore this property.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.