dmtf / redfish-usecase-checkers Goto Github PK

Hi,
After run the Query Parameter Checker, all the ErrorMessages show Query ({}) expected to expand resource {} on the result.json. But I try to GET /redfish/v1/?$expand=* or other supported syntax can expand resource like below:

I'm not sure if error messages also means anything else. Could you help to explain literal meaning if I misunderstand or our respond body doesn't meet specifications.
Test result and manual test respond content attached for you refer.
results and manual test.zip

Would like to have a usecase checker for applying BIOS settings. This type of test would involve the following:

1. Locate a BIOS resource
2. Apply a new attribute value to the Settings object for the BIOS resource
3. Adjust the Settings Apply Time to have the value applied on the next reboot
4. Reboot the Computer System
5. Check that the new Setting was applied in the BIOS resource, as well as an update to the Settings annotation to show when the new value was applied

Some things to watch for:

• It's very unlikely that a BIOS resource does not have a Settings object, however it's theoretically possible. We might need to consider this case in the test, which would mean simply a PATCH to the BIOS resource would take effect immediately
• There may not be a Settings Apply Time object, which means we can only assume the value will be applied on the next reboot
• There may need to be some timeout considerations for the last step
• Since different vendors will have different attributes, it will likely be necessary to have the specific attribute and value to apply in the test as a parameter
• It might be beneficial to add a step that sanity checks the BIOS resource against the advertised Attribute Registry
• Cleanup at the end of the test would be good to ensure the system returns to its previous state

Hi,
When I Sent the command like below:
python3 account_management.py -r 192.168.0.116 -u -p -S Always

The result is shown in the screenshot:

I checked PasswordChangeRequired Setting is true from new created account. Are there any methods to create an account that doesn't need to change the password? or create other RoleId account?

Thanks

cloned latest code this morning. power_control.py is failing on both Dell and Nokia servers. Results in doc.
power_control_results.docx

In top level Redfish-Usecase-Checkers repo, the Readme info needs to include info about behavior and input for One Time Boot checker.

A Power Action that shuts down an already powered off system, or powers on an already powered system, is expected not to change. Write in extra test to observe the message.

Testing on different server hardware, etc. power_control.py is successful at System Count, Reset Type Check, Reset Performed but fails at getting correct Power State Check;
hardware set one:
"ErrorMessages": [
"Power State Check: Self was not in the Off state after using GracefulShutdown as the reset type",
"Power State Check: Self was not in the Off state after using ForceOff as the reset type"

hardware set two:
"ErrorMessages": [
"Power State Check: System.Embedded.1 was not in the On state after using On as the reset type",
"Power State Check: System.Embedded.1 was not in the On state after using ForceRestart as the reset type",
"Power State Check: System.Embedded.1 was not in the On state after using PushPowerButton as the reset type"

1. Power Control Checker
   python3 power_control.py -r : -u -p -S Always
   Found 1 system instances
   Resetting Partition0 using On
   Task is Done!
   Monitoring power state for Partition0...
   Resetting Partition0 using ForceOff
   Task is Done!
   Monitoring power state for Partition0...
   Resetting Partition0 using ForceRestart
   Task is Done!
   Monitoring power state for Partition0...
   ERROR: Partition0 was not in the On state after using ForceRestart as the reset type.
   Resetting Partition0 using GracefulRestart
   Task is Done!
   Monitoring power state for Partition0...
   ERROR: Partition0 was not in the On state after using GracefulRestart as the reset type.
   Resetting Partition0 using GracefulShutdown
   Task is Done!
   Monitoring power state for Partition0...
   Resetting Partition0 using PowerCycle
   Task is Done!
   Monitoring power state for Partition0...

Note: In partition power off state, npar GracefulRestart and ForceRestart are not allowed. Need a script modification to bring partition to ON state before doing GracefulRestart, ForceRestart and GracefulShutdown

2. Account Management Checker
   python3 account_management.py --r : -u -p -S Always
   Creating new user 'testuser0'
   ERROR: Failed to find user 'testuser0' with the role 'Administrator'.
   Logging in as 'testuser0'
   Logging in as 'testuser0', but with the wrong password
   Setting user 'testuser0' to role 'ReadOnly'
   ERROR: Failed to set user 'testuser0' to 'ReadOnly' (Operation failed: HTTP 400
   A general error has occurred. See Resolution for information on how to resolve the error.
   The property RoleId is not in the list of valid properties for the resource.).
   Setting user 'testuser0' to role 'Operator'
   ERROR: Failed to set user 'testuser0' to 'Operator' (Operation failed: HTTP 400
   A general error has occurred. See Resolution for information on how to resolve the error.
   The property RoleId is not in the list of valid properties for the resource.).
   Setting user 'testuser0' to role 'Administrator'
   ERROR: Failed to set user 'testuser0' to 'Administrator' (Operation failed: HTTP 400
   A general error has occurred. See Resolution for information on how to resolve the error.
   The property RoleId is not in the list of valid properties for the resource.).
   Deleting user 'testuser0'

-Script is not parsing the password and it is appearing as 'null' while logging with new user created
-We do not support modifying user roles

3. Query Parameter Checker
   python3 query_parameters_check.py --r : -u -p -S Always
   Performing $expand=* on /redfish/v1/
   Performing $expand=. on /redfish/v1/
   Performing $expand=! on /redfish/v1/
   ERROR: Failed to perform $expand test (Operation failed: HTTP 400
   A general error has occurred. See Resolution for information on how to resolve the error.
   The value ! for the parameter $expand is of a different format than the parameter can accept.).

• Our product does not support $expand=! instead we support $expand=~

Hi,
When I sent power control command:
python3 power_control.py -r 192.168.0.116 -u -p -S Always

Found 1 system instances
Resetting Self using ForceOff
Task is Done!
Monitoring power state for Self...
Resetting Self using ForceRestart
Monitoring power state for Self...
ERROR: Self was not in the On state after using ForceRestart as the reset type.
Resetting Self using GracefulShutdown
Monitoring power state for Self...
Resetting Self using On
Task is Done!
Monitoring power state for Self...

I refer the schema ComputerSystem.v1_15_0.json mention "The ForceRestart value shall perform a ForceOff action, followed by an On action.". And refer an example in Specification:

So I think whether the ForceRestart should not be after ForceOff then avoid show Error message.

Thanks.

Test execution fails with error (below).

max@myepanes-bmc:~/openbmc/dmtf/Redfish-Usecase-Checkers/one_time_boot$ python3 one_time_boot.py 127.0.0.1:5000 Once Pxe -u root -p password123456
Traceback (most recent call last):
  File "one_time_boot.py", line 225, in <module>
    sys.exit(main(sys.argv))
  File "one_time_boot.py", line 197, in main
    base_url=args.rhost, username=args.user, password=args.password, default_prefix="/redfish/v1")
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 1027, in redfish_client
    max_retry=max_retry)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 952, in __init__
    max_retry=max_retry)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 422, in __init__
    self.get_root_object()
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 553, in get_root_object
    raise excp
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 551, in get_root_object
    resp = self.get('%s%s' % (self.__url.path, self.default_prefix))
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 585, in get
    headers=headers)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 979, in _rest_request
    args=args, body=body, headers=headers)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 844, in _rest_request
    raise RetriesExhaustedError()
redfish.rest.v1.RetriesExhaustedError

The same error is observed for the OpenBmc QEMU:
python3 one_time_boot.py 192.168.122.52:443 Once Pxe -u root -p 0penBmc

Would like to finalize the checker for using ComputerSystem Reset. It should look something like this:

• Find a ComputerSystem to test
• Invoke a Reset action based on the allowable values for ResetType

Some considerations:

• Should multiple resets be performed? Loops through all possible ResetType options?
• Reset might result in a Task and needs to be managed.
• Even though the action may be successful, is that enough to really go by to show the requested reset was performed?
  • Do we need something host side to verify the system went down, came back through BIOS, and booted to an OS?

When I run powercycle.py, the last two reset types are, in sequential order:
GracefulShutdown
PowerCycle

This test fails in the PowerCycle stage when the host state is initially in the power off state (because of the preceding GracefulShutdown) and then the host does not power up as expected by the test.

I suspect this is a test issue because in the case of IPMI, the PowerCycle command should not power up from the power off state:

Is the redfish design specification (sorry, I don't know where to check) consistent with the IPMI specification? If so, the test needs to be updated.

New Redfish-Usecase-Checkers fails if ServiceRoot/JsonSchemas is not available.
I was testing Account section.
In separate thread mraineri said
JsonSchemas is not required to be implemented; the property "JsonSchemas" in ServiceRoot_v1.xml is not annotated with Redfish.Required. This sounds like a bug in the Usecase Checkers.

https://github.com/DMTF/spmf/issues/2097

There is currently a 10 second timeout after each reset in the Power Control test. I'm finding that on some systems that is not long enough. I'd like to make changes to be able to use an input parameter to the script so that the user can define what's needed and it can default to 10 seconds if nothing is inputted. I'll make a pull request and link it to this issue

When we run the account_management.py, we get the result with failed to add user.
After changing code to check message, the root cause is that service doesn't support to modify the property Enabled so tool gets 400 Bad Request with message PropertyNotWritable.

1. Suggest to add the message of response in the result for helping to clarify error.
2. Suggest to modify more properties of ManagerAccount to avoid this case.

Thank you.

When we run the account_management.py, we get the result with failed to add user (Base.1.6.MalformedJSON).
After checking account_management.py as below:

        # Create a new user
            ...
            redfish_utilities.modify_user( redfish_obj, test_username, new_enabled = True )
            ...

Here should use new_enabled = true , right?

The value of BootSourceOverrideTarget is irrelevant when BootSourceOverrideEnabled is Disabled. BootSourceOverrideEnabled should change back to Disabled after one time boot. Isn't that what should be verified in the test?

account_manager.py and rf_accounts.py do not complete remove user with delete.. only remove name .

root@testingvm:/usr/local/bin# python3 rf_accounts.py -u root -p mypassword -r https://172.26.15.212

Name | Role | Locked | Enabled
root | Administrator | False | True
| Operator | False | True
| Administrator | False | True
| Administrator | False | True
| Administrator | False | True
| Administrator | False | True
| Operator | False | True
| Operator | False | True
testuser | Operator | False | True

root@testingvm:/usr/local/bin# rf_accounts.py -u root -p mypassword -r https://172.26.15.212 -delete testuser
Deleting user 'testuser'
Traceback (most recent call last):
File "/usr/local/bin/rf_accounts.py", line 45, in
redfish_utilities.delete_user( redfish_obj, args.delete )

root@testingvm:/usr/local/bin# python3 rf_accounts.py -u root -p mypassword -r https://172.26.15.212

Name | Role | Locked | Enabled
root | Administrator | False | True
| Operator | False | True
| Administrator | False | True
| Administrator | False | True
| Administrator | False | True
| Administrator | False | True
| Operator | False | True
| Operator | False | True

"TestResults": {
    "User Count": {
        "pass": 1,
        "fail": 0,
        "skip": 0
    },
    "Add User": {
        "pass": 2,
        "fail": 0,
        "skip": 0
    },
    "Credential Check": {
        "pass": 2,
        "fail": 0,
        "skip": 0
    },
    "Change Role": {
        "pass": 6,
        "fail": 0,
        "skip": 0
    },
    "Delete User": {
        "pass": 0,
        "fail": 1,
        "skip": 0
    },
    "ErrorMessages": [
        "Delete User: Failed to delete user 'alice73t'"
    ]
}

I'd like to propose a usecase for identifying components in repair locations, Google has published a public specification here: https://github.com/google/ecclesia-machine-management/blob/master/ecclesia/lib/redfish/g3doc/topology.md

We're hoping this usecase is general enough and uses only standard Redfish fields that this may be appropriate to be added to this tool.

Just wanted to ask what the process of doing so would be. Are there any requirements of consensus or cross-company agreements required for a usecase to be considered appropriate for adding?

Thanks,
Derek

While working on a different Redfishtool issue, I realized that the power_control checker should include the full set of match options that Redfishtool does for Systems commands.

The power_control checker does support the -I option already, but should also support -M, -L, -1 and -F.

The underlying redfishtool lib already has the logic to perform the matching; just need to add the arg parsing and RfTransport field initialization.

Hi, I tested the new files. The accounts are delete but the results.json say not deleted.

account_management_resutls.docx

Items such as the Power tools and User tools may react different based on how the service is currently in place. If possible, observe the current state of the machine before performing actions, print out errors if the state of the machine is not restored.

For exceptions, attempt to reverse operations.

Would like to see the One Time Boot Checker finished. We should have a flow similar to the following:

• Get a ComputerSystem on the service to test
• Examine the Boot object to see what one time boot options are allowed
• Set the boot option to USB
• Perform a reset of the ComputerSystem
• Verify the system booted via USB

Some things to consider:

• We might need to have a test image for booting to report back to the client that it booted successfully
  • There isn't anything in the data model today to indicate if the system is running off a given boot device, and this may not be something that can be realistically supported (unless others have ideas)
• Should we fall back on another interface (like PXE)
• Should we try to test both USB and PXE to show that the system is not "stuck" booting from the same device
• Reset may generate a Task and that needs to be handled
  • If supported, we could use OperationApplyTime to always have a Task spawn when there is a reset

In one_time_boot.py there are a few occurrences of various forms of "comply" rather than "conform" in the code. Use this issue to clean them up.

After adding the --all option to Redfishtool for the Systems commands, need to update the power_control usecase checker to also accept --all. So, if you call power_control with --all, it will issue the given reset command to all systems in the service.

Hi @mraineri ,

For Manager Ethernet Interface Checker, validator has a dependency on get_manager_ethernet_interface in managers.py in redfish_utilities that hardcodes interface_uri_pattern, which does not work on the device I have.

I changed the interface_uri_pattern to match the @odata.ids of the members in the EthernetInterfaceCollection I receive from the Redfish service on the device I have, and the Manager Ethernet Interface Checker completed the validation.

Is the hardcoded interface_uri_pattern on get_manager_ethernet_interface in managers.py in redfish_utilities the the only valid pattern? I wanted to ask this because I'm not sure whether the @odata.ids of the members in the EthernetInterfaceCollection I receive from the Redfish service on the device I have is correctly set.

Thanks,
Leo

the password in account_management needs at least one special character

I'm running into a failure on a PowerEdge when running the power_thermal_info test. Through debugging it looks like a Chassis endpoint does not have power/thermal modules (EnvironmentMetrics, PowerSubsystem, ThermalSubsystem, Power or Thermal). It is it valid that it fails here or should the chassis that does not have these modules just be skipped?

If this is a valid failure, should the logging be more descriptive to catch this error instead of returning a TypeError?