The test_dirs option should be available in config so it's not required in command line with every IPA invocation.

See SUSE-Enceladus/mash#461

The fixtures which are just functions are using camel case formatting and should instead be snake case to follow pep8 conventions.

Add CLI option to delete a specific history item.

For example:

ipa results --clear --show 3

or 

ipa results --clear-item 3

Deletes the history result at index 3.

It would be a nice enhancement to have a config setup option in CLI. This would help the user setup the ipa config and/or all config files with info for testing. Since a user may not have config setup for ec2utils, azurectl or Google.

ipa setup

Series of prompts requesting config data.

If there is a failure when getting ssh connection attempt to get console log from framework and print to test log.

Although it's duplicative to provide results file location in the results file output, it is helpful when calling IPA from code.

It's possible that a newly launched instance is not queryable via DescribeInstances right after launch. This will cause an unhandled exception in the _get_instance_state method that causes img-proof to crash without cleaning up properly.

ERROR 2019-12-10 19:40:03,634 TestingService
    Pass[1]: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_ec2.py", line 210, in _get_instance_state
    state = instance.state['Name']
  File "/usr/lib/python3.6/site-packages/boto3/resources/factory.py", line 339, in property_loader
    self.load()
  File "/usr/lib/python3.6/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidInstanceID.NotFound) when calling the DescribeInstances operation: The instance ID 'i-00c7aeb7ca53015f3' does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mash/services/testing/ec2_job.py", line 139, in run_job
    tests=self.tests
  File "/usr/lib/python3.6/site-packages/mash/services/testing/img_proof_helper.py", line 51, in img_proof_test
    timeout=img_proof_timeout
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_controller.py", line 132, in test_image
    return cloud.test_image()
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_cloud.py", line 669, in test_image
    self._launch_instance()
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_ec2.py", line 273, in _launch_instance
    self._wait_on_instance('running', self.timeout)
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_cloud.py", line 474, in _wait_on_instance
    current_state = self._get_instance_state()
  File "/usr/lib/python3.6/site-packages/img_proof/ipa_ec2.py", line 215, in _get_instance_state
    instance_id=self.running_instance_id
img_proof.ipa_exceptions.EC2CloudException: Instance with id: i-00c7aeb7ca53015f3, cannot be found.

This is hard to confirm given the error seems to be very infrequent. However, adding a slight delay before getting the instance state should resolve this. To prevent an instance being left in running state the exception should be handled gracefully which will allow img-proof to log the failure and do proper cleanup.

If ipa handles listing results from an archive then the user should just give us the archive name, i.e.

ipa results list --archive $PATH_TO_MY_ARCHIVE
ipa results show 5 --archive $PATH_TO_MY_ARCHIVE

And ipa should just extract the file on the fly to get the results.

Question is should this be to a temp dir which is deleted after the command or should it be cached somehow for subsequent calls?

Consider implementing some sort of test config files to simplify running tests multiple times and running similar tests.

This file would contain default command line args/options minimizing the length of command line statements.

Could be helpful if a user has many tests that use a similar set of options/args.

This is document the steps for the package rename which are as follows:

• Rename project in GitHub
• Setup project in Travis for CI
• Create a new read the docs project and hook to build docs
• Add new project to PyPI
• Deprecate / remove old python3-ipa project from PyPI
• Release a new version of python3-img-proof in OBS
• Propagate package to all projects in OBS
• Any usages of ipa config will need migrated to ~/.config/img_proof/config to prevent confusion from using ~/.config/ipa/config.
• Any usages of ipa will need to confirm the ~/img_proof directory is created.
• Any usages of the config file will need to change the ipa section to img_proof.

If a test fails prior to a sync point any subsequent tests are skipped and results are not aggregated to summary.

Scenario:
ipa test ... test_something_fails test_update test_another_thing

ipa results
>> FAILED tests=1|pass=0|fail=1|error=0

The second test is ignored.

There are cases where it would be helpful to run IPA tests against an instance via SSH without the requirement of provider credentials.

• The SSH backend would require a new instance-ip argument.
• All methods that require API authentication will do nothing (pass).
• A new flag will be required --ssh-only and provider will still be a required argument.

ipa test ... --instance-ip 10.0.0.1 --ssh-only ec2

If the repos test can pull CSP name from metadata it could be updated and this would then just be a new provider name. #112

ipa test ... --instance-ip 10.0.0.1 ssh

Consider calling metadata directly instead of from packaged script to support other distros.

https://github.com/SUSE-Enceladus/ipa/blob/master/usr/share/lib/ipa/tests/conftest.py#L129

If there is an error in a sync tests such as update the error is not logged. The exception is thrown and caught by CLI.

Problems:

• Error not logged
• Results file is not written
• Log file is not updated

Instead the error should be caught and logged and event loop should fail gracefully.

For the SLES tests we can infer the cloud environment from dmidecode.

e.g. (google, amazon, microsoft) in output.lower():

sudo dmidecode -t system

With this we can find the region from meta-data. This way the parameters don't need injected and tests can be run on an instance without the need of provider based class.

If an invalid service account json is provided for GCE raise a helpful error message instead of KeyError.

https://github.com/SUSE-Enceladus/ipa/blob/master/ipa/ipa_gce.py#L136

The --ssh-private-key option is confusing. Consider renaming it to --ssh-private-key-file.

I think the best route is to add a new arg --ssh-private-key-file and deprecate the old one. Or we can add another option name then remove the old one in a subsequent version.

@click.option(
    '--ssh-private-key',
    '--ssh-private-key-file',
    type=click.Path(exists=True),
    help='SSH private key file for accessing instance.'
)

Leap requires acceptance of new signing keys for repos:

• Cloud:Tools
• Devel:Languages:Python

Currently update fails when attempting zypper refresh.

In clear_cache method of utils the ssh clients should be closed prior to removing the reference. https://github.com/SUSE/ipa/blob/master/ipa/ipa_utils.py#L47

Should not rely on garbage collection to close the connections. http://docs.paramiko.org/en/2.4/api/client.html#paramiko.client.SSHClient.close

Consider adding a way to inject environment variables into the test instance.

In the EC2 class the ssh private key value has a different key in ec2utils config. This causes ipa and command line values to be overlooked.

If there is an error in a sync test such as test_update the output is not added to log. The output should be appended prior to err output in exception message. This makes debugging errors easier.

Problem: The timeouts for SSH connections and instance state changes are not always sufficient.

This depends on things like provider, network, region and instance size.

Make these timeouts configurable through (a) command line option(s).

The suse specific repository test should provide more details. With the switch to the modules and the various flavors of repositories, source, debuginfo etc. it has pretty much become impossible to manually verify registration success. IPA provides the necessary functionality. However, the registration test has a pass/fail status leaving the user/verifyer with the original problem. the test shold produce a diff list in the results if the repo data does not match the expected content. This should hopefully allow the person verifying the test results an opportunity to understand the failure and address it.

If either of the default test dirs are missing an exception is raised.

Instead of raising an error the dir should be ignored.

If there is an exception with the initial SSH connection after an instance is launched the instance is not cleaned up and the results are not properly logged.

Ensure that any exception during testing is handled properly. With the exception handling cleanup the instance if necessary and log results.

Instead of throwing an error when config file is not found attempt to run tests with provided command line args.

Config file not found: /{user-home}/.config/ipa/config

Currently an image cannot be tested in EC2 without the .ecutils.conf config file. Even if all values have been provided via command line.

Allow for testing images in EC2 without a configuration file so long as all required args have been supplied via command line.

Also if the config file is optional then the account_name should also be optional.

Distinguishing SAP images based on CPE_NAME instead of PRETTY_NAME.

The distinction in PRETTY_NAME has been removed in SP3.

https://github.com/SUSE/ipa/blob/master/usr/share/lib/ipa/tests/conftest.py#L73

CPE_NAME="cpe:/o:suse:sles_sap:12:sp3"

The determine region fixture returns the zone. E.g:

us-west1-a

This should return the region instead.

us-west1

Service account file is now used in Azure provider for authentication.

• Remove comment from https://github.com/SUSE/ipa/blob/master/ipa/ipa_azure.py#L57.
• Update docs and readme with instructions on generating service account files.

~> ipa list -v
ERROR: file not found: /home/ec2-user/ipa/tests/

expected behavior is to ignore the non-extant directory and list the available tests

Consider migrating EC2 and Azure to Apache Libcloud to minimize dependencies and make providers more uniform.

This would also make adding new providers much easier so long as they are supported by libcloud.

Todo:

• GCE
• EC2
• Azure

Attempting to launch azure instance in existing subnet uses the wrong names. The subnet and vent name are overridden by generated values.

These should be left as is if they are provided.

If ipa test is invoked with --history-log and the path does not exist yet an error is raised.

Error: Invalid value for "-h" / "--history-log": Path "foobar" does not exist.

Instead ipa should attempt to create path for new file.

With sync tests now in results if results are printed with verbose flag there is an error.

The sync tests have no path and should just be returned when parsing.

If an IPA test is started using the API instead of the CLI the distro name is not validated until after an instance is launched.

The set_distro method should instead be called prior to launching or connecting to the instance. This ensures the distro name is validated eagerly.

The results report plugin provides a simple dict output of pytest results. This is used for merging multiple results in a single ipa test run. For a more thorough report move to the pytest json plugin. This will require a patch submitted upstream and an option to make report accessible from code instead of saved to a json file.

ERROR 2019-05-20 20:14:03,773 TestingService
Pass[1]: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipa/ipa_cloud.py", line 636, in test_im
age
client
File "/usr/lib/python3.6/site-packages/ipa/ipa_utils.py", line 262, in get_hos
t_key_fingerprint
client.get_transport().get_remote_server_key().get_fingerprint()
File "/usr/lib/python3.6/site-packages/paramiko/transport.py", line 761, in ge
t_remote_server_key
raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/mash/services/testing/ipa_helper.py", l
ine 95, in ipa_test
timeout=ipa_timeout
File "/usr/lib/python3.6/site-packages/ipa/ipa_controller.py", line 131, in te
st_image
return cloud.test_image()
File "/usr/lib/python3.6/site-packages/ipa/ipa_cloud.py", line 644, in test_im
age
'An error occurred retrieving host key: %s' % error
ipa.ipa_exceptions.IpaCloudException: An error occurred retrieving host key: No
existing session

mash job id: 4f8c81dd-cfd4-4697-be5f-1777c6c4b220

When a running instance is provided with the -r option the instance is terminated if tests pass. This can be configured using the --cleanup/--no-cleanup however the instance by default should be left running.

Using -r should always leave the instance running.
Using -i should terminate the instance if all tests pass.
And --cleanup/--no-cleanup override the default functionality.

ipa needs to be able to archive results and the history file. For example

ipa results --create-archive --location my_network_drive/ipa_results/my_distro_version_tests --results 1,2,3,6,7,9,10

where the numbers correspond to the index of the results list from

ipa results --list

There continues to be random flaky paramiko errors during pytest runs. Should investigate a solution where we can re-run tests if they get ssh exceptions. Either retry via pytest mechanism or run all tests separately so img-proof can handle retries.

When the test_sles_wait_on_registration.py test was written there was an issue that required using pgrep to get service status. This should be re-tested to pinpoint the issue or cleanup the test.

This appears to be a bug in Click:

If the region shorthand is used without the running instance id the region value is put on running_instance_id.

ipa test ... -r us-west-1 -i ami-123456 ...

Instance with ID: us-west-1 not found.

Whereas this works fine:

ipa test ... -r us-west-1 -R i-1234567890 ...

For on demand test suite the update is not necessary. If refresh passes then things are good. Add a new sync test for refresh and use that instead of test_update in test_sles_on_demand.

If a test is skipped it is not accounted for by ipa results.

Number of skipped should be part of output:

PASSED tests=1|pass=0|fail=0|error=0

And the verbose styled output should be yellow or green not red.

test_sles_force_new_smt_reg::test_sles_force_new_smt_reg SKIPPED

SUSE-Enceladus/mash#493

Update the GCE docs to more explicitly state the steps necessary to setup and configure a service account for testing.