The above tests are failing due to pip is not installed on the test server. Either setup pip installation on the test server before running tests or change tests.

Foreman-maintain PR: theforeman/foreman_maintain#247

Failing because of FOREMAN_MAINTAIN_USE_BETA environment variable is not set properly.

use following code as template:

import pytest

@pytest.fixture(scope='function')
def setup_for_testcase(request):
    print('\nsync_plan_setup()')
    def teardown_for_testcase():
        print('\nsync_plan_teardown()')
    request.addfinalizer(teardown_for_testcase)

def test_testcasename(setup_for_testcase):
    print("testing")

tests.test_health.test_positive_check_hotfix_installed is failing in teardown step because it's unable to reinstall certain package because it's locked by version-locking.

New check added to foreman-maintain to detect hotfix on system.

Check if TMOUT environment variable is set: [OK]

Foreman-maintain PR: theforeman/foreman_maintain#261
description:

This patch reimplements hammer ping to use simple net::http
call instead of hammer. The main advantage is we can easily
skip the hammer auth setup which is not required for the call.
Another advantage is we can extend ping to pure foreman
and proxy.

Internaly the ping implementation moved to the instance feature,
related CLI option was kept for backward compatibility. The ping?
now returns true/false and the additional details such as error
message and failing services can be checked using additional
methods. Restrictions to :katello feature were removed as
the ping can query all types of instances now.

Maintain hammer configuration theforeman/foreman_maintain#165

cover scenarios having:
a. invalid_host
b. invalid_password
c. missing_password

test_health.test_negative_check_hammer_ping is failing for upstream FM because of "hammer-ping" check will not available any more in FM > v0.4.3 instead we will be using "server-ping".

currently test_positive_repositories_setup only tests repositories_setup scenario for 6.3, 6.4 and 6.5. But scenarios for 6.3.z, 6.4.z, 6.5.z, 6.6 and 6.6.z are missing.

It would be nice to check if the created backups contain the expected files and directories, this would allow us to find possible regressions with backup succeeding but not creating all necessary files.
Needs some research on differences in online/offline backup files and also between sat and capsule. As a first step this needs an way to extract the backup dir name from the ansible output

https://bugzilla.redhat.com/show_bug.cgi?id=1487559

theforeman/foreman_maintain#130

Backup (but also majority of other) tests rely solely on

assert "FAIL" not in result['stdout']

which can miss some problems, for example if a required command option changes, ansible_module.command produces no stdout therefore the above assert passes even if command actually fails. I suggest adding also assert result['rc'] == 0 wherever possible

Seems like the test_positive_restore_online_backup never succeeded, probably because the construct_command method slips some unwanted None to the foreman-maintain command, output looks as follows:

2018-08-24 02:55:43,677 - testfm.log - INFO - {'_ansible_parsed': True, 'stderr_lines': [u'ERROR: too many arguments', u'', u"See: 'foreman-maintain restore --help'"], u'cmd': [u'foreman-maintain', u'restore', u'None', u'-y', u'/tmp/online_backup_restore/'], u'end': u'2018-08-24 02:55:43.642736', '_ansible_no_log': False, u'stdout': u'', u'changed': True, u'rc': 1, u'start': u'2018-08-24 02:55:42.550026', u'stderr': u"ERROR: too many arguments\n\nSee: 'foreman-maintain restore --help'", u'delta': u'0:00:01.092710', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'_raw_params': u'foreman-maintain restore None -y /tmp/online_backup_restore/', u'removes': None, u'creates': None, u'chdir': None}}, 'stdout_lines': [], 'failed': True}

Scary thing is that did not prevent automation from passing, see test report for ttest_restore/test_positive_restore_online_backup in build no.37 but that was before the inclusion of exit code check in #5

Will investigate further, probably some additional method for base would be a good solution, similar to run_online_backup

theforeman/foreman_maintain#210 (comment)

# ./bin/foreman-maintain service stop --only foreman-tasks
# ./bin/foreman-maintain backup online /tmp
Starting backup: 2018-09-14 05:48:24 -0400
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------
Make sure Mongo DB is up: 
| Checking connection to the Mongo DB                                 [FAIL]    
undefined method `keys' for [SystemService(mongod [5])]:Array
--------------------------------------------------------------------------------
undefined method `keys' for [SystemService(mongod [5])]:Array

requirements installation fails for me in python 3 virtual env:

Collecting pytest-ansible==1.3.1 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/28/8a/1bc0e1473fb67a5854e5d3e4c625cda221c86805793faefe02d0b3944524/pytest-ansible-1.3.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-ms6oheq7/pytest-ansible/setup.py", line 67
        print "Removing '%s'" % rm

python 2 works ok, but I think we should either note in the readme that testfm is not py3 compatible, or maybe update pytest-ansible version if it doesn't break anything else

Now automation could not catch issue such as https://bugzilla.redhat.com/show_bug.cgi?id=1684730 until and unless it won't affect the upgrade. But we should catch such issues.

Add test case to verify correct respository enabled for beta

https://bugzilla.redhat.com/show_bug.cgi?id=1741909

upstream PR: theforeman/foreman_maintain#249
upstream issue: https://projects.theforeman.org/issues/26216

Scenarios to test in Testfm:

• satellite-installer --lock-package-versions

  • Check packages are locked or not using:
    • foreman-maintain packages is-locked
    • foreman-maintain packages status
    • Check using yum command
    • Check locked packages listed in file
  • Run satellite-installer --no-lock-package-versions
    • Check packages are unlocked or not using:
      • foreman-maintain packages is-locked
      • foreman-maintain packages status
      • Check using yum command
      • Check packages are not listed in file anymore

• foreman-maintain packages lock

  • Check packages are locked or not using:
    • foreman-maintain packages is-locked
    • foreman-maintain packages status
    • Check using yum command
    • Check locked packages listed in file
  • Run foreman-maintain packages unlock
    • Check packages are unlocked or not using:
      • foreman-maintain packages is-locked
      • foreman-maintain packages status
      • Check using yum command
      • Check packages are not listed in file anymore

Remove stubbed decorator for following tests because the checks which are used in tests are now available in foreman-maintain v0.4.3

• test_positive_check_upstream_repository
• test_positive_puppet_check_no_empty_cert_requests
• test_positive_puppet_check_empty_cert_requests

Also fix test_positive_foreman_maintain_health_check failing because of check puppet-check-no-empty-cert-requests which needs users permission. Fix the test by using --assumeyes option

Lets have a decorator for selecting test based on minimum or maximum version.

Ex:

1. test will run for version 6.6 and above then, start_in('6.6') /
2. test will run for version less than 6.5, ends_in('6.5')

https://bugzilla.redhat.com/show_bug.cgi?id=1624699

From user perspective, foreman_docker removal is done largely by foreman-maintain - relevant db migrations are run as part of upgrade procedure. I briefly checked code here, but it seems that testfm doesn't have any support for db migrations testing yet.

Is my assertion correct?

If it is, what would need to be done to enable db migrations testing here? Is this relatively small, or rather large task?

https://bugzilla.redhat.com/show_bug.cgi?id=1626651

failing due to missing 6.6 version

backup tests failed on capsule

>       assert set(files_list).issuperset(
                   ONLINE_BACKUP_FILES + CONTENT_FILES), assert_msg
E       AssertionError: All required backup files not found
E       assert False
E        +  where False = <built-in method issuperset of set object at 0x7f0705091588>((['candlepin.dump', 'config_files.tar.gz', '.config.snar', 'foreman.dump', 'metadata.yml', 'mongo_dump', ...] + ['pulp_data.tar', '.pulp.snar']))
E        +    where <built-in method issuperset of set object at 0x7f0705091588> = {'.', '..', '.config.snar', '.pulp.snar', 'config_files.tar.gz', 'metadata.yml', ...}.issuperset
E        +      where {'.', '..', '.config.snar', '.pulp.snar', 'config_files.tar.gz', 'metadata.yml', ...} = set(['.', '..', 'config_files.tar.gz', '.config.snar', 'metadata.yml', 'mongo_dump', ...])

Testcase should be able to find all the enabled sync-plans and should ensure that all of them gets disabled and enabled using foreman-maintain.

foreman-maintain PR: theforeman/foreman_maintain#268

Goal: To perform foreman-maintain testing with upstream nightly.

Description: Currently our automation run in Jenkins get triggered automatically according to satellite builds. Extend the support for upstream nightly as well so that we can catch the issues in early phase.
This will include setting up upstream nightly instance and running tests from testfm on top of it.

Limitations:

• Currently upstream nightly upgrade support is still in development
  theforeman/foreman_maintain#230

Till we have upgrade support in foreman-maintain, we can run tests related to backup, restore and services functionality on upstream.

Action Item:
See how can we utilize existing upstream nightly provisioning instance and trigger testfm automation.

stay with ansible 2.6

theforeman/foreman_maintain#244

As foreman-maintain has service module working, run service tests on capsule as well

would be nice to have at least flake8 run, never tried to set it before, though will investigate

https://bugzilla.redhat.com/show_bug.cgi?id=1632768

some of testcases are:
tests.test_advanced.test_positive_foreman_maintain_enable_maintenance_mode
tests.test_advanced.test_positive_foreman_maintain_disable_maintenance_mode
tests.test_health.test_positive_check_upstream_repository

Incremental backup should use the backup dir from previous backup, current tests use the parent dir instead, so essentially the backup is essentially full, not incremental. Strange that f-m does not complain about it, will file an issue also there

https://projects.theforeman.org/issues/24942

foreman-maintain health check --label check-upstream-repository

# ./bin/foreman-maintain health check --label check-upstream-repository
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check if any upstream repositories are enabled on system: 
- Checking for presence of upstream repositories                      [OK]      
--------------------------------------------------------------------------------