[WIP] Acceptance Criteria for peer review:

Acceptance Criteria:

Steps to test a CNF w/ a valid Helm chart:

• Pull down latest version of CNF conformance test code from develop branch
• Run shards install
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr helm_chart_valid
• Result: I would expect the output to be Pass with the CoreDNS CNF sample

Steps to test a CNF w/ an invalid Helm chart:

• Pull down latest version of CNF conformance test code from develop branch
• Run shard install
• Run crystal src/cnf-conformance.cr sample_coredns_cleanup
• Run crystal src/cnf-conformance.cr bad_helm_cnf_setup
• Run crystal src/cnf-conformance.cr helm_chart_valid
• Result: I would expect the output to be Failing

Steps to test a sample CNF w/ a valid Helm chart against all tests in the test category:

• Run crystal src/cnf-conformance.cr cleanup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr installability
• I expect to see PASSED for the helm valid test

Steps to test the entire CNF Conformance test suite with the sample CNF w/ a valid Helm chart:

• Follow Steps to test a CNF w/ a valid Helm chart from above
• Run crystal src/cnf-conformance.cr all
• I expect to see PASSED for the helm valid test

from testsuite.

Level of effort in pts: 0, 1, 2, 3, 5, 8, 13, 21, infinity: 5 pts

@wavell 8 pts
@williscool 5 pts
@lixuna 5 pts
@denverwilliams 5 pts
@taylor 5 pts
ashleigh 5 pts
nupejosh 8 pts

from testsuite.

Added new task above:

• If one test blows up (eg. CNF code not found for static) the suite should continue to run

from testsuite.

Added to A/C:

Steps to test a sample CNF w/ a valid Helm chart against all tests in the test category:

• Follow Steps to test a CNF w/ a valid Helm chart from above
• Run crystal src/cnf-conformance.cr installability
• I expect to see PASSED for the helm valid test

from testsuite.

Deferring support for testing multiple CNFs which are in the /cnfs folder.

Target CNF Tasks (eg. How do we use the sample CNF)

• Check /cnfs folder for CNFs.
• For each folder check inside for a cnf-conformance.yml. If the file is found run conformance tests for that CNF
• Update documentation to say that each CNF should be in a subfolder of the CNF

from testsuite.

@williscool please add a sample invalid CNF helm chart and test that the linter fails. The CoreDNS working helm chart can be modified to be invalid. See A/C for testing.

I tested on develop branch

rosenberg% git status
On branch develop
Your branch is up to date with 'origin/develop'.

I do not see a sample invalid cnf

rosenberg% ls sample-cnfs
sample-coredns-cnf

from testsuite.

@williscool, it looks like the setup task uses a different directory lfn-cnti/certification@89d81ce#diff-d8bebbf3115bd02eeb3a3e284a0535f7R75

Please match the cnf-conformance.yml to the cnf directory from setup

from testsuite.

@williscool Please update the CNF README to match the new sample cnf name, directory, etc

from testsuite.

@williscool It looks like some steps are not needed in this sample cnf including the helm chart install https://github.com/cncf/cnf-conformance/blob/develop/sample-cnfs/sample-bad_helm_coredns-cnf/README.md#core-dns-installation

Please review and remove the steps that are invalid for this sample

from testsuite.

@williscool the cnf-conformance.yml points to the coredns folder instead of a the "cnfs/bad_helm_chart" folder created in setup

From what I can tell it looks like the setup will copy that cnf-conformance.yml over and use it when the test runs.

from testsuite.

@wharris3, please add a comment with updated A/C steps for testing Steps to test a CNF w/ an invalid Helm chart:

It appears a new setup command is needed for that to run. Something like

rosenberg% crystal src/cnf-conformance.cr bad_helm_cnf_setup
Successfully created directories for cnf-conformance
Cloning into '/home/taylor/p/w/cncf/test/cnf-conformance/cnfs/bad_helm_coredns'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 55328 (delta 35), reused 30 (delta 19), pack-reused 55270
Receiving objects: 100% (55328/55328), 94.72 MiB | 3.04 MiB/s, done.
Resolving deltas: 100% (29285/29285), done.

Then run the test

rosenberg% crystal src/cnf-conformance.cr helm_chart_valid
FAILURE: Helm Chart stable/coredns Lint Failed

from testsuite.

@denverwilliams @wavell @williscool with the latest updates (PR 86, PR 87) is this ready for peer review again? If so please move to the peer review column and tag someone.

from testsuite.

A/C passes for helm chart. (PASS and FAIL scenarios)

from testsuite.