Investigate if prometheus-tester could be replaced with avalanche about prometheus-k8s-operator HOT 4 CLOSED

special test charm is half way between unit test and integration test

I disagree. It's perfectly fine to use test doubles in an integration test. What you are getting at sounds to me more like an end-to-end test, which while being valuable, is something that likely should live in the bundle repo, close to the final solution. Regardless, with those criteria, avalanche is a special test charm as well. 😛

if instead we can use existing charms, we should

This part I fully agree with, but not if it's not solving the problems we need it to address.

The developer does not have to bother if there is any other charm available to create his integration test with. The developer does not have to worry if some strange behaviour he is seeing is a problem with his charm under development or just a quirk of the other charm he happened to choose for his integration test.

I think you are both getting at the same thing here. We need to be able to test very particular conditions in our integration tests, and need a solid way to arrange that from the outside, and we don't want to take on additional maintenance burden unless it serves a purpose. However, I'm not sure we have the need of a full load generator for this scenario either.

I would suggest that once you're back from your vacation @balbirthomas, you and @sed-i sit down and have a look at your test charm, and see whether that behavior would be easily reproducible with avalanche as well.

If so, excellent - less for us to maintain. If not, still excellent - now we know.

from prometheus-k8s-operator.

I see no reason where there should not be functional or integration tests using more controlled test charms. Here are some of the benefits of tests in each repository using test charms

1. The test charms can be instrumented in arbitrary ways there by allowing very controlled tests.
2. The test charms can be designed to support a wide range of tests including, fuzzing, and detailed validation of state transitions.
3. The test charms tests can be made to be more lightweight and be capable of being run individually thereby facilitating the charm development process while the developer is creating a new charm or adding new features to the charm.
4. The tests charms support the creation of charm repositories that provide a self-contained validation of the functional behaviour of the charm irrespective of the current state of any other charm.

Finally I see no reason why such test charm based tests can not exist in addition to any bundle tests in the bundle repository.

from prometheus-k8s-operator.

• special test charm is half way between unit test and integration test
• having a single-purpose special charm is more difficult to maintain: if instead we can use existing charms, we should

from prometheus-k8s-operator.

Let us be honest here for any but the most trivial charms we have been creating special test charms during development or at least some of us have felt the need to do so. These special test charms are fairly trivial to create and hardly take any time. The are far more than a unit test. The charm under test is being tested with a real deployed workload unlike a unit tests. They test exactly the same functionality an integration test would. And they provide the developer true validation and insight into the behaviour of his charm while he is developing the charm. The developer does not have to bother if there is any other charm available to create his integration test with. The developer does not have to worry if some strange behaviour he is seeing is a problem with his charm under development or just a quirk of the other charm he happened to choose for his integration test.

from prometheus-k8s-operator.