Git Product home page Git Product logo

potemkin-decorator's Introduction

The potemkin decorator facilitates "integration testing" boto code by creating AWS resources using Cloudformation or Terraform. This provides a convenient way to setup initial conditions instead of having to develop boto code that is likely as complex as the "code under test".

Basic Usage

CloudFormation

Here is an example CloudFormation invocation from pytest:

import potemkin
import boto3


@potemkin.CloudFormationStack(
  'test/integration/test_templates/aes256_bucket.yml',
  stack_name_stem='TestStack',
  parameters={'BucketName': 'unclefreddie33388'},
  aws_profile='myprofile',
  teardown=False
)
def test_bucket_has_aes256_encryption(stack_outputs, stack_name):
  full_bucket_name = stack_outputs['BucketNameOut']

  s3 = boto3.Session(profile_name='myprofile').client('s3')
  get_bucket_encryption_response = s3.get_bucket_encryption(
    Bucket=full_bucket_name
  )

  assert get_bucket_encryption_response['ServerSideEncryptionConfiguration']['Rules'][0]['ApplyServerSideEncryptionByDefault']['SSEAlgorithm'] == 'AES256'

The CloudFormationStack creates the stack and binds the outputs to stack_outputs. The pytest method could invoke more boto code to manipulate the resources created by the decorator. In this case, the test just asserts that the initial condition is what is expected.

This is basically a python/pytest port of "aws-int-test-rspec-helper" that worked with Ruby/RSpec:

Terraform

Here is an example Terraform invocation from pytest:

import potemkin
import boto3


@potemkin.TerraformResources(
  'test/integration/test_templates/terraform',
  parameters={'BucketName': 'unclefreddie33388'},
  aws_profile='myprofile',
  teardown=False
)
def test_bucket_has_aes256_encryption(tf_outputs):
  full_bucket_name = tf_outputs['BucketNameOut']

  s3 = boto3.Session(profile_name='myprofile').client('s3')
  get_bucket_encryption_response = s3.get_bucket_encryption(
    Bucket=full_bucket_name
  )

  assert get_bucket_encryption_response['ServerSideEncryptionConfiguration']['Rules'][0]['ApplyServerSideEncryptionByDefault']['SSEAlgorithm'] == 'AES256'

TerraformResources creates the resources and binds the outputs to tf_outputs. The pytest method could invoke more boto code to manipulate the resources created by the decorator. In this case, the test just asserts that the initial condition is what is expected.

Service Specific Usage

The potemkin decorator has additional functions for interacting with specific AWS services

AWS Config

AWS Config initiates evaluations when a resource is created, but the evaluations are completed asynchronously. They can take several minutes to complete. The AWS config functions wait until the config rule has an evaluation for the resource, then returns the evaluation.

config_rule_wait_for_compliance_results

This function polls aws config until all resource_ids have evaluations. It then checks those evaluations against expected results and returns a truthy value. This can be used by both configuration change events and periodic events (by setting evaluate=True)

@potemkin.CloudFormationStack('test/integration/test_templates/eip.yml',
                              stack_name_stem='EipTestStack')
def test_wait_for_compliance_results(stack_outputs, stack_name):
    global expected_results
    configservice = boto3.Session().client('config')

    expected_results_success = {
        stack_outputs['EIPOutput']: "NON_COMPLIANT",
        stack_outputs['EIP2Output']: "NON_COMPLIANT"
    }

    assert config_rule_wait_for_compliance_results(
        configservice,
        rule_name='eip-attached',
        expected_results=expected_results_success)

config_rule_wait_for_absent_resources

This function is a companion to config_rule_wait_for_compliance_results and is used to validate that once resources are deleted they are removed from AWS config.

def test_wait_for_compliance_results_success_results():
    configservice = boto3.Session().client('config')
    resource_ids = list(expected_results.keys())

    assert [] == config_rule_wait_for_absent_resources(
        configservice, rule_name='eip-attached', resource_ids=resource_ids)

config_rule_wait_for_resource

This function polls aws config until there is an evaluation for the resource, then returns it. Use this function for config rules with a configuration change trigger. If you are checking more than one resource, consider using config_rule_wait_for_compliance_results.

import potemkin
import boto3


@potemkin.CloudFormationStack(
  'test/integration/test_templates/aes256_bucket.yml',
  stack_name_stem='TestStack',
  parameters={'BucketName': 'unclefreddie33388'}
)
def test_bucket_encryption_rule(stack_outputs, stack_name):
  configservice = boto3.Session().client('config')

  results = config_rule_wait_for_resource(configservice, 
                                          resource_id='unclefreddie33388', 
                                          rule_name='config-rule-s3-encryption')
  
  assert results['ComplianceType'] == 'NON_COMPLIANT'

evaluate_config_rule_and_wait_for_resource

This is similar to config_rule_wait_for_resource but it first initiates a config evaluation. Use this for config rules with a periodic trigger. If you are checking more than one resource, consider using config_rule_wait_for_compliance_results.

import potemkin
import boto3


@potemkin.CloudFormationStack(
  'test/integration/test_templates/aes256_bucket.yml',
  stack_name_stem='TestStack',
  parameters={'BucketName': 'unclefreddie33388'}
)
def test_bucket_encryption_rule(stack_outputs, stack_name):
  configservice = boto3.Session().client('config')

  results = evaluate_config_rule_and_wait_for_resource(configservice, 
                                                      resource_id='unclefreddie33388', 
                                                      rule_name='config-rule-s3-encryption')
  
  assert results['ComplianceType'] == 'NON_COMPLIANT'

potemkin-decorator's People

Contributors

builder-pluralstack avatar twellspring avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

bonya

potemkin-decorator's Issues

Basic decorator

Given a Cloudformation template in the local file system
And the "stem" of a stack name for the template
When potemkin decorator is specified on a pytest method
Then the Cloudformation template is converged in AWS with ambient credentials
And the stack is created with the stem name concatenated with an epoch timestamp suffix
And the stack is torn down after the pytest method runs

Given the Cloudformation template doesn't exist
When potemkin decorator is specified on a pytest method
Then the test method fails with a clear message that it could not located the template

Given the stem name contains characters that are illegal for Cloudformation stack names
When potemkin decorator is specified on a pytest method
Then the test method fails with a clear message for what is illegal and legal as far as stack names

Allow disabling tear down

When potemkin decorator is specified on a pytest method
And the tearDown argument is set to False
Then the Cloudformation template is converged in AWS with ambient credentials
And the stack is created with the stem name concatenated with an epoch timestamp suffix
And the stack is NOT torn down after the pytest method runs

When potemkin decorator is specified on a pytest method
And the tearDown argument is set to True or not specified
Then the Cloudformation template is converged in AWS with ambient credentials
And the stack is created with the stem name concatenated with an epoch timestamp suffix
And the stack is torn down after the pytest method runs

Don't always evaluate config rule

The evaluate_config_rule_and_wait_for_resource method always runs start_config_rules_evaluation which triggers a new evaluation of the rule. This is not generally needed as the creation of a resource will trigger a rule evaluation for that resource. Also, if one test has multiple calls to this method to test the results for multiple resource, it means the rule is potentially being triggered multiple times.

Can we change this method (and probably its name) to not trigger the rule by default?

Conventions/layout spike

See if there are any naming conventions that might make sense to reduce the number of explicit parameters necessary for the decorator. e.g. template X in directory layout will get loaded for test method name X.

Add support for aws profile to obtain AWS credentials

When potemkin decorator is specified on a pytest method
And the aws_profile argument is set to a name
Then the Cloudformation template is converged in AWS with credentials of that profile
And the stack is created with the stem name concatenated with an epoch timestamp suffix

// ambient creds if not set

The repository is private

The pypi homepage links to it... but you can't view it. Instead, you get a 404 page not found. How do I know you're not a haxor if I can't see the source?

Wait for deleted resource_ids absent from config

In order to fully test a config rule, we also need to be able to verify that resources that are delete are removed from config. Add a wait_for that returns when all resources are removed from config or timeout.

Better way to handle multiple resource checks

Creating a Cloudformation template that creates multiple resources to check all COMPLIANT/NON_COMPLIANT use cases requires multiple uses of config_rule_wait_for_resource

Since this is a common use case, create a method for checking multiple resources.

Cloudformation timeout hard coded to 5 minutes

The hard coded timeout of 5 minutes is not long enough for things like RDS. Make the cloudformation timeout configurable. The Waiter max tries also needs to be adjusted by the timeout setting

Handle Config NOT_APPLICABLE resources

Update config_rule_wait_for_compliance_results to be able to handle NA resources.

Wait until all other resources are found, then verify that the NA resources do not show up,

Teardown after failure determined by a flag.

If one of the assertions in a decorated test fails, the teardown functionality does not happen. This is reasonable for local use, but if potemkin-decorator is used in an automated pipeline, the teardown should always happen to prevent excessive resources from being left over.

Per discussion with Eric, we should add an flag for this with the default being to delete the stack if there is a failure in the pytest.

Note: if the stack creation fails, the failed stack should not be deleted.

Better output for stack failure

My integration test cloudformation template had a S3 bucket name that already existed. This caused the stack to fail. But the pytest output did not clearly state that the stack failed. Is there any way to make that more clear?

============================= test session starts ==============================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/wel0991/repos/aws-admins/custom-config-rules/s3-dr-encryption-at-rest-non-compliant
plugins: cov-2.8.1
collected 1 item

tests/integration/test_integration.py F                                  [100%]

=================================== FAILURES ===================================
________________________________ test_compliant ________________________________

    def decorated_test_function():
        with open(self._resolve_template_path(), 'r') as initial_condition_cfn_template_file:
            initial_condition_cfn_template_content = initial_condition_cfn_template_file.read()
    
        qualified_stack_name = self._unique_stack_name(self._stack_name)
    
        stack_outputs = self._create_stack(
            stack_name=qualified_stack_name,
            parameters=self._parameters,
>           template_body=initial_condition_cfn_template_content
        )

../../../../.local/share/virtualenvs/s3-dr-encryption-at-rest-non-compliant-PEQNnTXA/lib/python3.7/site-packages/potemkin/cloudformationstack.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../.local/share/virtualenvs/s3-dr-encryption-at-rest-non-compliant-PEQNnTXA/lib/python3.7/site-packages/potemkin/cloudformationstack.py:132: in _create_stack
    WaiterConfig=self._waiter_config()
../../../../.local/share/virtualenvs/s3-dr-encryption-at-rest-non-compliant-PEQNnTXA/lib/python3.7/site-packages/botocore/waiter.py:53: in wait
    Waiter.wait(self, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.waiter.CloudFormation.Waiter.StackCreateComplete object at 0x1104f9490>
kwargs = {'StackName': 'TestStack1585349054'}
acceptors = [<botocore.waiter.AcceptorConfig object at 0x1104f93d0>, <botocore.waiter.AcceptorConfig object at 0x1104f9550>, <boto...>, <botocore.waiter.AcceptorConfig object at 0x1104f9850>, <botocore.waiter.AcceptorConfig object at 0x1104f9950>, ...]
current_state = 'failure', config = {'Delay': 17, 'MaxAttempts': 15}
sleep_amount = 17, max_attempts = 15, num_attempts = 2
response = {'ResponseMetadata': {'HTTPHeaders': {'content-length': '1730', 'content-type': 'text/xml', 'date': 'Fri, 27 Mar 2020 ..., 16, 259000, tzinfo=tzutc()), 'DisableRollback': True, 'DriftInformation': {'StackDriftStatus': 'NOT_CHECKED'}, ...}]}

    def wait(self, **kwargs):
        acceptors = list(self.config.acceptors)
        current_state = 'waiting'
        # pop the invocation specific config
        config = kwargs.pop('WaiterConfig', {})
        sleep_amount = config.get('Delay', self.config.delay)
        max_attempts = config.get('MaxAttempts', self.config.max_attempts)
        num_attempts = 0
    
        while True:
            response = self._operation_method(**kwargs)
            num_attempts += 1
            for acceptor in acceptors:
                if acceptor.matcher_func(response):
                    current_state = acceptor.state
                    break
            else:
                # If none of the acceptors matched, we should
                # transition to the failure state if an error
                # response was received.
                if 'Error' in response:
                    # Transition to a failure state, which we
                    # can just handle here by raising an exception.
                    raise WaiterError(
                        name=self.name,
                        reason=response['Error'].get('Message', 'Unknown'),
                        last_response=response
                    )
            if current_state == 'success':
                logger.debug("Waiting complete, waiter matched the "
                             "success state.")
                return
            if current_state == 'failure':
                raise WaiterError(
                    name=self.name,
                    reason='Waiter encountered a terminal failure state',
>                   last_response=response,
                )
E               botocore.exceptions.WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state

../../../../.local/share/virtualenvs/s3-dr-encryption-at-rest-non-compliant-PEQNnTXA/lib/python3.7/site-packages/botocore/waiter.py:323: WaiterError
- generated xml file: /var/folders/qm/llkyl47s27l9nx2f5y7cn00h0000gn/T/tmp-3610L6vP3Ctvi676.xml -
========================== 1 failed in 21.39 seconds ===========================
python /Users/wel0991/.vscode/extensions/ms-python.python-2020.3.69010/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/wel0991/repos/aws-admins/custom-config-rules/s3-dr-encryption-at-rest-non-compliant -s --cache-clear tests

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.