Git Product home page Git Product logo

terraform-aws-shepherd's Introduction

App Shepherd Global

This module is used to configure AWS resources to work with the Shepherd project.

ETL Pipeline

etl-pipeline

Usage

Creates metric alarms for use with a Lambda Function

  • Success rate
module "shepherd" {
  source = "dod-iac/shepherd/aws"

  subscriber_buckets = [
    "bucket1",
    "bucket2",
  ]

  shepherd_users = [
    "iam_user1",
    "iam_user2",
  ]

  tags = {
    Project     = var.project
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

Terraform Version

Terraform 0.13. Pin module version to ~> 1.0.0 . Submit pull-requests to master branch.

Terraform 0.11 and 0.12 are not supported.

License

This project constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. However, because the project utilizes code licensed from contributors and other third parties, it therefore is licensed under the MIT License. See LICENSE file for more information.

Manual Operations Log

Athena Workgroups

For the Athena Workgroups it is required that the options "Queries with requester pays buckets" is set to "Enabled". Ensure that both the Athena primary workgroup and the Shepherd workgroups have this enabled. This will have to be done manually for any new workgroups added.

Create the Glue Tables

Each database needs a table with the data. There is a saved query in each workgroup for creating the table. After switching workgroups, and while checking the correct DB is selected, run the create-table query. This needs to be done for each database, remembering to switch workgroups each time. Confirm that the tables exist by looking in AWS Glue or in AWS Athena by selecting the appropriate database.

AWS IAM Roles

There are two roles that must be passed to the vendor and appear as outputs:

  • shepherd_glue_role_arn: The role used by AWS Glue to do ETL on the data
  • shepherd_users_role_arn: The role used by IAM users to work with the resources configured by this module

AWS SSM Parameters

Some data needs to be placed in AWS SSM Parameter store. They are:

  • salt: A random 32 character string used as a salt for hashing algorithms

To write a variable use the chamber tool:

SALT=$(tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 32 | head -n 1)
chamber write shepherd-global salt "${SALT}"

Requirements

No requirements.

Providers

Name Version
aws n/a
template n/a

Modules

Name Source Version
athena_results trussworks/s3-private-bucket/aws ~> 3.2.1
aws_logs trussworks/logs/aws ~> 10.0.0
glue_tmp_bucket trussworks/s3-private-bucket/aws ~> 3.2.1

Resources

Name Type
aws_athena_named_query.alter_table resource
aws_athena_named_query.create_proxy_table resource
aws_athena_named_query.create_table resource
aws_athena_named_query.create_view resource
aws_athena_named_query.create_view_proxy resource
aws_athena_named_query.repair_table resource
aws_athena_workgroup.shepherd resource
aws_glue_catalog_database.shepherd resource
aws_glue_job.create_csv resource
aws_glue_job.shepherd resource
aws_glue_job.shepherd_proxy resource
aws_glue_security_configuration.event_data resource
aws_glue_trigger.start_workflow_create_csv resource
aws_glue_trigger.start_workflow_loadpartitions resource
aws_glue_trigger.start_workflow_loadpartitions_proxy resource
aws_iam_group.shepherd_engineers resource
aws_iam_group.shepherd_redshift resource
aws_iam_group.shepherd_users resource
aws_iam_group_membership.shepherd_engineers resource
aws_iam_group_membership.shepherd_users resource
aws_iam_group_policy_attachment.shepherd_engineers_assume_role_policy_attachment resource
aws_iam_group_policy_attachment.shepherd_redshift_assume_role_policy_attachment resource
aws_iam_group_policy_attachment.shepherd_users_assume_role_policy_attachment resource
aws_iam_policy.assume_role_shepherd_engineers_policy resource
aws_iam_policy.assume_role_shepherd_redshift_policy resource
aws_iam_policy.assume_role_shepherd_users_policy resource
aws_iam_policy.glue_policy resource
aws_iam_policy.quicksight_service resource
aws_iam_policy.shepherd_athena_primarywg resource
aws_iam_policy.shepherd_engineers resource
aws_iam_policy.shepherd_redshift_athena resource
aws_iam_policy.shepherd_redshift_glue resource
aws_iam_policy.shepherd_redshift_kms resource
aws_iam_policy.shepherd_redshift_s3 resource
aws_iam_policy.shepherd_users_athena resource
aws_iam_policy.shepherd_users_glue resource
aws_iam_policy.shepherd_users_other resource
aws_iam_policy.shepherd_users_s3 resource
aws_iam_role.glue_role resource
aws_iam_role.quicksight_service resource
aws_iam_role.shepherd_engineers resource
aws_iam_role.shepherd_redshift resource
aws_iam_role.shepherd_users resource
aws_iam_role_policy_attachment.glue_policy_att resource
aws_iam_role_policy_attachment.glue_role_att resource
aws_iam_role_policy_attachment.quicksight_athena_policy_att resource
aws_iam_role_policy_attachment.quicksight_service_policy_attachment resource
aws_iam_role_policy_attachment.shepherd_engineers_policy_attachment resource
aws_iam_role_policy_attachment.shepherd_engineers_policy_attachment_primarywg resource
aws_iam_role_policy_attachment.shepherd_redshift resource
aws_iam_role_policy_attachment.shepherd_redshift_policy_attachment_athena resource
aws_iam_role_policy_attachment.shepherd_redshift_policy_attachment_glue resource
aws_iam_role_policy_attachment.shepherd_redshift_policy_attachment_kms resource
aws_iam_role_policy_attachment.shepherd_redshift_policy_attachment_s3 resource
aws_iam_role_policy_attachment.shepherd_users_policy_attachment_athena resource
aws_iam_role_policy_attachment.shepherd_users_policy_attachment_glue resource
aws_iam_role_policy_attachment.shepherd_users_policy_attachment_other resource
aws_iam_role_policy_attachment.shepherd_users_policy_attachment_primarywg resource
aws_iam_role_policy_attachment.shepherd_users_policy_attachment_s3 resource
aws_s3_bucket.csv_results resource
aws_s3_bucket_object.create_csv resource
aws_s3_bucket_object.index-html resource
aws_s3_bucket_object.loadpartition resource
aws_s3_bucket_object.not-found-html resource
aws_caller_identity.current data source
aws_iam_account_alias.current data source
aws_iam_policy_document.assume_redshift_role_policy data source
aws_iam_policy_document.assume_role_policy data source
aws_iam_policy_document.assume_role_shepherd_engineers_policy_doc data source
aws_iam_policy_document.assume_role_shepherd_redshift_policy_doc data source
aws_iam_policy_document.assume_role_shepherd_users_policy_doc data source
aws_iam_policy_document.csv_results_policy data source
aws_iam_policy_document.glue_assume_role_policy_document data source
aws_iam_policy_document.glue_policy_document data source
aws_iam_policy_document.quicksight_assume_role_policy data source
aws_iam_policy_document.quicksight_service data source
aws_iam_policy_document.shepherd_athena_primarywg data source
aws_iam_policy_document.shepherd_engineers data source
aws_iam_policy_document.shepherd_redshift_athena data source
aws_iam_policy_document.shepherd_redshift_glue data source
aws_iam_policy_document.shepherd_redshift_kms data source
aws_iam_policy_document.shepherd_redshift_s3 data source
aws_iam_policy_document.shepherd_users_athena data source
aws_iam_policy_document.shepherd_users_glue data source
aws_iam_policy_document.shepherd_users_other data source
aws_iam_policy_document.shepherd_users_s3 data source
aws_iam_user.shepherd_engineers data source
aws_iam_user.shepherd_users data source
aws_partition.current data source
aws_region.current data source
aws_ssm_parameter.salt data source
template_file.alter_table data source
template_file.create_proxy_table data source
template_file.create_table data source

Inputs

Name Description Type Default Required
application n/a string "shepherd" no
csv_bucket_allowed_ip_blocks List of CIDR blocks allowed to access the CSV bucket list(string)
[
"0.0.0.0/0"
]
no
csv_bucket_name The name of the S3 bucket hosting the publicly accessible CSV files. The name must be a valid DNS name. Best practice is to use a unique hash in the name, ie UNIQUEHASH.example.com string "" no
csv_jobs Details for each CSV job. See comments in code for details list(map(string)) [] no
environment n/a string "global" no
project n/a string "shepherd" no
region n/a string "us-gov-west-1" no
shepherd_engineers The set of IAM user names to add to the 'shepherd_engineers' group list(string) [] no
shepherd_users The set of IAM user names to add to the 'shepherd_users' group list(string) [] no
subscriber_buckets The set of AWS S3 buckets to subscribe too list(string) [] no
tags The tags for the project map(string) {} no

Outputs

Name Description
csv_results_bucket The CSV results bucket name
csv_website_domain The CSV domain of the website endpoint, if the bucket is configured with a website. This is used to create Route 53 alias records.
csv_website_endpoint The CSV website endpoint, if the bucket is configured with a website.
shepherd_glue_role_arn shepherd glue role arn
shepherd_users_role_arn shepherd-users role arn

terraform-aws-shepherd's People

Contributors

johnny-dds avatar chrisgilmerproj avatar kdibrown avatar rsivilli avatar austinlasseter avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.