Git Product home page Git Product logo

collie-hub's People

Contributors

felixzieger avatar florianow avatar github-actions[bot] avatar ishabakeh avatar j0g3sc avatar jelledb avatar johannesrudolph avatar malhussan avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

bogdandragosvasile infrazone muhammadumairsabir marre69

collie-hub's Issues

Buildingblocks with collie-hub and collie-cli

This issue collects various improvements for a simple and consistent workflow to deal with assembling tenants from buildingblocks using a pure IaC workflow using collie. Common examples of building blocks needed in a stage 2 (CFMM) cloudfoundation are a tenant building block (think subscription/account/project) that enforces tags + IAM role model, a budget alert (simple) and a hub+spoke vnet (complex).

The goals for the approach we want to support with collie are

  • plain and simple IaC composition using terraform: no meta-templating or file generation
  • isolation: building blocks should have well-defined inputs and outputs that enables them to be deployed and operated by teams outside the cloud foundation (this is common e.g. for DevOps toolchains and advanced network capabilities). This is also good practice for security, since automation will typically need to perform high privileged operations
  • automation-friendly: provide a clear ugprade path for plugging building blocks into a GitOps workflow or meshStack for automation

Concept: Building Blocks

In terragrunt parlance, building blocks are "shared service modules"

A Terraform module that is designed to be standalone and applied directly. These modules are not root modules in that they are still missing the key blocks like backend and provider, but aside from that do not need any additional configuration or composition to deploy

Concept: Backplanes

Backplanes add what's necessary to successfully deploy a building block to a tenant in your landing zones. From the perspective of an application team, a building block provides some sort of capability to their application's cloud environment, e.g. an on-prem connected spoke VNet. This spoke VNet however is only the "tip of the iceberg". Application teams can't (and should not have to care about) all of the mechanics that make this spoke VNet work, like the hub network, WAN connection, IPAM and firewall rules.

We will call everything that makes the application-team visible part of the building block work the "backplane". The backplane always includes

  • terraform backend (state storage)
  • terraform provider configuration (authentication and authorization)
  • any infrastructure shared between different building blocks (e.g. the hub network)

Packaging Building Blocks in collie-hub

We will settle for the following structure for packaging building blocks in collie-hub

  • kit/$platform/buildingblocks/$block/{README.md, main.tf, ...} building block backplanes will be normal kit modules

  • kit/$platform/buildingblocks/$block/buildingblock/{README.md, main.tf, ...} building block modules will be plain terraform modules

    • we might want to let building blocks come with their own metadata (e.g. an icon.svg) that can be used for self-service UIs and collie documentation
    • by convention, we let buildingblock backplanes emit a sensitive config_tf output. This output should contain provider and (optionally?) backend configuration block. This is the "missing" configuration required to deploy a building block and can be injected e.g. by terragrunt or other terraform automation

  •  collie-cli must learn to ignore **/buildingblock/README.md files when trying to parse configuration objects from the repo or learn to treat buildingblocks as its own concept

  • some BBs will have a very advanced backplane, e.g. the connectivity block requires the hub deployed from azure/kit/connectivity - should we go the trouble of always defining a dedicated backplane even if we could reuse the bootstraped SPN?

Deploying Building Blocks

Deploying building blocks to a cloud tenant becomes a simple terraform composition in a main.tf that invokes the building blocks as plain terraform modules like

module "subscription" {
  source = "github.com/likvid-bank/likvid-cloudfoundation/kit/azure/buildingblocks/subscription/buildingblock"
  subscription_name       = "glaskugel"
  parent_management_group = "likvid-corp"
}

module "connectivity" {
  source = "github.com/likvid-bank/likvid-cloudfoundation/kit/azure/buildingblocks/connectivity/buildingblock"

  providers = {
    azurerm.spoke = azurerm
    azurerm.hub   = azurerm.hub
  }

  location = "germanywestcentral"
  hub_rg   = "hub-vnet-rg"
  hub_vnet = "hub-vnet"

  name          = "glaskugel"
  address_space = ["10.1.0.0/24"]
}
  • as shown here, a tenant may need multiple building blocks that each come with their own provider/backend config. Providers can be aliased to disambiguate, but there must only be one backend per tenant... this is at odds with simple tenant composition via module calls in a main.tf

Testing Building Blocks

Once we have deployed a building block backplane, it's very useful to ensure that this backplane can successfully deploy the building block in isolation. This can be achieved with terraform test and terragrunt like so

dependency "buildingblock" {
  config_path = "../budget-alert"
}

dependency "glaskugel" {
  config_path = "../../tenants/glaskugel"
}

generate "config" {
  path      = "config.tf"
  if_exists = "overwrite"
  contents  = dependency.buildingblock.outputs.config_tf
}

terraform {
  source = "${get_repo_root()}//kit/azure/buildingblocks/budget-alert/buildingblock"
}

inputs = {
  subscription_id = dependency.glaskugel.outputs.subscription_id
  contact_emails = "[email protected], [email protected]"
}

open issues

  • it's using config.tf which works very well here, but is not similar to collie-style tenant composition (see above)
  •  we can't test backplane's backend configurations that way

Note: some of what's discussed in this issue here (especially the design principles) should end up on the collie documentation

Azure changed Permissions

Perhaps it would be more sensible to explore the available built-in roles instead of maintaining custom roles for the CloudFoundation group. The organization hierarchy cannot deploy definition sets due to the lack of a specific role for that purpose. Azure is updating roles more rapidly than we can keep up with through manual maintenance.

Azure Bootstrap module uses deprecated azuread_service_principal attributes

The azuread terraform provider is planing to deprecate some attributes, see hashicorp/terraform-provider-azuread#1214

Together with the lax version constraint we place on the provider this means new collie users see these deprecation warnings.

╷
│ Warning: Deprecated attribute
│ 
│   on outputs.tf line 2, in output "client_id":
│    2:   value = azuread_service_principal.cloudfoundation_deploy.application_id
│ 
│ The attribute "application_id" is deprecated. Refer to the provider
│ documentation for details.
│ 
│ (and 3 more similar warnings elsewhere)
╵
╷
│ Warning: Argument is deprecated
│ 
│   with data.azuread_service_principal.msgraph,
│   on resources.deploy-spn.tf line 50, in data "azuread_service_principal" "msgraph":
│   50:   application_id = data.azuread_application_published_app_ids.well_known.result.MicrosoftGraph
│ 
│ The `application_id` property has been replaced with the `client_id`
│ property and will be removed in version 3.0 of the AzureAD provider
│ 
│ (and one more similar warning elsewhere)
╵

New module: publish to GitHub pages

A new foundation level module could help us include a publishing workflow for "collie foundation docs" to deploy the generated vuepress page on GitHub pages.

This makes a lot of sense after
#109

At meshcloud we are already using this internally to publish documentation for our collie managed landing zones

azure/bootstrap: SPN-less deployment, but readonly SPN automation and deployment workflows

The current azure bootstrap module implements a very simple deployment workflow by delegating all permissions to an cloudfoundation_tf_deploy SPN and setting that up as the authentication for collie's platform modules (via terragrunt).

This means controlling access to the terraform state (which contains the secret for the SPN) = controlling access to who can deploy cloud foundation. This is currently done via an azure ad group (platform_engineers group). Bootstrapping requires an "out of band" admin user.

Where this solution falls short is the following

  • we regularly need to perform bootstrap in order to rotate SPN credential
  • it's difficult to give "read only" access to an automation pipeline (in order to generate terraform plans, but not have a CI/CD system wield power to actually deploy changes see #102 )

On GCP we can use ServiceAccount impersonation for this as it lets us get rid of the credential, but Azure does not have this.

foundation/docs mermaid plugin broken on windows 

PS C:\Users\Johannes\test\0922> collie foundation docs --preview
Tip: To set a default foundation run
        $ collie config set-foundation <foundation>"
 ? Select a foundation » f
generating docs foundation "f" ...
parsing kit modules ...
parsing kit modules DONE 70ms
parsing compliance controls ...
parsing compliance controls DONE 10ms
generating kit module documentation ...
could not find compliance control cfmm/security-and-compliance/shared-responsibility-model-alignment referenced in a compliance statement in kit\foundation\docs\README.md
generating kit module documentation DONE 20ms
generate documentation foundations\f ...
parsing platform modules foundations\f\platforms\az ...
parsing platform modules foundations\f\platforms\az DONE 22ms
generate documentation foundations\f\platforms\az ...
generate documentation foundations\f\platforms\az DONE 12810ms
generate documentation foundations\f DONE 12848ms
generating docs foundation "f" DONE 13174ms

up to date, audited 341 packages in 13s

42 packages are looking for funding
  run `npm fund` for details

16 vulnerabilities (5 moderate, 11 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues possible (including breaking changes), run:
  npm audit fix --force

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.

> [email protected] docs:dev
> vuepress dev docs

warning plugin @vuepress/plugin-git has been used multiple times, only the last one will take effect
info Initializing VuePress and preparing data...
X [ERROR] [plugin vite:dep-scan] The argument 'path' must be a string, Uint8Array, or URL without null bytes. Received 'C:\\Users\\Johannes\\test\\0922\\foundations\\f\\.docs\\docs\\.vuepress\\.temp\\internal\\node_modules\\C:UsersJohannes\test\x0...

    node_modules/resolve/lib/sync.js:31:22:
      31 │         var stat = fs.statSync(dir, { throwIfNoEntry: false });
         ╵                       ^

    at Object.statSync (node:fs:1706:10)
    at isDirectory (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:31:23)
    at loadNodeModulesSync (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:200:17)
    at Function.resolveSync [as sync] (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:107:17)   
    at resolveFrom (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:2077:29)   
    at resolvePackageData (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:28954:19)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29335:15
    at Array.find (<anonymous>)
    at tryNodeResolve (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29334:44)
    at Context.resolveId (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29166:28)

  This error came from the "onResolve" callback registered here:

    node_modules/vite/dist/node/chunks/dep-9c153816.js:38656:18:
      38656 │             build.onResolve({
            ╵                   ~~~~~~~~~

    at setup (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:38656:19)        
    at handlePlugins (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:855:23)
    at Object.buildOrServe (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1149:7)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:2110:17
    at new Promise (<anonymous>)
    at Object.build (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:2109:14)
    at Object.build (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1956:51)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:38432:54
    at Array.map (<anonymous>)

  The plugin "vite:dep-scan" was triggered by this import

    docs/.vuepress/.temp/internal/clientAppSetups.js:4:28:
      4 │ ...lientAppSetup3 from 'C:\Users\Johannes\test\0922\foundations\f\.docs\docs\.vuepress\theme\plugins\mermaid\client\...
        ╵                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~    

X [ERROR] [plugin vite:dep-scan] The argument 'path' must be a string, Uint8Array, or URL without null bytes. Received 'C:\\Users\\Johannes\\test\\0922\\foundations\\f\\.docs\\docs\\.vuepress\\.temp\\internal\\node_modules\\C:UsersJohannes\test\x0...

    node_modules/resolve/lib/sync.js:31:22:
      31 │         var stat = fs.statSync(dir, { throwIfNoEntry: false });
         ╵                       ^

    at Object.statSync (node:fs:1706:10)
    at isDirectory (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:31:23)
    at loadNodeModulesSync (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:200:17)
    at Function.resolveSync [as sync] (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\resolve\lib\sync.js:107:17)   
    at resolveFrom (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:2077:29)   
    at resolvePackageData (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:28954:19)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29335:15
    at Array.find (<anonymous>)
    at tryNodeResolve (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29334:44)
    at Context.resolveId (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:29166:28)

  This error came from the "onResolve" callback registered here:

    node_modules/vite/dist/node/chunks/dep-9c153816.js:38656:18:
      38656 │             build.onResolve({
            ╵                   ~~~~~~~~~

    at setup (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:38656:19)        
    at handlePlugins (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:855:23)
    at Object.buildOrServe (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1149:7)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:2110:17
    at new Promise (<anonymous>)
    at Object.build (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:2109:14)
    at Object.build (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1956:51)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\vite\dist\node\chunks\dep-9c153816.js:38432:54
    at Array.map (<anonymous>)

  The plugin "vite:dep-scan" was triggered by this import

    docs/.vuepress/.temp/internal/clientAppEnhances.js:5:30:
      5 │ ...entAppEnhance4 from 'C:\Users\Johannes\test\0922\foundations\f\.docs\docs\.vuepress\theme\plugins\mermaid\client\...
        ╵                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Error: Build failed with 2 errors:
node_modules/resolve/lib/sync.js:31:22: ERROR: [plugin: vite:dep-scan] The argument 'path' must be a string, Uint8Array, or URL without null bytes. Received 'C:\\Users\\Johannes\\test\\0922\\foundations\\f\\.docs\\docs\\.vuepress\\.temp\\internal\\node_modules\\C:UsersJohannes\test\x0...
node_modules/resolve/lib/sync.js:31:22: ERROR: [plugin: vite:dep-scan] The argument 'path' must be a string, Uint8Array, or URL without null bytes. Received 'C:\\Users\\Johannes\\test\\0922\\foundations\\f\\.docs\\docs\\.vuepress\\.temp\\internal\\node_modules\\C:UsersJohannes\test\x0...
    at failureErrorWithLog (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1624:15)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1266:28
    at runOnEndCallbacks (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1046:63)
    at buildResponseToResult (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1264:7)
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:1377:14
    at C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:678:9
    at handleIncomingPacket (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:775:9)
    at Socket.readFromStdout (C:\Users\Johannes\test\0922\foundations\f\.docs\node_modules\esbuild\lib\main.js:644:7)
    at Socket.emit (node:events:514:28)
    at addChunk (node:internal/streams/readable:343:12)
Unexpected error executing "(cd foundations\f\.docs && npm run docs:dev)"
Command exited with code 1
Tip: run collie with --verbose and --debug flags for more details

hub and spoke example missing logs

the spoke vnet has no logs similar to those set up in the networking kit. Diagnostic setting could also be switched on. You should also check what else can be activated, e.g. for peering

Docs pages publishing instructions or module

We should include instructions somewhere on how to best automate documentation publishing pipelines.

likvid-bank/likvid-cloudfoundation#13 for example added a workflow using terraform state storage on Azure and building on GitHub actions. This is probably not easy to deliver as a standard module because

  • has two axis of variation, a) which state storage system you use and b) which CI/CD platform is used
  • needs changes to quite a few different places in a collie repository to work (platform.hcl, bootstrap modules, adding a workflow file)

This is why this probably best delivered as a how-to guide in the docs

Azure Bootstrap, enable storage bucket versioning for tf states by default

It's a good practice to enable that so we should do it by default. There's then also the question of retention etc. coming up which is more complex, not sure I'd want to force that config decision on new users. Nonetheless just keeping all versions should be a reasonable default, esp. if users later run into a situation where they need to fix an issue with state.

colliedev pipeline integration

It would be really cool if collie missed a deployment pipeline. The kits are becoming more and more dependent on each other so that a deployment affects a kit that has not been touched.

I created a management group on the Azure side in which I work. Maybe we could use the likvid-demo account as a prod.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.