spinnaker / swabbie Goto Github PK

Spinnaker's clean up service

Kotlin 99.86% Slim 0.14%

swabbie's Introduction

Swabbie

IMPORTANT: This service is currently under development, and is actively being used at Netflix for deleting images, ebs snapshots and auto scaling groups.

Swabbie automates the cleanup of unused resources such as EBS Volumes and Images. As a Janitor Monkey replacement, it can also be extended to clean up a variety of resource types. Swabbie applies a set of rules to mark cleanup candidates. Once marked, a resource is scheduled for deletion, and an owner is notified. Before being deleted the resource is checked again to make sure it still qualifies for deletion. If so, it is deleted.

Deep Dive

For a more detailed understanding of how Swabbie works, visit the internals doc.

How it works

During initialization swabbie schedules work to routinely mark, notify and delete resources. The application configuration is flattened into work items that are placed on the work queue for processing:

YAML config -> Work Items -> Work Queue

Each visited resource is evaluated against the rules engine in order to determine if it should be deleted.

Rules in the rules engine are configurable and can be composed similar to an if/else branch. They can be defined with an AND (&&), or OR (||) operator:

AND: A branch applies if all contained rules apply to the resource being evaluated.
OR: A branch applies if any rule contained inside the branch applies.

resourceTypes:
- name: serverGroup
  enabled: true
  enabledRules:
  - operator: AND # branch(1)
    description: Empty Server Groups that have been disabled for more than than 45 days.
    rules:
      - name: ZeroInstanceRule
      - name: DisabledLoadBalancerRule
        parameters:
          moreThanDays: 45
  - operator: OR #branch(2)
    description: Expired Server Groups.
    rules:
      - name: ExpiredResourceRule

The above configuration translates to the following:

For every resource r of type serverGroup,

r.marked == true => (branch(1) || branch(2)) == true

Or more generally:

r.marked == true => (branch(1) || branch(2) || ... branch(n-1) || branch(n)) == true

As illustrated using defined rules:

if (((ZeroInstanceRule && DisabledLoadBalancerRule) || ExpiredResourceRule) == true)

Resource States:

Marked:

During the marking process, previously marked resources that no longer qualify for deletion are Unmarked.

Notified: Once marked, the resource owner is resolved and notified about the upcoming deletion.
Opted-Out: A resource can be explicitly opted out of deletion via API or exclusion policies. Opted out resources are exempt from swabbie actions.
Deleted:

Resources are re-evaluated before deletion to ensure they can be safely deleted.

What's supported today

Cloud Provider: AWS
- Netflix uses Edda
- Vanilla AWS is also supported
Resource Types:
- AMIs
- Server Groups
- Launch Configurations
- Launch Templates
- EBS Snapshots
- ELBs
Halyard: Not supported yet (PRs are welcome!)

Contributing

If you're interested in contributing support for other providers or resource types, open an issue or join the Spinnaker Team slack and post in #swabbie.

Areas:

Testing
Documentation
Other cloud provider
Extensibility
Plugin support

Running swabbie

Requirements:

Redis for storage
Copy config file to a directory of your choice Configuration (maps to SwabbieProperties)
Update config with your settings and run

./gradlew run -Dspring.config.location=/absolute/path/to/swabbie.yml

swabbie's People

Contributors

Stargazers

Watchers

swabbie's Issues

refactor(configuration): Move accounts & locations definitions to resourceType level

Current:

providers:
    - name: aws
      locations:
        - us-east-1
        - us-west-2
      accounts:
        - test
        - prod

Future:

resourceTypes:
  - name: image
     enabled: false
     .
     .
     accounts:
       - test
       - prod
      locations:
       - us-east-1
       - us-west-2

This should allow more fine grained control on how to scope swabbie actions for a resource type. Instead of relying on exclusions on accounts.

Will it be possible to run swabbie on GCP or other cloud providers?

Or will there be at least an abstraction layer so that it is easier to implement this later?

project no longer maintained?

Is this project no longer maintained? I noticed that there hasn't been a release in over a year

CC @jeyrschabu @aravindmd

Fix swabbie email template termination date

Echo template: <td align="left" style="padding: 4px 0; font-family: Helvetica, Arial, sans-serif; font-size: 12px;"> ${resourceData.resource.projectedDeletionStamp?number_to_date?string("EEE, d MMM yyyy")} </td>

Done When:

Swabbie formats the deletion date in human readable format
Update echo to read the value without any treatment to the format

Can we have a getting started documentation?

It will be nice to have a getting started documentation on steps of how to install and configure swabbie.

Add link in notification email to report wrong marking of a resource

I thought your idea @emjburns was awesome. We include a link in the email saying: If you believe this was a mistake click here. that endpoint will hit orca and create a Jira ticket via the Jira stage I wrote a while back. We can have it preassigned to a manager or the oncall person. Probably manager to start with or you since swabbie is your baby now:) (cc: @asher since we did something similar with OC)

mark endpoint should be expandable

Let's add an expand query param. By default we will return just a list of ids. When expend= true we return the full body. I suspect that list will grow so we should think about paginating results

Add ower resolution strategy for images that is not the field in Tag

@emjburns per our chat, let's add an owner resolution strategy that is not tag based. We discussed using Krieger to get the application associated with an ami id. We can then derive the name and retrieve the application object from Front50ApplicationCache and pluck out the email field. As we add more of these strategies, we need a way to discriminate based on resource type or some attribute on WorkConfiguration.
This will ensure the correct users are getting the notifications rather than the person who created the ami

Fix testing controller notification endpoints

@aravindmd

OrphanedSnapshotRule does not work for vanilla aws

I current have swabbie successfully running against AMI and launch-configurations , but for snapshots, seems swabbie can not correctly process the OrphanedSnapshotRule

I saw in the code if looks for the info as follow :

const val IMAGE_EXISTS = "imageExists"

where can I go deeper to debug this ? also what is the reference of this "imageExists" ? seems not clear in the code that where swabbie got this

I am trying to cleanup some resources ex: if a microservice(inside a pod) is not getting much traffic ,i want to kill the pod after marking , notifying .

is it possible with swabbie?

thanks,
hari

Add time restrictions so swabbie runs between 9-3pm business days

in ScheduledAgent we would add additional check to the current discovery check that will decide whether or not to actuate. Preferably, something configurable.
Big bonus is to hook up DynamicConfigServicefor aswabbie.quiet.time=trueor swabbie.killSwitch=trueFast Props. The latter will halt currently running tasks and will require a manual kick via an endpoint. @robzienert may have some thoughts on this. Thinking along the line of the orcaQOS`