Right now, we have a skeleton for inventory email notification. For improvements:

(1) We should add more data to this, such as listing each resources that is queued for loading, and what is the final status (success, failed, not loaded, etc).

(2) Use email template.

A design and some thought are required here.

As reported by ahoying:

The pipelines ran by the inventory loader should be configurable and loadable as plugins, which should be self contained with their database schema, loaders, fetches, transformers, etc all bundled along with dependencies.

Then the inventory_runner can get the list of enabled pipelines, figure out the order to process based on dependencies, raise exceptions if dependencies are missing, optionally launch different pipelines in parallel if they are not dependent on each other, etc.

Which would help migrate to dataflow (if we decide that's a good idea) down the road.

In addition to our snapshot batch mode of running inventory and scanner, we should read from audit logs or something that has more realtime visibility into what's happening on GCP and perform necessary Forseti audits/scans.

e.g., A firewall rule is changed by an account that isn't used by the enforcer and/or whitelisted

• Configurable to send to admin and/or project owner
• A configurable html/email that can be customized to configure what's emailed.

Currently, enforcer only outputs its results to console. It would be nice if the output can be emailed as a notification.

• Currently there is no API to expand membership

Due to timing constraints in creating the replica the database can't be created, despite having a finished response from the instance API saying the task was complete.

As a workaround, re-executing the deployment manager script after the replica is created will succeed.

This will be resolved in a future version of the API.

• Currently enforcer code is the chief offender

Because it looks a little unhappy.

Address the problem if the json file contains a single rule, either raise an exception early or properly format it as a single item list transparently.

• Remove protoc duplication between deployment scripts and travis requirements

Create a new docs/ directory in the root of the repo.

Suggested flow of the documentation (this is just a skeleton):

README.md - intro, table of contents

prerequisites.md

• Create GCP project
• Set up billing
• Install toolchain/third party software for setting up forseti-security
• Run the setup.py installation
• Link to "What's next"
  • Deploy to GCP
  • Run locally

deployment_manager.md - use the current instructions from the deployment-templates/README.md, minus the prereq steps

dev_setup.md - Links to appropriate setup instructions for each module

For each functional module (i.e. probably don't have to do this for common modules) in Forseti Security:

• In google/cloud/security/ reuse the README.md for additional setup steps but do not include the prereq steps
• Show usage of the CLI
• If there is additional documentation related to configuration (e.g. writing rules, policies, etc.), link to them where needed.

• Configurable to send to admin and/or project owner
• A configurable html/email that can be customized to include "rules of the road" for new projects.

Data Studio can prepare some very cool looking reports & charts, and it can also use Cloud SQL as a datasource. So, we should see if we can use it to visualize things like how much inventory has been collected, how many violations have been found, and how many things have been enforced.

Also see if the basic reports can be given to users.

Would be cool to have some screenshots or sample outputs (e.g. CSVs) without installing and seeing what it provides. My 2c.

• e.g., What role, member, project, and the required action

This applies to the config files in /config directory.

/config/db.yaml.sample
/config/inventory.yaml.sample

Currently, they are stored and read from local disk. But we run the risk of losing them on re-deployment. So to prevent this and with the benefit of easier deployment, we should allow an option to store and read them from Google Cloud Storage.

This should include

• Firewall rules
• Network configuration(s)

For instance in g/c/f/services/scanner/dao.py

https://github.com/GoogleCloudPlatform/forseti-security/blob/2.0-dev/google/cloud/forseti/services/scanner/dao.py#L138

https://github.com/GoogleCloudPlatform/forseti-security/blob/2.0-dev/google/cloud/forseti/services/scanner/dao.py#L154

• What keys are generated, exported
• Who has access to the account
• IAM permissions

Also integrate with Travis

Story

Monitor for anything that someone should respond when the basic health of the service changes. Such as if the VM host goes down, an alert should be sent. Customers need to know if Forseti is running/healthy which is difficult to determine today.

Possible Solution(s)

Few more things to consider:

• Is Forseti healthy (is the Forseti process running, is Forseti meeting its SLO?). This can be done with Stackdriver process health; I'm not sure if this is configurable via Terraform. Terraform supports an uptime check with http calls.
• Is the Forseti VM healthy (is the VM running, is the CPU, memory, disk used up?). This can be done with Stackdriver metrics monitoring; I'm not sure if this is configurable via Terraform.
• Is the CloudSQL DB VM healthy (ditto as above).
• Notify on Forseti errors in the logs, which is available via Terraform. Is this the same as turning on notifications for new errors?

Consider adding an uptime check and alerting when errors occur.