sredocs

sredocs is to turn postmortems, charters and other SRE docs into machine readable data for analysis while retaining human readability properties.

This repository contains code for downloading postmortems and/or charters following certain template(s) from Google Docs, parsing the data to CSV and support for uploading the result to Google BigQuery.

Here is a demo dashboard for incident analysis using this, Google Data Studio and BigQuery. It contains an interactive report with information related to five dummy postmortems.

sredocs currently parses postmortems using a single specific template (sample) and charter that will be published separately later.

Download

Here is how to use the download mode:

1. Go to https://console.developers.google.com. I'd recommend creating a new project called "sredocs".
2. Switch to that project. Search for "Google Drive API" then enable it.
3. Go to APIs & Services -> Credentials.
4. Click on "Create credentials" then select Service account key.
5. Give your Service account key a name, e.g "sredocs".
6. Take note of its address (looks like an email address). Do not assign it a role while creating it.
7. Save the private key (it will be automatically downloaded).
8. Switch to Google Drive.
9. Give your service account address, e.g [email protected]... view access to the folder with SRE docs.
10. Pass the private key to sredocs via -download_credentials_path.
11. Run mkdir <download_dir>
12. Run sredocs -mode=download -cloud_credentials=<creds.json> -download_folder=<docs_folder> -download_destination=<download_dir>

Parse

Here is how to use the parse mode:

1. Run mkdir <parsed_dir>
2. Run sredocs -mode=parse -parse_kind=auto -parse_path=<download_dir> -parse_output_path=<parsed_dir>

Upload

Here is how to use the upload mode:

1. Follow the instructions for Download first.
2. Go to https://console.developers.google.com.
3. Navigate to IAM & Admin.
4. Click on the Add button.
5. Search for your service account, e.g sredocs@... Give it the BigQuery Admin role.
6. Run sredocs -mode=upload -cloud_credentials=<creds.json> -upload_path=<parsed_dir> -upload_project=<org:project> -upload_dataset=<sredocs> -upload_table=<sredocs_20190603> -upload_truncate=true

BYOP

You can also bring your own parser.

type Parser interface {
        CompileRegex(fields []string) ([]*regexp.Regexp, error)
        Parse(fields []string, b []byte) (*bytes.Buffer, error)
        CSVHeader(regexps []*regexp.Regexp) []string
        NamedGroup(field string) string
        Save(buf *bytes.Buffer, filename string) error
}

You will have to modify charter/* and postmortem/* to switch away from the DefaultParser.

You can also extend sredocs by adding a new kind of document by using charter/* as a template.