The main idea behind Monocle is to detect anomalies in the way changes are produced in your project on GitHub and Gerrit.
Each team is unique in its way of working: how to do code reviews, how many reviewers, how to do CI, is self merge allowed...
So the philosophy behind Monocle is to let you visualize and explore metrics and data that are relevant to the way you work by navigating and filtering in the web user interface.
Browse changes metrics on the demo instance. Follow us on @change_metrics on Twitter.
For example, your team may want to know:
- what is the ratio of created changes vs merged changes?
- is there a good balance between change creations and change reviews?
- what is the ratio of abandoned changes?
- what are the collaboration patterns between the team members?
- how long does it take to merge a change?
- average delay for the first comment or review?
- long standing changes?
- do we have new contributors?
Here is a graph representing what has been going on during a period of time:
Here is the graph of collaboration patterns:
Here is the graph of the complexity versus time to merge changes:
Monocle supports GitHub Pull Requests and Gerrit Reviews. Monocle provides a set of crawlers designed to fetch Pull Requests and Reviews data from the GitHub or Gerrit APIs and to store changes and changes related events into Elasticsearch. These changes and events are exposed via a JSON API. Furthermore Monocle implements ready to use queries and a React based web UI.
To summarize we have the following components in Monocle:
- an Elasticsearch data store.
- an api service.
- a crawler service.
- a web UI.
Monocle is in an early phase of development. Feedback is highly welcome.
The process below describes how to index changes from a GitHub repository, a full GitHub organisation and Gerrit repositories, and then how to start the web UI to browse metrics using docker-compose
.
$ git clone https://github.com/morucci/monocle.git
$ cd monocle
$ mkdir data etc dump
$ ln -s docker-compose.yml.img docker-compose.yml
By default docker-compose will fetch the latest published container images. Indeed, we produce Docker container images for the master version of Monocle. If running master does not git your needs, you could still use the last release by setting the MONOCLE_VERSION to 0.7.0 in the .env file. Please refer to Configuration of the containers.
The config.yaml
file is used by the crawler and api services.
If you want to crawl GitHub repositories, generate a personal access token on GitHub (w/o any specific rights) at https://github.com/settings/tokens.
Then create the config file etc/config.yaml
. Here is an example your could start with. Make sure to replace <github_token>
by your personal access token:
---
tenants:
- index: monocle
crawler:
loop_delay: 10
github_orgs:
- name: tektoncd
repository: pipeline
updated_since: "2020-05-01"
token: <github_token>
base_url: https://github.com
To crawl the full tektoncd GitHub organization then remove the repository entry from the file. A more complete example is available in the section Full configuration file example.
Start Monocle:
$ docker-compose up -d
Ensure services are running:
$ docker-compose ps
monocle_api_1 uwsgi --uid guest --gid no ... Up 0.0.0.0:9876->9876/tcp, 0.0.0.0:9877->9877/tcp
monocle_crawler_1 monocle --elastic-conn ela ... Up
monocle_elastic_1 /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp
monocle_web_1 docker-entrypoint.sh /bin/ ... Up 0.0.0.0:3000->3000/tcp
You might need to check the crawler logs to ensure the crawler started to fetch changes:
$ docker-compose logs -f crawler
You should be able to access the web UI at http://localhost:3000.
After a change in the configuration file, the api and crawler services need to be restarted:
$ docker-compose restart api
$ docker-compose restart crawler
ElasticSearch could need some capabilities to run in container mode. Take a look at the logs to see if it started correctly:
$ docker-compose logs elastic
For example, you could need to increase this system parameter:
$ sudo sysctl -w vm.max_map_count=262144
or make the data directory writable for other:
$ chmod o+w data
You might want to wipe a Monocle index:
docker-compose run --no-deps crawler /usr/local/bin/monocle \
--elastic-conn elastic:9200 dbmanage --index <index-name> --delete-repository ".*"
or delete an index:
docker-compose run --no-deps crawler /usr/local/bin/monocle \
--elastic-conn elastic:9200 dbmanage --index <index-name> --delete-index
For a local deployment, default settings are fine.
The following settings are available in the .env
file:
MONOCLE_URL=<host or ip>
to configure the URL serving the Web UI (defaulthttp://localhost:3000
).MONOCLE_API_URL=<host or ip>
to configure the URL serving the API (defaulthttp://localhost:9876
).MONOCLE_VERSION=<version>
to use a specific version. By default it useslatest
.MONOCLE_TITLE=<title>
to change the title of the web application. By default it isMonocle
.ES_XMS and ES_XMX
to change the ElasticSearch JVM HEAP SIZE. By default 512m.MONOCLE_API_ADDR=<ip>
to change the IP address the API service is listening to (default0.0.0.0
).MONOCLE_WEB_ADDR=<ip>
to change the IP address the Web service is listening to (default0.0.0.0
).MONOCLE_ELASTIC_ADDR=<ip>
to change the IP address the ElasticSearch service is listening to (default0.0.0.0
). This is only exposed in the development version of the docker-compose (docker-compose.yml.dev
).
If you want to protect the access to your indices, you can require a
GitHub login to access and the people able to use the indices will be
the ones listed in the users
section in config.yaml
.
Configure the GitHub Oauth authentication to secure private indexes
- Create an Oauth APP in your GitHub user settings page
- Add "http://$MONOCLE_HOST:9876/api/0/authorize" in "User authorization callback URL"
- Save the
CLIENT_ID
andCLIENT_SECRET
into.env
asGITHUB_CLIENT_ID=<CLIENT_ID>
andGITHUB_CLIENT_SECRET=<CLIENT_SECRET>
.
The authentication and authorization support is new and only provides
a solution to control access to private indexes. Only login users
part of users
will be authorized to access the related index.
Note that the GitHub application can be also used as a Oauth APP.
Monocle can interact with a GitHub application to create and use installed application token to query the API.
Once the application is created and Monocle started with application id and
private key. If a github_orgs
entry's token attribute is missing Monocle will
search accross the application installations for an installed application
on the related GitHub organization. If any, it will generate an installation token
for the matching installation and use it to query the GitHub API.
- Register new GitHub App
- In
Repository permissions
setMetadata
asRead-Only
,Pull requests
asRead-Only
andContents
asRead-Only
- Click
Create the GitHub App
- Click
Generate a private key
and download the key - Save the
App ID
- Save the private key into
etc/app_key.rsa
- Into the
.env
file addGITHUB_APP_ID=<APP_ID>
andGITHUB_APP_KEY_PATH=/etc/monocle/app_key.rsa
---
tenants:
- index: monocle
crawler:
loop_delay: 10
github_orgs:
- name: tektoncd
repository: pipeline
updated_since: "2020-03-15"
token: <github_token>
base_url: https://github.com
- name: spinnaker
updated_since: "2020-03-15"
token: <github_token>
base_url: https://github.com
gerrit_repositories:
- name: ^zuul/.*
updated_since: "2020-03-15"
base_url: https://review.opendev.org
# A private index only whitelisted users are authorized to access
# See "Advanced deployment configuration" section
- index: monocle-private
users:
- <github_login1>
- <github_login2>
crawler:
loop_delay: 10
github_orgs:
- name: containers
repository: libpod
updated_since: "2020-03-15"
token: <github_token>
base_url: https://github.com
A new field self_merged
has been added. Previously indexed changes can be updated by running the self-merge
migration process.
docker-compose run --no-deps crawler /usr/local/bin/monocle --elastic-conn elastic:9200 dbmanage --index <index-name> --run-migrate self-merge
Follow our contributing guide.