ocurrent / current-bench Goto Github PK

Experimental benchmarking infrastructure using OCurrent pipelines

License: Apache License 2.0

OCaml 37.22% Dockerfile 2.31% HTML 0.16% CSS 0.81% JavaScript 0.16% ReScript 54.50% Makefile 1.50% Standard ML 0.33% Shell 2.76% Python 0.24%

current-bench's Introduction

OCurrent

OCurrent allows you to specify a workflow / pipeline for keeping things up-to-date.

For example, the pipeline shown above fetches the head of a GitHub repository's master branch, builds it, runs the tests, and deploys the binary if the tests pass. When a new commit is pushed, it runs the pipeline again.

Another use might be to keep the GitHub build status of each PR in your Git repository showing the result of fetching, building and testing the PR's head commit. If the head commit changes, the result must be recalculated.

An OCurrent pipeline is written using an OCaml eDSL. When OCurrent evaluates it, it records the inputs used (e.g. the current set of open PRs and the head of each one), monitors them, and automatically recalculates when an input changes.

Larger uses of OCurrent include the OCaml Docker base image builder and ocaml-ci, which is the CI that tests this repository itself.

Documentation

The OCurrent docs contains user documentation and examples. In particular, you might like to start by reading about the example pipelines or how to write your own plugins.

For technical docs, see the API Documentation.

Licensing

OCurrent is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

current-bench's People

Contributors

Stargazers

Watchers

Forkers

gs0510 rizo tatchi d4hines magnuss shubhamkumar13 jsthomas punchagan art-w ps-19 zineb-ada gridbugs misterda 3rafal rikusilvola julow tmattio

current-bench's Issues

Allow configurable benchmark targets in the pipeline to allow easier enrolling.

Add unit information to the metrics

It would be great to have a label to explain what the Y-axis actually is.

Ensure the opam repository is up to date before building the project

The docker image produced by current-bench to run the benchmarks might not be using an up to date opam repository.

This can be causing errors during package installation and terminating the benchmark execution with "Docker build exited with status 20".

@samoht

I think the base ocaml-ci images are tracking opam-repository and trigger a rebuild whenever a new version of a dependency is merged in opam-repository. Would be nice if ocaml-bench could do something similar?

Use `---privileged` with Docker run_args for Multicore OCaml

Multicore OCaml Sandmark environment uses bwrap and we need to use the --privileged option with Docker run for the same in pipeline/lib/pipeline.ml:

-   [ "--security-opt"; "seccomp=./aslr_seccomp.json" ]
+   [ "--security-opt"; "seccomp=./aslr_seccomp.json"; "--privileged" ]

Otherwise, we get a Operation not permitted error.

Fix error reporting when db insertion fails

When the db is not correctly set up or has an incorrect schema applied, the result insertion will fail with an error similar to the following:

pipeline_1  | Result status PGRES_FATAL_ERROR unexpected (expected status:PGRES_COMMAND_OK); ERROR:  relation "benchmarks" does not exist
pipeline_1  | LINE 3:   benchmarks(run_at, duration, repo_id, commit, branch, pull...
pipeline_1  |           ^

At the same time the job is marked as successful and the PR is marked as "passed".

This is not a benchmark execution problem, but at the very least we should mark the result as undetermined.

When the support for in-progress benchmarks is added, it should be easier to report this kind of errors.

Add URL for commit SHA

At the moment, the benchmarks are displayed with the commit SHA but to see the corresponding commit, one has to go and search the repo for it. Linking it would take the user directly to the commit.

Make it easy to see the benchmark's output

Hasura Admin Secret

Hi 👋!
What is the best way to handle the X-Hasura-Admin-Secret value? It's currently hard-coded into the repo. Can this be used to do anything destructive? I'm not familiar with Hasura but it seems great, and I want to use Hasura Cloud for my own implementation of current-bench. I'll hard code into the client for now, but I don't want some jerk to DOS me if it can be easily avoided 😅

Specify if higher values are better in metric results

Currently the UI shows a +/- signs for the relative comparison values with between master and PRs. It would be nice to be able to know know if higher values are better or worse to add color clues.

One way to achieve this is by adding an extra field to the benchmark output. Another would be to have the ability to set this information in the UI and save it in the db.

Run benchmarks for old commits

For new projects that start using current-bench, we might want to have the ability to compute benchmarks for old commits.

Of course, this could result in significant resource usage and should not be done by default.

Add support for multi benchmarks in the UI

We need to complete the frontend changes for #17.

This is my suggestion on what needs to be done:

Fetch all available benchmarks with GraphQL in the UI.
Add benchmarks menu to the sidebar.
Request benchmark data for a specific benchmark.
Add benchmark to the URI.

Issues with Null Pull numbers

The following request from CommitInfo.res, line 3 is failing:

query ($repoId: String!, $pullNumber: Int, $isMaster: Boolean!) {
  lastCommitInfo: benchmarks(
    limit: 1
    where: {_and: [{pull_number: {_eq: $pullNumber}}, {pull_number: {_is_null: $isMaster}}, {repo_id: {_eq: $repoId}}]}
    order_by: [{run_at: desc_nulls_last}]
  ) {
    run_at
    pull_number
    branch
    commit
    build_job_id
    run_job_id
    __typename
  }
}

It gives the error:

{
  "errors": [
    {
      "extensions": {
        "path": "$.selectionSet.benchmarks.args.where._and[0].pull_number._eq",
        "code": "validation-failed"
      },
      "message": "expected a 32-bit integer for type \"Int\", but found null"
    }
  ]
}

I'm not proficient enough in GraphQL yet to know how to fix this, so for now I'll just monkey patch my data to avoid this case.

Link text is unclear

The text "Learn" in the nav-bar and the footer is unclear without context and may be confusing to screen readers. You can add a special screen reader text than can help with accessibility.

Relevant code:
Learn

Show benchmarks logs

It's useful to see the benchmarks logs when the benchmarks are running or failed (or just if someone wants to reproduce the results and see what have been built). Currently these logs are not accessible.

Add abiility to run `Quick` and `Slow` benchmarks

Some benchmarks might run faster than others and it might be useful for the pipeline to report the Quick benchmarks and report that and run Slow benchmarks in the background as well.

Parse json from `make bench` output

For some of the repos, decompress for example, the benchmarks use a randomly generated file using cat as the input, so current-bench currently fails because the pipeline can handle only json. If we could parse json from the pread log in the pipeline, we could handle extraneous data better.

Ability to set scale on UI to start at 0

Is it possible to change the way the raw results are plotted such that they always include 0 on the y-axis (that is [0, y_max+delta])? (see screenshot at end)

Right now you have them plotted as [y_min-delta, y_max+delta] for each run which makes the graphs not easy to interpret across a large number of benchmarks, the reader has to inspect the axis scale because the shape might not be significant. For example the first time_secs plot has a range of [101.9, 102.1] that is 102 +/- 0.1% but visually the changes looks proportionally more significant because of the zoomed y-axis. Also when comparing graphs, we could have a graph with range [91, 101] that could have the same visual shape as [101.9, 102.1] but the two could have very different implications.

I’m open to other scaling proposals that also keep things easy to compare: e.g. display as normalised deviation from median/mean with all axes for the plots on a common y-scale. However these other scaling proposals might not be as easy to implement as just making 0 the minimum for all our positive valued metrics.

@shakthimaan requested I file the issue here

Allow configurable Dockerfile

Currently, the docker DSL specifies the steps needed to run benchmarks, but for different repositories the packages might differ, someone repositories might need pinning some packages for the benchmarks to work, hence a configurable dockerfile which can be plugged into the pipeline will be useful.

Last commit tick is not shown

See title.

This is a known issue: danvk/dygraphs#506

Improve precision in graphs

Both the graphs and the metrics table currently have rounding issues as illustrated in the following graph:

Show smaller time interval on PRs

Sometimes changes to benchmarks might be made but since the data is immutable, it shows up as NA. If we load smaller data from the database, this problem of displaying dead benchmarks will go away.

Index benchmarks introduced a change and the PR shows the data for the dead metric as well.

Benchmarks are not ordered correctly on the graphs

Currently, the benchmarks are sorted by the timestamp they were created at which doesn't always result in the correct graph. Sometimes the master & pull request commits are interspersed because the pipeline ran on the master more than once. One way to make it better would be to topologically sort the commits or display master benchmarks first followed by displaying the benchmarks of the commit in the PR.

Detect when we try to access a PR that does not exist

There's a Javascript error and the whole page becomes blank when we try to access the benchmark results from a PR that does not exist. This can happen for instance when we manually modify the PR number in the URL.

We should handle that in a better way, like showing a message that tells the user that the PR does not exist (yet).

Stable builds / show dependency changes

Currently we use the latest dependencies from opam repository to build, but as the dependency versions may change over time they can affect the benchmark results. It would be useful if it was possible to use the same dependencies for each PR rebuild and/or highlight if there's a dependency change (both over time on branches and on a PR).

Fix overlapping graph labels

This:

From my understanding dygraphs does not have built-in support for vertical labels. We apply a CSS transform to rotate the labels. For some reason the position calculation for labels used in dygraphs results in this strange overlapping.

Another problem I noticed is that the last data point on the graph does not (always?) have a commit label.

Add health-checks for db migrations in development mode

Similar to how it's done in #91

Confusing in-progress benchmarks

When you click on the benchmark page (from a PR status) you land on a page which shows you outdated results without any clear indication that the benchmark result are still being computed. It's probably better to show either nothing or partial progress (if that's possible), or just the logs currently being produced by the benchmark.

When multiple benchmarks are produced ensure that there are no name collisions

By default benchmark_name is set to NULL. We do this even if there are multiple objects produced by make bench. This most certainly is a mistake and we should require the output to have the benchmark_name field with multiple benchmark results.

Limit the numbers of datapoints displayed on the graph

The data from last 50 commits would be a good number, more than that clutters the graph.

Database not Populating

Hi,

I'm trying to use the production version of the app, but something's not working. My database never gets any inserts. I can't decipher what the problem is, as I don't see any errors in the logs. I tried troubleshooting with print_endline, but the control flow elludes me - I'm unfamiliar with the let+ and and+ syntax and I think there is magic in there. Any help would be appreciated. Here's the debug logs (hopefully there aren't any secrets in there 😛):

 application [INFO] Logging ready.
current.github [DEBUG] API call on https://api.github.com/app/installations
   current_web [INFO] Starting web server: (TCP (Port 8081))
 current.cache [DEBUG] set: "docker" "pull" "ocaml/opam"
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031300-docker-pull-bcb363.log
       current [INFO] Exec: "docker" "pull" "ocaml/opam"
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [DEBUG] Got response:
                         `List ([`Assoc ([("id", `Int (16069792));
                                           ("account",
                                            `Assoc ([("login",
                                                      `String ("d4hines"));
                                                      ("id", `Int (25231383));
                                                      ("node_id",
                                                       `String ("MDQ6VXNlcjI1MjMxMzgz"));
                                                      ("avatar_url",
                                                       `String ("https://avatars.githubusercontent.com/u/25231383?v=4"));
                                                      ("gravatar_id",
                                                       `String (""));
                                                      ("url",
                                                       `String ("https://api.github.com/users/d4hines"));
                                                      ("html_url",
                                                       `String ("https://github.com/d4hines"));
                                                      ("followers_url",
                                                       `String ("https://api.github.com/users/d4hines/followers"));
                                                      ("following_url",
                                                       `String ("https://api.github.com/users/d4hines/following{/other_user}"));
                                                      ("gists_url",
                                                       `String ("https://api.github.com/users/d4hines/gists{/gist_id}"));
                                                      ("starred_url",
                                                       `String ("https://api.github.com/users/d4hines/starred{/owner}{/repo}"));
                                                      ("subscriptions_url",
                                                       `String ("https://api.github.com/users/d4hines/subscriptions"));
                                                      ("organizations_url",
                                                       `String ("https://api.github.com/users/d4hines/orgs"));
                                                      ("repos_url",
                                                       `String ("https://api.github.com/users/d4hines/repos"));
                                                      ("events_url",
                                                       `String ("https://api.github.com/users/d4hines/events{/privacy}"));
                                                      ("received_events_url",
                                                       `String ("https://api.github.com/users/d4hines/received_events"));
                                                      ("type",
                                                       `String ("User"));
                                                      ("site_admin",
                                                       `Bool (false))
                                                      ]));
                                           ("repository_selection",
                                            `String ("selected"));
                                           ("access_tokens_url",
                                            `String ("https://api.github.com/app/installations/16069792/access_tokens"));
                                           ("repositories_url",
                                            `String ("https://api.github.com/installation/repositories"));
                                           ("html_url",
                                            `String ("https://github.com/settings/installations/16069792"));
                                           ("app_id", `Int (109289));
                                           ("app_slug",
                                            `String ("test-occurent-bench"));
                                           ("target_id", `Int (25231383));
                                           ("target_type", `String ("User"));
                                           ("permissions",
                                            `Assoc ([("contents",
                                                      `String ("write"));
                                                      ("metadata",
                                                       `String ("read"));
                                                      ("statuses",
                                                       `String ("write"));
                                                      ("repository_hooks",
                                                       `String ("write"))
                                                      ]));
                                           ("events",
                                            `List ([`String ("create");
                                                     `String ("delete");
                                                     `String ("fork");
                                                     `String ("label");
                                                     `String ("push")]));
                                           ("created_at",
                                            `String ("2021-04-08T14:39:09.000Z"));
                                           ("updated_at",
                                            `String ("2021-04-14T22:52:38.000Z"));
                                           ("single_file_name", `Null);
                                           ("has_multiple_single_files",
                                            `Bool (false));
                                           ("single_file_paths", `List ([]));
                                           ("suspended_by", `Null);
                                           ("suspended_at", `Null)])
                                  ])
current.github [INFO] Found installation 16069792 for "d4hines"
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [INFO] Getting API token
current.github [DEBUG] API call on https://api.github.com/app/installations/16069792/access_tokens
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [DEBUG] Got response:
                         `Assoc ([("token",
                                   `String ("ghs_qmFkuP4WeBVq3M4ZsRhZbL93hxKec83jmCKQ"));
                                   ("expires_at",
                                    `String ("2021-04-15T04:13:02Z"));
                                   ("permissions",
                                    `Assoc ([("contents", `String ("write"));
                                              ("metadata", `String ("read"));
                                              ("repository_hooks",
                                               `String ("write"));
                                              ("statuses", `String ("write"))
                                              ]));
                                   ("repository_selection",
                                    `String ("selected"))
                                   ])
current.github [DEBUG] Get repositories for "d4hines" from https://api.github.com/installation/repositories
       current [INFO] Exec: "docker" "image" "inspect" "ocaml/opam" "-f" 
                            "{{index .RepoDigests 0}}"
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [DEBUG] Got response:
                         `Assoc ([("total_count", `Int (1));
                                   ("repository_selection",
                                    `String ("selected"));
                                   ("repositories",
                                    `List ([`Assoc ([("id", `Int (353847111));
                                                      ("node_id",
                                                       `String ("MDEwOlJlcG9zaXRvcnkzNTM4NDcxMTE="));
                                                      ("name",
                                                       `String ("marigold-tezos"));
                                                      ("full_name",
                                                       `String ("d4hines/marigold-tezos"));
                                                      ("private",
                                                       `Bool (false));
                                                      ("owner",
                                                       `Assoc ([("login",
                                                                 `String (
                                                                 "d4hines"));
                                                                 ("id",
                                                                  `Int (
                                                                  25231383));
                                                                 ("node_id",
                                                                  `String (
                                                                  "MDQ6VXNlcjI1MjMxMzgz"));
                                                                 ("avatar_url",
                                                                  `String (
                                                                  "https://avatars.githubusercontent.com/u/25231383?v=4"));
                                                                 ("gravatar_id",
                                                                  `String (
                                                                  ""));
                                                                 ("url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines"));
                                                                 ("html_url",
                                                                  `String (
                                                                  "https://github.com/d4hines"));
                                                                 ("followers_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/followers"));
                                                                 ("following_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/following{/other_user}"));
                                                                 ("gists_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/gists{/gist_id}"));
                                                                 ("starred_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/starred{/owner}{/repo}"));
                                                                 ("subscriptions_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/subscriptions"));
                                                                 ("organizations_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/orgs"));
                                                                 ("repos_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/repos"));
                                                                 ("events_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/events{/privacy}"));
                                                                 ("received_events_url",
                                                                  `String (
                                                                  "https://api.github.com/users/d4hines/received_events"));
                                                                 ("type",
                                                                  `String (
                                                                  "User"));
                                                                 ("site_admin",
                                                                  `Bool (
                                                                  false))
                                                                 ]));
                                                      ("html_url",
                                                       `String ("https://github.com/d4hines/marigold-tezos"));
                                                      ("description",
                                                       `String ("A mirror of https://gitlab.com/marigold/tezos"));
                                                      ("fork", `Bool (false));
                                                      ("url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos"));
                                                      ("forks_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/forks"));
                                                      ("keys_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/keys{/key_id}"));
                                                      ("collaborators_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/collaborators{/collaborator}"));
                                                      ("teams_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/teams"));
                                                      ("hooks_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/hooks"));
                                                      ("issue_events_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/issues/events{/number}"));
                                                      ("events_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/events"));
                                                      ("assignees_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/assignees{/user}"));
                                                      ("branches_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/branches{/branch}"));
                                                      ("tags_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/tags"));
                                                      ("blobs_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/git/blobs{/sha}"));
                                                      ("git_tags_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/git/tags{/sha}"));
                                                      ("git_refs_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/git/refs{/sha}"));
                                                      ("trees_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/git/trees{/sha}"));
                                                      ("statuses_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/statuses/{sha}"));
                                                      ("languages_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/languages"));
                                                      ("stargazers_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/stargazers"));
                                                      ("contributors_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/contributors"));
                                                      ("subscribers_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/subscribers"));
                                                      ("subscription_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/subscription"));
                                                      ("commits_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/commits{/sha}"));
                                                      ("git_commits_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/git/commits{/sha}"));
                                                      ("comments_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/comments{/number}"));
                                                      ("issue_comment_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/issues/comments{/number}"));
                                                      ("contents_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/contents/{+path}"));
                                                      ("compare_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/compare/{base}...{head}"));
                                                      ("merges_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/merges"));
                                                      ("archive_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/{archive_format}{/ref}"));
                                                      ("downloads_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/downloads"));
                                                      ("issues_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/issues{/number}"));
                                                      ("pulls_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/pulls{/number}"));
                                                      ("milestones_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/milestones{/number}"));
                                                      ("notifications_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/notifications{?since,all,participating}"));
                                                      ("labels_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/labels{/name}"));
                                                      ("releases_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/releases{/id}"));
                                                      ("deployments_url",
                                                       `String ("https://api.github.com/repos/d4hines/marigold-tezos/deployments"));
                                                      ("created_at",
                                                       `String ("2021-04-01T23:01:10Z"));
                                                      ("updated_at",
                                                       `String ("2021-04-15T03:10:29Z"));
                                                      ("pushed_at",
                                                       `String ("2021-04-15T03:10:27Z"));
                                                      ("git_url",
                                                       `String ("git://github.com/d4hines/marigold-tezos.git"));
                                                      ("ssh_url",
                                                       `String ("[email protected]:d4hines/marigold-tezos.git"));
                                                      ("clone_url",
                                                       `String ("https://github.com/d4hines/marigold-tezos.git"));
                                                      ("svn_url",
                                                       `String ("https://github.com/d4hines/marigold-tezos"));
                                                      ("homepage",
                                                       `String (""));
                                                      ("size", `Int (101816));
                                                      ("stargazers_count",
                                                       `Int (0));
                                                      ("watchers_count",
                                                       `Int (0));
                                                      ("language",
                                                       `String ("OCaml"));
                                                      ("has_issues",
                                                       `Bool (true));
                                                      ("has_projects",
                                                       `Bool (true));
                                                      ("has_downloads",
                                                       `Bool (true));
                                                      ("has_wiki",
                                                       `Bool (true));
                                                      ("has_pages",
                                                       `Bool (false));
                                                      ("forks_count",
                                                       `Int (0));
                                                      ("mirror_url", `Null);
                                                      ("archived",
                                                       `Bool (false));
                                                      ("disabled",
                                                       `Bool (false));
                                                      ("open_issues_count",
                                                       `Int (0));
                                                      ("license",
                                                       `Assoc ([("key",
                                                                 `String (
                                                                 "other"));
                                                                 ("name",
                                                                  `String (
                                                                  "Other"));
                                                                 ("spdx_id",
                                                                  `String (
                                                                  "NOASSERTION"));
                                                                 ("url",
                                                                  `Null);
                                                                 ("node_id",
                                                                  `String (
                                                                  "MDc6TGljZW5zZTA="))
                                                                 ]));
                                                      ("forks", `Int (0));
                                                      ("open_issues",
                                                       `Int (0));
                                                      ("watchers", `Int (0));
                                                      ("default_branch",
                                                       `String ("master"));
                                                      ("permissions",
                                                       `Assoc ([("admin",
                                                                 `Bool (
                                                                 false));
                                                                 ("push",
                                                                  `Bool (
                                                                  false));
                                                                 ("pull",
                                                                  `Bool (
                                                                  false))
                                                                 ]))
                                                      ])
                                             ]))
                                   ])
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [INFO] GraphQL(default_ref): cost:1 remaining:4973 resetAt:2021-04-15T03:15:12Z
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: git fetch https://github.com/d4hines/marigold-tezos.git#refs/heads/master (12fa1c421fdef59948cca13109665d11bee31292)
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031301-git-fetch-c2346d.log
       current [INFO] Exec: "git" "clone" "--recursive" "-q" "https://github.com/d4hines/marigold-tezos.git" 
                            "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031301-github-set-status-ac74aa.log
       current [INFO] Result: Ok: ()
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [INFO] Exec: "git" "-C" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f" 
                            "branch" "-f" "fetch-12fa1c421fdef59948cca13109665d11bee31292" 
                            "12fa1c421fdef59948cca13109665d11bee31292"
       current [INFO] Exec: "git" "-C" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f" 
                            "branch" "-f" "fetch-12fa1c421fdef59948cca13109665d11bee31292" 
                            "12fa1c421fdef59948cca13109665d11bee31292"
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: docker build {
                                           "commit":
                                             "12fa1c421fdef59948cca13109665d11bee31292",
                                           "dockerfile": {
                                             "contents":
                                               "0bdebea505c40f2924759d9a5f3888ab"
                                           },
                                           "docker_context": null,
                                           "squash": false,
                                           "build_args": []
                                         }
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031319-docker-build-bfe0e0.log
       current [INFO] Exec: "cp" "-a" "--" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f/.git" 
                            "/tmp/git-checkout10657da"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [INFO] Exec: "git" "-C" "/tmp/git-checkout10657da" "reset" 
                            "--hard" "12fa1c421fdef59948cca13109665d11bee31292"
       current [INFO] Exec: "git" "-C" "/tmp/git-checkout10657da" "submodule" 
                            "update" "--init" "--recursive"
       current [INFO] Exec: "docker" "build" "--iidfile" "/tmp/git-checkout10657da/docker-iid" 
                            "--" "/tmp/git-checkout10657da"
current.docker [INFO] Built docker image sha256:a682159e9bb570c8dc189947102d2a332d789c09f9a43ffbbdc7b664ef21038b
       current [DEBUG] Switch.turn_off: already off
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: "docker" "run" "--security-opt"
                       "seccomp=./aslr_seccomp.json" "--tmpfs"
                       "/dev/shm:rw,noexec,nosuid,size=4g" "--rm" "-i"
                       "sha256:a682159e9bb570c8dc189947102d2a332d789c09f9a43ffbbdc7b664ef21038b"
                       "/usr/bin/setarch" "x86_64" "--addr-no-randomize"
                       "make" "bench"
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031322-docker-pread-771b68.log
       current [INFO] Exec: "docker" "run" "--security-opt" "seccomp=./aslr_seccomp.json" 
                            "--tmpfs" "/dev/shm:rw,noexec,nosuid,size=4g" 
                            "--rm" "-i" "sha256:a682159e9bb570c8dc189947102d2a332d789c09f9a43ffbbdc7b664ef21038b" 
                            "/usr/bin/setarch" "x86_64" "--addr-no-randomize" 
                            "make" "bench"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       12fa1c42/ocaml-benchmarks to
                       {"state":"failure","description":"Command \"docker\" \"run\" \"--security-opt\" \"seccomp=./aslr_seccomp.json\" \n\"--tmpfs\" \"/dev/shm:rw,noexec,nosuid,size=4g\" \"--rm\" \"-i\" \"sha256:a68","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031322-github-set-status-75573b.log
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
   current_web [INFO] HTTP POST "/webhooks/github"
current.github [INFO] input_webhook: ((headers
  ((accept */*) (connection close) (content-length 7492)
   (content-type application/json) (host ocurrent.marigold.dev:8081)
   (user-agent GitHub-Hookshot/64aa546)
   (x-github-delivery 8d4ece0a-9d98-11eb-840e-f26b4baf96a4)
   (x-github-event push) (x-github-hook-id 290932103)
   (x-github-hook-installation-target-id 109289)
   (x-github-hook-installation-target-type integration)))
 (meth POST) (resource /webhooks/github) (version HTTP_1_1)
 (encoding (Fixed 7492)))
current.github [INFO] Got GitHub event "push"
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
current.github [INFO] GraphQL(default_ref): cost:1 remaining:4972 resetAt:2021-04-15T03:15:12Z
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: git fetch https://github.com/d4hines/marigold-tezos.git#refs/heads/master (e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4)
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031329-git-fetch-affd23.log
       current [INFO] Exec: "git" "-C" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f" 
                            "branch" "-f" "fetch-e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4" 
                            "e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031329-github-set-status-9f87c3.log
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [INFO] Exec: "git" "-C" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f" 
                            "fetch" "-f" "https://github.com/d4hines/marigold-tezos.git" 
                            "refs/heads/master"
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [INFO] Exec: "git" "-C" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f" 
                            "branch" "-f" "fetch-e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4" 
                            "e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4"
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: docker build {
                                           "commit":
                                             "e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4",
                                           "dockerfile": {
                                             "contents":
                                               "0bdebea505c40f2924759d9a5f3888ab"
                                           },
                                           "docker_context": null,
                                           "squash": false,
                                           "build_args": []
                                         }
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031329-docker-build-844220.log
       current [INFO] Exec: "cp" "-a" "--" "/home/d4hines/repos/current-bench/var/git/marigold-tezos.git-ecf80115d780255b0108dda6b07dc18f/.git" 
                            "/tmp/git-checkout37965990"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [INFO] Exec: "git" "-C" "/tmp/git-checkout37965990" "reset" 
                            "--hard" "e8f2f7784ed31e0343dc26b2d71996a19c5f8ff4"
       current [INFO] Exec: "git" "-C" "/tmp/git-checkout37965990" "submodule" 
                            "update" "--init" "--recursive"
       current [INFO] Exec: "docker" "build" "--iidfile" "/tmp/git-checkout37965990/docker-iid" 
                            "--" "/tmp/git-checkout37965990"
current.docker [INFO] Built docker image sha256:9cf87199b7065eda0b16f547bd861e07736f18a30906d9dd1b564d3692c16a89
       current [DEBUG] Switch.turn_off: already off
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: "docker" "run" "--security-opt"
                       "seccomp=./aslr_seccomp.json" "--tmpfs"
                       "/dev/shm:rw,noexec,nosuid,size=4g" "--rm" "-i"
                       "sha256:9cf87199b7065eda0b16f547bd861e07736f18a30906d9dd1b564d3692c16a89"
                       "/usr/bin/setarch" "x86_64" "--addr-no-randomize"
                       "make" "bench"
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031347-docker-pread-b5b209.log
       current [INFO] Exec: "docker" "run" "--security-opt" "seccomp=./aslr_seccomp.json" 
                            "--tmpfs" "/dev/shm:rw,noexec,nosuid,size=4g" 
                            "--rm" "-i" "sha256:9cf87199b7065eda0b16f547bd861e07736f18a30906d9dd1b564d3692c16a89" 
                            "/usr/bin/setarch" "x86_64" "--addr-no-randomize" 
                            "make" "bench"
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"pending","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
 current.cache [DEBUG] set: Set d4hines/marigold-tezos refs/heads/master
                       e8f2f778/ocaml-benchmarks to
                       {"state":"failure","description":"Command \"docker\" \"run\" \"--security-opt\" \"seccomp=./aslr_seccomp.json\" \n\"--tmpfs\" \"/dev/shm:rw,noexec,nosuid,size=4g\" \"--rm\" \"-i\" \"sha256:9cf","target_url":"https://tezos-current-bench.hasura.app/d4hines/marigold-tezos"}
       current [INFO] Created new log file at
                      /home/d4hines/repos/current-bench/var/job/2021-04-15/031347-github-set-status-02564c.log
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event...
       current [DEBUG] Evaluating...
       current [DEBUG] Waiting for an external event..

Support external datasets

Some benchmarks will require external datasets to work. If the dataset is large it might not be feasible to store it in the git repo of the project. In this case, before running the benchmarks, the data needs to be fetched.

A few notes:

We could have a docker volume accessible from the running container.
Invalidating the cached dataset could be difficult.
There should be limits on the size of the dataset.

Automatically detect the backend URI

@tatchi :

When I started the frontend, I was a bit surprised to actually see data from http://autumn.ocamllabs.io. That would be nice to grab the backend URL from env (it's currently hardcoded in the code). Vite allows reading .env files but from what I understand, they should be located at the root of the project (within frontend/ in our case). Maybe we could copy (and rename) the env file e.g development.env within the frontend folder during one of the docker steps? Not sure though if that's a good idea 😅

Another (very simplistic) option is to decide the backend URI based on the the current window location in the app. If it's localhost, use localhost, if it's not, use the production URI.

wrong commit links

When clicking on a commit id in a graph, this points to a truncated url (which I think works fine because GH support this but would be nice to have the full hash).
But the main issue is that it always point to https://github.com/mirage/index, e.g. clicking on a commit ID on:

http://autumn.ocamllabs.io/mirage/irmin/pull/1312

leads to:

mirage/index@49facfe

Read stderr from the docker container.

Currently we expect the benchmarks to silence all stderr output and only produce valid JSON output on stdout.

This can lead to situations where debugging errors during the benchmark execution is very difficult.

For example, the dune benchmarks are currently failing with:

$ docker run -it --rm current-bench-dune-error-2 make bench
make: *** [Makefile:129: bench] Error 2

But in reality the problem is:

$ dune exec -- ./bench/bench.exe
cloning ocaml-dune/dune-bench
        dune (internal) (exit 1)
/home/opam/.opam/4.12/bin/dune build @install --root . &> /dev/null
Fatal error: exception Dune_util.Report_error.Already_reported
Raised at Dune_engine__Scheduler.Run.go in file "src/dune_engine/scheduler.ml", line 790, characters 27-40
Called from Dune__exe__Bench in file "bench/bench.ml", line 131, characters 4-168

We should forward the stderr to the current logs.

Data API for Jupyter notebooks

Create a POC implementation of a Jupiter notebook fetching the data from our benchmarking backend.

Correctly handle metrics with multiple values.

Abstract out Docker image name from pipeline/lib/pipeline.ml to environments

The "ocaml/opam" Docker image name is hard-coded in pipeline/lib/pipeline.ml. Kindly abstract it out to an environment variable, so that Multicore OCaml can specify ocaml/opam:ubuntu-20.10-ocaml-4.10.

  let dockerfile =
    let+ base = Docker.pull ~schedule:weekly "ocaml/opam" in
    `Contents (dockerfile ~base)
  in

Periodically Clean docker images on autumn

Every few weeks, autumn runs out of memory and all PRs start failing with Docker exit status 99. One solution would also be to ensure that the images are on /data and periodically clean them up.

The status 99 is from here:

Fatal error:
opam: "mkdir" failed on /home/opam/.opam/repo/origin: No space left on device
The command '/bin/sh -c opam remote add origin https://opam.ocaml.org' returned a non-zero code: 99

Include more information on the PR in the current logs

It's difficult to know to what PR a given job belongs to.

Show min/avg/max in the tooltip for multi-value datapoints.

From #74,

Related to the multi-value datapoints: it would be good to show min/avg/max in the tooltip.

Ability to start and run only PostgreSQL and frontend

For Multicore OCaml, we provision our hardware with specific configuration, and will re-run the experiments with different settings for compiler analysis. For such cases, we would like to use an ETL tool to just load the benchmark results and use the PostgreSQL and frontend Docker containers only for visualization.

Consider running the benchmarks with compiler variants

This could include different compiler versions and variants:

flambda
multicore
best-fit alloc strategy

Remote Runners

We're benchmarking Tezos on a Raspberry Pi for a number of reasons. It would be great if there was a notion of "remote runners" - install some software on the Pi that allows it to run the benchmarks, while the rest is running in the cloud somewhere.

Of course, if we go too far in this direction we've recreated other full-fledged CI systems out there (Jenkins, Circle CI, etc).

Thoughts?

Support for native builds for bare metals instead of Docker container

For Multicore OCaml, we would like to be able to build and run benchmarks on bare metal machines. It will be good to create an OCurrent pipeline just for native builds, while, still using the PostgreSQL database and frontend containers.

Set OPAMVAR_bench=1 before installing dependencies

Suggested by @MagnusS

This will allow users to specify benchmarks-only dependencies in opam files:

$ OPAMVAR_bench=1 opam install --deps-only .

depends: [
  "logs" {bench}
]

The current alternative approach is adding a separate opam file just for benchmark dependencies.

Fails to build development environment

Hello,

current-bench fails to start for me.

I'm on Windows 10, running Docker version 20.10.5.

The contents of ./environments/development.env are:

OCAML_BENCH_DOCKER_CPU=1
OCAML_BENCH_DB_PASSWORD=docker
OCAML_BENCH_GRAPHQL_KEY=secret
OCAML_BENCH_TARGET_ARCH=amd64

I run make start-development and get the following output (errors way down at bottom):

PS C:\Users\d4hin\repos\current-bench> make start-development
docker-compose \
        --project-name="current-bench" \
        --file=./environments/development.docker-compose.yaml \
        --env-file=./development.env \
        up \
        --remove-orphans \
        --build
Building db
failed to get console mode for stdout: The handle is invalid.
[+] Building 11.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s 
 => => transferring dockerfile: 151B                                       0.0s 
 => [internal] load .dockerignore                                          0.0s 
 => => transferring context: 2B                                            0.0s 
 => [internal] load metadata for docker.io/library/postgres:12             1.7s 
 => [internal] load build context                                          0.0s 
 => => transferring context: 87B                                           0.0s 
 => [1/2] FROM docker.io/library/postgres:12@sha256:2f3f78532c9cc5435d1cf  9.2s 
 => => resolve docker.io/library/postgres:12@sha256:2f3f78532c9cc5435d1cf  0.0s 
 => => sha256:2f3f78532c9cc5435d1cf9c8f5fb1409f9abd43e5b7 1.86kB / 1.86kB  0.0s
 => => sha256:75646c2fb4101d306585c9b106be1dfa7d82720ba 27.14MB / 27.14MB  1.0s
 => => sha256:7e98825f6d67159fc035c0db985619c0635cc7edb89 1.80kB / 1.80kB  0.3s
 => => sha256:6ef141d922bc14123d24ddec34aa11d0f360702a2 10.63kB / 10.63kB  0.0s
 => => sha256:4217f50ff2ca28b7c3dddb25372379c97093df532c7 3.25kB / 3.25kB  0.0s
 => => sha256:2355d0ffeb5531909b8910928bcbaa874e9464657c7 4.18MB / 4.18MB  0.4s
 => => sha256:cfd3ce06be45a467d795e78a52a17330a56a7b85a3b 1.42MB / 1.42MB  0.5s
 => => sha256:c7b7bb83e8f78baa3e4e402b1704b6df53db00bec8b 7.97MB / 7.97MB  0.9s
 => => sha256:c6786930510827ff4067b1103131548886adcd7 391.17kB / 391.17kB  0.6s
 => => sha256:19614baa7ddd150dc7115b541a85cec0e8178fc9c6e62c8 149B / 149B  0.6s
 => => sha256:af508737d8137d59629c343294044cb7f854220431d 3.05kB / 3.05kB  0.7s
 => => sha256:562171e5e21c288ecc3ae718a64e16c2f17902deb 73.34MB / 73.34MB  6.1s
 => => sha256:9a1da7929f851b2f7db64c04a23cf5b2c7377b0db64 9.02kB / 9.02kB  1.0s
 => => extracting sha256:75646c2fb4101d306585c9b106be1dfa7d82720baabe1c75  1.3s
 => => sha256:83302896f1018b739b0cdcdd817cbc82c823cf8651cffd6 200B / 200B  1.2s
 => => sha256:1f2e5b2f6e536af3111e90491c622a3bd93179b70459f09 129B / 129B  1.2s
 => => sha256:8892c210e4afc8a8f8ece7cd0bcccfee3b234dd09b6 4.40kB / 4.40kB  1.4s
 => => sha256:46ddd2f5c65450c2987d3d2526b0a0b7d6f1655d312fb31 120B / 120B  1.4s
 => => extracting sha256:2355d0ffeb5531909b8910928bcbaa874e9464657c71213a  0.2s
 => => extracting sha256:7e98825f6d67159fc035c0db985619c0635cc7edb89aee89  0.0s
 => => extracting sha256:cfd3ce06be45a467d795e78a52a17330a56a7b85a3b3a25b  0.1s
 => => extracting sha256:c7b7bb83e8f78baa3e4e402b1704b6df53db00bec8b57266  0.3s
 => => extracting sha256:c6786930510827ff4067b1103131548886adcd7a25a012ba  0.1s
 => => extracting sha256:19614baa7ddd150dc7115b541a85cec0e8178fc9c6e62c82  0.0s
 => => extracting sha256:af508737d8137d59629c343294044cb7f854220431d23fc4  0.0s
 => => extracting sha256:562171e5e21c288ecc3ae718a64e16c2f17902deb1011cad  2.5s
 => => extracting sha256:9a1da7929f851b2f7db64c04a23cf5b2c7377b0db64b3b67  0.0s
 => => extracting sha256:1f2e5b2f6e536af3111e90491c622a3bd93179b70459f098  0.0s
 => => extracting sha256:83302896f1018b739b0cdcdd817cbc82c823cf8651cffd68  0.0s
 => => extracting sha256:8892c210e4afc8a8f8ece7cd0bcccfee3b234dd09b6b2637  0.0s
 => => extracting sha256:46ddd2f5c65450c2987d3d2526b0a0b7d6f1655d312fb310  0.0s
 => [2/2] COPY ./init.sql /docker-entrypoint-initdb.d/init.sql             0.1s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:65222942325e8b01767e984270bbcbc7150b551d35577  0.0s
 => => naming to docker.io/library/current-bench_db                        0.0s
Successfully built 65222942325e8b01767e984270bbcbc7150b551d35577e19aa4f3177c8caa3e6
Pulling graphql-engine (hasura/graphql-engine:v1.3.3)...
v1.3.3: Pulling from hasura/graphql-engine
fcf970b8ac66: Pull complete
Digest: sha256:814e36b355177046195edd795ca561fe762a2ae117129f185a26f9d3f609efb9
Status: Downloaded newer image for hasura/graphql-engine:v1.3.3
Building frontend
failed to get console mode for stdout: The handle is invalid.
[+] Building 42.9s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s 
 => => transferring dockerfile: 554B                                       0.0s 
 => [internal] load .dockerignore                                          0.0s 
 => => transferring context: 52B                                           0.0s 
 => [internal] load metadata for docker.io/library/nginx:stable-alpine     0.7s 
 => [internal] load metadata for docker.io/library/node:lts                0.8s 
 => [builder 1/7] FROM docker.io/library/node:lts@sha256:f6b9ff4caca9d4f  18.1s 
 => => resolve docker.io/library/node:lts@sha256:f6b9ff4caca9d4f0a331a882  0.0s 
 => => sha256:abea835c0b3b0ef244d92a1076602ed0891a35b1b77 7.83kB / 7.83kB  0.0s 
 => => sha256:da61ad49fa9961d6dfe53dc067fd690133f205498 11.29MB / 11.29MB  0.7s
 => => sha256:af62e04c0f85ed10801df4314d8cb9ba5e6c391f6a3 4.34MB / 4.34MB  0.5s
 => => sha256:f6b9ff4caca9d4f0a331a882e560df242eb332b7bbbed2f 776B / 776B  0.0s
 => => sha256:61e6ae4231ac78eb737a10e70d8925e92e7a35ec0c3 2.21kB / 2.21kB  0.0s
 => => sha256:00168f89dbe8f3c9985e536784c27517f6cc35ea5 45.38MB / 45.38MB  2.9s
 => => sha256:aae835e2c68ef1d48a4ffef72547a79926019aec6 49.79MB / 49.79MB  3.6s
 => => sha256:5254dc317968c943c4321cdbf0364a6a47c8332 214.35MB / 214.35MB  8.3s
 => => sha256:b3f76fdfa8e8784238b141aaa54fa1862aef5348665 4.19kB / 4.19kB  2.9s
 => => extracting sha256:00168f89dbe8f3c9985e536784c27517f6cc35ea56263469  1.9s
 => => sha256:5681b30a17f656444f1bd37c07ee6f594c7e2e26c 34.58MB / 34.58MB  5.5s
 => => sha256:da9a613e2daeaae78b9cd247e52f3f731b701730d7a 2.38MB / 2.38MB  3.9s
 => => sha256:67891041a0a2d985480f4934fc94bc0b180f9f27741bd5c 292B / 292B  4.1s
 => => extracting sha256:da61ad49fa9961d6dfe53dc067fd690133f205498e8045e8  0.4s
 => => extracting sha256:af62e04c0f85ed10801df4314d8cb9ba5e6c391f6a3c1db1  0.2s
 => => extracting sha256:aae835e2c68ef1d48a4ffef72547a79926019aec6a2bbeb5  2.4s
 => => extracting sha256:5254dc317968c943c4321cdbf0364a6a47c833296b3f45e3  7.6s
 => => extracting sha256:b3f76fdfa8e8784238b141aaa54fa1862aef534866543116  0.0s
 => => extracting sha256:5681b30a17f656444f1bd37c07ee6f594c7e2e26c4ec796e  1.6s
 => => extracting sha256:da9a613e2daeaae78b9cd247e52f3f731b701730d7ab8bf6  0.1s
 => => extracting sha256:67891041a0a2d985480f4934fc94bc0b180f9f27741bd5c2  0.0s
 => [stage-1 1/4] FROM docker.io/library/nginx:stable-alpine@sha256:ec3f6  5.7s
 => => resolve docker.io/library/nginx:stable-alpine@sha256:ec3f6e4587a20  0.0s
 => => sha256:ec3f6e4587a2053f406b1b4e027c260e82e4e32c256 1.65kB / 1.65kB  0.0s
 => => sha256:7409b874fed478250a841bc67aa1451df7163543cf6 1.36kB / 1.36kB  0.0s
 => => sha256:d0d03989cd8e044493922e446203a9893eb662a3279 7.98kB / 7.98kB  0.0s
 => => sha256:9b794450f7b6db7c944ba1f4161edb68cb535052fe7 2.82MB / 2.82MB  4.4s
 => => extracting sha256:9b794450f7b6db7c944ba1f4161edb68cb535052fe7db8ac  0.1s
 => => sha256:71c851db865893ed287bbecc2934e756a25e4e78bab 6.67MB / 6.67MB  4.9s
 => => extracting sha256:71c851db865893ed287bbecc2934e756a25e4e78babcd419  0.3s
 => => sha256:fc9aed49b78e299c67c0ee890e26326969df6c564fda2b2 902B / 902B  5.0s
 => => sha256:bb846ca9b45ab911d714856abaca670ed6a798711e73325 665B / 665B  5.1s
 => => sha256:281095db834a17364db464c05304d89414f2ab118283136 600B / 600B  5.0s
 => => extracting sha256:281095db834a17364db464c05304d89414f2ab118283136b  0.0s
 => => extracting sha256:fc9aed49b78e299c67c0ee890e26326969df6c564fda2b28  0.0s
 => => extracting sha256:bb846ca9b45ab911d714856abaca670ed6a798711e73325d  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 332.87kB                                      0.0s
 => [stage-1 2/4] COPY ./nginx.conf /etc/nginx/nginx.conf                  0.2s
 => [stage-1 3/4] RUN rm -rf /usr/share/nginx/html/*                       0.3s
 => [builder 2/7] WORKDIR /app                                             0.4s
 => [builder 3/7] COPY package.json yarn.lock /app/                        0.0s
 => [builder 4/7] RUN yarn install                                        14.7s
 => [builder 5/7] COPY . /app/                                             0.0s
 => [builder 6/7] RUN yarn build                                           2.1s
 => [builder 7/7] RUN yarn bundle                                          6.5s
 => [stage-1 4/4] COPY --from=builder /app/dist /usr/share/nginx/html      0.0s
 => exporting to image                                                     0.1s
 => => exporting layers                                                    0.0s
 => => writing image sha256:c536e74e0967b7344ff7988a302a7ae2068e7bdfcd00f  0.0s
 => => naming to docker.io/library/current-bench_frontend                  0.0s
Successfully built c536e74e0967b7344ff7988a302a7ae2068e7bdfcd00fa74233b8a508ada4d16
Building pipeline
failed to get console mode for stdout: The handle is invalid.
[+] Building 170.7s (22/22) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s 
 => => transferring dockerfile: 1.78kB                                     0.0s 
 => [internal] load .dockerignore                                          0.0s 
 => => transferring context: 2B                                            0.0s 
 => [internal] load metadata for docker.io/library/debian:stable-slim      0.9s 
 => [internal] load metadata for docker.io/ocaml/opam:debian-ocaml-4.12    0.9s 
 => [internal] load build context                                          0.0s 
 => => transferring context: 54.32kB                                       0.0s 
 => [build 1/8] FROM docker.io/ocaml/opam:debian-ocaml-4.12@sha256:e39cf  49.1s 
 => => resolve docker.io/ocaml/opam:debian-ocaml-4.12@sha256:e39cf5ddf42c  0.0s
 => => sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d130 50.43MB / 50.43MB  2.8s
 => => sha256:b704dfdc404aa80ee420ceb91d1974441bb41b 286.70MB / 286.70MB  20.9s
 => => sha256:dbb52d71bac56d7b6969ac8b0310d33588f700 189.72MB / 189.72MB  14.2s
 => => sha256:e39cf5ddf42ceff246eee0402466a81e47a1c29567a 1.66kB / 1.66kB  0.0s
 => => sha256:7671537e58b4c6a388caee4d445b9a16476fbb3eb6fd1d2 956B / 956B  0.0s
 => => sha256:f7c1669319a4c7bcd2269e33e7bf6950679512069 10.47kB / 10.47kB  0.0s
 => => extracting sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0  2.2s
 => => extracting sha256:b704dfdc404aa80ee420ceb91d1974441bb41b37fce81fc  20.2s
 => => extracting sha256:dbb52d71bac56d7b6969ac8b0310d33588f7009a3692b933  6.4s
 => [stage-1 1/8] FROM docker.io/library/debian:stable-slim@sha256:f31a1f  6.4s
 => => resolve docker.io/library/debian:stable-slim@sha256:f31a1f9e274ebf  0.0s
 => => sha256:f31a1f9e274ebf9a97b964c9aed3386e1720d56a3d2 1.85kB / 1.85kB  0.0s
 => => sha256:65c240929acadafb6cdf7447d7e3ba1abc214a8b008fe4e 529B / 529B  0.0s
 => => sha256:f052c451335940bea31ab0e58feff9fc657959c4915 1.46kB / 1.46kB  0.0s
 => => sha256:7fa62122c34635d215ecb61e3d515d9eb1676a0f2 27.14MB / 27.14MB  5.0s
 => => extracting sha256:7fa62122c34635d215ecb61e3d515d9eb1676a0f258d352e  1.3s
 => [stage-1 2/8] RUN apt-get update     && apt-get install --no-install  26.2s
 => [stage-1 3/8] WORKDIR /app                                             0.0s
 => [stage-1 4/8] COPY ./aslr_seccomp.json /app/aslr_seccomp.json          0.0s
 => [stage-1 5/8] COPY ./db/migrations /app/db/migrations                  0.0s
 => [build 2/8] RUN sudo apt-get update &&     sudo apt-get install -qq   16.0s
 => [build 3/8] WORKDIR /mnt/project                                       0.0s
 => [build 4/8] COPY --chown=opam:opam pipeline.opam pipeline.opam         0.0s
 => [build 5/8] RUN opam install -y --deps-only -t .                     100.1s
 => [build 6/8] COPY --chown=opam . .                                      0.0s
 => [build 7/8] RUN sudo chown opam .                                      0.5s
 => [build 8/8] RUN opam exec -- dune build --profile=release bin/main.ex  2.2s
 => [stage-1 6/8] COPY --from=build /home/opam/.opam/4.12/bin/omigrate /a  0.1s
 => [stage-1 7/8] COPY ./scripts/wait-for /app/bin/wait-for                0.0s
 => [stage-1 8/8] COPY --from=build /mnt/project/_build/default/bin/main.  0.1s
 => exporting to image                                                     1.5s
 => => exporting layers                                                    1.5s
 => => writing image sha256:943cfc7ba022701594600f0d26fd96ada9ff5fbbac566  0.0s
 => => naming to docker.io/library/current-bench_pipeline                  0.0s
Successfully built 943cfc7ba022701594600f0d26fd96ada9ff5fbbac566a9c7ad3c80558708e94
Creating current-bench_db_1 ... done
Creating current-bench_pipeline_1       ... done
Creating current-bench_db-migrate_1     ... done
Creating current-bench_graphql-engine_1 ... done
Creating current-bench_frontend_1       ... done
Attaching to current-bench_db_1, current-bench_graphql-engine_1, current-bench_db-migrate_1, current-bench_pipeline_1, current-bench_frontend_1
db-migrate_1      | standard_init_linux.go:219: exec user process caused: no such file or directory
db_1              | The files belonging to this database system will be owned by user "postgres".
db_1              | This user must also own the server process.
db_1              |
db_1              | The database cluster will be initialized with locale "en_US.utf8".
db_1              | The default database encoding has accordingly been set to "UTF8".
db_1              | The default text search configuration will be set to "english".
db_1              |
db_1              | Data page checksums are disabled.
db_1              |
db_1              | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1              | creating subdirectories ... ok
db_1              | selecting dynamic shared memory implementation ... posix
db_1              | selecting default max_connections ... 100
db_1              | selecting default shared_buffers ... 128MB
db_1              | selecting default time zone ... Etc/UTC
db_1              | creating configuration files ... ok
db_1              | running bootstrap script ... ok
db_1              | performing post-bootstrap initialization ... ok
db_1              | syncing data to disk ... ok
db_1              |
db_1              |
db_1              | Success. You can now start the database server using:
db_1              |
db_1              |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1              |
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":4000},"enabled_log_types":["http-log","websocket-log","startup","webhook-log","query-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":true,"port":8080}}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://docker:...@db:5432/docker"}}}
graphql-engine_1  | {"type":"pg-client","timestamp":"2021-04-02T20:12:48.629+0000","level":"warn","detail":{"message":"postgres 
connection failed, retrying(0)."}}
graphql-engine_1  | {"type":"pg-client","timestamp":"2021-04-02T20:12:48.629+0000","level":"warn","detail":{"message":"postgres 
connection failed, retrying(1)."}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"db\" (172.18.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
graphql-engine_1  | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"db\" (172.18.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}       
db_1              | initdb: warning: enabling "trust" authentication for local connections
db_1              | You can change this by editing pg_hba.conf or using the option -A, or
db_1              | --auth-local and --auth-host, the next time you run initdb.
pipeline_1        |    application [INFO] Logging ready.
db_1              | waiting for server to start....2021-04-02 20:12:48.618 UTC [47] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
pipeline_1        |    application [ERROR] Database connection error:
pipeline_1        | Connection failure: could not connect to server: Connection refused
pipeline_1        |     Is the server running on host "db" (172.18.0.2) and accepting
pipeline_1        |     TCP/IP connections on port 5432?
pipeline_1        |
db_1              | 2021-04-02 20:12:48.620 UTC [47] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:12:48.634 UTC [48] LOG:  database system was shut down at 2021-04-02 20:12:48 UTC
pipeline_1        | pipeline: internal error, uncaught exception:
pipeline_1        |           Postgresql.Error(_)
pipeline_1        |           Raised at Pipeline__Db_util.check_connection in file "lib/db_util.ml", line 10, characters 6-15   
pipeline_1        |           Called from Pipeline.v in file "lib/pipeline.ml", line 245, characters 2-36
pipeline_1        |           Called from Cmdliner_term.app.(fun) in file "cmdliner_term.ml", line 25, characters 19-24
pipeline_1        |           Called from Cmdliner.Term.run in file "cmdliner.ml", line 117, characters 32-39
db_1              | 2021-04-02 20:12:48.637 UTC [47] LOG:  database system is ready to accept connections
db_1              |  done
db_1              | server started
db_1              | CREATE DATABASE
db_1              |
db_1              |
db_1              | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
db_1              | GRANT
db_1              |
db_1              |
current-bench_db-migrate_1 exited with code 1
db_1              | 2021-04-02 20:12:48.919 UTC [47] LOG:  received fast shutdown request
db_1              | waiting for server to shut down....2021-04-02 20:12:48.921 UTC [47] LOG:  aborting any active transactions  
db_1              | 2021-04-02 20:12:48.922 UTC [47] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
db_1              | 2021-04-02 20:12:48.922 UTC [49] LOG:  shutting down
db_1              | 2021-04-02 20:12:48.935 UTC [47] LOG:  database system is shut down
db_1              |  done
db_1              | server stopped
db_1              |
db_1              | PostgreSQL init process complete; ready for start up.
db_1              |
db_1              | 2021-04-02 20:12:49.027 UTC [1] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bitgnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1              | 2021-04-02 20:12:49.028 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1              | 2021-04-02 20:12:49.028 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1              | 2021-04-02 20:12:49.033 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:12:49.043 UTC [84] LOG:  database system was shut down at 2021-04-02 20:12:48 UTC
db_1              | 2021-04-02 20:12:49.047 UTC [1] LOG:  database system is ready to accept connections

Any help would be greatly appreciated 🙏

Hard-Coded URI's

There are several hard-coded URI's in the frontend and the pipeline. These should be moved to configuration, but for now they are easy enough to change in my own fork. However, there is another source of hard-coded URI's that I can't locate. My production pipeline (installed as a GH app) is failing. It's giving this warning:

current.github [WARNING] Set d4hines/marigold-tezos
                           refs/heads/master
                           902910d2/ocaml-benchmarks to
                           {"state":"failure","description":"Command \"docker\" \"run\" \"--security-opt\" \"seccomp=./aslr_seccomp.json\" \n\"--tmpfs\" \"/dev/shm:rw,noexec,nosuid,size=4g\" \"--rm\" \"-i\" \"sha256:98a","target_url":"http://autumn.ocamllabs.io/d4hines/marigold-tezos"} failed: 403 Forbidden
                           {"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/reference/repos#create-a-commit-status"}

I can't track down where it's getting the URL http://autumn.ocamllabs.io/d4hines/marigold-tezos from. Any ideas?

Investigate why there are repeated entries for the same commit

And decide what to do with the repeated entries.

In-progress PRs are not shown on the prs page.

Support graph descriptions

Since all the graphs are displayed in the same way right now it could be useful with support for a description of each graph that is displayed below the title in the UI. It could then be possible to include additional information, such as a short explanation of what the graph shows, higher/lower is better etc.