This enables the common desktop app crash reporting semantics of prompting the user once a crash has occurred to check if they would like to report the issue back to the developer.

Some of our customers build software that's intended to run in CI/CD pipelines. It is important for them to have a good sense of which CI/CD tools their users are using their software in.

Most of our customers would also like to be able to filter on events fired from CI/CD or filter them out of their analyses.

I propose that, when we generate system tags, we try to detect whether we are running in a CI/CD environment and add the (inferred) name of the CI/CD provider (e.g. GitHub Actions, CircleCI, TravisCI, etc.) as a tag of the form ci:<provider>.

Each CI/CD platform has certain environment variables that it uses to configure internal processes:

1. GitHub Actions
2. Circle CI
3. Travis CI

Note that all of them seem to set CI=true. So at the very least we should use that. We can also use the presence or absence of other environment variables to infer which specific CI provider we are working with.

These should be function/method decorators. They should allow the caller to specify that certain kinds of information get reported when a function is called:

1. That the function was called (with args + kwargs?)
2. If there was an exception raised by the function
3. When the function returns (with return value?)

The function should have a signature like this:

func (reporter *HumbugReporter) PublishAsync(report Report) (func, <-bool) {
    ...
}

The return values should be:

1. A function with no arguments and no return which can be run as a goroutine. when run, this routine will publish the given report.
2. A channel to which the go routine will write true when publication is complete.

Instead of making this a method on HumbugReporter, we could also implement it with signature:

func PublishAsync(reporter *HumbugReporter, report Report) (func, <-bool) {
    ...
}

Add a mode which gives callers a job queue to submit reports (rather than request/response).

For people who want to opt out of any Humbug reporting across any tool with a single setting.

All of the methods for HumbugReporter work on a *HumbugReporter. They should all do a nil check and, if nil, should print a warning and noop.

This will allow Humbug customers to hide sensitive parameter values from usage reports.

Currently, we have no way of grouping errors together (e.g. by name in Python) in the Javascript library.

The tokens being used for reporting do not have the full power of Bugout API tokens, and the naming should reflect this.

Relevant line of code:

User request was to identify when Humbug is run in colab. There are a few environment variables available there that will help us. E.g. COLAB_GPU.

For other Jupyter environments, this may be a little tougher. Requires some investigation on Kaggle and in local Jupyter notebooks and in Jupyter lab.

Consent is checked on every report publication anyway, so it makes sense for consent.ConsentMechanism functions to take an optional Report argument. This can be used to present users with the contents of the report for approval, so they can see exactly what they are consenting to.

Future flows along these lines: users are presented with a raw report and they can modify it prior to giving consent.

Something to think about, the only change required here would be to change the definition of ConsentMechanism from Callable[[], bool] to Callable[[Optional[Report]], bool].

As it stands now, this would introduce a circular dependency between humbug.consent and humbug.report, so we would have to move Report into another file (e.g. data.py).

We should clean up paths (for example in stack traces) which look like /home/zomglings/blahblahblah to not include things like username.

One way to make progress in Python is to make all paths relative to current directory or to project root directory.

For example, system report + error report.

e.g.

consent.yes = ["1", "y", "Y", ...]

and

consent.no = ["0", "n", "N", ...]

Some options. We should implement at least one of the following options, but we do not have to implement them all.

Using inspect

Here is an example of how to detect this automatically:

def is_in_library():
    stack = inspect.stack()[:-1]
    if all("site-packages" not in finfo.filename for finfo in stack):
        print("(debug) Not in library!")
        return False
    return True  # don’t send statistics!

Taken from Ray telemetry proposal: https://docs.google.com/document/d/1gZut2v52xDd3bNBaw2PxilUBWQRixIQcOJPx16FIoAw/edit#

Now, I don't like using inspect.stack (see this Stack Overflow post and this Python mailing list thread for some reasons why).

However, we can implement this and leave it up to the creator of the consent mechanism whether or not they should use it.

Stateful consent switch - off when used as library, on when used as standalone tool

Create a stateful consent mechanism with a boolean switch. By default, the switch will be False and the mechanism will not grant consent. This way, if a tool which integrates with Humbug can be used as a library, any dependents of that library will not send reporting.

However, if the tool can also be used from the command line, the CLI can switch the state of the mechanism to True to enable reporting.

On GitHub Actions, we can use a matrix.

The way HumbugConsent is implemented in the javascript library doesn't allow for rich consent flows which resolve consent dynamically at runtime. They only allow for consent resolution at the time that the reporter is created.

Make it look like the Python or Go version of HumbugConsent.

It should be possible to define a user prompt (using humbug.consent.prompt_user) with a default value and a timeout period. If the prompt times out, consent should move forward with the default value.

Our Python 3.6 tests did not catch the issue that dataclasses is not a standard package.

(See this PR: #84)

It means that we don't have any tests for HumbugReporter. We need to add such tests.

Current, error reports only take the additional tag: type:error.

This doesn't help to understand the distribution of specific errors (for example ValueErorr in Python).

We should also add the name of the error as a tag. For example, in Python, any ValueError reported to Bugout should also get the tag: error:ValueError.

For errors that are imported from other libraries (e.g. fastapi.HTTPException), we should try to add the fully qualified error path: error:fastapi.HTTPException rather than error:HTTPException).

For errors defined in the package integrating with Humbug, we can use relative paths to start with.

See: https://github.com/bugout-dev/moonstream/blob/dd9954094ccaee2dc6c5d51121f84f37e097fff5/backend/setup.py#L21

This can be done by creating a custom_report method and using the composition to be implemented in #16

See the workflow in the recipe: https://github.com/bugout-dev/humbug/blob/main/python/recipes/error_reporting.py#L89

Only caveat is that you need to store a reference to the original excepthook and call it after you do whatever custom stuff is required for the Reporter part of the excepthook.

The sample code handles it correctly, so you can just follow that pattern.

https://github.com/bugout-dev/humbug/blob/main/python/humbug/report.py#L128

See comment by u/ElevenPhonons on Reddit: https://www.reddit.com/r/Python/comments/m84h10/humbug_usage_and_crash_reports_for_python/grh7ads?utm_source=share&utm_medium=web2x&context=3

We should at least handle KeyboardInterrupt in a more responsible manner than just eating them up. This should probably apply to everything that gets caught that isn't a subclass of Exception in the exception hierarchy: https://docs.python.org/3/library/exceptions.html#exception-hierarchy

See: https://consoledonottrack.com/

We have already implemented BUGGER_OFF=1. We should just add DO_NOT_TRACK=1 as well with the same functionality.

We have to do this in every client library.

This is a report from a user. I will verify the issue and add the stack trace here.

For Python, basically a report which sends a pip freeze.