Create a custom trace exporter than prints direct links to traces in Honeycomb UI. This exporter can be enabled either via option or env var, with the default set to off.

Option name: EnableLocalVisualizations
Env var: HONEYCOMB_LOCAL_VISUALIZATIONS

It might be nice to have type annotations and run a type checker setup for once we get into custom ad-ons to the base distro. This might take some discovery, I haven't gotten a chance to use python types before but I'd love to if we can!

If we decide to use mypy here's a cheatsheet and don't forget to include a py.typed file for packaging. (poetry may do this by default?)

I'm not sure if this is something off with how I'm holding it, but after #56 , I cannot get the distro to work properly. It says it cannot find module honeycomb. If I revert and move honeycomb/opentelemetry out of the src folder, and update the pyproject package reference, it works as expected.

To reproduce, it may be necessary to remove the __pycache__ and reinstall. If so as a follow-up I'd like to add that to the Makefile as well, a clean target that removes pycache and dist.

When running tests, this error shows up in the console:

Failed to export metrics, error code: StatusCode.UNAUTHENTICATED
Failed to export metrics, error code: StatusCode.UNAUTHENTICATED
Failed to export traces, error code: StatusCode.UNAUTHENTICATED
Failed to export traces, error code: StatusCode.UNAUTHENTICATED

Versions
v0.1.2b0

Steps to reproduce

I am trying to run a Honeycomb + OpenTelemetry instrumented Flask app following the documentation on the docs here, as of April 4th 2023.
https://docs.honeycomb.io/getting-data-in/opentelemetry/python-distro/#automatic-instrumentation

I have built a Docker image as described here:

Dockerfile:

FROM python:3.9

COPY ./requirements.txt /app/requirements.txt

WORKDIR /app

RUN pip install -r requirements.txt
RUN pip install honeycomb-opentelemetry --pre
RUN opentelemetry-bootstrap --action=install

COPY . /app

ENTRYPOINT ["opentelemetry-instrument", "python","/app/app.py"]
EXPOSE 5000

requirements.txt:

Flask==2.0.2

app.py:

from flask import Flask, jsonify, request
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

# Set up tracer
resource = Resource.create({"service.name": "my-service"})
trace.set_tracer_provider(TracerProvider(resource=resource))

# Define routes for the Flask app
app = Flask(__name__)


@app.route("/")
def index():
    return "Welcome to my Flask app!"


@app.route("/hello")
def hello():
    return jsonify({"message": "Hello, world!"})


@app.route("/user/<username>")
def user_profile(username):
    user_agent = request.headers.get("User-Agent")
    with trace.get_tracer(__name__).start_as_current_span("user_profile"):
        # Do some work here...
        return jsonify({"username": username, "user_agent": user_agent})


if __name__ == "__main__":
    app.run(host='0.0.0.0')

When running the application with this command, with Honeycomb debugging active:
docker container run -p 5000:5000 -e DEBUG=true flask_test

the following inconsistency with versions appears even before Flask starts:

Distribution distro configuration failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 37, in _load_distros
    distro = entry_point.load()()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2449, in load
    self.require(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2472, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (opentelemetry-sdk 1.16.0 (/usr/local/lib/python3.9/site-packages), Requirement.parse('opentelemetry-sdk~=1.17.0'), {'opentelemetry-exporter-otlp-proto-grpc', 'opentelemetry-exporter-otlp-proto-
http'})
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 119, in initialize
    distro = _load_distros()
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 52, in _load_distros
    raise exc
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 37, in _load_distros
    distro = entry_point.load()()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2449, in load
    self.require(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2472, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (opentelemetry-sdk 1.16.0 (/usr/local/lib/python3.9/site-packages), Requirement.parse('opentelemetry-sdk~=1.17.0'), {'opentelemetry-exporter-otlp-proto-grpc', 'opentelemetry-exporter-otlp-proto-
http'})

WIP PR: https://github.com/honeycombio/hound/pull/13069

Django's a thing. I think in our docs we should at least acknowledge it and mention or link to the appropriate instrumentation to install for Django apps. Probably not worth a whole-hog sample and docs section though.

We have a dockerized version of the plain python app, but this ticket is to dockerize the flask app to test more instrumentation and baggage with smoke tests

Versions

current

Steps to reproduce

1. run the flask app, see no automatic instrumentation for flask.

Additional context

The flask app should be automatically instrumented with the opentelemetry-instrument command line argument, but only manual instrumentation is being picked up. Smoke tests for the flask app should also be updated to include tests for auto-instr spans.

Create a new trace sampler that is based on the parent based sampler & ratio based sampler and returns appends the honeycomb sample rate as an additional attribute. This sampler should be added by to the tracer provider and use the configured sample rate from the HoneycombOptions class.

• Create new sampler
• Add sampler as part of tracer provider creation

Is your feature request related to a problem? Please describe.
Add makefile commands for the most common actions. See DEVELOPING for commands.

✔ Project Setup

• #4
  • pyproject setup; poetry
  • linting: pylint, pycodestyle
  • testing suite setup hello world; coverage, pytest
  • readme dev instructions
• #20
• #2
• #5

✔ Instrumentation Setup - Tracing

• #10
• #11
• #12
• #13
• #14
• #98

✔ Project Maintenance / Hygiene

• #38
• #31
• #64
• #84

✔ Honeycomb Add-ons

• #15
• #16
• #17
• #18

✔ Packaging

• #6
  • explore packaging a py package locally
  • try it out in EGS
• #7
  • setup packaging and release with circleCI + PyPI
  • bump2version like libhoney-py?
  • releasing.md

✔ Smoke Tests

• #72
• #73
• #74
• #75

Documentation

• #76
• #77
• #54
• #78
• #79
• #80
• #88
• #67
• #57

TODO:

README:

• include minimum supported Python (like upstream) and a link to DEVELOPING if the reader wants to tinker
  • #87
• describe (very high level diffs) of examples and link to their READMEs
  • #78
  • #105

DEVELOPING:

• replace poetry commands with make commands for a dev experience consistent with our other projects and CI
  • #87

EXAMPLES READMES:

• use poetry commands in the examples READMEs to speak to a Pythony audience that don't care about our dev workflow

Our logging is currently done using print(). We should instead use the defacto Logging packages.

Is your feature request related to a problem? Please describe.
There are a lot of linting errors when running make lint. These should be resolved before enabling the lint to run in CI in #31.

We'd like the project to be set up for local development, this includes:

• pyproject setup and poetry
• a system for linting. see: pylint, pycodestyle
• a testing suite setup with a hello world example; code coverage
• and a readme / dev instructions for each of these aspects!

See also #2 required for local setup for development and troubleshooting.

We initially built this with a Parent-based approach to sampling, however in discussion there have been several shortcomings of this approach.

• Parent based would potentially make a decision that differs from the sample rate we're setting as an attribute
• We intentionally removed the parent-based from java... "parent-based sampler would not work because we need to know what SampleRate to set"
• If we are to follow the behavior in our other distros, we should not be considering a parent span's sampling decision when sampling.

This is a ticket to discuss :) See if these shortcomings still exist. Once we decide and document all concerns, PR #96 is available to toggle it off if needed!

Is your feature request related to a problem? Please describe.

Unexpected behavior and failing tests when running tests with some environment variables set locally.

Describe the solution you'd like

Find a way to clear environment variables before test runs, similar to how we used the setup-jest file for the node distro.

Describe alternatives you've considered

currently just manually unsetting environment variables

Additional context

Something like monkeypatching environment variables seems to work but it involves a lot of extra copy/paste:

import pytest

@pytest.fixture
def mock_env_missing(monkeypatch):
    monkeypatch.delenv("OTEL_SERVICE_NAME", raising=False)

def test_defaults(mock_env_missing):

...
def test_can_set_service_name_with_envvar(monkeypatch):

In #5 we set up the steps to run linting, but found it to be too cumbersome at this stage of development.

We should modify pylint to do what we'd like for linting, and ignore any rules we don't really care about.

Once we get to the "Honey add ons" section of the work, we can re-enable and require linting. I imagine this goes hand-in-hand with also setting up types #19 and docstrings.

When DEBUG=true, a warning shows in the console that an invalid a protocol is detected provided:

WARNING:honeycomb.opentelemetry.options:Invalid OTLP exporter protocol detected. Must be one of ['grpc', 'http/protbuf']. Defaulting to grpc

This warning should not be displaying if I have not set any specific protocol. Is the warning valid? If so, is our protocol not being set properly?

Steps to reproduce

1. export HONEYCOMB_API_KEY=mykey
2. export DEBUG=true
3. make run_example

• Requirements
• Examples
• Initialize
• Automatic Instrumentation
  • Install Packages
    • double-check there are no dependency conflicts with installing latest release
  • Configure and Run
  • GRPC vs HTTP
  • Advanced Configuration
• Manual Instrumentation
  • Creating Spans
  • Adding Attributes to Spans
  • Propagating Baggage
• Sampling
  • Deterministic Sampler, Sample Rate Attribute.
• Distributed Trace Propagation
• Troubleshooting
  • (ensure existing sections are still relevant)
  • Exporting to the console

https://github.com/honeycombio/docs/pull/1684

We noticed in setting up the smoke tests that setting a HONEYCOMB_API_ENDPOINT isn't picked up from the opentelemetry-instrument command that parses that distro options. #91 (comment)

Setting this as an environment variable should overwrite the DEFAULT_API_ENDPOINT.

We've got coverage in as a dev dependency.

Options:

1. Leave it as an interesting easter egg for a developer to discover.
2. Add coverage reporting to the test suite and, in CI, upload the result as an HTML report in a test job's artifacts.
3. Remove coverage as a dependency.

Update HoneycombOptions to add the following options and include trace & metrics alternatives as appropriate.

• API key
• Traces API key
• Metrics API key
• Endpoint
• Traces Endpoint
• Metrics Endpoint
• Dataset
• Traces Dataset
• Metrics Dataset
• SampleRate
• ServiceName
• ServiceVersion
• LocalVisualizations
• Debug

Collection team recently added the ability to auto-instrument Python using the OTel Operator. We should validate that they can use the Pythin Distro to add manual instrumentation in that use case. (@TylerHelmuth has context on using the operator)

Outcome: we can use the distro with operator! --> file a follow-up issue to add docs (post python distro beta)
Outcome: we can't use the distro with operator :( --> file a follow-up issue to make it work as part of the python distro beta release

Add option to enable HTTP/protobuf based exporters instead of the default gRPC exporters. The protocol can be updated uing the following environment variables.

Supported values are grpc & http/protobuf with grpc being default.

See OpenTelemetry Protocol Exporter for more details.

• Configure tracer provider exporter using OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
• Configure metrics provider exporter using OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL

We'd like to have an example flask app in this repo that does the following:

• uses latest code version of the Honey SDK (local dependency)
• exercises at least one auto-instrumentation
• creates at least one manual span
• has a curl-able endpoint that generates telemetry
• can be configured to send traces to HNY or to a local console

I run with opentelemetry-instrument, it doesn’t pick up my code configuration in this app.

poetry run opentelemetry-instrument python3 app.py results in unknown_service:python and debug is not true.

poetry run python3 app.py correctly picks up those env vars 🤔

I guess the question is… should it? Or does using opentelemetry-instrument mean you can only use env vars and not config in code with configure_opentelemetry?

While reviewing some Python PRs, @JamieDanielson and I came across a couple of things with the example app that could be improved or maybe we were missing something! Wanted to log this issue before the holidays so that everything isn't washed away in January

1. Get the example app to use our distro.
2. The command poetry run opentelemetry-instrument flask run runs the app but results in some errors from OTel when traces are being sent

Transient error StatusCode.UNAVAILABLE encountered while exporting traces, retrying in 1s.

Setup a config.yml for circleCI that runs

• the tests
• and lint
• and codestyle.

We also would like:

• the tests to have a test summary
• and the lint and codestyle to be relevant to our project

• explain the differences in examples (in the main readme? and also in example readmes)

• flask --- uses environment variable options and opentelemetry-instrument

  • has the baggage examples for context vs token

• hello-world --- uses configure_opentelemetry and options parameters

• Add an example of local exporter.

• Use MOAR options

• Metrics?

Please move the example info from developing.md to readme.md

Ensure docs describe both auto and manual SDK configuration using the distro.

For example:

• Auto instrumentation with opentelemetry-instrument
• Manual configuration with configure_opentelemetry

Note: our hello-world-flask example only show auto instrumentation.

Create a new span processor that sets all currently set attrbutes from trace context baggage on newly started spans. This span procesor should also be added to the tracer provider.

• Create new baggage span processor
• Add processor to tracer provider

Is your feature request related to a problem? Please describe.

We want to exercise the new distro in EGS!

Describe the solution you'd like

Switch a python service in EGS to use the Python Distro.

Audit:

• Options - TODO fallback,
• Sampler - docstring for the class
• Distro
• Local Exporter - use logger rather than print
• Baggage
• Resource
• Trace (commit, and it's done)

We'd like to follow good, modern python packaging guidelines so let's explore what those are.
Before setting packaging and releasing up on PyPI (#7) we're going to want to test it out locally.

This ticket might be done when:

• We can check out how the packaging works locally and test against it
• You feel good about the experience of including this module in external code (i.e. EGS)

This might be a good place to start? https://packaging.python.org/en/latest/tutorials/packaging-projects/.
It's worth noting that we're using poetry so best practices might differ from pip recommendations.

In some following scenarios, we should print warnings for the user to inform them of bad / invalid configuration:

• No Honeycomb API key set
• Legacy API key and no dataset us set
• New API key and dataset is set

Note: Depoends on the following PR to merge

• #37