Git Product home page Git Product logo

cloud-native-observability's Introduction

Cloud-Native Observability with OpenTelemetry

Cloud-Native Observability with OpenTelemetry

This is the code repository for Cloud-Native Observability with OpenTelemetry, published by Packt.

Learn to gain visibility into systems by combining tracing, metrics, and logging with OpenTelemetry

What is this book about?

Cloud-Native Observability with OpenTelemetry is a guide to helping you look for answers to questions about your applications. This book teaches you how to produce telemetry from your applications using an open standard to retain control of data. OpenTelemetry provides the tools necessary for you to gain visibility into the performance of your services. It allows you to instrument your application code through vendor-neutral APIs, libraries and tools.

This book covers the following exciting features:

  • Understand the core concepts of OpenTelemetry
  • Explore concepts in distributed tracing, metrics, and logging
  • Discover the APIs and SDKs necessary to instrument an application using OpenTelemetry
  • Explore what auto-instrumentation is and how it can help accelerate application instrumentation
  • Configure and deploy the OpenTelemetry Collector Get to grips with how different open-source backends can be used to analyze telemetry data Understand how to correlate telemetry in common scenarios to get to the root cause of a problem

If you feel this book is for you, get your copy today!

https://www.packtpub.com/

Instructions and Navigations

All of the code is organized into folders. For example, Chapter02.

The code will look like the following:

from opentelemetry._metrics import set_meter_provider
from opentelemetry.sdk._metrics import MeterProvider
from opentelemetry.sdk.resources import Resource

def configure_meter_provider():
  provider = MeterProvider(resource=Resource.create())
  set_meter_provider(provider)

if __name__ == "__main__":
  configure_meter_provider()

Following is what you need for this book: This book is for software engineers, library authors, and systems operators looking to better understand their infrastructure, services and applications by leveraging telemetry data like never before. Working knowledge of Python programming is assumed for the example applications that you’ll be building and instrumenting using the OpenTelemetry API and SDK. Some familiarity with Go programming, Linux, and Docker is preferable to help you set up additional components in various examples throughout the book.

With the following software and hardware list you can run all code files present in the book (Chapter 1-12).

Software and Hardware List

Chapter Software required OS required
1-12 OpenTelemetry for Python 1.9.0+ Windows, Mac OS X, and Linux (Any)
1-12 OpenTelemetry Collector v0.42.0+ Windows, Mac OS X, and Linux (Any)

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. Click here to download it.

Errata

  • Page 29 (Figure 2.1): The port for the legacy-inventory should say 5001
  • Page 55 (Figure 2.5): The bubble between inventory API and inventory store should be labeled 4 instead of 3.

Related products

Get to Know the Author

Alex Boten is a senior staff software engineer at Lightstep and has spent the last 10 years helping organizations adapt to a cloud-native landscape. From building core network infrastructure to mobile client applications and everything in between, Alex has first-hand knowledge of how complex troubleshooting distributed applications is. This led him to the domain of observability and contributing to open source projects in the space. A contributor, approver, and maintainer in several aspects of OpenTelemetry, Alex has helped evolve the project from its early days in 2019 into the massive community effort that it is today. More than anything, Alex loves making sense of the technology around us and sharing his learnings with others.

Download a free PDF

If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF.

https://packt.link/free-ebook/9781801077705

cloud-native-observability's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

cloud-native-observability's Issues

/usr/bin/env: ‘python3\r’: No such file or directory in chapter03

Hello,

Only to know, I just execute the chapter03 on Windows with WSL (Ubuntu) and I have this error when trying to start the brokentelephone-python container.

/usr/bin/env: ‘python3\r’: No such file or directory

I have had to execute dos2unix command in the docker-compose file and python files to change the ending characters and rebuild again and its works.

macos monterry: port 5000 in use by airplay receiver

Less of an issue and more of an FYI....

In case others hit this problem and are not aware, port 5000 is now in use by airplay receiver in this version of MacOS. One can only assume nobody at Apple uses Flask?

The easiest fix is to use a different port. However, you can also go into the sharing systems preference and uncheck the airplay receiver.

More detail here: https://medium.com/pythonistas/port-5000-already-in-use-macos-monterey-issue-d86b02edd36c

chapter 02: resource limits for grocery-store and inventory too stringent

Both containers start and stop.

Had to raise the memory limit (set at 50M) to 80M to have them running. Below is a docker stats view to show the actual usage is above the limits set in the docker-compose.yml

CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
7afa474a8f1b   shopper                   0.69%     32.3MiB / 30.76GiB    0.10%     59.4kB / 106kB    0B / 360kB       8
b6fe39fa5ef5   grocery-store             4.28%     66.84MiB / 80MiB      83.55%    104kB / 153kB     0B / 381kB       17
c414d81d1f71   inventory                 4.45%     60.35MiB / 80MiB      75.44%    65.9kB / 97.6kB   0B / 385kB       17
6e0675111661   opentelemetry-collector   0.56%     42.8MiB / 30.76GiB    0.14%     743kB / 977kB     946kB / 0B       21
2325027f0e93   promtail                  0.90%     18.57MiB / 30.76GiB   0.06%     14.6kB / 35.7kB   209kB / 102kB    21
d24b5a7fc636   prometheus                0.00%     31.75MiB / 30.76GiB   0.10%     233kB / 1.98MB    807kB / 266kB    21
dcf11c6fe1f5   loki                      0.34%     23.09MiB / 30.76GiB   0.07%     163kB / 23.9kB    287kB / 49.2kB   20
3ba52c18e8da   jaeger                    0.22%     16.91MiB / 30.76GiB   0.05%     604kB / 1.5MB     20.5kB / 0B      21
6a9886319e0c   grafana                   0.06%     27.5MiB / 30.76GiB    0.09%     22.8kB / 4.62kB   426kB / 184kB    21

Images for Mac M1

Could you make the images available for arm architecture (Mac M1)?
In chapter 2 the shopper, grocery-store and legacy-inventory images are not running in my environment.

Logger initialization failure on Chapter 07 - grocery_store.py

As OTLPHandler was already changed to LogginHandler, I'm trying to update grocery_store.py code on Chapter 07 to working with new handler.
I'm using opentelemetry-sdk and opentelemetry-api version 1.18.0, which looks stable version at the moment.
I've updated code like as below, but resulted in logger provider initializing error.

Code changed (import statement and dictConfig object statement)

import requests
from flask import Flask
from opentelemetry.instrumentation.wsgi import OpenTelemetryMiddleware

dictConfig(
    {
        "version": 1,
        "handlers": {
            "otlp": {
                "class": "opentelemetry.sdk._logs.LoggingHandler"
            }
        },
        "root": {"level": "DEBUG", "handlers": ["otlp"]}
    }
)

app = Flask(__name__)
app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app)

@app.route("/")
def welcome():
    return "Welcome to the grocery store!"

@app.route("/products")
def products():
    url = "http://localhost:5001/inventory"
    resp = requests.get(url)
    return resp.text

if __name__ == "__main__":
    app.run(port=5000)

What I tried env variables are as below

OTEL_RESOURCE_ATTRIBUTES="service.name=grocery-store,
                       service.version=0.1.2,
                       net.host.name='hostname',
                       net.host.ip='ipconfig getifaddr en0'" \
  OTEL_TRACES_EXPORTER=console \
  OTEL_PYTHON_TRACER_PROVIDER=sdk \
  OTEL_METRICS_EXPORTER=console \
  OTEL_PYTHON_METER_PROVIDER=sdk \
  OTEL_LOGS_EXPORTER=console \
  OTEL_PYTHON_LOGGER_PROVIDER=sdk \
  OTEL_PROPAGATORS=b3,tracecontext \
  opentelemetry-instrument python grocery_store.py

And main part of runtime error shows Failed to load configured provider logger_provider and here's full error messages.

Failed to load configured provider logger_provider
Traceback (most recent call last):
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/cloud_native_observability/lib/python3.9/site-packages/opentelemetry/util/_providers.py", line 43, in _load_provider
    next(  # type: ignore
StopIteration
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/config.py", line 564, in configure
    handler = self.configure_handler(handlers[name])
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/config.py", line 745, in configure_handler
    result = factory(**kwargs)
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/cloud_native_observability/lib/python3.9/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 330, in __init__
    self._logger_provider = logger_provider or get_logger_provider()
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/cloud_native_observability/lib/python3.9/site-packages/opentelemetry/_logs/_internal/__init__.py", line 182, in get_logger_provider
    logger_provider: LoggerProvider = _load_provider(  # type: ignore
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/cloud_native_observability/lib/python3.9/site-packages/opentelemetry/util/_providers.py", line 43, in _load_provider
    next(  # type: ignore
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/ch04/grocery_store.py", line 7, in <module>
    dictConfig(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/config.py", line 809, in dictConfig
    dictConfigClass(config).configure()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/config.py", line 571, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'otlp'
{
    "resource_metrics": [
        {
            "resource": {
                "attributes": {
                    "telemetry.sdk.language": "python",
                    "telemetry.sdk.name": "opentelemetry",
                    "telemetry.sdk.version": "1.18.0",
                    "service.name": "grocery-store",
                    "service.version": "0.1.2",
                    "net.host.name": "'hostname'",
                    "net.host.ip": "'ipconfig getifaddr en0'",
                    "telemetry.auto.version": "0.39b0"
                },
                "schema_url": ""
            },
            "scope_metrics": [],
            "schema_url": ""
        }
    ]
}
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 2141, in shutdown
    h.flush()
  File "/Users/nopd/dev/opentelemetry_book/Cloud-Native-Observability/cloud_native_observability/lib/python3.9/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 388, in flush
    self._logger_provider.force_flush()
AttributeError: 'LoggingHandler' object has no attribute '_logger_provider'

I'm not quiet sure why initialization is failing.

troubleshooting chapter08 with opentelemetry-sdk 1.12.0 and opentelemetry-exporter-otlp 1.12.0

Hello 👋 just adding a friendly note here that if you're also running the book examples with the latest OTel libraries (1.12.0 as opposed to 1.9.0 in the book) you may need to make some quick changes

in common.py change:

  • from opentelemetry.sdk._logs import OTLPHandler is now LoggingHandler
  • from opentelemetry.exporter.otlp.proto.grpc._metric_exporter is now from opentelemetry.exporter.otlp.proto.grpc.metric_exporter

in grocery_store.py, also change references of OTLPHandler to LoggingHandler

chapter 02: Application exit due to OOM on MacBook Pro M1

I think it's fine to re-open this issue #19. I am using Macbook Pro M1 16' with Ventura 13.2.1.

It seems the example application is consuming more resources than expected. They are easy to exit due to OOM.

I think it's more likely to be the issue of the M1 processor and arm64/amd64. The application itself should be fine enough to be an example.

So it might be better to demonstrate the known issue in README.md in chapter02, and lift the memory limit a little bit higher.

CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
55b6edc83153   shopper                   0.51%     84.52MiB / 3.841GiB   2.15%     682kB / 1.41MB    0B / 360kB       9
c8bba4c53200   grocery-store             14.43%    209.1MiB / 256MiB     81.70%    1.21MB / 1.91MB   0B / 381kB       19
aeba0817fdbe   inventory                 12.79%    199.7MiB / 256MiB     78.01%    739kB / 1.19MB    190kB / 385kB    19
0abd7428313d   grafana                   0.03%     25.57MiB / 3.841GiB   0.65%     27.9kB / 9.72kB   594kB / 217kB    9
90f93f3fe692   promtail                  0.97%     14.59MiB / 3.841GiB   0.37%     3.24kB / 1.82kB   307kB / 377kB    11
2785d432f8e2   prometheus                0.69%     25.89MiB / 3.841GiB   0.66%     654kB / 141kB     36.9kB / 463kB   11
a59b518fbc3f   opentelemetry-collector   1.52%     37.02MiB / 3.841GiB   0.94%     2.65MB / 3.48MB   0B / 0B          10
c3bda125a3b1   loki                      0.13%     20.86MiB / 3.841GiB   0.53%     429kB / 73.5kB    184kB / 106kB    9
48f806f3fdc3   jaeger                    0.21%     21.04MiB / 3.841GiB   0.53%     2.04MB / 187kB    0B / 0B          10
docker version
➜  chapter02 git:(main) ✗ docker version
Client:
 Cloud integration: v1.0.31
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:21:21 2023
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.18.0 (104112)
 Engine:
  Version:          20.10.24
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.7
  Git commit:       5d6db84
  Built:            Tue Apr  4 18:17:07 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker compose version
➜  chapter02 git:(main) ✗ docker compose version
Docker Compose version v2.17.2

chapter02 docker-compose error

docker-compose up Error Message
ERROR: for opentelemetry-collector Cannot create container for service opentelemetry-collector: json: cannot unmarshal number 0.1 into Go struct field ContainerConfigWrapper.StopTimeout of type int
ERROR: compose.cli.main.exit_with_metrics: Encountered errors while bringing up the project.

docker-compose --version
docker-compose version 1.29.2, build unknown

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.