eclipse / microprofile-service-mesh Goto Github PK

I'm experimenting with OpenTracing and Istio and I'd like to understand whether the results I'm seeing are the expected behavior with regard to how headers and spans are propagated in the context of Istio with the JAX-RS REST client. In short, I have three microservices, each one in an OpenLiberty container (with sidecar). The call sequence is:
ingress-gateway --> A --> B --> C. When viewing the spans in Jaeger or Zipkin I expect to see a hierarchical view, with a "tree" style visualization showing at a glance what the call sequence looks like. Here's what I see in my experiment - showing the same trace in both Jaeger and then Zipkin as I have both running in the cluster:

In Zipkin:

(In this example, the "instrument-craft-shop" service makes two POST calls to "maker-bot" which in turn makes POST calls to the "dbwrapper" service.)

The Bookinfo example (https://istio.io/docs/tasks/telemetry/distributed-tracing/) shows a clear parent/child relationship between services that are calling each other - is that possible to achieve with OpenTracing/Microprofile and JAX-RS client? (Jaeger shows this kind of hierarchy only within the istio-* services, while Zipkin does not.) When I look at headers I see that x-b3-traceid is preserved across all calls in the sequence but x-b3-parentspanid is unique for every call.

The code is available here https://github.com/ykoyfman/tracing-demo (the places where the REST client is being used is https://github.com/ykoyfman/tracing-demo/blob/master/maker-bot/src/main/java/com/sebastian_daschner/maker_bot/Persist.java and in https://github.com/ykoyfman/tracing-demo/blob/master/instrument-craft-shop/src/main/java/com/sebastian_daschner/instrument_craft_shop/control/MakerBot.java. I did manually add the x-request-id header to all calls, following the bookinfo example of header propagation.

Istio 1.0.6
OpenLiberty 19.0.0.2 / mpOpenTracing-1.2

Multiple deployments of service b should be created against Open Liberty, Thorntail, Payara, KumuluzEE and TomEE. In the Istio config rules, we should configure a round-robin rule, so that the services in different docker image can be used to demonstrate the portable aspect of MicroProfile programming model.

Create maven project and spec sub model to ensure this project can build, so that further spec and configure rules can be added.

In reviewing the various service mesh efforts there are logical abstractions that have overloaded meanings when you speak of a particular concrete notion in a given part of a targeted service mesh. The concept of a service being one example where it can be the actual business logic endpoint deployed to a runtime, a set of replicated container runtimes and access policy, and a DNS style of name.

The goal is to define a service mesh API/SPI that allows enterprise developers to continue to focus on business logic concerns exposed as a set of endpoints via a deployment containing configuration and metadata that is used to wire the service into a mesh by a devop/administrator who is responsible for completing the various security, reliability, performance and monitoring requirements. The MicroProfile runtimes will need to bridge between the target service mesh and the MicroProfile service deployment to avoid leakage of the target mesh details into the MicroProfile service deployment.

I'll have some form of initial working draft by the Feb 5 kickoff meeting.

We need to create Fault Injection rules between service a and b invocation to simulate the request invocation failure.

We need to investigate whether we need to do any interaction between reactive spec and Istio architecture.

Update the deployment yaml to use MicroProfile Health 2.0

Create 2 versions of service B (e.g. one optimised for IE6 and another one for other browsers). Then use Istio destination rules to route traffic to those.

Will MP LRA has any impact on Istio? Hopefully not. We need to demonstrate this feature on Istio platform to make sure.

Istio and MicroProfile both can do Fault Tolerance
Istio can perform Retry - if both enabled, there will be multiplication of retries. e.g. MP specify 3 retries and Istio configures to do 4 retries. The invocation will do 12 retries
Istio can perform circuit breaker, similar to MP Circuit break. Both circuit break sometimes do not work well.
Istio can control the maximum number of concurrent requests, which is part of Bulkhead. Similar to semaphore style of bulkhead.

At the moment, if both Istio and MP FT are enabled, the behavior is confusing.

In some cases, the apps use libraries that use HTTPS only. In this cases, Istio can only report and control source and destination IP, and also SNI. Istio cannot report and control paths, headers, methods, content (L7).

If Istio will be able to perform TLS interception, Istio will be able to provide full support for HTTPS traffic.

MicroProfile Config can understand URL configuration.
Service Meshes provide some service discovery mechanisms.

It would be good if there were a default config source of type ServiceMesh (for example) which could hook in to the service discovery and inject the URL of a dependent service. I'm not sure yet exactly how this would work (or the feasibility) but something like:

    @Inject
    @ConfigProperty(name="microprofile.service.myAppName")
    private URL myAppEndpoint;

Here, the name would be a standard, well-known format which would be reliably and predictably derived from the Service Mesh, but also be in a format which can be set by a system property in a dev/test environment which may not be in a Service Mesh.

While JWT is an ongoing work in progress in Istio, there is the question of how service identity as well as end user identity fits in with the current MP-JWT RBAC access.

How can a MicroProfile service have visibility into the service mesh environment configmap and secrets information associated with the service.

We need to create Istio fault tolerance config rule to do Retry, Circuit breaker, limit the concurrent requests, timeout. set the MP_Fault_Tolerance_NonFallback_Enabled in configmap to disable MP FT annotations except fallback.

Related to #8.

If using Rest Client API to call into other services, maybe Istio can utilise this.

Scott said that this is the area we need to look into more. Header propagation might be configured from here.

MP Health check should be used by Kube's health check as MP Health check is more accurate.

Metrics provides Prometheus format metrics.
Istio provides an optional Prometheus instance for metrics.

Does Metrics need to provide anything so that Prometheus in a k8s/Service Mesh environment can play nicely? Metrics currently publishes metrics to the endpoint that the default Prometheus scraper already expects, so outside of a service mesh environment, the configuration is very simple. It may be that this is already optimal.

With mTLS enabled the only way for the endpoint to be accessible is by adding an Ingress Gateway rule for it

How to push open tracing semantics to Istio system

Istio has to require headers in application code. MicorProfile open tracing is the ideal place to add headers??? tracing header propagation so that service invocation chain can inherit the header in the request.

Investigate how to use Open API to publish servcies/apis to Istio.

Upload some features to Istio.
Open API might be used to publish apis/services to the central area looked at by Istio.