google / ts-bridge Goto Github PK
View Code? Open in Web Editor NEWTime Series Bridge: a tool to import metrics from one monitoring system to another
License: Apache License 2.0
Time Series Bridge: a tool to import metrics from one monitoring system to another
License: Apache License 2.0
We're using Travis-CI.org, which is getting shut down December 2020: https://docs.travis-ci.com/user/migrate/open-source-repository-migration
As such, we'll need to migrate to githubActions.
ts-bridge is currently implemented as GAE application, which is fine for most of the potential users, but we've already faced with challenges trying to deploy ts-bridge in environments that are included into Google VPC Service Controls service perimeters. An attempt to deploy ts-bridge with end up with an error:
{"errors":[{"code":"DENIED","message":"Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: "}]}.
GAE is currently not supported by VPC service controls.
The proposal is to containerize ts-bridge and remove any GAE-specific dependencies (ideally - Cloud Datastore dependencies as well). As a result it will be possible to deploy ts-bridge inside K8S cluster (e.g. in GKE) or as GAE Flex application or in any other environment that support running containers.
Currently, the GitHub Action (previously Travis CI) only runs tests with the latest version of Go. We should extend this in the future to run for different versions of Go.
InfluxDB metric source is configured by specifying a query, this query must have 1 column, so that it returns just 1 value.
We're not checking this in validateQuery , as a result we throw errors that are not very intuitive (like "cannot create metric '<metric_name>': not an App Engine context "
version/version.go
should read in the project version from .ts-bridge-version
May be useful to document how someone can create alerts on missing timeseries in Service -> Alerts
cc @Dnefedkin
Currently query results get written to Stackdriver as GAUGE metrics.
This does not work well for importing SLIs, since an SLI is typically a ratio of two counters (good_events / valid_events). To allow calculating a ratio over an arbitrary long time window, we'll need ts-bridge to import both counters (good_events and valid_events) to Stackdriver separately as cumulative metrics.
This is not trivial, since CUMULATIVE metrics in Stackdriver require the client to report start_time (metric reset time) with each point. This timestamp is usually set to the start time of the process exporting a counter, and is used by Stackdriver to track counter resets.
I believe we should be able to use Datadog's cumsum function to get a cumulative counter, and then export it to Stackdriver as a CUMULATIVE metric.
This will require keeping some internal state per counter, and I think we can use metric records for that.
In all the funcs inside tasks/tasks.go
, the variables tsbridge.MetricConfig
, tsbridge.Metrics
, tsbridge.Config
and storage.Manager
are required repeatedly. Instead of passing them every time, a struct could be made to store these values. This would make the function calls much cleaner to read.
There is already a struct named web.Handler
which stores these values, so the code should be refactored to use the same struct in both handlers.go
and tasks.go
. Currently, handlers.go
is only used for AppEngine, whereas tasks.go
functions are used in both AppEngine and non-AppEngine cases. This must be accounted for when refactoring.
See also: post by @nerdinary in #103 (comment)
Some folks from Datadog that I talked to mentioned that the most recent point returned by the API might contain incomplete data if a query aggregates values reported from multiple machines and data coming from some of the machines is slightly delayed.
Since it's impossible to edit values that have already been submitted to Stackdriver, I suggest ts-bridge ignore points from Datadog that are less than 1 minute old to allow data to be fully processed and aggregated.
We will need to perform security scans TS-Bridge images before they are published to GCR.
Trivy will be used as it is open source and does not require account permissions to use.
Currently TSBridge just sends requests with standard google-cloud-go
user agent, leaving the application bit empty.
Let's pick user agent from packer:
google-api-go-client/0.5 Packer/1.6.3 (+https://www.packer.io/; go1.15.2; linux/amd64),gzip(gfe)
google-api-go-client/%VERSION% ts-bridge/%VERSION% (go%VERSION%; %PLATFORM%)
, e.g.:
google-api-go-client/0.5 ts-bridge/0.1 (go1.13.2; linux/amd64)
Currently, changelogs and release descriptions are written manually for every release. We would like to use GitHub Actions to do the following:
tar.gz
formatIt would be great to support Datadog metrics labels in the query
field, or translate Datadog metric labels to Cloud Monitoring metric labels.
args: ['-c', 'docker pull gcr.io/$PROJECT_ID/ts-bridge:latest || exit 0']
This step (originally copied from https://cloud.google.com/cloud-build/docs/speeding-up-builds#using_a_cached_docker_image) will mask all errors from docker pull, not just when image doesn't exist. Consider using a more optimal approach which can parse the output from docker pull to handle errors better.
Current version of google.golang.org/appengine leads to "not an App Engine context" errors for all Cloud Datastore calls that ts-bridge is doing
Add CI configuration (GithubActions or CloudBuild) that would build ready-to-use containers for all major platforms (e.g. linux/amd64, linux/arm, linux/arm64)
Tag images with:
git rev-parse --short HEAD
) Cross compile images for linux/amd64, linux/arm, linux/arm64
App Engine Go runtimes below 1.11 are getting deprecated, we should migrate the app to support Go 1.11.
Currently we don't support passing credentials to ts-bridge directly. It's all done through google-cloud-go
library support, so classic application default credentials.
Design:
This should require just piping through the options from the lib, described here: https://github.com/googleapis/google-cloud-go#authorization
So 1.16 just released and it seems that something has changed to make a couple of tests fail only for 1.16.x
build, notably:
See: https://github.com/google/ts-bridge/runs/1933024386
Assigning to Adi to take a look later (this is not very urgent). Sorry for not doing it myself, but 1.16 just released and this didn't fail in RC so I couldn't have known.
Connections to external dependencies, such as Prometheus & StackDriver exporters are established only after the initial sync loop is started.
A few examples:
https://github.com/google/ts-bridge/blob/master/tasks/tasks.go#L61
To prevent multiple dependency connections being opened, this is wrapped in a sync.Once function,
Line 102 in 32ca319
There might be other examples - see the sync loop for all external dependencies established.
TODO:
Occasionally seeing the following errors from Stackdriver:
total_bytes_rcvd: failed to write to Stackdriver: CreateTimeSeries error: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0], timeseries: metric:<type:"custom.googleapis.com/datadog/total_bytes_rcvd" > resource:<type:"global" > metric_kind:GAUGE value_type:DOUBLE points:<interval:<end_time:<seconds:1540994261 > > value:<double_value:14838.339990234375 > >
Datadog returns more than 1 point per minute (for this specific metric points are 15sec apart), while Stackdriver generally recommends sending 1 point per minute.
Stackdriver adapter should probably just drop points that are less than 1 minute apart from the one previously written.
I am occasionally seeing the following error message in logs:
StatsCollector: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0-2]
What I believe is happening:
There is no way to stop the default view worker, however we can call SetReportingPeriod
to reset its timer. By setting a very long reporting period in the beginning of every sync operation (every time we register the exporter) we will effectively disable regular flushing of metrics and will only rely on flushing that happens at the end of the sync operation.
Currently we don't support passing credentials to ts-bridge directly. It's all done through google-cloud-go
library support, so classic application default credentials.
Design:
This should require just piping through the options from the lib, described here: https://github.com/googleapis/google-cloud-go#authorization
In introspection dashboard, we still use the old Stackdriver link, which takes us through a series of re-directs. We can migrate to the new cloud monitoring endpoint now.
cc @Dnefedkin
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.