takuti / datadog-anomaly-detector Goto Github PK

Fix timezone of replay feature from JST to UTC.

Catch the errors properly and output better logging messages into the log file.

It can be obtained from: http://docs.datadoghq.com/api/#graph-snapshot

The URLs are used in slack notifications.

Even though a matrix condition is checked at https://github.com/takuti/datadog-anomaly-detector/blob/master/dd_anomaly_detector/changefinder/changefinder_1d.py#L114-L115, sometimes LinAlgError occurs because of ln.inv for a singular matrix.

This must be handled appropriately as:

try:
    C = toeplitz(self.c[:self.k])
    if not np.all(C == 0.0) and np.isfinite(ln.cond(C)): 
        a = np.dot(ln.inv(C), self.c[1:])
except ln.LinAlgError as err:
    # do something

What will caused when Datadog API returns API outages or some errors? Daemon will down w/ errors?

Better to avoid tentative API errors and full error massages in log files and post a succinct error message to slack channel.

SELECT (timestamp * 1000).format() as date can be used, but it is not flexible enough in terms of timezone setting and formatting.

Write Norikra/Esper UDF:

• https://github.com/norikra/norikra-udf-mock
• http://stackoverflow.com/questions/36281078/how-can-i-add-timezone-to-esper-queries

but what is the way to use only python installed locally?

At the beginning of each interval, our daemon must reload .ini file in order to check if a new query is added to the config file.

• Create an init script and provide a better interface as sudo service changefinder start
• Create ansible playbooks to setup whole detector system
• Set API keys on the ini file, rather than hard coded
• Remove dependency with a python-daemon-3K package

References:

• Original python-daemon package
• How to set up a service
• Japanese articles to create Python daemon scripts
  • http://qiita.com/okhiroyuki/items/d19e8b0c4599767955a8
  • http://dreamerdream.hateblo.jp/entry/2016/01/08/000000

When a user did not specify any hyperparameters in the .ini file, DD have to automatically set default values of T1, T2, and k.

How to choose the default values for each metric? Unique values for all metrics may lead poor results.

TODOs:

• Run model selection if k is not set
• Meaningful default (or optimal) values for r, T1 and T2

ref #11

• AR(k) estimation method
  • Burg's method
  • Yule-Walker
• Scoring function
  • Logloss
  • Hellinger distance

Which combination is the best on the real-world DD metrics and use-cases?

• Both of them should be configurable
• Test all possible cases on the synthetic and twitter data
• Upload a notebook for further comparison
• Test on the aggregated real DD data points
• Deploy and monitor whether the combination works as expected
  • If not, why? How can we solve the problem?

datadog-anomaly-detector SHOULD have a safe limitation on Datadog API calls per hour.
https://help.datadoghq.com/hc/en-us/articles/205060139-API-rate-limit

The limit is better to be configurable.
Also, warning messages are going to be reported on Slack channel similar to other errors.

from #7 (cc: @myui)

Due to API rate limit, snapshots must be taken by Fluentd-side (i.e. DD itself) rather than the daemon-side API-based implementation.

• Try fluent-plugin-datadog_event to send events for each anomaly
  • https://cloudpack.media/12494 (in Japanese)

Relationship scope and host should be implemented carefully.

https://github.com/takuti/datadog-anomaly-detector/blob/master/dd_anomaly_detector/base_detector.py#L78

Current detection logic ChangeFinder is highly sensitive to the hyperparameters and estimation techniques as discussed on #11 and #14 . In addition, it is unclear if ChangeFinder is theoretically appropriate to our problem.

Thus, additional survey and considering different change-point detection algorithms is also necessary in practice.

• Knowledge Discovery from Time-series Data using Nonlinear Transformations (Japanese)
• Change-point detection and failure analysis of sensor data using sparse structure learning (Japanese)
• Knowledge Discovery from Heterogeneous Dynamic Systems using Change-Point Correlations

• create a new script which is similar to replay.py
  • test on time series in a specific time range
  • the script returns optimal k of an AR(k) model for each query given in a config file
• online (i.e. adaptive) model selection
  • Dynamic Model Selection With its Applications to Novelty Detection
  • Japanese slides

The Burg's method can be better than Yule-Walker. Implement it and compare each other.

http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/YWSourceFiles/WhyNotToUseYW.pdf

References:

• arburg() on MATLAB
  • Implementation
• Burg's method
• http://mikeo410.minim.ne.jp/cms/~audiolinearpredictionarburgarburg (in jp)

TODOs:

• Replace vectorization with for-loop
• Compare arburg() and aryule()