ambersjones / pyhydroqc Goto Github PK

need to consider how to implement a linear drift correction procedure.

Detect calibration events based on time of day, length of persistence, change in slope, other sensor behavior.

Consider improving the ARIMA detection by 1. apply detection as is 2. linear interpolate 3. build model again 4. detect again

line 84 of anomaly_utilities.py

ARIMA functions need to be moved from the example script to a utilities script

any anomaly needs to be factored into the results

Need to apply dynamic threshold to model detection.

write a function to use an edge detection like filter to determine likely candidates for calibration events

After loops run, there are sometimes individual items left - need to be deleted/removed from code.

The sklearn package has been depreciated. It might be helpful to replace 'sklearn' with 'sci-kit-learn' in your pip requirements file setup.py setup.cfg, files

https://github.com/scikit-learn/sklearn-pypi-package

Thank you

everything to current

add to the calibration detection function to ensure that output event dates are at least 3 in length.

there are separate anomaly detection functions for ARIMA and LSTM that should be merged to one

currently, the LSTM correction proceeds in only one direction. need to assess if this results in significant nonlinearities, and if so, consider how to implement a cross-fade.

Need to

• create iPython notebook with steps for running through all functionality.
• share on HydroShare
• execute on CUAHSI's JupyterHub environment

add a function to determine the sensor output difference after calibration

create_training_dataset - and rename create_clean_training_dataset
create_bidir_training_dataset - rename create_bidir_clean_training_dataset
create_autoen_model

The performance and speed of ARIMA correction might be improved by limiting the window of piecewise model development to 2-6 times the length of the gap.

Need to move the repo to be under @AmberSJones.

there are cases where warnings in model development, etc. need to be suppressed.

Improve the ARIMA correction script by ordering the gaps that are corrected by length - shortest gaps first- rather than ordered chronologically.

consider training LSTM to classify and detect linear drift/calibration

FBP_detect
LSTM_detect
ARIMA_detect
ScriptArchive files?

Check that dataframes are being copied so that changes aren't being made on original copies

Need to consider implementation of feature based anomaly detection.