Git Product home page Git Product logo

neuralrdes's Introduction

Neural Rough Differential Equations for Long Time Series
(ICML 2021)
[arXiv]

Overview

Neural Controlled Differential Equations (Neural CDEs) are the continuous-time analogue of an RNN. However, as with RNNs, training can quickly become impractical for long time series. Here we use rough path theory to extend this formulation through application of a pre-existing mathematical tool from rough analysis - the log-ODE method - which allows us to take integration steps larger than the discretisation of the data, resulting in significantly faster training times, with retainment (and often even improvements) in model performance.

Loosely speaking, this corresponds to binning the data prior to running a Neural CDE, with bin statistics carefully chosen to extract precisely the information most relevant to solving a CDE. We have traded in length for additional input channels, with this trade off coming at the cost of having two additional hyperparameters to tune.


Paper

This repository contains all the code for reproducing the experiments from the Neural Rough Differential Equations for Long Time Series paper.

Code

Code for constructing the log-signature NCDEs is in the ncdes/ folder and can be adapted to run your own models.

Library

We recommended checking out the torchcde project, which is a well maintained library for doing all things NCDE related, and now includes a log-ODE implementation.


Reproducing the Experiments

Setup the Environment

First to setup the environment, install the requirements with

  • pip install -r requirements.txt
  • pip install signatory==1.2.0.1.4.0 --no-cache-dir --force-reinstall

(Signatory has to be installed after PyTorch, due to limitations with pip).

Downloading the Data

Now setup a folder to save the data in at data/raw/. Here data/ should be made a symlink if you do not want to store the data in the project root.

Navigate into get_data/ and run

  • python get_data/uea.py
  • python get_data/tsr.py This will download the EigenWorms and BIDMC data to data/raw/ that was used in the paper (note that this can take a couple of hours as it downloads the entire UEA and TSR archives).

Run the Experiments

Finally, run the experiments. We provide a convenient script experiments/runs.py for doing this. Note that we need to run a hyperparameter optimization search and then the main run for each dataset. To run a hyperoptimzation followed and then the main experiment run navigate into experiments, open python and run

>>> import runs
>>> runs.run('UEA', 'EigenWorms', 'hyperopt')   # Syntax is run(dataset_folder, dataset_name, configuration_name)
# Once completed
>>> runs.run('UEA', 'EigenWorms', 'main')

This being an example for EigenWorms. For the BIDMC data it is just:

>>> import runs
>>> runs.run('TSR', 'BIDMC32{*}', 'hyperopt')   # Where {*} is one of {RR, HR, SpO2}
>>> runs.run('TSR', 'BIDM32{*}', 'main')

This will run on the cpu. If you have GPUs available and wish to run on the GPU (strongly recommended) we have an optional gpu_idxs argument that takes a list of gpu indexes. For example,

>>> runs.run('TSR', 'BIDM32RR', 'main', gpu_idxs=[0, 1])

will run the respiration rate runs on GPUs with indexes [0, 1].

We do have an option to run all the runs in one go

>>> runs.run(gpu_idxs=[0, 1])    # With user specified GPUs

but this is not recommended as it may take ~1-2 weeks.

Analysis of Results

Results will be saved to experiments/models/. To properly analyse them we recommend opening them in a jupyter notebook (see notebooks/{results, plots}.ipynb to reproduce the tables and figures from the paper), however for an overview one can use the parse_results.py file in experiments/ by running the following in the terminal

> python parse_results.py TSR BIDMC32RR hyperopt    # Arguments are the same as the runs.run function

this will print an overview of the results to the console.


Citation

@article{morrill2021neuralrough,
  title={Neural Rough Differential Equations for Long Time Series},
  author={Morrill, James and Salvi, Cristopher and Kidger, Patrick and Foster, James and Lyons, Terry},
  journal={International Conference on Machine Learning},
  year={2021}
}

neuralrdes's People

Contributors

jambo6 avatar patrick-kidger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

neuralrdes's Issues

Suggest to loosen the dependency on sacred

Hi, your project neuralRDEs requires "sacred==0.8.1" in its dependency. After analyzing the source code, we found that the following versions of sacred can also be suitable without affecting your project, i.e., sacred 0.8.0. Therefore, we suggest to loosen the dependency on sacred from "sacred==0.8.1" to "sacred>=0.8.0,<=0.8.1" to avoid any possible conflict for importing more packages or for downstream projects that may use neuralRDEs.

May I pull a request to further loosen the dependency on sacred?

By the way, could you please tell us whether such dependency analysis may be potentially helpful for maintaining dependencies easier during your development?



We also give our detailed analysis as follows for your reference:

Your project neuralRDEs directly uses 3 APIs from package sacred.

sacred.experiment.Experiment.__init__, sacred.ingredient.Ingredient.__init__, sacred.observers.file_storage.FileStorageObserver.__init__

Beginning from the 3 APIs above, 44 functions are then indirectly called, including 27 sacred's internal APIs and 17 outsider APIs. The specific call graph is listed as follows (neglecting some repeated function occurrences).

[/jambo6/neuralRDEs]
+--sacred.experiment.Experiment.__init__
|      +--sacred.host_info.check_additional_host_info
|      +--sacred.experiment.gather_command_line_options
|      |      +--sacred.utils.get_inheritors
|      |      +--warnings.warn
|      +--inspect.stack
|      +--os.path.basename
|      +--sacred.ingredient.Ingredient.__init__
|      |      +--collections.OrderedDict
|      |      +--inspect.stack
|      |      +--os.path.dirname
|      |      +--os.path.abspath
|      |      +--sacred.dependencies.gather_sources_and_dependencies
|      |      |      +--sacred.dependencies.get_main_file
|      |      |      |      +--os.path.abspath
|      |      |      |      +--sacred.dependencies.Source.create
|      |      |      |      |      +--os.path.exists
|      |      |      |      |      +--sacred.dependencies.get_py_file_if_possible
|      |      |      |      |      |      +--os.path.exists
|      |      |      |      |      +--os.path.abspath
|      |      |      |      |      +--sacred.dependencies.get_commit_if_possible
|      |      |      |      |      |      +--os.path.dirname
|      |      |      |      |      |      +--git.Repo
|      |      |      |      |      |      +--git.Repo.remote
|      |      |      |      |      |      +--git.Repo.is_dirty
|      |      |      |      |      +--sacred.dependencies.Source.__init__
|      |      |      |      |      +--sacred.dependencies.get_digest
|      |      |      |      |      |      +--hashlib.md5
|      |      |      |      +--os.path.dirname
|      |      |      +--sacred.dependencies.PackageDependency.create
|      |      |      |      +--sacred.dependencies.PackageDependency.__init__
|      +--sacred.ingredient.Ingredient.command
|      |      +--sacred.ingredient.Ingredient.capture
|      |      |      +--sacred.config.captured_function.create_captured_function
|      |      |      |      +--sacred.config.signature.Signature.__init__
|      |      |      |      |      +--sacred.config.signature.get_argspec
|      |      |      |      |      |      +--inspect.signature
|      |      |      |      |      |      +--collections.OrderedDict
|      |      |      |      +--sacred.config.captured_function.captured_function
|      |      |      |      |      +--sacred.config.custom_containers.fallback_dict
|      |      |      |      |      +--sacred.randomness.get_seed
|      |      |      |      |      |      +--random.randint
|      |      |      |      |      +--sacred.randomness.create_rnd
|      |      |      |      |      |      +--random.Random
|      |      |      |      |      +--time.time
|      |      |      |      |      +--sacred.utils.ConfigError.track
|      |      |      |      |      |      +--sacred.utils.join_paths
|      |      |      |      |      +--datetime.timedelta
|      +--sacred.commands.print_named_configs
|      |      +--collections.OrderedDict
|      |      +--sacred.commands._format_named_configs
|      |      |      +--sacred.commands._format_named_config
+--sacred.ingredient.Ingredient.__init__
+--sacred.observers.file_storage.FileStorageObserver.__init__
|      +--pathlib.Path
|      +--os.path.exists
|      +--sacred.observers.file_storage.FileStorageObserver.initialize

We scan sacred's versions and observe that during its evolution between any version from [0.8.0] and 0.8.1, the changing functions (diffs being listed below) have none intersection with any function or API we mentioned above (either directly or indirectly called by this project).

diff: 0.8.1(original) 0.8.0
['sacred.observers.mongo.MongoObserver.insert', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver._determine_run_dir', 'sacred.serializer.NumpyGenericHandler.flatten', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.log_metrics', 'sacred.serializer.NumpyArrayHandler.flatten', 'sacred.observers.mongo.QueuedMongoObserver', 'sacred.serializer.PandasDataframeHandler', 'sacred.observers.mongo.QueuedMongoObserver.__init__', 'sacred.observers.gcs_observer._is_valid_bucket', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.interrupted_event', 'sacred.observers.gcs_observer.gcs_option', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.heartbeat_event', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.failed_event', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.artifact_event', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.resource_event', 'sacred.serializer.NumpyArrayHandler.restore', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver', 'sacred.experiment.Experiment', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.find_or_save', 'sacred.observers.queue.QueueObserver', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_file_to_base', 'sacred.serializer.NumpyArrayHandler', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.started_event', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver._objects_exist_in_dir', 'sacred.observers.mongo.QueueCompatibleMongoObserver.save', 'sacred.serializer.PandasDataframeHandler.flatten', 'sacred.observers.mongo.QueueCompatibleMongoObserver', 'sacred.observers.mongo.MongoObserver', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver._list_gcs_subdirs', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.__eq__', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.completed_event', 'sacred.serializer.NumpyGenericHandler.restore', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_directory', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.put_data', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.queued_event', 'sacred.serializer.NumpyGenericHandler', 'sacred.observers.queue.QueueObserver.__init__', 'sacred.observers.queue.QueueObserver._run', 'sacred.experiment.Experiment._check_command', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_cout', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_file', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.__init__', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_sources', 'sacred.serializer.PandasDataframeHandler.restore', 'sacred.observers.gcs_observer.gcs_join', 'sacred.observers.gcs_observer.GoogleCloudStorageObserver.save_json', 'sacred.observers.mongo.MongoObserver._try_to_detect_content_type']

As for other packages, the APIs of inspect, os, collections, pathlib, warnings, time, datetime, git, hashlib and random are called by sacred in the call graph and the dependencies on these packages also stay the same in our suggested versions, thus avoiding any outside conflict.

Therefore, we believe that it is quite safe to loose your dependency on sacred from "sacred==0.8.1" to "sacred>=0.8.0,<=0.8.1". This will improve the applicability of neuralRDEs and reduce the possibility of any further dependency conflict with other projects.

Batch size for eigenworms dataset

Why is the batch size for eigenworms 1024? There are only 131 samples in the train dataset and 128 samples in the test dataset, am I missing something?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.