Online-Recurrent-Extreme-Learning-Machine (OR-ELM) for time-series prediction, implemented in python

Python 2.08% Jupyter Notebook 97.92%

oselm elm os-elm or-elm extreme-learning-machine online-sequential-elm time-series time-series-prediction online-learning online-learning-algorithms

online-recurrent-extreme-learning-machine's Introduction

Online-Recurrent-Extreme-Learning-Machine

Online-Recurrent-Extreme-Learning-Machine (OR-ELM) for time-series prediction, implemented in python.

Requirements

Python 2.7
Numpy
Matplotlib
pandas
Expsuite (included in this repository)

Dataset

NYC taxi passenger count
Prediction of the New York City taxi passenger data. left. Example portion of taxi passenger data (aggregated at 30 min intervals).
public data stream provided by the New York City Transportation Authority
preprocessed (aggregated at 30 min intervals) by Cui, Yuwei, et al. in "A comparative study of HTM and other neural network models for online sequence learning with streaming data." Neural Networks (IJCNN), 2016 International Joint Conference on. IEEE, 2016. , code

Implemented Algorithms

Online Sequential Extreme Learning Machine (OS-ELM)
- Liang, Nan-Ying, et al. "A fast and accurate online sequential learning algorithm for feedforward networks." IEEE Transactions on neural networks 17.6 (2006): 1411-1423.
Fully Online Sequential Extreme Learning Machine (FOS-ELM)
- Wong, Pak Kin, et al. "Adaptive control using fully online sequential-extreme learning machine and a case study on engine air-fuel ratio regulation." Mathematical Problems in Engineering 2014 (2014).
Normalized FOS-ELM (NFOS-ELM) (proposed)
- FOS-ELM + Layer Normalization + forgetting factor
Normalized Auto-encoded FOS-ELM (NAOS-ELM) (proposed)
- FOS-ELM + Layer Normalization + forgetting factor + weight auto-encoding (input->hidden)
Online Recurrent Extreme Learning Machine (OR-ELM) (proposed)
- FOS-ELM + Layer Normalization + forgetting factor + weight auto-encoding (input->hidden, hidden->hidden)
- This is for training recurrent neural networks (RNNs)

Example of usage

Run prediction code:

python run.py -a ORELM

Plot performance comparison:

python plotResults.py

Result

Prediction from OR-ELM

Performance comparison
- FOS-ELM and proposed variants including OR-ELM

To do

Rewrite this code with Pytorch for GPU acceleration

If you use this code, please cite our paper "Online Recurrent Extreme Learning Machine and its Application to time-series Prediction" in IEEE Access.

Paper URL: http://ieeexplore.ieee.org/abstract/document/7966094/ http://rit.kaist.ac.kr/home/International_Conference?action=AttachFile&do=get&target=paper_0411.pdf

Park, Jin-Man, and Jong-Hwan Kim. "Online recurrent extreme learning machine and its application to time-series prediction." Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE, 2017.

Acknowledgement

This work was supported by the ICT R&D program of MSIP/IITP. [2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion]

online-recurrent-extreme-learning-machine's People

Stargazers

Watchers

online-recurrent-extreme-learning-machine's Issues

Dividing the dataset into train and test

Two issues related to each other:

As I can see in the code, self.beta is directly responsible for prediction. However, this matrix is being updated recursively up to the end of the process (to be more exact len(sequence)-predictionStep.
For instance in FOS_ELM we have:
self.beta = self.beta + np.dot(self.M, np.dot(Ht, targets - np.dot(H, self.beta)))
and this train process is in a for loop with prediction of the next:
for i in range(numLags, len(sequence)-predictionStep-1):
net.train(X[[i], :], T[[i], :])
Y = net.predict(X[[i+1], :])
predictedInput[i+1] = Y[-1]
I believe this update should be continued up to the end of the training, then one needs to feed the testdata only to prediction function.
The whole data is normalized but if you divide the dataset to test and train, you cannot normalize test set. (you can use mean and variance of train set for normalizing test).

Based on these two issues, the NRMSE mentioned at the end of the process is not reliable.

How to make to run or-elm classification with iris database

Hello, i would like to learn, how to make to run or-elm classification with iris database.

Is it possible ?

OR-ELM python3 (or R)

Dear Jin,

First of all I would like to thank you for developing this repo. I read with great pleasure your paper on the OR-ELM algorithm and I would like to use it as part of my PhD thesis.
I was wondering if your code will work on Python3 or if you are aware of any R implementation of OR-ELM.

Thank you.

Best Wishes,
Ahmed

A question about the training process

Hi, I have successfully ran through your code, but I have a question I want to communicate with you.

As shown in codes below, why you choose to compare the gap between predictedinput [i+1] and targetInput [i+1] instead of comparing the gap between predictedinput [i+1] and trueData [i+1] .

I think predictedinput [i+1] is the prediction at time i+1 which is correspond to trueData [i+1]

Ramp Function Diverges

As a low-effort test of the ORELM, I substituted for NYC Taxi traffic, the attached ramp function from 0 to 20000 in increments of 10000, backed up the nyc_taxi.csv and copied therampified_nyc_taxi.csv in its place. It seems not to converge.

This isn't necessarily an issue with the code. It is interesting in that it raises the issue of parameter seeking in the ORELM algorithm. What is it about the ramp function that causes ORELM to diverge and how would one find the parameter(s) values that converge on a solution?

rampified_nyc_taxi.zip

For comparison, this is with a sine function rather than ramp function -- although in this case, I synchronized the time of day with the phase:

sine_nyc_taxi.zip

chickenbestlover / online-recurrent-extreme-learning-machine Goto Github PK