Git Product home page Git Product logo

mfdfa's Introduction

DOI:10.1016/j.cpc.2021.108254 arXiv zenodo PyPI - License PyPI PyPI - Python Version Build Status codecov Documentation Status

MFDFA

Multifractal Detrended Fluctuation Analysis MFDFA is a model-independent method to uncover the self-similarity of a stochastic process or auto-regressive model. DFA was first developed by Peng et al.1 and later extended to study multifractality MFDFA by Kandelhardt et al.2.

In the latest release there is as well added a moving window system, especially useful for short timeseries, a recent extension to DFA called extended DFA, and the extra feature of Empirical Mode Decomposition as detrending method.

Installation

To install MFDFA you can simply use

pip install MFDFA

And on your favourite editor simply import MFDFA as

from MFDFA import MFDFA

There is an added library fgn to generate fractional Gaussian noise.

You can find the latest published paper of this library in Computer Physics Communications L. Rydin Gorjão, G. Hassan, J. Kurths, and D. Witthaut, MFDFA: Efficient multifractal detrended fluctuation analysis in python, Computer Physics Communications 273, 108254 2022. You can find the paper here.

The MFDFA library

MFDFA basis is solely dependent on numpy, especially numpy's polynomial. In version 0.3 a Empirical Mode Decomposition method was added for an alternative method of detrending timeseries, relying on Dawid Laszuk's PyEMD.

Employing the MFDFA library

An exemplary one-dimensional fractional Ornstein–Uhlenbeck process

The rationale here is simple: Numerically integrate a stochastic process in which we know exactly the fractal properties, characterised by the Hurst coefficient, and recover this with MFDFA. We will use a fractional Ornstein–Uhlenbeck, a commonly employ stochastic process with mean-reverting properties. For a more detailed explanation on how to integrate an Ornstein–Uhlenbeck process, see the kramersmoyal's package. You can also follow the fOU.ipynb

Generating a fractional Ornstein–Uhlenbeck process

This is one method of generating a (fractional) Ornstein–Uhlenbeck process with H=0.7, employing a simple Euler–Maruyama integration method

# Imports
from MFDFA import MFDFA
from MFDFA import fgn
# where this second library is to generate fractional Gaussian noises

# integration time and time sampling
t_final = 2000
delta_t = 0.001

# Some drift theta and diffusion sigma parameters
theta = 0.3
sigma = 0.1

# The time array of the trajectory
time = np.arange(0, t_final, delta_t)

# The fractional Gaussian noise
H = 0.7
dB = (t_final ** H) * fgn(N = time.size, H = H)

# Initialise the array y
y = np.zeros([time.size])

# Integrate the process
for i in range(1, time.size):
    y[i] = y[i-1] - theta * y[i-1] * delta_t + sigma * dB[i]

And now you have a fractional process with a self-similarity exponent H=0.7

Using the MFDFA

To now utilise the MFDFA, we take this exemplary process and run the (multifractal) detrended fluctuation analysis. For now lets consider only the monofractal case, so we need only q=2.

# Select a band of lags, which usually ranges from
# very small segments of data, to very long ones, as
lag = np.unique(np.logspace(0.5, 3, 100).astype(int))
# Notice these must be ints, since these will segment
# the data into chucks of lag size

# Select the power q
q = 2

# The order of the polynomial fitting
order = 1

# Obtain the (MF)DFA as
lag, dfa = MFDFA(y, lag = lag, q = q, order = order)

Now we need to visualise the results, which can be understood in a log-log scale. To find H we need to fit a line to the results in the log-log plot

# To uncover the Hurst index, lets get some log-log plots
plt.loglog(lag, dfa, 'o', label='fOU: MFDFA q=2')

# And now we need to fit the line to find the slope. Don't
# forget that since you are plotting in a double logarithmic
# scales, you need to fit the logs of the results
H_hat = np.polyfit(np.log(lag)[4:20],np.log(dfa[4:20]),1)[0]

# Now what you should obtain is: slope = H + 1
print('Estimated H = '+'{:.3f}'.format(H_hat[0]))

Uncovering multifractality in stochastic processes

You can find more about multifractality in the documentation.

Changelog

  • Version 0.4.3 - Reverting negative values in the estimation of the singularity strenght α.
  • Version 0.4.2 - Corrected spectral plots. Added examples from the paper.
  • Version 0.4.1 - Added conventional spectral plots as h(q) vs q, τ(q) vs q, and f(α) vs α.
  • Version 0.4 - EMD is now optional. Restored back compatibility: py3.3 to py3.9. For EMD py3.6 or larger is needed.
  • Version 0.3 - Adding EMD detrending. First release. PyPI code.
  • Version 0.2 - Removed experimental features. Added documentation
  • Version 0.1 - Uploaded initial working code

Contributions

I welcome reviews and ideas from everyone. If you want to share your ideas or report a bug, open an issue here on GitHub, or contact me directly. If you need help with the code, the theory, or the implementation, do not hesitate to reach out, I am here to help. This package abides to a Conduct of Fairness.

Literature and Support

Submission history

This library has been submitted for publication at The Journal of Open Source Software in December 2019. It was rejected. The review process can be found here on GitHub. The plan is to extend the library and find another publisher.

History

This project was started in 2019 at the Faculty of Mathematics, University of Oslo in the Risk and Stochastics section by Leonardo Rydin Gorjão and is supported by Dirk Witthaut and the Institute of Energy and Climate Research Systems Analysis and Technology Evaluation. I'm very thankful to all the folk in Section 3 in the Faculty of Mathematics, University of Oslo, for helping me getting around the world of stochastic processes: Dennis, Anton, Michele, Fabian, Marc, Prof. Benth and Prof. di Nunno. In April 2020 Galib Hassan joined in extending MFDFA, particularly the implementation of EMD.

Funding

Helmholtz Association Initiative Energy System 2050 - A Contribution of the Research Field Energy and the grant No. VH-NG-1025; STORM - Stochastics for Time-Space Risk Models project of the Research Council of Norway (RCN) No. 274410, and the E-ON Stipendienfonds.

References

1Peng, C.-K., Buldyrev, S. V., Havlin, S., Simons, M., Stanley, H. E., & Goldberger, A. L. (1994). Mosaic organization of DNA nucleotides. Physical Review E, 49(2), 1685–1689
2Kantelhardt, J. W., Zschiegner, S. A., Koscielny-Bunde, E., Havlin, S., Bunde, A., & Stanley, H. E. (2002). Multifractal detrended fluctuation analysis of nonstationary time series. Physica A: Statistical Mechanics and Its Applications, 316(1-4), 87–114

mfdfa's People

Contributors

galibhassan avatar lrydin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mfdfa's Issues

question: About estimation of hurst exponent.

I have a question about

  1. estimation procedure of the Hurst exponent.

In README.md, you argued ``We will fit the first points, since the results are more accurate there'', and used the first points to estimate Hurst exponent. Is there any evidence, such as papers or results of numerical simulations, that you can say so? (I thought this description was intuitively different from the methods described in the original paper of DFA.)

Negative H values when running mfdfa

Hi,
Thank you for all of the hard work you've done to make this library. I've tried following the instructions in the documentation as well as the instructions in this post #20 in order to analyze some of my own data and have encountered an issue.
For reference, the code I am using is in this repository in the main.py and file_reader.py files. https://github.com/FuadYoussef/ChineseSentenceLengths

When I run the MFDFA and attempt to plot the data, I am able to generate the following graph:
Figure_1

However the H values I get by subtracting 1 from each of the slopes is this: [0.24513458672164035, 0.2186072164297126, 0.1926072404229875, 0.17172103657125448, -0.34609042498381115, -0.3619098348631248, -0.375020070287282, -0.38637649591195067]

I obtain these H values using this code:
slopes = np.polyfit(np.log(lag), np.log(dfa), 1)[0]
slopes = slopes.tolist()
hExponents = []
for slope in slopes:
hExponents.append(slope - 1)

My confusion is about why I am getting negative H values from this. Any help or suggestions you could give would be greatly appreciated. Thank you.

[Testing] Performance tests

We should run some performance tests to see how MFDFA with all the extensions is running.
This should include:

  • standard MFDFA (i.e., no extensions)
  • with EMD
  • with eDFA

We should test this for timeseries starting from 10³ to 10⁷ datapoints, and do at least 20 iterations, to get the mean and a reasonable standard deviation.
We should generate stochastic data with H=0.2, H=0.5, and H=0.8, for comparison.

Implement a moving window for short timeseries

Implement a moving window to analyse shorter timeseries or improve results at large lag times.

At this stage I don't recall where I first read this, so I have to find the original publication and add it later.

Asymmetric DFA (A-DFA)

In A DFA approach for assessing asymmetric correlations, by Jose Alvarez-Ramirez, Eduardo Rodriguez, and Juan Carlos Echeverria, have "[...] developed a DFA extension for exploring the existence of asymmetries in the scaling behaviour of time-series". This is by now denote A-DFA. Could be interesting to implement.
The paper can be found here: https://doi.org/10.1016/j.physa.2009.03.007

Qingju Fan proposes an extension to A-DFA based on EMD in http://dx.doi.org/10.1016/j.physa.2015.08.056. This could also be rather interesting to include.

Preparing for v0.4: Separating EMD and MFDFA

I have been deliberating over the presence of the Empirical Mode Decomposition EMD in v0.3, i.e., the PyEMD by Dawid Laszuk.
The EMD package is phenomenal, but comprises an overhead of ~700 kb (and usually some additional packages).
My plan now is to create a separate repository (likely to be named MFDFA-EMD) which adds the EMD functionality included in v0.3.

The cause is simple: the original MFDFA is meant to be lightweight, i.e., depend only of numpy, so that one day maybe someone can make a parallel-isable version of it.

[I'm adding this to the readme of the repository]

Implement also modification when q = 0

In Faini 2021, they suggest a "seemingly simple" modification of the equation for when q = 0

image

Currently it seems like we do:

fluctuation = np.float_power(np.mean(np.float_power(var, q / 2), axis=1), 1 / q.T)

My fluency with equations is not there yet to be able to implement the adaptation, but if you have any ideas or leads I'm all ears :)

Examples on electricity prices

Hello Dear Professor, I am very interested in the MFDFA package you have written. I would like to reproduce the example of electricity price in your MFDFA package, but I do not see the dataset and code for electricity price fluctuation in your public code package. I would like to ask you if you could add this one example so that I can learn it. I look forward to hearing from you!

Minor typo

MFDFA.py

line 167: ... it runs*
line 168: Did you mean discarded

Hey @00stat, sorry for the delay, I was away last week. Well, if you'd like to help, we need create a new function similar to MFDFA and fundamentally "wrap" [L210-211](https://github.com/LRydin/MFDFA/blob/ef652e12174541e5b2ad96260b919b40f8b99d9e/MFDFA/MFDFA.py#L210-L211) with a `curve_fit` function from `scipy`. `curve_fit` can then take all the detrendings suggested in the paper.

          Hey @00stat, sorry for the delay, I was away last week. Well, if you'd like to help, we need create a new function similar to MFDFA and fundamentally "wrap"  [L210-211](https://github.com/LRydin/MFDFA/blob/ef652e12174541e5b2ad96260b919b40f8b99d9e/MFDFA/MFDFA.py#L210-L211) with a `curve_fit` function from `scipy`. `curve_fit` can then take all the detrendings suggested in the paper.

The primary issue here is that a large amount of these functions will perfectly fit the data, so we have to test what is the minimum lag that can be accepted for each, which will likely have to be a bit of guesswork.

What do you think?

Originally posted by @LRydin in #24 (comment)

Document the use of `numpy` `masked`

When I designed MFDFA I had an eye on how to handle missing data, so the code if capable of handling numpy masked arrays, which is pretty nice, since we all have to deal with missing or corrupted data. I should document this in the docs :)

Quick question

Hi @LRydin! Just a quick check,

If I'm not mistaken, here you compute Tau(q) as :

return q, (q * slopes) - 1

However, in Orozco-Duque, A., Novak, D., Kremen, V., & Bustamante, J. (2015). Multifractal analysis for grading complex fractionated electrograms in atrial fibrillation. Physiological Measurement, 36(11), 2269–2284. doi:10.1088/0967-3334/36/11/2269 they compute it as:

image

which would be (q-1) * slopes?

Is it equivalent or is there something I missed?

Multifractal Cross-Correlation Analysis (MFCXA)

An interesting, similar (in nature), Multifractal Cross-Correlation Analysis (MFCXA) is presented by Qingju Fan and Dan Li, Modeling electricity prices: jump diffusion and regime switching, Physica A, 429, 17-27, 2015 (possibly not the original article where MFCXA is proposed).

It involves a more complicated functional, but it seems like something we could add to MFDFA in a later version.

Fluctuation Analysis (and Multifractal FA)

Somehow after all this time I have simply forgotten that in the end DFA stems from FA (Fluctuation Analysis). Naturally this should also be included.

This just be as simple as discarding any form of detrending, i.e., discarding the polynomial (other forms of fitting). This would be just circumventing L128-L129.

Then one needs to not detrend lines L132-L133 (i.e., not do the polyval)

F = np.var(Y_ - polyval(X[:i], p), axis = 1)
F_r = np.var(Y_r - polyval(X[:i], p_r), axis = 1)

Implement an Empirical Mode Decomposition method for detrending

This is a method I first encountered in A Modified Multifractal Detrended Fluctuation Analysis (MFDFA) Approach for Multifractal Analysis of Precipitation in Dongting Lake Basin, China. The idea is simple, replace the polynomial fittings by an Empirical Mode Decomposition, a.k.a., Hilbert–Huang transform (HHT)

We thus have the choice of including as many of the intrinsic mode functions to perform the fitting. Naturally we have three fundamental things to find out:

  • is there a decent python code for EMD already?
  • if yes: can we port it? if no: can we build one?
  • what do we need to neatly integrate this into our code?

A modified Multifractal Detrended Fluctuation Analysis (MFDFA) approach for multifractal analysis of precipitation

A recent paper in Physica A, entitled A modified Multifractal Detrended Fluctuation Analysis (MFDFA) approach for multifractal analysis of precipitation (Physica A 565, p125611, 2021) suggests a set of "complicated" functions to serve as fitting for each segmentation, replacing the simple polynomial fittings. These include a combination of polynomial and trignometric functions, which are meant to minimise the mean absolute error for each fit of a segment.

We can consider implementing this, likely at the cost of performance as we likely require scipy's fit, which I believe in single-threaded. The approach seems simple enough to constitute an interesting addition to MFDFA.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.