Git Product home page Git Product logo

lstasiak / ising_nn_pca Goto Github PK

View Code? Open in Web Editor NEW
2.0 2.0 1.0 62.54 MB

The repository contains programs and models for investigation of estimating phase-transitions in 2D Ising model using Deep Neural Networks and comparison of the results with data compressed by PCA algorithm. The results are part of Master Thesis being the completition of the study on Big Data Analytics at WUST

License: GNU General Public License v3.0

CMake 0.01% C++ 1.94% Jupyter Notebook 97.87% Python 0.19%
pca-analysis neural-networks keras-tensorflow ising-model academic-project machine-learning neural-network

ising_nn_pca's Introduction

The role of dimensionality reduction in Neural Network based identification of phase transitions

The repository contains programs and models for investigation of estimating phase-transitions in 2D Ising model using Deep Neural Networks and comparison of the results with data compressed by PCA algorithm. The results are part of Master Thesis written on Big Data Analytics at Wrocław University of Science and Technology (WUST) In short:

  • Ising model and Monte-Carlo simulations are written in C++ 20. We used pcg random number generator.
  • Data being collected are spin configurations (binary matrices) for different matrix size.
  • For every dataset, a Deep Neural Network model there was created, separately trained and analyzed. ML Models are written in Python 3.8 with help of Tensorflow and Keras.
  • Predictions were repeated on data transformed by dimensionality reduction technique - PCA (from scikit-learn) and compared with result on original data.
  • The main goal was to estimate the critical temperature - indicating phase transition in Ising model using Neural Networks and check the impact of dimensionality reduction on estimations.

Phase transitions & Ising Model

Statistical mechanics explains macroscopic behaviour of complex systems from the behaviour of large assemblies of microscopic entities. One of the main topics in this area is the research on phase transitions, i.e., the abrupt changes of the system (typically behaved dramatically), due to the change of a macroscopic variable.

The Ising (or Lenz-Ising) model is the most popular and most unversal model of analyzing phase transitions. We consider 2-dimensional model, since in 1D there is no phase transitions to observe. In the Ising model we consider L x L lattice (or graph in general) of spins (or atoms/agents etc). By analogy to magnetic moments, every spin can take one of the two possible values: "up" / "down" (e.g. 1 or -1). The lattice with spins set in such a way is called spin configuration.

Basically, we observe the change of the order parameter, which in this case is called magnetization, i.e., average spin in single configuration:

formula

The order parameter is different than zero in low temperatures, that is below the critical point (Curie Temperature) -- which indicates ordered, ferromagnetic phase. If we cross the critical point, i.e., the temperature of the system will be sufficiently high, the order parameter is approximately zero and the system is in paramagnetic (disordered) phase. For Ising model, we assume the critical temperature near T ~ 2.269.


Monte Carlo simulations of the Ising Model

Data to train and test machine learning model consists of the spin configurations of size: L = 10, 20, 30, 40, 50, 60 sampled by monte carlo steps (MCS). The method used to update configuration is called Metropolis-Hastings Algorithm. For every configuration size, we perform simulation for 30 000 MCS as thermalization time and collect data for 200 000 MCS. Temperature range, for which simulation was ran, includes 176 different values from: linspace(0.5, 4.0, 0.02). In summary, dataset for single configuration size consists of 2000 samples per temperature, 352176 in total. Here you can see the change of the system during the simulation time for three different temperatures: above critical temperature (Tc), approximately in Tc and below critical value: ising_configurations We can see that above critical temperature, the system is in high fluctuations that do not disappear with time. On the other hand, below Tc the system ends up in established state -- approximately all spins take the same sign (are in the same direction).


Deep Neural Networks in identification of phase transition

For collected datasets, modeling and training of neural networks was perfomed in separate notebooks, but in similar way. Since every L x L configuration is basically binary matrix, there is no need to normalize data, but small preprocessing including change of numerical type and creating array of labels [0 - for T<2.26, 1 - otherwise] and converting it to categorical format was applied. Networks are built using Keras Functional API. We have 2 Dense layers, every layer has 100 neurons in case of smaller configuration size (<30) and 200 for higher dimensional data. The output layer consists of 2 neurons for two classes: low-temperatures and high-temperatures. Other parameters:

  1. Activation functions: ReLU for hidden layers, softmax for output layer.
  2. Loss: Categorical Cross Entropy
  3. Optimizer: Adam
  4. Regularization method: L_2 norm + early stopping in callbacks.

The obtained accuracy of training on the test set is around 99% in general.

Results of predictions are output probabilities, averaged over 1000 samples. The trajectories are presented below (red color for high-temperatue class).

Here is the estimation of the critical point based on the predictions. We use finite size scaling to stack all crossing points from the plot above and fit the linear regression and read the intercept value. In the diagram below, the green line indicates range of standard error of intercept.

Estimated value of the critical temperature: T ~= 2.2695 +/- 0.0026

Now, if we are sure, that our system predicts phase of the system based on binary matrix, we can apply dimensionality reduction on original data and repeat training and predictions of models described above. Basically, the PCA method was applied for 75% of retained explained variance, 50% and below 35% of variance. The most reduced case is for keeping 2 principal components. In general, the result of dimensionality reduction is possible without significant loss of training accuracy (about 0.5-1% in comparison to training on original data), which is caused by sparsity of data -- a flagship case of curse of dimensionality.

Projections of 1000 samples on first two principal components

Output layer averaged over validation set after PCA and 2 components as function of temperature

Finite size scaling on crossing temperatures for data reduced by PCA to 2 components

Estimated value of the critical temperature based on low-dimensional data: T ~= 2.2685 +/- 0.0019

Notice that all estimated critical points are within the error range of theoretical value.

ising_nn_pca's People

Contributors

lstasiak avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.