This is the code used to optimize the weights of an under-parameterized neural network. Training is done via gradient flow using MLPGradientFlow.jl (https://arxiv.org/abs/2301.10638). In this repo, we release the code to train and visualize the result of training for the erf activation function and standard Gaussian input data (https://arxiv.org/abs/2311.01644).
- This file README.md
- Simulation file
erf50/erf_sims.jl
- Script to see the loss curves and gradient norms
plot-training.py
- Script to see the summary of training
plot-training-summary.py
for all widths - Script to visualize the weights at convergence
plot-results.py
- Helper functions
helper.py
To visualize the results using Python as done in this repo, need to install
- juliacall
- numpy
- matplotlib
We find that gradient flow converges to either one of two minima depending on the direction of initialization when the student width is about one-half of teacher width.
We plot the results for
![configs](https://private-user-images.githubusercontent.com/37277437/296828193-bf4a51bc-7539-4ac6-a7e8-b8cf6a1351a2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUxMDI4MjgsIm5iZiI6MTcyNTEwMjUyOCwicGF0aCI6Ii8zNzI3NzQzNy8yOTY4MjgxOTMtYmY0YTUxYmMtNzUzOS00YWM2LWE3ZTgtYjhjZjZhMTM1MWEyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMxVDExMDg0OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJlN2Q2NGYxMTEwNWYzYzEyZTIxMjQ2ZDg3NDE1ZjM4ZTg4NjViMzkyZGJmOTZlNjc3NDIyNWQwMTJiMGZjODgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.KUlQQ1zmTYZbfwjymEdDedKVVDl4cSqr6FQfkLvmjms)
Jan 15, 2024