train.py --hybrid --vr --iter 4500

Training of Neural Network using Particle Swarm Optimization.

• Losses: [8.012271, 8.002567, 8.042032, 8.00327] Iteration: 0
• Losses: [6.176217, 4.361371, 8.0, 4.4884667] Iteration: 10000
• Losses: [1.7860475, 1.6976731, 4.1016097, 1.7039604] Iteration: 20000
• Losses: [1.6730804, 1.6676208, 1.702151, 1.6678984] Iteration: 30000
• Losses: [0.2826424, 1.6667058, 1.6677036, 1.0983028] Iteration: 40000
• Losses: [0.10317229, 1.6666698, 1.6667093, 0.16274434] Iteration: 50000
• Losses: [0.038965203, 1.6666677, 1.6666709, 0.062464874] Iteration: 60000
• Losses: [0.01462996, 1.6666673, 1.6666676, 0.023709508] Iteration: 70000
• Losses: [0.0054597864, 1.6666669, 1.6666669, 0.008893641] Iteration: 80000
• Losses: [0.002021174, 1.666667, 1.6666667, 0.003305968] Iteration: 90000

With Regular Neural Networks you just hope that you hadn't chosen the second network. If 20000 iterations took 20 days. Even after 20 days are you really sure that you got the best optimum loss and would further training improve network performance.

Thus we propose a new hybrid approach one that scales up crazy parallely. Particle Swarm Optimization with Neural Networks.

• Easy multi GPU training. More GPU's more training. Only the global best weights need to be synchronised. That can be done asynchronously on the CPU. Each GPU could be a single particle is case of huge networks or each GPU could instead be driving 100's of smaller particles
• Higher Learning Rates are Possible. ( > 100x when compared to traditional training). We literally trained with learning rate = 0.1. When's the last time you used that high a learning rate.
• Network is far less likely to get stuck on local optimum
• Faster on GPU depending upon how much GPU power you have and the network size.
• Better initialization. Use this for initialization and then use regular old gradient decent if network size is huge.

• GlobalBest Factor : The higher it is the more the particles stick together. Control Particles from going from too further. Based on my tests it is better to set Global Best Factor > Local Best Factor for training purposes.
• LocalBest Factor: The higher it is the less the particle will move towards the global best. And lesser chance of convergence. Set it to higher values if you wish to increase the exploration. Set a high value if using for initialization and not training
• Velocity Decay: Velocity Decay is best between 0.75 -1. The decay prevents the network weights going too far away.
• Velocity Max : Maximum velocity for a particle along a dimension. It useful to prevent network from seeing a pendulum effect.
• Velocity Max Decay : It is a good idea to have the velocity max go down with iteration for finer control especially if you are not using the hybrid approach
• Learning Rate: A much higher learning rate can be used with hybrid approach for learning weights.
• Number of Particles : The more the merrier. As much as your GPU can support
• Number of Iterations: Until you get bored.

Not that a great idea. Current testing based on fully connected networks, training with only PSO isn't a sufficient. Works great for Initializations. Increasing hidden layers however makes pso converge quicker. An increase in network size is compensated with lesser number of iterations.

Way to go. Faster than traditional approaches if you have sufficient power. Robust against Local Optimas

With the 32 particles the sample network can be trained under 20K iteration with a final loss of 1.9013219e-06 using a Hybrid Approach. Oops did we mention the learning rate 0.1 (Ouch). And yes it never gets stuck up on local minimas. A much bigger search space is explored by the particles as compared to a single particle.

While using traditional approaches we get stuck up 50% of the times with a learning rate of 0.001. Even after 90000 iteration our losses for the best particle were at 0.002021174.

Cite this work if used for academic purposes. For commercial purposes contact me preferably via mail.

• Testing : Rigorous
• Comments : More the Merrier
• Modules : Split em up
• Variable Names : Big is OKay, Non-informative ain't
• Code Format: Python PEP 8

• Trying out other PSO variants
• Testing on MNIST
• Adding support for more Layers

• Local Best PSO without backprop
• More memory efficient atleast Theoratically
• Doesn't store activation of all Layers during Feed Forward