This is an edited version of the go-deep library except it has been converted to 32-bit for better performance and some extra activation functions have been added (Elu, Mish and Swish, RootX, MulDiv and DoubleRoot)
Update: concurrency is now used more to increase performance and enabled multiple activation functions in the one network. (one activation function type per layer)
Feed forward/backpropagation neural network implementation. Currently supports:
- Activation functions: sigmoid, hyperbolic, ReLU, Elu, Mish, Swish, also activations I created (RootX, DivX, DoublePow, DoubleRoot and DoubleDiv).. RootX is particularly effective.
- Double Root is looks like a combination of the sqrt function and tanh (double Div is similar except using division), they can be used to squash numbers inside your network to prevent the network from exploding... Boom!
- I designed DoubleDiv, DoublePow & DoubleRoot to help the neural networks solve mathematical equations, usually used with the linear activation function
- RootX (combining sqrt with relu) seems to solve problems facter than Mish and Swish... Still testing DivX (combining division with relu) but should produce similar results to RootX
- Solvers: SGD, SGD with momentum/nesterov, Adam
- Classification modes: regression, multi-class, multi-label, binary
- Supports batch training in parallel
- Bias nodes
Networks are modeled as a set of neurons connected through synapses. No GPU computations - don't use this for any large scale applications.
go get -u github.com/nathanleary/neural-net
Import the go-deep package
import (
"fmt"
deep "github.com/nathanleary/neural-net"
"github.com/nathanleary/neural-net/training"
)Define some data...
var data = training.Examples{
{[]float32{2.7810836, 2.550537003}, []float32{0}},
{[]float32{1.465489372, 2.362125076}, []float32{0}},
{[]float32{3.396561688, 4.400293529}, []float32{0}},
{[]float32{1.38807019, 1.850220317}, []float32{0}},
{[]float32{7.627531214, 2.759262235}, []float32{1}},
{[]float32{5.332441248, 2.088626775}, []float32{1}},
{[]float32{6.922596716, 1.77106367}, []float32{1}},
{[]float32{8.675418651, -0.242068655}, []float32{1}},
}Create a network with two hidden layers of size 2 and 2 respectively:
n := deep.NewNeural(&deep.Config{
/* Input dimensionality */
Inputs: 2,
/* Three hidden layers consisting of two neurons each, and a single output */
Layout: []int{2, 2, 2, 2, 1},
/* Activation functions: Sigmoid, Tanh, ReLU, Linear, Elu, Mish, Swish, RootX, DoubleRoot */
/*Defining the three hidden layer's Activation function*/
Activation: []deep.ActivationType{
deep.ActivationMulDiv,
deep.ActivationRootX,
deep.ActivationDoubleRoot,
deep.ActivationMish,
},
/* Determines output layer activation & loss function:
ModeRegression: linear outputs with MSE loss
ModeMultiClass: softmax output with Cross Entropy loss
ModeMultiLabel: sigmoid output with Cross Entropy loss
ModeBinary: sigmoid output with binary CE loss */
Mode: deep.ModeBinary,
/* Weight initializers: {deep.NewNormal(μ, σ), deep.NewUniform(μ, σ)} */
Weight: deep.NewNormal(1.0, 0.0),
/* Apply bias */
Bias: true,
})Train:
// params: learning rate, momentum, alpha decay, nesterov
optimizer := training.NewSGD(0.05, 0.1, 1e-6, true)
// params: optimizer, verbosity (print stats at every 50th iteration)
trainer := training.NewTrainer(optimizer, 50)
training, heldout := data.Split(0.5)
trainer.Train(n, training, heldout, 1000) // training, validation, iterationsresulting in:
Epochs Elapsed Error
--- --- ---
5 12.938µs 0.36438
10 125.691µs 0.02261
15 177.194µs 0.00404
...
1000 10.703839ms 0.00000
Finally, make some predictions:
fmt.Println(data[0].Input, "=>", n.Predict(data[0].Input))
fmt.Println(data[5].Input, "=>", n.Predict(data[5].Input))Alternatively, batch training can be performed in parallell:
optimizer := NewAdam(0.001, 0.9, 0.999, 1e-8)
// params: optimizer, verbosity (print info at every n:th iteration), batch-size, number of workers
trainer := training.NewBatchTrainer(optimizer, 1, 200, 4)
training, heldout := data.Split(0.75)
trainer.Train(n, training, heldout, 1000) // training, validation, iterationsSee training/trainer_test.go for a variety of toy examples of regression, multi-class classification, binary classification, etc.
See examples/ for more realistic examples:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent