A self-designed neural network, written entirely in Python and the NumPy library. Heavily inspired by Joel Grus' joelnet. Check the .ipynb notebooks out for implementation.

• testing_real_data has a neural network successfully implemented on the Banknote Authentication Data Set.
• testing_custom_data contains a neural network implemented on the XOR logic function. And a custom dataset with an XOR-like distribution.

.
├── dataset/
|   └── data_banknote_authentication.txt
├── numpyNET/
|   ├── activation.py
|   ├── data_loader.py
|   ├── error.py
|   ├── layers.py
|   ├── nn.py
|   ├── optimizer.py
|   └── train.py
├── README.md
├── requirements.txt
├── testing_custom_data.ipynb
└── testing_real_data.ipynb

from numpyNET.nn import Model
from numpyNET.layers import Dense, Sigmoid
from numpyNET.optimizer import SGD
from numpyNET.error import Error, MSE, BinaryCrossEntropy
from numpyNET.data_loader import BatchIterator
from numpyNET.train import train, predict

# Designing a Model
model = Model([
    Dense(input_size=..., output_size=...),
    Sigmoid(),
    Dense(input_size=..., output_size=...),
    Sigmoid()
])

# Model Hyperparameters
num_epochs = ...
optim = SGD(learn_rate=...)
batch_size = ...
err = MSE()

# Training
train_features = ...
train_labels = ...
history = train(
    model, train_features, train_labels, plotting=True,
    epochs=num_epochs, optimizer=optim, err=err,
    iterator=BatchIterator(batch_size=batch_size)
)

# Predictions
unknown_features = ...
prediction = predict(model, unknown_features)

The classic linear layer that takes an input matrix and returns its matrix product with a weight matrix and a bias term added. Each neuron in the Dense layer receives input from all neurons of its previous layer. Activations are implemented as separate layers.

The rectified linear activation function, or ReLU activation function, is perhaps the most common function used for hidden layers. In a ReLU unit, if the input value is negative, then a value 0.0 is obtained, otherwise, the input is returned.

Another classic, it applies a Sigmoid function to the input such that the output is bounded in the interval (0, 1). The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to 0.0.

This layer applies a Tanh function to the input. The output is bounded in the interval (-1, 1). The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1.0.

Incomplete. Might have some issues, as the gradient either explodes, or vanishes to nan.

Untested.

Implmented. Well tested.

The batch sizes are implemented in data_loader.py. Will work on implementing Momentum / Adagrad / Adam optimizers.