The second and third week's programming assignment for the UCSanDiego online course is centred around the generative models and classifiers.

The project includes the univariate and bivariate models.

The Wine data set is the running example for our discussion of the generative approach to classification.

The data can be downloaded from the UCI repository, but we have included it in our package. It contains 178 labeled data points, each corresponding to a bottle of wine:

The features (x): a 13-dimensional vector consisting of visual and chemical features for the bottle of wine The label (y): the winery from which the bottle came (1,2,3)

We use a shuffle of the dataset from a fixed permutation. Next, we split the dataset into train and test sections including 130 and 48 data entries respectively.

The model runs based on three parameters:

• The probability of each class pi
• The mean of each class mu
• The variance of each class var

Then the prediction is made by comparing the value of probability of class i times probability of point x belonging to class i:

• Probability of class i: pi
• Probability of point x belonging to class i: Derived from the normal distribution based on the values of mu and var.

The formulae are as follows:

• Univariate:

• Bivariate:

• Multivariate:

This project uses Python 3.10.12 to run and for a list of requirements consult the requirements.txt list.

To run the project, configure the conf.yaml with data about the preprocessing method and dataset features. Then run the entry point main.py. To get a list of programme arguments, run the entry point with argument help.

Minimum train error and the corresponding test error are respectively 0.2076923076923077 0.16666666666666666, that belongs to feature 6 (Flavanoids).

As an example, the histogram and the normal distribution of this feature for the three classes are:

The amount of separation of the distributions of the classes for this feature are as follows:

Which separates the classes quite better relative to other features.

Train errors:

Test errors:

Minimum train error and the corresponding test error are respectively 0.069 and 0.083, that belong to the feature combination (0, 6) (Alcohol, Flavanoids)

Also the decision boundary for the above feature combination turns out to be:

The distribution separation for the three classes based on this selected feature combination is depicted in the following figure.

Conversely, the minimum test error and the corresponding train error are respectively 0.062 and 0.077, that belong to the feature combination (6, 9) (Flavanoids, Color intensity). But, since we only derive the model from train data, this is not a valid choice.

We finally, involve all 13 features to perform the classification and this time we have achieved the following train and test errors:

Train error using all features: 0.015384615384615385

Test error using all features: 0.041666666666666664

The MNIST dataset consists of 60000 images of handwritten digits for the train set and an additional 10000 for the test set.

We have also trained the model on the MNIST train dataset and calculated the test error on the test dataset.

In this section, the covariance turned out to be singular, which made it impossible to work with. To resolve singularity we add a diagonal matrix c.I to the covariance matrix to regularise it. This trick makes sure that the covariance matrix is non-singular and invertible.

At this point, the scalar term c is another parameter of the model that we have to optimise.

Testing the model for different values of c resulted in the following:

Therefore, the value of 4050 was best suiting for the regularising term c.