Authors:

• Meir Goldberg
• Almog Hadad

A Novel Approach for Neural Network Compression. Using ALDS compression to improve IMP compression Maintain accuracy.

Per layer compression:

• Weight tensor is folded into a matrix

• Uses an SVD to compress the weights

Global compression:

• Ensure global compression rate across all layers

• Maintain model accuracy

• Demonstrated to achieve accurate compression of over 60% on known neural networks

• Based on the lottery ticket hypothesis: A dense randomly initialized feed-forward network contains subnetworks (known as winning tickets) that when trained in isolation reach a test accuracy comparable to the original network in a similar number of iterations.

Identify the winning ticket:

• Train the network

• Prune the smallest magnitude weights

• Winning tickets can be less than 10-20% the size of the full network

• 6 convolutional layers

• 3 FC layers

-Using CIFAR-10 as our dataset

• Run compression with a fixed value

• Reset the net

• Repeat

Real understanding of the results – further analysis requires more resources Our assumption:

• ALDS reduced the values of insignificant parameters

• Maybe also increased the values of significant parameters

• This assisted IMP in pruning the more insignificant parameters

• also likely: ALDS prevented IMP from removing entire layers