Bug in the MP fits of the randomized ESDs about weightwatcher HOT 8 CLOSED

Thanks for submitting

The MP solver has some issues that sometimes cause the residual to off for the first case

I will try to fix in the next minor release, 0.4.8

from weightwatcher.

This is a scaling issue

From what I can tell, different frameworks set the normalization of W differently, and I have not figured out a way to automatically detect and fix this yet

In RMT (random matrix theory), we expect W to say a Gaussian random matrix, with variance 1.0, and normalized by 1/N. But the frameworks don't actually do this--they might use Glorot normalization (which is not 100% the same ) , and they frequently rescale W by say 1/(N+M), or sqrt(2/(N+M)), rather than 1/N. So when we initialize the DNN layers, we need to figure out what was done, and then move back to this normalization (see below)

My best idea right now is to simply let the user enter their own normalization function, i.e:
watcher.analyze(..., normalizer=(lambda N, M, rf: (N/(N+M))))
So that in the code we can, for every layer, we can fix the normalization using:
W = W*normalized(N,M,rf) I think this will work for the Transformers (not sure about the Conv2D layers though)

Here is an footnote from our long paper which discusses this
(its buried in a footnote because it is a minor tweak to the theory--but for for the MP fits to work , its critical to get this right)

from weightwatcher.

The following seems to help in my case.

First, change the range of sigma in the following function.

Second, change the following condition.

Third, change the following normalization code.

I test locally on a bunch of NLP models with only linear layers. The results seem to be fine.

There are still some cases that I do not know how to solve. For example, for CV models, it is possible that the eigenvalues do not follow the quarter circle function. In some other cases, the ESD is not very similar to any MP distribution or QC distribution, such as the following figure.

from weightwatcher.

I have implemented your fix, but the VGG model does not look correct
Can you please confirm that the fix works on all plots in the VGG notebooks provided
WeightWatcher-VGG.ipynb
WeightWatcher-VGG-Random.ipynb
and the GPT notebooks
The fix appears to break the QC plots, when N=M, Q=1 for VGG

But the GPT models seems ok when Q=1

from weightwatcher.

For example, for CV models, it is possible that the eigenvalues do not follow the quarter circle function.
If the matrix is randomized, it will mostly follow the QC law
I suspect this is a scaling issue

from weightwatcher.

Ok, so I think the previous fix still has a problem. On my side, I tried to change

Previously, I suggested Wscale = np.sqrt(to_plot.shape[0]*rf)/Wnorm. I did not understand the meaning of rf and assumed that is always 1.

Now, the VGG seems to give a proper result.

Could you see if that works?

from weightwatcher.

Another potential issue is that Q can be < 1.0 in the following line.

I think one should add an if-condition to check if N and M should be flipped.

from weightwatcher.

I think this is now fixed, see #60
closing

from weightwatcher.