method of composing reductions across layers about laser HOT 2 CLOSED

Hello, thank you for your interest in our work!

1. There might be a dependence on whether a reduction at a certain layer is helpful, given there is a reduction made in another layer. Therefore, we cannot just compose reductions after individually searching for the best rate per layer, while holding other layers fixed.

2. Added a script that walks through our procedure for this.

We wanted to check if the benefits of performing laser across layers are additive or not, so we employed a simple strategy. The current strategy we use is as follows:
1. Initialize a vector that represents different amounts of reduction across each of the different layers
2. Edit the model with Laser starting from the final layer (restricted over a set $\rho$ values)
3. Validate over the validation set of the dataset
4. use the signal from step 2 to reduce or increase the amount of reduction
5. repeat until convergence
6. return the vector of reductions

I have also added a script that walks through this procedure under the scripts directory.

One thing to note: This search procedure is probably not the most efficient and only performs a sparse search over possible $\rho$ values and only over the encoder MLP layers ('MLP_FC_IN') layers. A more thorough search might result in additional increased improvements!

from laser.

MANY THANKS!!!!

from laser.