How to visualize Hyena Matrix H(u) from the Paper about safari HOT 1 OPEN

Thank you for the kind words. I'll use a notation close to the paper to make it easier to connect the dots, and set the number of projections to 3. In the first version of Hyena, you will have a different matrix for each channel. To visualize H(u) for a target channel you'll need to materialize the following matrices:

• T_v: toeplitz matrix corresponding to the causal short 1d convolution applied to the target channel of the v projection
• T_x1: same as above, but for a different projection
• T_x2: same as above, but for a different projection
• D_v: the L x L matrix with v at the given channel on the diagonal. Note this is also L x L
• D_x1: same as above
• D_x2: same as above
• T: Toeplitz matrix corresponding to the long implicit convolution.

Once you have these matrices, you can multiply them in the same order you would apply them to v, e.g,:

• H(u) := T_x2 D_x2 T T_x1 D_x1 T_v v

Once you have the matrix you can also check whether applying H(u) gives you the same output via a direct matrix multiply. When you generate the Toeplitz matrices with the convolutional filters, be careful to take into account padding to keep all convolutions causal.

from safari.