mlx 0.13 very slow with q8 and fp16 about mlx-examples HOT 5 OPEN

Can this be related to model size? 70B?

Oh yes, it's almost certainly related to the size / amount of RAM required. There seems to be a performance cliff for very large models. It shouldn't be swapping because its still not using all the RAM on the machine but it does seem related to memory page demand. Still debugging.. this one might take a little while to iron out.

from mlx-examples.

I tested on M3 Max and got same result. with even lower power consumption and GPU frequency. Really strange.

from mlx-examples.

It's pretty slow for me also in 8-bit (and presumably 16 as well, but didn't test). Not sure why yet.

from mlx-examples.

@ivanfioravanti just curious you phrased this issue as if it used to be faster in previous MLX versions. Is that the case?

from mlx-examples.

Sorry for the delay I was out the whole week and happy to be back playing with MLX 🛝
I thought the issue was impacting all models using q8 but this is not the case. It seems issue is present with Llama 70B only.

• Mistral-7B-Instruct-v0.2 works like a charm
• mlx-community/dolphin-2.9.1-yi-1.5-34b-8bit works like a charm

Can this be related to model size? 70B? I will try comparing another large model.

from mlx-examples.