gptq 4bit avg loss is large about autogptq HOT 3 OPEN

Make sure you have proper dataset that is similar to the training dataset.

from autogptq.

@moseshu Have you figure this problem out?

from autogptq.

1. From my experience, moe models are much harder to quant due to gate/routers.

2. You need good dataset, as close to originals training as possible with high enough of nsamples. 128 for every 7B is my rule.

3. Later layers have always had much higher losses than earlier layers.

Based on what you posted, can't tell if quant is good or not. Do ppl post quant and compare running avg loss of early vs later layers.

from autogptq.