Paper：ADAPTING LARGE LANGUAGE MODELS VIA READING COMPREHENSION about lmops HOT 3 CLOSED

Hi, after constructing the data, we DID continue with the standard NEXT TOKEN PREDICT Task on the original model.

from lmops.

I'm sorry to disturb you. I'd like to ask you again. Following the approach in your article, the difference is that I will using the data in the supervised fine-tuning period (use Lora or other PEFT methods), will it have a certain effect?(Constrained by hardware limitations, I am unable to perform continue pre-training. I intend to utilize your approach inject the domain data through supervised training. )

from lmops.

Hi, I think PEFT methods should be effective, as our approach can be seen as a form of supervised fine-tuning. Just make sure to compute the NEXT TOKEN PREDICT Task loss on all the tokens.

from lmops.