ctarnold / jpllm Goto Github PK

Instruct model requires chat history of user, assistant, user, assistant alternations. Trying to implement zero-shot prompting while somehow conforming to this requirement is proving difficult.

Given sagor sarker's LID model trained on the LinCE twitter/X data, fine-tune it on the human speech in the bangor miami corpus.

Make slurm scripts consistent in time, memory requested, etc. 30 minutes is shown to be a reasonable time for a few messages to the model. 190GB a reasonable CPU limit. 80GB a reasonable GPU limit.

In web mistral interfaces, the model code-switches to a limited, but consistent, degree. In my implementation, I am seeing a lack of code-switching. I likely have temp to a default of 1, making the model deterministic with a default to English. A preliminary attempt to set temp to some value between 0 and 1 resulted in an error thrown.

Because we need to conform to user, assistant response templates, we can alternate each portion of dialogue. Whenever the speaker switches, switch the user to assistant and vice versa. Steps:

Clean corpus data into chat template
Provide chat template to model
Collect and Save output

Eventually scale to large instruction datasets, needs parallel computations for maximum efficiency. (ex: split 1000 prompts into sets of 50, 100, 250, 500, depending on cluster availability. Test experimentally).

When model with code-switching behavior (at least some code-switching as in the web interface) is implemented, add the component of lang id via the BERT model. Test on the miami corpus, or the LinCE framework.

Compare the success of the LID/POS Tagging Tools to corpus data.

From the linked prompting data in the Southeast Asia code-mixing study, transcribe and modify it to an English/Spanish context.