pipeline

pretraining -> fine tunning -> (learn to rank answers) -> rlhf

1. huge amount of data, 99% of computation here. The basic idea is through lots of iterations try to predict the next token given the context window and the weight of the neural net. Adjust the weight accordingly. The model can be trik into being a useful asssistant but is primaliry a next work predictor.
2. smaller amount of data, 10k-100k samples of high quality prompts -> answer. Dataset cuareted by contractors.
3. given a number of answers to the same prompt, rank them and teach the model how to predict the score.
4. given the model that knows how to predict the score of a useful anwser, generate answers that have a good score by adjusting the asking prompts and adjusting the weight based on score.

human vs llm brains

The human is able to combine a lot of thins, like off-tooling, check for correctness and try again vs llms that are, like train trying to build the road ahead, predicting the next word. They have a big fact-based knowledge "database" in the weihght of the nn and a "perfect" working memory in the context window. =

prompting

In order to improve llms brain, the following strategies appear.

Chain of thought, is a useful idea that enables the llm to work with more token while generating the answer rather than only relying in the ones in the prompt. The basic idea is to provide examples or prompt the model to answer in a particular way, like step by step thinking (zero-shot cot) of showing the model how to reason (cot). Another usefull way is tree of thought or siomply asking the model to produce a better answer, like, imagine you are 120IQ or an expert in the field.

The model spends the same amount of computationper token, so if the model is able to produce the answer in more token, the answer is porobably going to be better as is not trying to pack a huge amount of computation/interpratation intoa smaller amount of tokens.

side notes

1. llama is a great pre-training model.

from hdc.