theunodb / spaza-ai-simple-bpe-tokenizer Goto Github PK
View Code? Open in Web Editor NEWThis package provides a set of tools for processing text using the GPT family of models. The GPT models process text using tokens, which are common sequences of characters found in text. The models understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens.
License: BSD 3-Clause "New" or "Revised" License