sbatten / tokenizer Goto Github PK
View Code? Open in Web Editor NEWThis project forked from microsoft/tokenizer
.NET and Typescript implementation of BPE tokenizer for OpenAI LLMs.
Home Page: https://github.com/microsoft/Tokenizer/
License: MIT License