coderevolutionplugins / gpt-3-encoder-php Goto Github PK

PHP BPE Text Encoder / Decoder for GPT-2 / GPT-3

License: MIT License

PHP 100.00%

gpt-3 gpt-2 gpt-2-simple gpt-2-text-generation gpt-3-prompt gpt-3-prompts gpt-3-text-generation gpt-2-tokenizer gpt-3-tokenizer decoder

gpt-3-encoder-php's Introduction

GPT-3-Encoder-Decoder-PHP

PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a PHP implementation of OpenAI's original python encoder and decoder which can be found here. The main source of inspiration for writing this encoder was the NodeJS version of this encoder, found here.

You can test the results, by comparing the output generated by this script, with the official tokenizer page from OpenAI.

This specific encoder and decoder is used in the Aiomatic WordPress plugin, to count the number of tokens a string will use when sent to OpenAI API. Check more of my work on my website.

Usage

The mbstring PHP extension is needed for this tool to work correctly (in case non-ASCII characters are present in the tokenized text): details here on how to install mbstring

$prompt = "Many words map to one token, but some don't: indivisible. Unicode characters like emojis may be split into many tokens containing the underlying bytes: 🤚🏾 Sequences of characters commonly found next to each other may be grouped together: 1234567890";

$token_array = gpt_encode($prompt);

$original_text = gpt_decode($token_array);

gpt-3-encoder-php's People

Contributors

Stargazers

Watchers

Forkers

myystigri semji searchgpt tuyenlaptrinh adrienbrault imlhb ddebowczyk ger86 lpuzzolokodear tomasvts xiaojian520 andy-sh ethicalsecurity-agency antonybenjiro tw-stephen xiaoguijoker dexit caotico25

gpt-3-encoder-php's Issues

Support gpt-3.5-turbo (cl100k_base encoding)

Support gpt-3.5-turbo (cl100k_base encoding)?

Any chance of this being updated to support 3.5 and 4?

Hi, I'm just wondering if there is any possibility this will be updated for the new cl200k_base encoding? Does it currently cover the p50k_base?

utf8-encode deprecation

utf8-encode deperacation warning for PHP 8.2+

https://www.php.net/manual/en/function.utf8-encode.php

The result of gpt_encode for Chinese characters may not be correct

The expected result was generated by the OpenAI API tokenizer ( https://platform.openai.com/tokenizer ) , while the actual result was produced by the gpt_encode function.

Wrong cases:

一天一蘋果，醫生遠離我！ Expected: 25, Actual: 24
一天一蘋果，醫生遠離我 Expected: 22, Actual: 21
一天一蘋果醫生遠離我 Expected: 19, Actual: 18
一天一蘋果，醫生遠離我 apple a day keeps the doctor away Expected: 29, Actual: 28

Correct cases:

蘋果 Expected: 6, Actual: 6

Extremely slow and causing time-outs?

Hi,

Thanks again for the script.

I do have an issue though, it seems to be very slow and causing our scripts to time-out even after increasing time-out to 120 seconds on a 200 row array loop. We count tokens for each array value of max 200-500 words or so.

When we turned the script/checks off, our scripts ran within 2-4 seconds. Is this a known issue? Anything we can do to help troubleshoot and see what is lagging, and where?

Cheers.

problem with Chinese

English is no problem,
But there is a problem when switching to Chinese

$prompt = "我是 GPT-4，一款先进的自然语言生成 AI。";
echo ('Count: ' . gpt_encode($prompt) );

Count: 34 tokens

in reality: 19 tokens

Error in gpt_utf8_encode function

Function has parameter $s, which is not used in the body. Additionally, the function starts with
$str .= $str;

The parameter should be simply $str instead of $s

[suggest] Any plan to implement GPT-4 encoder

I noticed that the token count differs between GPT-3 and GPT-4. Is there any plan to develop a GPT-4 encoder in PHP?

Python example: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

Thank you for your time.

Move sample code to a different file

It would be nice to move the sample code from the end of gpt3-encoder.php to an independent file. That way we can use the functions without getting that sample execution.