This microservice takes a string of content and returns the token count using the tiktoken
library. It is built using Flask.
- Python 3.x
- Flask
- tiktoken
- Poetry
-
Clone the repository:
git clone https://github.com/haha-systems/toll.git cd toll
-
Install the required packages:
pip install flask tiktoken poetry
-
Create a
requirements.txt
file:pip freeze > requirements.txt
-
Run the Flask app:
python main.py
-
Send a POST request to the
/token_count
endpoint with the content you want to tokenize. For example, usingcurl
:curl -X POST -H "Content-Type: application/json" -d '{"content": "Your content here"}' http://127.0.0.1:4000/token_count
You should get a JSON response with the token count:
{ "token_count": 4 }
- Description: Returns the token count for the provided content.
- Request:
- Headers:
Content-Type: application/json
- Body: JSON object with a
content
field containing the string to be tokenized.
- Headers:
- Response:
- Success: JSON object with a
token_count
field. - Error: JSON object with an
error
field and an appropriate error message.
- Success: JSON object with a
{
"content": "Your content here"
}
{
"token_count": 4
}
This project is licensed under the MIT License.