berriai / litellm-proxy Goto Github PK

License: MIT License

Dockerfile 2.89% Python 97.11%

litellm-proxy's Introduction

liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching

⚠️ DEPRECATION WARNING: LiteLLM is our new home. You can find the LiteLLM Proxy there. Thank you for checking us out! ❤️

Azure, Llama2, OpenAI, Claude, Hugging Face, Replicate Models

Usage

Step 1: Put your API keys in .env Copy the .env.template and put in the relevant keys (e.g. OPENAI_API_KEY="sk-..")

Step 2: Test your proxy Start your proxy server

$ cd litellm-proxy && python3 main.py

Make your first call

import openai 

openai.api_key = "sk-litellm-master-key"
openai.api_base = "http://0.0.0.0:8080"

response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey"}])

print(response)

What does liteLLM proxy do

Make /chat/completions requests for 50+ LLM models Azure, OpenAI, Replicate, Anthropic, Hugging Face

Example: for model use claude-2, gpt-3.5, gpt-4, command-nightly, stabilityai/stablecode-completion-alpha-3b-4k

{
  "model": "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1",
  "messages": [
    {
      "content": "Hello, whats the weather in San Francisco??",
      "role": "user"
    }
  ]
}

Consistent Input/Output Format
- Call all models using the OpenAI format - completion(model, messages)
- Text responses will always be available at ['choices'][0]['message']['content']
Error Handling Using Model Fallbacks (if GPT-4 fails, try llama2)
Logging - Log Requests, Responses and Errors to Supabase, Posthog, Mixpanel, Sentry, LLMonitor, Traceloop, Helicone (Any of the supported providers here: https://docs.litellm.ai/docs/

Example: Logs sent to Supabase
Token Usage & Spend - Track Input + Completion tokens used + Spend/model
Caching - Implementation of Semantic Caching
Streaming & Async Support - Return generators to stream text responses

API Endpoints

`/chat/completions` (POST)

This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc

Input

This API endpoint accepts all inputs in raw JSON and expects the following inputs

model (string, required): ID of the model to use for chat completions. See all supported models [here]: (https://docs.litellm.ai/docs/): eg gpt-3.5-turbo, gpt-4, claude-2, command-nightly, stabilityai/stablecode-completion-alpha-3b-4k
messages (array, required): A list of messages representing the conversation context. Each message should have a role (system, user, assistant, or function), content (message text), and name (for function role).
Additional Optional parameters: temperature, functions, function_call, top_p, n, stream. See the full list of supported inputs here: https://docs.litellm.ai/docs/

Example JSON body

For claude-2

{
  "model": "claude-2",
  "messages": [
    {
      "content": "Hello, whats the weather in San Francisco??",
      "role": "user"
    }
  ]
}

Making an API request to the Proxy Server

import requests
import json

# TODO: use your URL
url = "http://localhost:5000/chat/completions"

payload = json.dumps({
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "content": "Hello, whats the weather in San Francisco??",
      "role": "user"
    }
  ]
})
headers = {
  'Content-Type': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)

Output [Response Format]

Responses from the server are given in the following format. All responses from the server are returned in the following format (for all LLM models). More info on output here: https://docs.litellm.ai/docs/

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I'm sorry, but I don't have the capability to provide real-time weather information. However, you can easily check the weather in San Francisco by searching online or using a weather app on your phone.",
        "role": "assistant"
      }
    }
  ],
  "created": 1691790381,
  "id": "chatcmpl-7mUFZlOEgdohHRDx2UpYPRTejirzb",
  "model": "gpt-3.5-turbo-0613",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 41,
    "prompt_tokens": 16,
    "total_tokens": 57
  }
}

Installation & Usage

Running Locally

Clone liteLLM repository to your local machine:

git clone https://github.com/BerriAI/liteLLM-proxy

Install the required dependencies using pip
```
pip install requirements.txt
```

(optional)Set your LiteLLM proxy master key

os.environ['LITELLM_PROXY_MASTER_KEY]` = "YOUR_LITELLM_PROXY_MASTER_KEY"
or
set LITELLM_PROXY_MASTER_KEY in your .env file

Set your LLM API keys

os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
or
set OPENAI_API_KEY in your .env file

Run the server:
```
python main.py
```

Deploying

Quick Start: Deploy on Railway
GCP, AWS, Azure This project includes a Dockerfile allowing you to build and deploy a Docker Project on your providers

Support / Talk with founders

Our calendar 👋
Community Discord 💭
Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
Our emails ✉️ [email protected] / [email protected]

Roadmap

Support hosted db (e.g. Supabase)
Easily send data to places like posthog and sentry.
Add a hot-cache for project spend logs - enables fast checks for user + project limitings
Implement user-based rate-limiting
Spending controls per project - expose key creation endpoint
Need to store a keys db -> mapping created keys to their alias (i.e. project name)
Easily add new models as backups / as the entry-point (add this to the available model list)

litellm-proxy's People

Contributors

Stargazers

Watchers

litellm-proxy's Issues

Server failing to start while setting for Redis Semantic Cache

Getting the following error while deploying the proxy server on kubernetes cluster with semantic cache set as true.

INFO: Started server process [1]
INFO: Waiting for application startup.
ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 608, in aenter
await self._router.startup()
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 709, in startup
await handler()
File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3061, in startup_event
await initialize(**worker_config)
File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2905, in initialize
) = await proxy_config.load_config(router=llm_router, config_file_path=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2059, in load_config
self._init_cache(cache_params=cache_params)
File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 1960, in _init_cache
litellm.cache = Cache(**cache_params)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/litellm/caching.py", line 1326, in init
self.cache = RedisSemanticCache(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/litellm/caching.py", line 582, in init
from redisvl.index import SearchIndex
ModuleNotFoundError: No module named 'redisvl'

ERROR: Application startup failed. Exiting.

#------------------------------------------------------------#

'This feature doesn't meet my needs because...'

https://github.com/BerriAI/litellm/issues/new

#------------------------------------------------------------#

Thank you for using LiteLLM! - Krrish & Ishaan

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new

key_management_settings error in 1.40.19

Seeing the following stack trace in 1.40.19

Traceback (most recent call last):
  File "/opt/casemark/litellm/bin/litellm", line 8, in <module>
    sys.exit(run_server())
             ^^^^^^^^^^^^
  File "/opt/casemark/litellm/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/casemark/litellm/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/casemark/litellm/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/casemark/litellm/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/casemark/litellm/lib/python3.11/site-packages/litellm/proxy/proxy_cli.py", line 498, in run_server
    if key_management_settings is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'key_management_settings' where it is not associated with a value

Looking at the code it kind of makes sense that this would error,
Line 498 is trying to access key_management_settings, but it's undefined here based on the indention level.
Even though key_management_system is defined on 472 in scope at 498

I would create a PR, but I'm not sure what the intention is here

ModuleNotFoundError: No module named 'openai.error'

$ python main.py
Python-dotenv could not parse statement starting at line 1
Traceback (most recent call last):
File "/Users/icourt/PythonProjects/liteLLM-proxy-main/main.py", line 8, in
import llm as llm
File "/Users/icourt/PythonProjects/liteLLM-proxy-main/llm.py", line 10, in
import openai.error
ModuleNotFoundError: No module named 'openai.error'
(litellm-proxy-main-3.11)

Add helm chart for kubernetes

The Feature
add helm charts for litellm proxy

Motivation, pitch
user request

Twitter / LinkedIn details
https://www.linkedin.com/in/rjmath/

Unauthorized

No matter what I do, I get an 401 Unauthorized response back.

I tried using the .env and setting up my OpenAI key.
I tried using Bearer Auth header with my OpenAI key.

Here is the json body I'm sending via PostMan:

{
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "content": "Hello, whats the weather in San Francisco??",
      "role": "user"
    }
  ]
}

I've also tried accessing TogtherAI models and am having the same problem.
Using the latest git version.

it fails on the [completions] or [chat/completions] for JSONDecodeError

after run [ litellm --model ollama/xxx], it works fine on the swagger URL.
and it works well on [GET /models].
but it fails on the [completions] or [chat/completions]. it

{
"error": {
"message": "Expecting value: line 1 column 1 (char 0)\n\nTraceback (most recent call last):\n File "/home/ubuntu/.local/lib/python3.8/site-packages/litellm/proxy/proxy_server.py", line 1456, in completion\n data = json.loads(body_str)\n File "/usr/lib/python3.8/json/init.py", line 357, in loads\n return _default_decoder.decode(s)\n File "/usr/lib/python3.8/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n",
"type": "None",
"param": "None",
"code": 500
}
}

Deploy LiteLLM-Proxy on AWS Lambda

Hello,

Thanks for this nice proxy. I created a demo for deploying it on AWS Lambda with Lambda Web Adapter, and using Lambda Function URL to provide a HTTPS endpoint.

Hope this is useful.

Info Endpoint in Openai v1/model style

It would be helpful to provide the information on which model is running.

Update railway button

railway button is out of sync - causes deployment issues. need to fix this.

Issue with docker image and calls

I have been using the following docker file for start up with issue but just started getting problems

services:

  tgi:
    image: ghcr.io/huggingface/text-generation-inference:1.2
    command: --model-id TheBloke/zephyr-7B-beta-AWQ --max-batch-prefill-tokens 2048 --quantize awq
    volumes:
      - ./models:/data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  llm-api:
    image: ghcr.io/berriai/litellm:main-v1.10.3
    
    command:
      - /bin/sh
      - -c
      - |
        pip install async_generator
        litellm --model huggingface/TheBloke/zephyr-7B-beta-AWQ --api_base http://tgi/generate_stream --host 0.0.0.0 --port 3000
    entrypoint: []
    ports:
      - 7766:3000    
    platform: linux/amd64

I do the following curl call

curl http://localhost:7766/v1/chat/completions    -H "Content-Type: application/json"   -d '{  "model": "zephyr-7B-beta-AWQ", "messages": [{"role": "user", "content": "what is the capital of england"}]}'

and get the following. While the question gets answered at the end of the response it also throws an error - "detail":"HuggingfaceException - Expecting value: line 1 column 1 (char 0)

{"detail":"HuggingfaceException - Expecting value: line 1 column 1 (char 0)\n\nOriginal Response: data:{\"token\":{\"id\":28789,\"text\":\"<\",\"logprob\":-0.0015649796,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-0.0000034570694,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":489,\"text\":\"ass\",\"logprob\":-0.0000010728836,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11143,\"text\":\"istant\",\"logprob\":-0.0000023841858,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-8.34465e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28767,\"text\":\">\",\"logprob\":-0.0000011920929,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":13,\"text\":\"\\n\",\"logprob\":-0.00062561035,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1014,\"text\":\"The\",\"logprob\":-0.24304199,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.023544312,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":302,\"text\":\" of\",\"logprob\":-0.3972168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.0012617111,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.0013980865,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":4222,\"text\":\" London\",\"logprob\":-0.019927979,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.19641113,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2993,\"text\":\" However\",\"logprob\":-0.26660156,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.000010609627,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.46411133,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.1472168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-1.1308594,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.054351807,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.0000051259995,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2939,\"text\":\" country\",\"logprob\":-0.0051498413,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2373,\"text\":\" within\",\"logprob\":-0.51416016,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.000021338463,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-0.017425537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000008225441,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.024337769,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.37060547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.40722656,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-1.0527344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000028848648,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.0693359,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.00016498566,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2894,\"text\":\" whole\",\"logprob\":-0.000089645386,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1235,\"text\":\" does\",\"logprob\":-0.38671875,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":459,\"text\":\" not\",\"logprob\":-0.00006234646,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":506,\"text\":\" have\",\"logprob\":-0.0008454323,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.0059127808,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.03540039,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.23498535,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.1435547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1259,\"text\":\" such\",\"logprob\":-0.14331055,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.3293457,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":415,\"text\":\" The\",\"logprob\":-0.6118164,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":6194,\"text\":\" UK\",\"logprob\":-0.96777344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28742,\"text\":\"'\",\"logprob\":-0.39111328,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28713,\"text\":\"s\",\"logprob\":-0.0000022649765,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":799,\"text\":\" other\",\"logprob\":-0.39526367,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.02331543,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.000024080276,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5780,\"text\":\" countries\",\"logprob\":-0.00116539,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":460,\"text\":\" are\",\"logprob\":-0.3400879,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14322,\"text\":\" Scotland\",\"logprob\":-0.033111572,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0056495667,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14831,\"text\":\" Wales\",\"logprob\":-0.0023822784,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0059928894,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.00001168251,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":12781,\"text\":\" Northern\",\"logprob\":-0.00004386902,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11170,\"text\":\" Ireland\",\"logprob\":-0.0000026226044,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.14685059,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1430,\"text\":\" each\",\"logprob\":-0.26757812,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":395,\"text\":\" with\",\"logprob\":-0.05065918,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":871,\"text\":\" its\",\"logprob\":-0.4020996,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1216,\"text\":\" own\",\"logprob\":-0.007835388,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.027450562,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.07928467,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28747,\"text\":\":\",\"logprob\":-0.6953125,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":25970,\"text\":\" Edinburgh\",\"logprob\":-0.00020945072,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.011047363,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8564,\"text\":\" Card\",\"logprob\":-0.00029182434,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2728,\"text\":\"iff\",\"logprob\":-0.00000333786,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.00059080124,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.0000019073486,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":365,\"text\":\" B\",\"logprob\":-0.000027298927,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":599,\"text\":\"elf\",\"logprob\":-2.3841858e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":529,\"text\":\"ast\",\"logprob\":-0.000002026558,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.04925537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8628,\"text\":\" respectively\",\"logprob\":-0.000021219254,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.0000834465,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2,\"text\":\"</s>\",\"logprob\":-0.18835449,\"special\":true},\"generated_text\":\"<|assistant|>\\nThe capital of England is London. However, England is a constituent country within the United Kingdom, and the United Kingdom as a whole does not have a capital city as such. The UK's other constituent countries are Scotland, Wales, and Northern Ireland, each with its own capital city: Edinburgh, Cardiff, and Belfast, respectively.\",\"details\":{\"finish_reason\":\"eos_token\",\"generated_tokens\":80,\"seed\":null}}\n\n\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/litellm/llms/huggingface_restapi.py\", line 456, in acompletion\n    response_json = response.json()\n  File \"/usr/local/lib/python3.9/site-packages/httpx/_models.py\", line 761, in json\n    return jsonlib.loads(self.content, **kwargs)\n  File \"/usr/local/lib/python3.9/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/local/lib/python3.9/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/local/lib/python3.9/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/litellm/main.py\", line 185, in acompletion\n    response = await init_response\n  File \"/usr/local/lib/python3.9/site-packages/litellm/llms/huggingface_restapi.py\", line 472, in acompletion\n    raise HuggingfaceError(status_code=500, message=f\"{str(e)}\\n\\nOriginal Response: {response.text}\")\nlitellm.llms.huggingface_restapi.HuggingfaceError: Expecting value: line 1 column 1 (char 0)\n\nOriginal Response: data:{\"token\":{\"id\":28789,\"text\":\"<\",\"logprob\":-0.0015649796,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-0.0000034570694,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":489,\"text\":\"ass\",\"logprob\":-0.0000010728836,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11143,\"text\":\"istant\",\"logprob\":-0.0000023841858,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-8.34465e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28767,\"text\":\">\",\"logprob\":-0.0000011920929,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":13,\"text\":\"\\n\",\"logprob\":-0.00062561035,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1014,\"text\":\"The\",\"logprob\":-0.24304199,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.023544312,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":302,\"text\":\" of\",\"logprob\":-0.3972168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.0012617111,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.0013980865,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":4222,\"text\":\" London\",\"logprob\":-0.019927979,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.19641113,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2993,\"text\":\" However\",\"logprob\":-0.26660156,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.000010609627,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.46411133,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.1472168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-1.1308594,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.054351807,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.0000051259995,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2939,\"text\":\" country\",\"logprob\":-0.0051498413,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2373,\"text\":\" within\",\"logprob\":-0.51416016,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.000021338463,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-0.017425537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000008225441,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.024337769,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.37060547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.40722656,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-1.0527344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000028848648,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.0693359,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.00016498566,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2894,\"text\":\" whole\",\"logprob\":-0.000089645386,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1235,\"text\":\" does\",\"logprob\":-0.38671875,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":459,\"text\":\" not\",\"logprob\":-0.00006234646,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":506,\"text\":\" have\",\"logprob\":-0.0008454323,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.0059127808,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.03540039,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.23498535,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.1435547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1259,\"text\":\" such\",\"logprob\":-0.14331055,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.3293457,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":415,\"text\":\" The\",\"logprob\":-0.6118164,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":6194,\"text\":\" UK\",\"logprob\":-0.96777344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28742,\"text\":\"'\",\"logprob\":-0.39111328,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28713,\"text\":\"s\",\"logprob\":-0.0000022649765,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":799,\"text\":\" other\",\"logprob\":-0.39526367,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.02331543,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.000024080276,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5780,\"text\":\" countries\",\"logprob\":-0.00116539,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":460,\"text\":\" are\",\"logprob\":-0.3400879,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14322,\"text\":\" Scotland\",\"logprob\":-0.033111572,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0056495667,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14831,\"text\":\" Wales\",\"logprob\":-0.0023822784,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0059928894,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.00001168251,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":12781,\"text\":\" Northern\",\"logprob\":-0.00004386902,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11170,\"text\":\" Ireland\",\"logprob\":-0.0000026226044,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.14685059,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1430,\"text\":\" each\",\"logprob\":-0.26757812,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":395,\"text\":\" with\",\"logprob\":-0.05065918,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":871,\"text\":\" its\",\"logprob\":-0.4020996,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1216,\"text\":\" own\",\"logprob\":-0.007835388,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.027450562,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.07928467,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28747,\"text\":\":\",\"logprob\":-0.6953125,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":25970,\"text\":\" Edinburgh\",\"logprob\":-0.00020945072,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.011047363,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8564,\"text\":\" Card\",\"logprob\":-0.00029182434,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2728,\"text\":\"iff\",\"logprob\":-0.00000333786,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.00059080124,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.0000019073486,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":365,\"text\":\" B\",\"logprob\":-0.000027298927,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":599,\"text\":\"elf\",\"logprob\":-2.3841858e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":529,\"text\":\"ast\",\"logprob\":-0.000002026558,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.04925537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8628,\"text\":\" respectively\",\"logprob\":-0.000021219254,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.0000834465,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2,\"text\":\"</s>\",\"logprob\":-0.18835449,\"special\":true},\"generated_text\":\"<|assistant|>\\nThe capital of England is London. However, England is a constituent country within the United Kingdom, and the United Kingdom as a whole does not have a capital city as such. The UK's other constituent countries are Scotland, Wales, and Northern Ireland, each with its own capital city: Edinburgh, Cardiff, and Belfast, respectively.\",\"details\":{\"finish_reason\":\"eos_token\",\"generated_tokens\":80,\"seed\":null}}\n\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/litellm/proxy/proxy_server.py\", line 873, in chat_completion\n    response = await litellm.acompletion(**data)\n  File \"/usr/local/lib/python3.9/site-packages/litellm/utils.py\", line 1465, in wrapper_async\n    raise e\n  File \"/usr/local/lib/python3.9/site-packages/litellm/utils.py\", line 1411, in wrapper_async\n    result = await original_function(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/litellm/main.py\", line 195, in acompletion\n    raise exception_type(\n  File \"/usr/local/lib/python3.9/site-packages/litellm/utils.py\", line 4634, in exception_type\n    raise e\n  File \"/usr/local/lib/python3.9/site-packages/litellm/utils.py\", line 4199, in exception_type\n    raise APIError(\nlitellm.exceptions.APIError: HuggingfaceException - Expecting value: line 1 column 1 (char 0)\n\nOriginal Response: data:{\"token\":{\"id\":28789,\"text\":\"<\",\"logprob\":-0.0015649796,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-0.0000034570694,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":489,\"text\":\"ass\",\"logprob\":-0.0000010728836,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11143,\"text\":\"istant\",\"logprob\":-0.0000023841858,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28766,\"text\":\"|\",\"logprob\":-8.34465e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28767,\"text\":\">\",\"logprob\":-0.0000011920929,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":13,\"text\":\"\\n\",\"logprob\":-0.00062561035,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1014,\"text\":\"The\",\"logprob\":-0.24304199,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.023544312,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":302,\"text\":\" of\",\"logprob\":-0.3972168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.0012617111,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.0013980865,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":4222,\"text\":\" London\",\"logprob\":-0.019927979,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.19641113,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2993,\"text\":\" However\",\"logprob\":-0.26660156,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.000010609627,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5783,\"text\":\" England\",\"logprob\":-0.46411133,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":349,\"text\":\" is\",\"logprob\":-0.1472168,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-1.1308594,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.054351807,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.0000051259995,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2939,\"text\":\" country\",\"logprob\":-0.0051498413,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2373,\"text\":\" within\",\"logprob\":-0.51416016,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.000021338463,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-0.017425537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000008225441,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.024337769,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.37060547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":272,\"text\":\" the\",\"logprob\":-0.40722656,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2969,\"text\":\" United\",\"logprob\":-1.0527344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11508,\"text\":\" Kingdom\",\"logprob\":-0.000028848648,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.0693359,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.00016498566,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2894,\"text\":\" whole\",\"logprob\":-0.000089645386,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1235,\"text\":\" does\",\"logprob\":-0.38671875,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":459,\"text\":\" not\",\"logprob\":-0.00006234646,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":506,\"text\":\" have\",\"logprob\":-0.0008454323,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":264,\"text\":\" a\",\"logprob\":-0.0059127808,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.03540039,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.23498535,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":390,\"text\":\" as\",\"logprob\":-1.1435547,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1259,\"text\":\" such\",\"logprob\":-0.14331055,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.3293457,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":415,\"text\":\" The\",\"logprob\":-0.6118164,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":6194,\"text\":\" UK\",\"logprob\":-0.96777344,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28742,\"text\":\"'\",\"logprob\":-0.39111328,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28713,\"text\":\"s\",\"logprob\":-0.0000022649765,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":799,\"text\":\" other\",\"logprob\":-0.39526367,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14769,\"text\":\" constitu\",\"logprob\":-0.02331543,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":308,\"text\":\"ent\",\"logprob\":-0.000024080276,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5780,\"text\":\" countries\",\"logprob\":-0.00116539,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":460,\"text\":\" are\",\"logprob\":-0.3400879,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14322,\"text\":\" Scotland\",\"logprob\":-0.033111572,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0056495667,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":14831,\"text\":\" Wales\",\"logprob\":-0.0023822784,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.0059928894,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.00001168251,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":12781,\"text\":\" Northern\",\"logprob\":-0.00004386902,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":11170,\"text\":\" Ireland\",\"logprob\":-0.0000026226044,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.14685059,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1430,\"text\":\" each\",\"logprob\":-0.26757812,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":395,\"text\":\" with\",\"logprob\":-0.05065918,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":871,\"text\":\" its\",\"logprob\":-0.4020996,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":1216,\"text\":\" own\",\"logprob\":-0.007835388,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":5565,\"text\":\" capital\",\"logprob\":-0.027450562,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2990,\"text\":\" city\",\"logprob\":-0.07928467,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28747,\"text\":\":\",\"logprob\":-0.6953125,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":25970,\"text\":\" Edinburgh\",\"logprob\":-0.00020945072,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.011047363,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8564,\"text\":\" Card\",\"logprob\":-0.00029182434,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2728,\"text\":\"iff\",\"logprob\":-0.00000333786,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.00059080124,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":304,\"text\":\" and\",\"logprob\":-0.0000019073486,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":365,\"text\":\" B\",\"logprob\":-0.000027298927,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":599,\"text\":\"elf\",\"logprob\":-2.3841858e-7,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":529,\"text\":\"ast\",\"logprob\":-0.000002026558,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28725,\"text\":\",\",\"logprob\":-0.04925537,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":8628,\"text\":\" respectively\",\"logprob\":-0.000021219254,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":28723,\"text\":\".\",\"logprob\":-0.0000834465,\"special\":false},\"generated_text\":null,\"details\":null}\n\ndata:{\"token\":{\"id\":2,\"text\":\"</s>\",\"logprob\":-0.18835449,\"special\":true},\"generated_text\":\"<|assistant|>\\nThe capital of England is London. However, England is a constituent country within the United Kingdom, and the United Kingdom as a whole does not have a capital city as such. The UK's other constituent countries are Scotland, Wales, and Northern Ireland, each with its own capital city: Edinburgh, Cardiff, and Belfast, respectively.\",\"details\":{\"finish_reason\":\"eos_token\",\"generated_tokens\":80,\"seed\":null}}\n\n\n"}

I have tried with different models and still get the same.
I appreciate this is the latest docker image but this has been working for us hence why we kept it stable.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.