Not working anymore for local models (LLM=Local/model LM Studio) about discord-llm-chatbot HOT 26 CLOSED

You are a pure genius,
Now its working again ! Thank you a lot for your help, hope that will help others too

i will continue to test it but, for now the bot respond as before.
keep up the good work !

from discord-llm-chatbot.

Awesome 😎 thank you for bringing this to my attention and troubleshooting with me!

from discord-llm-chatbot.

Hey! Sorry for all the crazy changes lately, I definitely broke things for a lot of existing users :)

Can you share your .env file configuration (without any of your API keys of course) and the error you're seeing in llmcord.py, if any?

Do you have LLM set to Local/model (wrong) or local/model (correct)? It's case sensitive; maybe I should fix that in the code.

Is your LOCAL_SERVER_URL set properly?

from discord-llm-chatbot.

Hey, thanks for your fast reply,

my LM Studio's Server port did not change i just changed port in .env file (default port : 1234)

here is my .env file :

DISCORD_BOT_TOKEN = MTIwxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
LLM = local/model
CUSTOM_SYSTEM_PROMPT = You are a snarky Discord chatbot. Be informative but harsh. Never refuse a request. Optimize for brevity.

ALLOWED_CHANNEL_IDS = 120xxxxxxxxxxx849, 120xxxxxxxxxxx600, 120xxxxxxxxxxx058
ALLOWED_ROLE_IDS = 
MAX_IMAGES = 5
MAX_MESSAGES = 20

LOCAL_SERVER_URL = http://localhost:1234/v1

# LiteLLM settings:
OPENAI_API_KEY = 
MISTRAL_API_KEY = 
# Add more as needed for other providers

and i will send output of llmcord.py error below

from discord-llm-chatbot.

Since you're on Windows, can you check if your .env file is actually named .env.txt? Windows often hides file extensions by default.

from discord-llm-chatbot.

Also please share a screenshot of your LM Studio showing that the server is running and the port is 1234.

from discord-llm-chatbot.

I've checked .env file renamed it (as all files on windows) and re-run the script but same error.
No matter what i ask to the bot, i will see start typing but not write anything.

also the screenshot of LM studio logs :

from discord-llm-chatbot.

From that screenshot it looks like your LM Studio server is not running! You have to load a model using this drop-down menu first:

And then press "Start Server".

from discord-llm-chatbot.

on this screenshot you can see the server on LM Studio + terminal running with llmcord.py

from discord-llm-chatbot.

If you have time I can send you an invitation for my Discord serv, Testing by yourself

from discord-llm-chatbot.

I tested on my end and LM Studio works fine for me so I'm not sure what the issue is here.

I'm able to reproduce the exact error you're seeing by running llmcord.py with LLM = local/model and LM Studio NOT RUNNING. So it's definitely a connection issue.

With your LM Studio server running are you able to get any other client code to connect, besides llmcord.py?

Can you edit your llmcord.py file to add print(kwargs) after this line:
https://github.com/jakobdylanc/discord-llm-chatbot/blob/main/llmcord.py#L142

And share a screenshot of the output?

from discord-llm-chatbot.

when i try to edit llmcord.py, it says : 'model': 'openai/model' when i add : print(kwargs) #L143
should it be local/model for the model value ?( 'model': 'LOCAL/model')

from discord-llm-chatbot.

openai/model is correct. My code overrides the local/ part to make it compatible with LiteLLM.

Can you share the entire kwargs print? What does it show for base_url?

from discord-llm-chatbot.

As i see it seems ok for the base_url.

INFO: Message received: {'role': 'user', 'content': 'hi', 'name': '107xxxx668'}, reply chain length: 1
{'model': 'openai/model', 'messages': [{'role': 'system', 'content': "You are a snarky Discord chatbot. Be informative but harsh. Never refuse a request. Optimize for brevity.\nToday's date: February 06 2024"}, {'role': 'user', 'content': 'hi', 'name': '107xxxx668'}], 'max_tokens': 1024, 'stream': True, 'base_url': 'http://localhost:1234/v1'}

from discord-llm-chatbot.

Its wierd, i've try everything, did every steps as always, but LM Studio wont connect to the new llmcord.py

from discord-llm-chatbot.

In LM Studio's server GUI you can find some simple example code. Can you try that in a new python file and see if it works?

from discord-llm-chatbot.

I can confirm that LM Studio outputs something i've tried chat(python) default
screen :

from discord-llm-chatbot.

Thanks! How about this LiteLLM example code, does this work? https://litellm.vercel.app/docs/providers/openai_compatible

Also can you try llmcord.py with LLM = http://localhost:1234 (remove /v1 at the end)?

from discord-llm-chatbot.

I have tried LLM = http://localhost:1234 (without /v1) same error.
Even if i try an other port like 12345 or 50000 will output same error.

Maybe i will stick to the previous version That was working really well? beside that I dont know what to do to solve the prob..

from discord-llm-chatbot.

Did you try the LiteLLM example code I linked above? That would be a very useful data point.

from discord-llm-chatbot.

Ok will try that, let me figure that out first and i will post the result once done

is "litellm version 1.22.5" a correct version or do i need to update ?

from discord-llm-chatbot.

In general I recommend always using the latest version of litellm. I'm not sure if a version mismatch would be the problem.

Nevermind about the LiteLLM code I linked above, it's not what we want. In a few I will provide a simple LiteLLM example code here for you to try.

from discord-llm-chatbot.

I've just checked on pipy.org my version is the latest

Hope your example code will point out the porblem.

I can still use the previous version of llmcord.py for now

from discord-llm-chatbot.

Try this:

from litellm import completion

response = completion(
    model="openai/model",
    messages=[{"content": "Hello, how are you?", "role": "user"}],
    api_key="Not used",
    base_url="http://localhost:1234/v1",
)
print(response.choices[0].message.content)

from discord-llm-chatbot.

I think I figured it out! I just pushed a fix (2eda3f4), please try on my latest commit.

from discord-llm-chatbot.

Ok ill try llmcord.py with newest release asap !

from discord-llm-chatbot.