[Feature] support for OpenAI-like mock servers & OpenAI proxy servers about chat-llamaindex HOT 6 OPEN

Thanks tranhoangnguyen03 for this request, I am just right now trying to figure out how to do this too! :)

Edit:
As a temporary workaround you can edit "chat-llamaindex\node_modules.pnpm\node_modules\openai\index.js" line 58 and change that URL to point "http://localhost:5001/v1" if you are using koboldcpp-rocm or koboldcpp.

It's nothing perfect for sure and the token generation limits are way too high for me but I hope that helps someone.

from chat-llamaindex.

+1

Also looking at how to connect to e.g. Azure OpenAI endpoints. Thinking there needs to be somewhat significant code changes to make it support those endpoints.

If somebody has figured this out, let it be known in this issue!

from chat-llamaindex.

I second this. For most developers in the corporate world Azure is the only compliant way to access OpenAI models. Or using an open source model deployed on their cloud infrastructure. In either case we simply need all OpenAI API connection options to be configurable - that's all. Best just evaluate the same environment variables as the OpenAI Python module is doing.

from chat-llamaindex.

LlamaIndexTS should use Azure if the following environment variables are set:

• AZURE_OPENAI_ENDPOINT
• AZURE_OPENAI_API_INSTANCE_NAME
• OPENAI_API_TYPE (set to azure)

(see: https://github.com/run-llama/LlamaIndexTS/blob/dfd22aac464fed862c39c45c01717a15ced6c3ad/packages/core/src/llm/azure.ts#L90-L96)

Can you set these variables in env.development.local and try again?

from chat-llamaindex.

But I am not able to access the embedding model, couldn't find the respective variable in .env as well

    code: 'OperationNotSupported',
    message: 'The embeddings operation does not work with the specified model, gpt-4-32k. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'

Any idea how can I use this ?

from chat-llamaindex.

对于这种观点我很认同，Azure 是访问 OpenAI 模型的唯一合规方式。或者使用部署在其云基础架构上的开源模型，最好只评估与 OpenAI Python 模块相同的环境变量

from chat-llamaindex.