Comments (1)
Here are the ways the 3 main model providers support alternate endpoints.
OpenAI
For OpenAI, the AzureOpenAI
offers a parameter to select the endpoint.
import os
from openai import AzureOpenAI
client = AzureOpenAI(
api_key=os.getenv("AZURE_OPENAI_API_KEY"),
api_version="2023-12-01-preview",
azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT")
)
Anthropic
For Anthropic, you can use the base_url
parameter.
from anthropic import Anthropic, DefaultHttpxClient
client = Anthropic(
# Or use the `ANTHROPIC_BASE_URL` env var
base_url="http://my.test.server.example.com:8083",
http_client=DefaultHttpxClient(
proxies="http://my.test.proxy.example.com",
transport=httpx.HTTPTransport(local_address="0.0.0.0"),
),
)
source : https://github.com/anthropics/anthropic-sdk-python?tab=readme-ov-file#configuring-the-http-client
Gemini
For Gemini, this is a bit more complicated.
from google.api_core.client_options import ClientOptions
import google.generativeai as genai
import PIL.Image
import os
def get_client_cert():
# code to load client certificate and private key.
return client_cert_bytes, client_private_key_bytes
options = ClientOptions(api_endpoint="foo.googleapis.com",
client_cert_source=get_client_cert)
client = ImageAnnotatorClient(client_options=options)
genai.configure(api_key=os.environ["GOOGLE_API_KEY"], ClientOptions=options)
img = PIL.Image.open('path/to/image.png')
model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(["What is in this photo?", img])
print(response.text)
Source = https://ai.google.dev/
Source = https://googleapis.dev/python/google-api-core/latest/client_options.html
from llm.
Related Issues (20)
- Log token performance stats
- Plugin for HF serverless inference
- Autocomplete online cli HOT 1
- Limit max tokens in response
- Parsing JSON in reponses
- `llm logs` cannot combine `-q` and `-m` options HOT 1
- backslash cannot be used in `llm chat`
- Best practice for remote APIs that require a proxy HOT 1
- Pydantic failures HOT 1
- keys set command should provide better feedback
- File upload support?
- [FR] Using Whisper.cpp, add a voice chat mode
- Don't require quoting the prompt HOT 1
- Discussion: Should `llm` keep `pip` as a dependency?
- Change default CLI command from `prompt` to `--help` HOT 1
- "AttributeError: module 'numpy' has no attribute 'int8'" on fresh brew install HOT 5
- Support for new gpt4o-mini model HOT 1
- Remove obsolete DEFAULT_EMBEDDING_MODEL code
- Remove warning about PyTorch HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llm.