Ask LLM

This is a straightforward, zero-dependency CLI tool to interact with any LLM service.

It is available in several flavors:

Python version. Compatible with CPython or PyPy, v3.10 or higher.
JavaScript version. Compatible with Node.js (>= v18) or Bun (>= v1.0).
Clojure version. Compatible with Babashka (>= 1.3).
Go version. Compatible with Go, v1.19 or higher.

Ask LLM is compatible with either a cloud-based (managed) LLM service (e.g. OpenAI GPT model, Grog, OpenRouter, etc) or with a locally hosted LLM server (e.g. llama.cpp, LocalAI, Ollama, etc). Please continue reading for detailed instructions.

Interact with the LLM with:

./ask-llm.py         # for Python user
./ask-llm.js         # for Node.js user
./ask-llm.clj        # for Clojure user
go run ask-llm.go    # for Go user

or pipe the question directly to get an immediate answer:

echo "Why is the sky blue?" | ./ask-llm.py

or request the LLM to perform a certain task:

echo "Translate into German: thank you" | ./ask-llm.py

Using Local LLM Servers

Supported local LLM servers include llama.cpp, Nitro, Ollama, and LocalAI.

To utilize llama.cpp locally with its inference engine, ensure to load a quantized model such as Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5. Adjust the environment variable LLM_API_BASE_URL accordingly:

/path/to/llama.cpp/server -m Phi-3-mini-4k-instruct-q4.gguf
export LLM_API_BASE_URL=http://127.0.0.1:8080/v1

To utilize Nitro locally, refer to its Quickstart guide for loading a model like Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5 and set the environment variable LLM_API_BASE_URL:

export LLM_API_BASE_URL=http://localhost:3928/v1

To use Ollama locally, load a model and configure the environment variable LLM_API_BASE_URL:

ollama pull phi3
export LLM_API_BASE_URL=http://127.0.0.1:11434/v1
export LLM_CHAT_MODEL='phi3'

For LocalAI, initiate its container and adjust the environment variable LLM_API_BASE_URL:

docker run -ti -p 8080:8080 localai/localai tinyllama-chat
export LLM_API_BASE_URL=http://localhost:3928/v1

Using Managed LLM Services

To use OpenAI GPT model, configure the environment variable OPENAI_API_KEY with your API key:

export OPENAI_API_KEY="sk-yourownapikey"

To utilize other LLM services, populate the relevant environment variables as demonstrated in the following examples:

Anyscale

export LLM_API_BASE_URL=https://api.endpoints.anyscale.com/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"

Deep Infra

export LLM_API_BASE_URL=https://api.deepinfra.com/v1/openai
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/Mistral-7B-Instruct-v0.1"

Fireworks

export LLM_API_BASE_URL=https://api.fireworks.ai/inference/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="accounts/fireworks/models/llama-v3-8b-instruct"

Grog

export LLM_API_BASE_URL=https://api.groq.com/openai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="gemma-7b-it"

Lepton

export LLM_API_BASE_URL=https://mixtral-8x7b.lepton.run/api/v1/
export LLM_API_KEY="yourownapikey"

OpenRouter

export LLM_API_BASE_URL=https://openrouter.ai/api/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/mistral-7b-instruct:free"

Together

export LLM_API_BASE_URL=https://api.together.xyz/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"

ariya / ask-llm Goto Github PK

ask-llm's Introduction

Ask LLM

Using Local LLM Servers

Using Managed LLM Services

ask-llm's People

Contributors

Stargazers

Watchers

Forkers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent