Run local LLMs on iGPU, APU and CPU (AMD , Intel, and Qualcomm (Coming Soon)). Easiest way to launch OpenAI API Compatible Server on Windows, Linux and MacOS

* The Llama model architecture supports similar model families such as CodeLlama, Vicuna, Yi, and more.

+ The Mistral model architecture supports similar model families such as Zephyr.

• [2024/06] Support Phi-3 (mini, small, medium), Phi-3-Vision-Mini, Llama-2, Llama-3, Gemma (v1), Mistral v0.3, Starling-LM, Yi-1.5.
• [2024/06] Support vision/chat inference on iGPU, APU, CPU and CUDA.

• Supported Models
  • Onnxruntime Models
  • Ipex-LLM Models
• Getting Started
  • Installation From Source
  • Launch OpenAI API Compatible Server
  • Launch Chatbot Web UI
  • Launch Model Management UI
• Compile OpenAI-API Compatible Server into Windows Executable
• Acknowledgements

• Windows

  1. Custom Setup:
  • XPU: Requires anaconda environment. conda create -n ellm python=3.10 libuv; conda activate llm.
  • DirectML: If you are using Conda Environment. Install additional dependencies: conda install conda-forge::vs2015_runtime.

  2. Install embeddedllm package. $env:ELLM_TARGET_DEVICE='directml'; pip install -e .. Note: currently support cpu, directml and cuda.

     • DirectML: $env:ELLM_TARGET_DEVICE='directml'; pip install -e .[directml]
     • CPU: $env:ELLM_TARGET_DEVICE='cpu'; pip install -e .[cpu]
     • CUDA: $env:ELLM_TARGET_DEVICE='cuda'; pip install -e .[cuda]
     • XPU: $env:ELLM_TARGET_DEVICE='xpu'; pip install -e .[xpu]
     • With Web UI:
       • DirectML: $env:ELLM_TARGET_DEVICE='directml'; pip install -e .[directml,webui]
       • CPU: $env:ELLM_TARGET_DEVICE='cpu'; pip install -e .[cpu,webui]
       • CUDA: $env:ELLM_TARGET_DEVICE='cuda'; pip install -e .[cuda,webui]
       • XPU: $env:ELLM_TARGET_DEVICE='xpu'; pip install -e .[xpu,webui]

• Linux

  1. Custom Setup:
  • XPU: Requires anaconda environment. conda create -n ellm python=3.10 libuv; conda activate llm.
  • DirectML: If you are using Conda Environment. Install additional dependencies: conda install conda-forge::vs2015_runtime.

  2. Install embeddedllm package. ELLM_TARGET_DEVICE='directml' pip install -e .. Note: currently support cpu, directml and cuda.

     • DirectML: ELLM_TARGET_DEVICE='directml' pip install -e .[directml]
     • CPU: ELLM_TARGET_DEVICE='cpu' pip install -e .[cpu]
     • CUDA: ELLM_TARGET_DEVICE='cuda' pip install -e .[cuda]
     • XPU: ELLM_TARGET_DEVICE='xpu' pip install -e .[xpu]
     • With Web UI:
       • DirectML: ELLM_TARGET_DEVICE='directml' pip install -e .[directml,webui]
       • CPU: ELLM_TARGET_DEVICE='cpu' pip install -e .[cpu,webui]
       • CUDA: ELLM_TARGET_DEVICE='cuda' pip install -e .[cuda,webui]
       • XPU: ELLM_TARGET_DEVICE='xpu' pip install -e .[xpu,webui]

1. Custom Setup:

   • Ipex

     • For Intel iGPU:

       set SYCL_CACHE_PERSISTENT=1
       set BIGDL_LLM_XMX_DISABLED=1

     • For Intel Arc™ A-Series Graphics:

       set SYCL_CACHE_PERSISTENT=1

2. ellm_server --model_path <path/to/model/weight>.

3. Example code to connect to the api server can be found in scripts/python. Note: To find out more of the supported arguments. ellm_server --help.

1. ellm_chatbot --port 7788 --host localhost --server_port <ellm_server_port> --server_host localhost. Note: To find out more of the supported arguments. ellm_chatbot --help.

   ![Chatbot Web UI](asset/ellm_chatbot_vid.webp)
   

It is an interface that allows you to download and deploy OpenAI API compatible server. You can find out the disk space required to download the model in the UI.

1. ellm_modelui --port 6678. Note: To find out more of the supported arguments. ellm_modelui --help.

   ![Model Management UI](asset/ellm_modelui.png)
   

1. Install embeddedllm.
2. Install PyInstaller: pip install pyinstaller.
3. Compile Windows Executable: pyinstaller .\ellm_api_server.spec.
4. You can find the executable in the dist\ellm_api_server.

• Excellent open-source projects: vLLM, onnxruntime-genai, Ipex-LLM and many others.