Git Product home page Git Product logo

ashleykleynhans / text-generation-docker Goto Github PK

View Code? Open in Web Editor NEW
20.0 3.0 9.0 223 KB

Docker image for the Text Generation Web UI: A Gradio web UI for Large Language Models. Supports Transformers, AWQ, GPTQ, llama.cpp (GGUF), Llama models

License: GNU General Public License v3.0

Dockerfile 6.71% Shell 25.53% HTML 3.36% Python 58.45% HCL 5.95%
docker docker-image llm runpod text-generation-webui galactica gpt-j llama llama-cpp opt

text-generation-docker's Introduction

Docker image for the Text Generation Web UI: A Gradio web UI for Large Language Models. Supports Transformers, AWQ, GPTQ, llama.cpp (GGUF), Llama models

Docker Pulls Template Version

Note

The legacy APIs no longer work with the latest version of the Text Generation Web UI. They were deprecated in November 2023 and have now been completely removed. If you want to use the LEGACY APIs, please set the image tag to 1.9.5. You will also have to add port 6000 for the legacy REST API and/or port 6005 for the legacy Websockets API.

Installs

Available on RunPod

This image is designed to work on RunPod. You can use my custom RunPod template to launch it on RunPod.

Building the Docker image

Note

You will need to edit the docker-bake.hcl file and update REGISTRY_USER, and RELEASE. You can obviously edit the other values too, but these are the most important ones.

# Clone the repo
git clone https://github.com/ashleykleynhans/text-generation-docker.git

# Log in to Docker Hub
docker login

# Build the image, tag the image, and push the image to Docker Hub
cd text-generation-docker
docker buildx bake -f docker-bake.hcl --push

# Same as above but customize registry/user/release:
REGISTRY=ghcr.io REGISTRY_USER=myuser RELEASE=my-release docker buildx \
    bake -f docker-bake.hcl --push

Running Locally

Install Nvidia CUDA Driver

Start the Docker container

docker run -d \
  --gpus all \
  -v /workspace \
  -p 3000:3001 \
  -p 5000:5001 \
  -p 7777:7777 \
  -p 8888:8888 \
  -p 2999:2999 \
  -e VENV_PATH="/workspace/venvs/text-generation-webui" \
  ashleykza/oobabooga:latest

You can obviously substitute the image name and tag with your own.

Ports

Connect Port Internal Port Description
3000 3001 Text Generation Web UI
5000 5001 Open AI Compatible API
7777 7777 Code Server
8888 8888 Jupyter Lab
2999 2999 RunPod File Uploader

Environment Variables

Variable Description Default
VENV_PATH Set the path for the Python venv for the app /workspace/venvs/text-generation-webui
JUPYTER_LAB_PASSWORD Set a password for Jupyter lab not set - no password
DISABLE_AUTOLAUNCH Disable Web UI from launching automatically (not set)
DISABLE_SYNC Disable syncing if using a RunPod network volume (not set)

Logs

Text Generation Web UI creates a log file, and you can tail the log instead of killing the service to view the logs.

Application Log file
Text Generation Web UI /workspace/logs/textgen.log

For example:

tail -f /workspace/logs/textgen.log

Community and Contributing

Pull requests and issues on GitHub are welcome. Bug fixes and new features are encouraged.

Appreciate my work?

Buy Me A Coffee

text-generation-docker's People

Contributors

ashleykleynhans avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

tripathiarpan20 travischristy kunibald413 dkvsl smet4ik njbbaer dvejmz alifar

text-generation-docker's Issues

Updating To Latest Version of textgen

How do you update this RunPod template to the latest version of text-generation-webui by @oobabooga?

Running ./update_linux.sh encourages me to install conda by @conda:

/update_linux.sh: line 22: /workspace/text-generation-webui/installer_files/conda/etc/profile.d/conda.sh: No such file or directory
./update_linux.sh: line 23: conda: command not found
Conda is not installed. Exiting...

Kill and restart text generation webui

Hi,

Thanks for providing this wonderful template for runpod. I wonder whether there is a way to kill and restart text generation webui when it hangs without stopping and starting a new instance?

Runpod stuck syncing venv to workspace, please wait...

I may be misunderstanding how to use it, am I supposed to just run text gen from the root or should there be something in the workspace. I'm not really sure what's going on.

This is the runpod container log:

2023-11-12T14:33:58.120484823Z 
2023-11-12T14:33:58.120650403Z ==========
2023-11-12T14:33:58.120705223Z == CUDA ==
2023-11-12T14:33:58.120841251Z ==========
2023-11-12T14:33:58.124930389Z 
2023-11-12T14:33:58.124961969Z CUDA Version 12.1.1
2023-11-12T14:33:58.126921808Z 
2023-11-12T14:33:58.126934818Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2023-11-12T14:33:58.128504057Z 
2023-11-12T14:33:58.128511257Z This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2023-11-12T14:33:58.128515067Z By pulling and using the container, you accept the terms and conditions of this license:
2023-11-12T14:33:58.128518887Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2023-11-12T14:33:58.128522357Z 
2023-11-12T14:33:58.128524967Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2023-11-12T14:33:58.141952208Z 
2023-11-12T14:33:58.144042427Z Starting Nginx service...
2023-11-12T14:33:58.161578175Z  * Starting nginx nginx
2023-11-12T14:33:58.172530437Z    ...done.
2023-11-12T14:33:58.172997667Z Running pre-start script...
2023-11-12T14:33:58.174982735Z Container is running
2023-11-12T14:33:58.174994035Z Syncing venv to workspace, please wait...

I connected to the terminal and ran start.sh and it got stuck again, I pressed cntrl+c and then it seemed to continue, but the ports still are not used.

root@05433a96ca8b:/# ./start.sh
Starting Nginx service...
* Starting nginx nginx
Running pre-start script...
Container is running
Syncing venv to workspace, please wait...
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(713) [sender=3.2
rsync error: received SIGUSR1 (code 19) at main.c(1613) [generator=3.2.7]
Syncing Oobabooga Text Generation Web UI to workspace, please wait...
Fixing venv...
Starting Oobabooga Text Generation Web UI
Oobabooga Text Generation Web UI started
Log file: /workspace/logs/textgen.log
All services have been started
Pod Started
Setting up SSH...
* Starting OpenBSD Secure Shell server sshd
Starting Jupyter Lab...
Jupyter Lab started
Exporting environment variables...
Container is READY!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.