The pipe is a multimodal-first tool for feeding real-world information into large language models. It is built on top of dozens of carefully-crafted heuristics to create sensible text and image prompts from files, directories, web pages, papers, github repos, etc.

• Prepare prompts from dozens of complex file types 📄
• Visual document extraction for complex PDFs, markdown, etc 🧠
• Outputs optimized for multimodal LLMs 🖼️ + 💬
• Multi-threaded ⚡️
• Works with missing file extensions, in-memory data streams 💾
• Works with directories, URL, git repos, and more 🌐

To use the pipe with Python, simply append the output to the start of your prompt:

import openai
import thepipe
openai_client = openai.OpenAI()
response = openai_client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages = thepipe.make_prompt_from_source("https://github.com/emcf/thepipe"),
)

The pipe is accessible from the command line or from Python. The input source is either a file path, a URL, or a directory (or zip file) path. The pipe will extract information from the source and process it for downstream use with language models, vision transformers, or vision-language models. The output from the pipe is a sensible text-based (or multimodal) representation of the extracted information, carefully crafted to fit within context windows for any models from gemma-7b to GPT-4. It uses a variety of heuristics for optimal performance with vision-language models, including AI filetype detection with filetype detection, AI PDF extraction, efficient token compression, automatic image encoding, reranking for lost-in-the-middle effects, and more, all pre-built to work out-of-the-box.

To use The Pipe, clone the repository and install the requirements:

git clone https://github.com/emcf/thepipe
pip install -r requirements.txt
npm install
npx playwright install --with-deps

Linux users can install ctags with

sudo apt-get install -y universal-ctags

Windows users must ensure ctags.exe is in their PATH environment variable.

To use The Pipe from the command line, simply run

python thepipe.py path/to/directory

This command will process all supported files within the specified directory, compressing any information over the token limit if necessary, and outputting the resulting prompt and images to a folder.

Arguments are:

• The input source (required): can be a file path, a URL, or a directory path.
• --match (optional): Regex pattern to match files in the directory.
• --ignore (optional): Regex pattern to ignore files in the directory.
• --limit (optional): The token limit for the output prompt, defaults to 100K. Prompts exceeding the limit will be compressed.
• --mathpix (optional): Extract images, tables, and math from PDFs using Mathpix.
• --text_only (optional): Do not extract images from documents or websites. Additionally, image files will be represented with OCR instead of as images.

You can use the pipe's output with other LLM providers via LiteLLM.