txtai builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems.

NeuML uses txtai and/or the concepts behind it to power all of our Natural Language Processing (NLP) applications. Example applications:

• cord19q - COVID-19 literature analysis
• paperai - AI-powered literature discovery and review engine for medical/scientific papers
• neuspo - a fact-driven, real-time sports event and news site
• codequestion - Ask coding questions directly from the terminal

txtai is built on the following stack:

• sentence-transformers
• transformers
• faiss
• Python 3.6+

The easiest way to install is via pip and PyPI

pip install txtai

You can also install txtai directly from GitHub. Using a Python Virtual Environment is recommended.

pip install git+https://github.com/neuml/txtai

Python 3.6+ is supported

This project has dependencies that require compiling native code. Windows and macOS systems require the following additional steps. Most Linux environments will install without any additional steps.

• Install C++ Build Tools - https://visualstudio.microsoft.com/visual-cpp-build-tools/

• PyTorch Windows binaries are not on PyPI, the following url link must be added when installing

  pip install txtai -f https://download.pytorch.org/whl/torch_stable.html
  

  See pytorch.org for more information.

• Run the following before installing

  brew install libomp
  

  See this link for more information.

The examples directory has a series of examples and notebooks giving an overview of txtai. See the list of notebooks below.

Notebook	Description
Introducing txtai	Overview of the functionality provided by txtai
Extractive QA with txtai	Extractive question-answering with txtai
Build an Embeddings index from a data source	Embeddings index from a data source backed by word embeddings
Extractive QA with Elasticsearch	Extractive question-answering with Elasticsearch