maheshsql / genai-batch-ingester Goto Github PK

The GenAI-Batch-Ingester pattern is an innovative solution designed to address the complexities and challenges associated with the bulk processing and ingestion of documents from historic and ongoing perspective, ranging from a few hundred to tens of millions.

This pattern is built upon the robust foundation provided by Azure Cloud services, harnessing the power of Azure Functions, Storage Queues, Azure Document Intelligence, Azure OpenAI, CosmosDB, and Azure Storage. It is designed to streamline the process of data ingestion, offering both scalability and reliability, is well-suited for operations that require the generative AI application of language model prompts and the subsequent storage for analysis or processing.

This pattern provides a versatile framework that can be adapted to various scenarios, including the extension into vectorization and ingestion into vector stores, such as the Azure AI Search service.

Current version supports PDF file ingestion.

Caution

• Do not run directly on 1000s of documents.
• Get this solution working locally through Azure Function Core tools as the first step.
• Run on a very small set (1-2) of smaller documents (< 10 pages) first.
• This pattern assumes incremental addition of new document batch and track progress one batch at a time.
• Incorrect configuration, updates to the code may lead to unexpected Azure bill.

• Clone this repository and open in VS code container
• Provision Azure services
  • Azure Storage Account
    • 2 containers and 4 queues (Please refer storage setup shown below)
  • Azure CosmosDB
  • Azure OpenAI Service
  • Azure Function App [Python 3.10] (not required unless deploying to Azure)
  • You will have to create Python virtual environment and install packages given in requirements.txt file before running func start command. Scripts to install miniconda are available in ./azure_functions/scripts directory. Run these scripts / code in bash.
• Renamed to template_local.settings.json file to local.settings.json
• Update values / keys in local.settings.json

Template file is included which can be renamed to local.settings.json and configuration values need to be updated in this file.

Please note that the solution supports single or multiple Azure OpenAI and Document Intelligence endpoints to spread calls across them. For using multiple endpoints, please separate the values using pipe '|' delimiter. This includes keys and deployment names.

Example below.

"AZURE_OPENAI_DEPLOYMENT_ID": "gpt-35-turbo-1106|gpt-35-turbo-0613|gpt-35-turbo-0125"

Open terminal in VSCode where you have opened this project as a dev container

az login --tenant <TENANT_ID>

az account set --subscription <SUBSCRIPTION_ID>

cd azure_functions

func azure functionapp publish <NAME_OF_YOUR_FUNCTION_APP> --publish-local-settings

Each chunk's processing is logged into CosmosDB and handy to review process completion status / troubleshoot / benchmark processing performance. Below CosmosDB NOSQL queries can help with this.

--merged_chunks processed so far

SELECT VALUE COUNT(1) FROM c WHERE c.doc_type = 'chunk_log' and c.chunk_state = 'Complete'

--merged_chunks with errors

SELECT * FROM c WHERE c.doc_type = 'chunk_log' and c.chunk_state = 'Error'

--merged_chunks that were Throttled

SELECT * FROM c WHERE c.doc_type = 'chunk_log' and c.chunk_state = 'Throttled'

--merged_chunks that were Skipped due to any other reasons / unexpected response from LLM API

SELECT * FROM c WHERE c.doc_type = 'chunk_log' and c.chunk_state = 'Skipped'