OpeAIBatcher is a Python wrapper for the OpenAI Batch API, designed to streamline the process of handling large datasets with OpenAI's powerful language models. This utility facilitates tasks such as file uploads, batch creation, status tracking, and result retrieval, making it easier to manage extensive API requests efficiently.

• File Upload: Easily upload files to OpenAI for batch processing.
• Batch Creation: Create batches from uploaded files for processing with OpenAI's API.
• Status Tracking: Track the status of your batches in real-time.
• Result Retrieval: Retrieve and save batch results once processing is complete.
• Content Extraction: Extract relevant content from batch responses and save it in a structured format.(Just json surppoted for now)

To use OpeAIBatcher, ensure you have the necessary dependencies installed. You can install them using pip:

pip install openai tqdm jsonlines

Here's a basic example of how to use OpeAIBatcher:

from opeai_batcher import OpeAIBatcher

# Initialize the batcher with your OpenAI API key
batcher = OpeAIBatcher(api_key="your-api-key")

# Upload a file for batch processing
input_file = batcher.file_upload("example.jsonl")

# Create a batch from the uploaded file
batch = batcher.create_batch(input_file)

# Track the status of your batch
batcher.retrieve_batch_status(batch)

# Retrieve and save the batch results
batcher.retrieve_batch_results(batch, retrive_path="output")

# Extract content from the batch responses
batcher.extract_content_from_response("output/" + batch.output_file_id + ".jsonl", "output")

Initializes the OpeAIBatcher with the given parameters.

• base_url: The base URL for the OpenAI API.
• api_key: Your OpenAI API key.
• headers: Optional headers for the API requests.
• log_level: Logging level (default: "INFO").

Uploads a file to OpenAI for batch processing.

• file_path: Path to the file to be uploaded.

Lists all files uploaded for batch processing.

Creates a batch from the uploaded file.

• file: The file object returned from the file_upload method.

Lists all batches created for processing.

Retrieves the status of the specified batch and tracks its progress.

• batch: The batch object or batch ID.

Retrieves the results of the specified batch and saves them to the specified path.

• batch: The batch object or batch ID.
• retrive_path: Path to save the retrieved results.

Extracts content from batch responses and saves it in a structured format.

• source_jsonl_path: Path to the source JSONL file containing the batch responses.
• target_jsonl_path: Path to save the extracted content.

This project is licensed under the MIT License.

Feel free to contribute, open issues, or submit pull requests to improve OpeAIBatcher!