The Evaluation-AI repository is dedicated to building a comprehensive collection of program evaluation reports and leveraging state-of-the-art techniques in large language models (LLMs) to create a powerful tool for assessing government programs. Our mission is to develop a model that can effectively understand and analyze policy frameworks using retrieval-augmented generation (RAG) and a hybrid search strategy. We are committed to transparency and open-source fairness measurement to ensure the reliability and equity of our models.

To run a Streamlit app of a local LLM specialized in program evaluation critiques, go to our page to run the app codes: https://github.com/casualcomputer/evaluation-ai-pro. Our backend design is as follows:

Repository of Program Evaluation Reports:

• Curate and maintain a rich collection of program evaluation reports, serving as a valuable resource for researchers and practitioners.
• Advanced LLM Training: Utilize the latest advancements in LLMs to train a model that comprehensively understands policy frameworks and program evaluation contexts.
• Retrieval-Augmented Generation (RAG): Employ a hybrid search strategy combining traditional retrieval methods with generative capabilities to enhance information extraction and comprehension.
• Contextual Application: Develop and tailor the tool for various tasks within the program evaluation context, ensuring its relevance and effectiveness in real-world applications.

Backend pipelines:

1. Scrape and clean reports
2. RAG for document retrival and summarization a. retrival b. summarization c. LLM validation/critique
3. Human-in-the-loop for user feedback
4. Model evaluation and online/batch learning

Frontend:

1. Chat interface

git clone https://github.com/casualcomputer/evaluation-ai.git

you can use -h with the scripts to see the help messages.

cd evaluation-ai/src/data/
python 00_load_raw_data.py
python 01_extract_text.py

• Reports are named as <department acronym>_<id>_<title>.<extension>