VR-Therapist-Virtual-Mental-Health-Experience (VR-Therapist) is a cutting-edge application integrating Virtual Reality (VR) technology and Large Language Models (LLMs) to simulate therapeutic interactions for mental health rehabilitation. This project provides a virtual therapist environment where users can experience immersive and interactive therapy sessions.

• Immersive Therapy: Offers a virtual therapist in a realistic VR environment.
• AI-Driven Responses: Utilizes LLMs to generate therapeutic dialogue in real-time.
• Customizable Interaction: Allows for dynamic interaction based on user input.
• Virtual Environments: Provides various virtual settings for different therapeutic needs.

• Figures: Important figures illustrating the VR environment and system architecture.
• PDF: The research paper detailing the project.
• Presentation: Project presentation slides.
• Demo: A video demo.

The project consists of two main components:

• Flask Python Server: Handles the backend processing, including speech-to-text conversion and LLM interaction.
• Unity Project: Manages the VR environment and user interaction.

• Python 3.x
• Flask
• Google Speech-to-Text API
• AWS Polly

1. Clone the repository:

   git clone https://github.com/mahmoud1yaser/VR-Therapist-Virtual-Mental-Health-Experience
   
   cd VR-Therapist-Virtual-Mental-Health-Experience/Server

2. Install the required packages:

   pip install -r req.txt

3. Create a config.json file based on the provided example in Server/config.json.example.

   • You will need to obtain the following keys:
     • POE Session Key (Note: The POE API service is currently under maintenance and may not be functional at this time.)
     • AWS Polly ACCESS-Key and SECRET-ACCESS-Key

4. Run the Flask server:

   python app.py

• Unity 2022.x or later
• VR Headset (Oculus Rift, HTC Vive, etc.)

1. Open Unity Hub and add the project:

   Open Unity Hub > Add Project > Select `VR-Therapist-Virtual-Mental-Health-Experience/VR-Game`

2. Go to Assets/Scenes/Menu and run the scene:

   In Unity Editor > Add Project > Assets > Scenes > Menu > Open and Run

1. Start the Flask Server: Ensure the Flask server is running by following the steps in the Flask Python Server section.
2. Run the Unity Project: Open and run the MainMenuScene in Unity as described.
3. Initiate Therapy Session: Wear the VR headset, and follow the on-screen instructions to start a therapy session. The virtual therapist will interact with you based on your inputs.

• System Diagram:

• Top and Side View of the Environment:

• Isometric View of the Environment:

• Patient Eye Tracking:

• Therapist Lip Sync:

• Research Paper (PDF)
• Project Presentation (PDF)

We have prepared a demo video to showcase the features and functionality of the VR-Therapist-Virtual-Mental-Health-Experience. The video includes a walkthrough of the system, including setting up the environment, starting a therapy session, and interacting with the virtual therapist.

• LLM Fine-Tuning: Enhance LLM performance for more accurate therapeutic responses.
• Integration of Retrieval-Augmented Generation (RAG): Incorporate real-time information retrieval to improve response relevance.
• Expansion of Therapy Modules: Add more psychological treatment methodologies.
• Creation of Diverse Virtual Environments: Design various customizable virtual settings.
• Customization of Virtual Therapist: Allow personalization of the therapist's appearance and interaction style.
• Patient History Management and Analysis: Implement secure data handling for personalized therapy.

• Mahmoud Yaser - [email protected]
• Ibrahim Mohamed - [email protected]
• Ahmed Hassan - [email protected]
• Maha Medhat - [email protected]
• Mohamed Ismail - [email protected]

This project is licensed under the MIT License.