What can be said can be solved.

Get early access to the PTA model‎ ‎ |‎ ‎ Scale on our shoulders

IntentPilot is an orchestration of two tools: AskUI's object detector with OpenAI's GPT-4v to achieve automation. It is designed to automate repetitive tasks, and to assist users in performing complex tasks with ease. This repository is our attempt to understand the GPT-4v's potential in automation and building an end-to-end automation tool.

We are inspired by and improve upon Self-Operating-Computer by a more accurate object detection model and an improved prompting strategy. We also provide a more user-friendly interface, and a more intuitive way to interact with the tool. For example, the notification feature to let users know what is happening and what to do next. Also, our tool works across all keyboard layouts - US, German, etc, which was one of the limitations of similar tools.

• Python 3.11 or later
• OpenAI Key
• AskUI credentials
  • ASKUI_WORKSPACE_ID and ASKUI_TOKEN are needed in .env file to get the product running.
  • You can get your own AskUI credentials by signing up at AskUI
• You can also copy the .env.example file to .env and fill in the required details OR You can enter the credentials in the terminal when you start the app.

⚠️IMPORTANT: If you saved the credentials with the flag -c --config, you MUST delete them with the flag -d --deleteconfig again for the local .env file to be read again.

• In case of linux, you may need to install the following packages:

sudo apt-get install xsel xclip python3-tk python3-dev

• In case of MacOS, you will have to grant permissions to the terminal to access the clipboard. You can do this by going to System Preferences -> Security & Privacy -> Privacy -> Accessibility and then adding the terminal to the list of apps that can control your computer.

We are currently working on the Windows version of the tool. It will be available soon.

Quick Fix: The package also works on Windows but the Windows Defender is deleting the src/intent_pilot/utils/screenshot.py file. You have to restore the file from the quarantine and add it to the exclusion list.

pip install intent-pilot

After installation, run intent in your terminal:

intent

In case, you are unable to run the command, try running the following command:

python -m intent_pilot

We recommend using PDM to build and run Intent Pilot

Run the following command in a terminal to install PDM:

curl -sSL https://pdm-project.org/install-pdm.py | python3 -

Run the following command to install all the dependencies needed for Intent Pilot:

pdm install

Start the Intent Pilot by running this command:

pdm run intent

• --debug: Prints debug output to the console
• --model -m <modelname>: The model to use - llava or default gpt4v
• -c --config: Prompts to save the credentials configuration to ~/.askui/intent-pilot.env
• -d --deleteconfig: Prompts to delete the credentials from ~/.askui/intent-pilot.env

Install Ollama for your system from their Website.

• ollama serve to start the API locally
• ollama run llava to start the model locally
• Start the Intent Pilot with the flag -m llava

For real-time discussions and community support, join our Discord server:

• Join our Discord Server and then navigate to the #intent-pilot channel.

Thank you for your interest in contributing! We welcome involvement from the community.

We are currently in the process of building PTA (Prompt-to-Automation) model, a Multi-Modal Model that can understand and execute commands in natural language, in real-time and faster than any Virtual Personal Assistant (VPA) in the market.