Factory Internship Take-Home

Generate detailed scopes for new tasks created in Linear using DSPy and the Linear API.

For optimization, a dataset of tasks for training and testing generated by OpenAI's GPT-4. Then, the program was compiled using OpenAI's GPT-3.5 and used for inference.

To evaluate the program, scopes were generated using the test set and scored based on the number of pre-defined questions answered correctly. The vanilla GPT-3.5 achieved 90% accuracy, the uncompiled program achieved 92.5% accuracy, and the compiled program achieved 98.75% accuracy.

Create + setup the conda environment:

make env
conda activate factory
make install
make setup

Create a .env file:

echo "OPENAI_API_KEY=<your OpenAI API key>" > .env
echo "LINEAR_API_TOKEN=<your LINEAR API token>" >> .env
echo "LINEAR_WH_TOKEN=<your LINEAR webhook token>" >> .env

First compile the model (optional):

make compile

Then run the API:

make run

Serve over https for the webhook:

make serve

To lint + format the code manually:

make fix

To bump transitive dependencies:

make upgrade