chicagohai / hypothesis_generation Goto Github PK

Currently when adapting a new dataset with hypogenic, one needs to implement

1. New data processor class
2. New Prompt class
3. New Task class
   Which is not easy to use. Ideally, we should make this process automated and new users can add their dataset just by preprocessing their data to a specific format, write their prompts, and run a script.

With the current version of our algorithm, we only do single-hypothesis inference to update their rewards.

Essentially, we could extend the algorithm such that for each training example, we can evaluate a group of hypotheses with some multiple hypotheses inference method, and update the hypotheses accordingly.

This could be beneficial because at downstream inference, we will likely use multiple hypotheses inference instead of using a single one for every test example.

Our hypothesis generation shows promising results, but the hypothesis-based inference still faces several bottlenecks.

One possible improvement we can do to get better hypothesis selection.

In particular, if the model is able to pick the most relevant hypothesis from the top 5 generated hypotheses for each test example, we observe ~20% improvement in accuracies for the real datasets. This shows there are lots of room for improvement in the hypothesis selection part of hypothesis-baed inference.

We currently have one synthetic dataset and three real datasets in social science. It would be great to have some new datasets, potentially in different regimes.

One common issue for LLMs is their instability with different prompts. It would be great to have optimized prompt for Hypogenic.

Possible direction: adapt DSP (https://github.com/stanfordnlp/dspy) for Hypogenic.