biocompute-objects / bco-rag Goto Github PK

Thinking about creating separate scripts solely for identifying potentially erroneous evals saves and then separately building the evaluations file on "high quality reviews" only (will have to determine that criteria).

handle the logging directly in the objects rather than in the main.py entrypoint.

The standardized queries contain two parts:

• The query part that contains what is in the domain and what the domain represents.
• The schema part contains the output formatting for the return response.

The first part of the query is obviously important for the semantic retrieval process but I have a hunch that including the second part of the query for the semantic retrieval is polluting the retrieval. Going to try splitting the query and re-injecting the data schema part before the data is sent to the llm.

Some domains are dependent on other domains. The big one is the parametric domain has to be generated after the description domain as the step numbers have to match up.

Will have to hold the description domain in memory and pass it in as part of the query for the parametric domain.

Llamaindex's SemanticSplitterNodeParser can sometimes produce chunks that are too large for the embedding model. Unfortunately there is no max length option for the semantic chunking to avoid this issue.

Will have to eventually subclass the SemanticSplitterNodeParser and create a two level safety net that will naively split large chunks into sub-chunks in order to stay under the embedding model input token limits.

Reference:
run-llama/llama_index#12270

Right now the default score check prevents erroneously saving default evaluations. Should re-work the logic to exclude some fields such as:

• Score (being implemented here: #9)
• json format error (which I want to automatically set when falling back to the raw txt file)

Need to update some documentation for the migration and new features that have been added.

Should probably remove the local evaluation data. That was meant for the proof of concept and a true backend should probably be setup, potentially in the form of MongoDB and a simple Flask API. The domain generations (for now) can probably be kept in the repository.

Should store the score along with the score eval. Because the scoring will iterate/change, need to keep track of which score the score eval was submitted against.