Ideally, we want the model to justify the choices made by the semantic search system. There are several approaches:

1. A RAG QA system with a prompt that queries 'Why are the following {documents} relevant to {query}?'

   • e.g. A query for 'diabetes' returns health related documents; why are these relevant? Can specific columns be named by the LLM?

2. Document Store split into small chunks; returned chunks can be highlighted in the main document as relevant to the query.

   • Unsure whether this negatively impacts retreival performance.
   • Can be combined with RAG.

3. Extractive QA - returns string taken directly from document.

4. Summariser - prompt: 'summarise the returned documents'

There is a list of existing queries that use the CDRC keyword-based search. Evaluation could make use of these.

1. Find most common searches and compare semantic search results to website results.
   • List retrieved datasets
2. Should semantic search encourage longer form questioning? e.g. 'which datasets could help study diabetes?' rather than 'diabetes'.

It is worth noting that the most frequent queries tend to search with a known dataset in mind, e.g. 'imd', 'ahah'. Semantic search would be more focussed on dataset discovery.