The Ranking Evaluator offers an executor that can evaluate the performance of a search system by comparing the matches obtained by a query with the matches expected inside a groundtruth.

• metric (str, default precision): The evaluation metric to compute (Possible values are: precision, recall, ndcg, reciprocal_rank, average_precision, fscore)
• eval_at (Optional[int], default None): the point at which evaluation is computed, if None give, will consider all the input to evaluate. Not relevant for reciprocal_rank metric
• beta (int or float, default 1): Parameter to weight differently precision and recall. Only used when metric is reciprocal_rank
• power_relevance (bool, default True): The power relevance places stronger emphasis on retrieving relevant documents. Only used when metric is ndcg
• is_relevance_score (bool, default True): Boolean indicating if the actual scores are to be considered relevance. Highest value is better. If True, the information coming from the actual system results willbe sorted in descending order, otherwise in ascending order. Only used when metric is ndcg
• attribute_fields (Union[str, Tuple[str]], default ('tags__id', )): The fields names to be extracted from the matches. This relies in DocumentArray get_attributes API to extract these fields. For every metric this field should be the field where the Document identifier can be found. For ndcg the relevance field is also needed.
• evaluation_name (str): The name of the evaluation to set for the documents evaluated
• default_traversal_paths (str): Fallback traversal path in case there is not traversal path sent in the request.

None

Use the prebuilt images from JinaHub in your python codes,

from jina import Flow
f = Flow().add(
        uses='jinahub+docker://RankingEvaluator')

or in the .yml config.

jtype: Flow
pods:
  - name: evaluator
    uses: 'jinahub+docker://RankingEvaluator'

1. Install the jinahub-ranking-evaluator

   pip install git+https://github.com/jina-ai/executors-evaluator-ranking.git

2. Use jinahub-image in your code

   from jinahub.evaluators.rank.ranking_evaluator import RankingEvaluator
   from jina import Flow
   
   f = Flow().add(uses=RankingEvaluator)

1. Clone the repo and build the docker image

   git clone https://github.com/jina-ai/executors-evaluator-ranking.git
   cd executors-evaluator-ranking
   docker build -t jinahub-ranking-evaluator-image .

2. Use jinahub-image in your codes

   from jina import Flow
   
   f = Flow().add(
           uses='docker://jinahub-ranking-evaluator-image:latest')

from jina import Flow, Document, DocumentArray

def check_resp(resp):
    for doc in resp.docs:
        print(f'evaluations: {doc.evaluations}')

def get_doc_groundtruth_pairs():
    matches = DocumentArray([Document(tags={'id': i}) for i in range(5)])
    matches_gt = DocumentArray(
        [Document(tags={'id': 1}), Document(tags={'id': 0}), Document(tags={'id': 20}), Document(tags={'id': 30}),
         Document(tags={'id': 40})])
    query = Document()
    query.matches = matches
    gt = Document()
    gt.matches = matches_gt
    yield query, gt

with Flow().add(uses='jinahub+docker://RankingEvaluator') as f:
    f.post(on='foo',
           inputs=get_doc_groundtruth_pairs(),
           on_done=check_resp)

Documents with sorted matches plus groundtruth with sorted expected matches.

Documents with evaluations field filled with a score value by the name of the computed evaluation metric