Evaluation for long_context_tasks failed with a KeyError: 'continuation_indices' about llm-foundry HOT 3 CLOSED

We used this branch. hf_eval.yaml, which I've included below, is an example of how we performed long_context evals. Please note this branch is under active development and might change without warning.

seed: 1
max_seq_len: 16000
device_eval_batch_size: 1
models:
  -
    model_name: openai/gpt-3.5-turbo
    model:
      name: openai_chat
      version: gpt-3.5-turbo
    tokenizer:
      name: tiktoken
      kwargs:
        model_name: gpt-3.5-turbo

icl_tasks: "eval/yamls/long_context_tasks.yaml"
eval_gauntlet: "eval/yamls/eval_gauntlet_long_context.yaml"

from llm-foundry.

Howdy! Right now, the openai client is only compatible with language modeling tasks, and the tasks in long_context_tasks.yaml are all question_answering tasks. That being said, we're currently working on supporting the entire gauntlet for the openai client.

from llm-foundry.

@maxisawesome Thanks. I'm wondering how I can reproduce the results reported in the blog. Could you please give some kind advice?
From: https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

from llm-foundry.