Comments (6)
Hi, @13416157913 有可能是因为指定的输出文件夹里已经包含了全部的测试结果,框架不会重复测试。如果想要重新跑的话,可以尝试换一个输出路径。
from agentbench.
因为指定的输出文件夹里已经包含了
os-std输出文件目录已经删除了,再运行的代码的;
from agentbench.
可以看一下你的配置文件吗?
from agentbench.
可以看一下你的配置文件吗?
assignments:
- agent: pci-chatgpt
task: os-std
concurrency:
agent:
pci-chatgpt: 5
task:
os-std: 5
definition:
agent:
pci-chatgpt:
module: src.client.agents.FastChatAgent
parameters:
body:
max_tokens: 4096
model: pci-chatgpt
repetition_penalty: 1.2
temperature: 0.05
top_k: 30
top_p: 0.8
controller_address: http://xxx.xxx.xxx.xxx:10001
headers:
Content-Type: application/json
max_new_tokens: 4096
model_name: pci-chatgpt
name: FastChat
prompter:
args:
agent_role: assistant
name: role_content_dict
repetition_penalty: 1.2
return_format: '{response[choices][0][message][content]}'
temperature: 0.05
top_k: 30
top_p: 0.8
url: http://xxx.xxx.xxx.xxx:10001/api/gpt/chat/completions
task:
os-std:
module: src.client.TaskClient
parameters:
concurrency: 24
controller_address: http://localhost:5000/api
data_config:
bk: []
files:
- index_prefix: std-001-
problem_file: ../data/os_interaction/data/1/.json
script_dir: ../data/os_interaction/scripts/1/
- index_prefix: std-002-
problem_file: ../data/os_interaction/data/2/.json
script_dir: ../data/os_interaction/scripts/2/
- index_prefix: std-003-
problem_file: ../data/os_interaction/data/3/.json
script_dir: ../data/os_interaction/scripts/3/
- index_prefix: std-004-
problem_file: ../data/os_interaction/data/4/.json
script_dir: ../data/os_interaction/scripts/4/
- index_prefix: std-005-
problem_file: ../data/os_interaction/data/5/.json
script_dir: ../data/os_interaction/scripts/5/
- index_prefix: std-006-
problem_file: ../data/os_interaction/data/6/.json
script_dir: ../data/os_interaction/scripts/6/
- index_prefix: std-007-
problem_file: ../data/os_interaction/data/7/*.json
script_dir: ../data/os_interaction/scripts/7/
ignore: []
docker_config:
directory: ../data/os_interaction/res/dockerfiles
localhost: local-os
name: os-std
round_limit: 8
scripts:
directory: ../data/os_interaction/scripts
output: outputs/2024-02-02-18-27-06
from agentbench.
可以看一下你的配置文件吗?
您好,请问是什么原因,os任务提示Message: 0 samples remaining. db任务能正常运行。
from agentbench.
可以看一下你的配置文件吗?
问题已解决,是配置问题。
from agentbench.
Related Issues (20)
- [Bug/Assistance] OS任务报错AttributeError: 'NpipeSocket' object has no attribute '_sock' HOT 2
- [Bug/Assistance] 为什么dbbench任务,在mysql数据库中指创建一个unkown数据库名,而且里面只有一张表名称也是unkown,是不是初始化有问题? HOT 1
- [Bug/Assistance] 测试kg-std任务时,输出文件中全部状态都是task limit reached HOT 1
- [Bug/Assistance] kg-std任务运行的runs.jsonl文件中问题在数据集中找不到 HOT 4
- [Feature] Use for benchmarking agents like AutoGPT? HOT 1
- 我该怎么解决这个问题,跑mind2web,不太清楚该如何操作这个任务,能给出一些具体的指导吗,谢谢 HOT 17
- Card_Game这个任务跑不起来 HOT 4
- Benchmark for mistral models HOT 1
- Connection error HOT 3
- 增加对Cluade3的评测 HOT 2
- [Bug/Assistance] - Reproducing Results on Alfworld (HH) (vs. ReAct paper) HOT 4
- OS std 测试集结果 HOT 1
- [Bug/Assistance] mind2web的unknown是怎么回事? HOT 1
- Excellent Job! Well, no offense, it seems LLM-Bench rather than AgentBench in essence. HOT 1
- 请问支持使用openai的tool_call接口进行测试吗? HOT 1
- 请问如何使用本地的llama-2-hf模型进行测试呢,希望得到一些明确的指导![Bug/Assistance] HOT 1
- [Feature] 请问每个任务的分是怎么计算的呢?比如OS任务中得到的只是一个准确率,但是在论文中Table3每个任务对应的都是分数,这中间的映射过程我在文中并没有找到,可以提示一下吗 HOT 1
- Would llama3 wizardlm2 and other latest models be tested and published in leaderboard? 请求添加llama3 wizardlm等24年4-5月大模型的测试结果 HOT 3
- [Feature] Add a LICENSE to the project HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from agentbench.