Comments (8)
zhc7
commented on July 19, 2024
Hi, I guess you wish to set up the environment by yourself. Based on the error message you provided, I think you should check "../search_engine/{indexes}" directory. There should be some folders starting with indexes_. Maybe you've missed some steps. If you executed setup.sh, make sure you used -d all. And maybe you should check if there were any error messages omitted when you executed setup.sh.
By the way, we recommend you to use our prebuilt docker images.
from agentbench.
harshraj172
commented on July 19, 2024
Thanks for this, I got the issue. It seems easier to do with the docker container.
Thanks You
from agentbench.
Z-ZHHH
commented on July 19, 2024
Faced the below error when I ran the webshop task. It seems the code is running in the docker, can anyone pls help?
jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException
Here is the full one:
(agentbench) GP-TRT-2:~/AgentBench$ bash .assignments/2023-09-14-10-16-52.sh
Evaluating in docker localhost/task:webshop, Parameters: --task outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev/task.yaml --agent outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev/agent.yaml --output outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev
> [Warning] FastChat agent not available
{'module': 'src.tasks.WebShop', 'parameters': {'end': 280, 'name': 'WebShop-dev', 'num_envs': 3, 'start': 200, 'worker_limit': 3, 'workers': 6}}
{'module': 'src.agents.HTTPAgent', 'parameters': {'body': {'Key1': 'Value1', 'Key2': 'Value2'}, 'headers': {'Content-Type': 'application/json'}, 'name': 'llama2-7b', 'prompter': {'args': {'agent_role': 'assistant'}, 'name': 'role_content_dict'}, 'url': 'http://localhost:8000/v1/chat/completions'}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
[Evaluation] Successfully loaded Task.
Evaluating task 'WebShop-dev' ...
Start Predicting All ...
0%| | 0/80 [00:00<?, ?it/s]> [Warning] FastChat agent not available
> [Warning] OSInteraction task not available
> [Warning] FastChat agent not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] FastChat agent not available
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
Attributes loaded.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 70730.25it/s]
Process SpawnProcess-1:
Traceback (most recent call last):
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/workspace/src/tasks/webshop_docker/__init__.py", line 38, in predict
env = WebAgentTextEnv(observation_mode="text", human_goals=True)
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 61, in __init__
self.server = SimServer(
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 299, in __init__
self.search_engine = init_search_engine(num_products=num_products)
File "/root/workspace/src/tasks/webshop/web_agent_site/engine/engine.py", line 206, in init_search_engine
search_engine = LuceneSearcher(os.path.join(BASE_DIR, f'../search_engine/{indexes}'))
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pyserini/search/lucene/_searcher.py", line 51, in __init__
self.object = JLuceneSearcher(index_dir)
File "jnius/jnius_export_class.pxi", line 270, in jnius.JavaClass.__init__
File "jnius/jnius_export_class.pxi", line 384, in jnius.JavaClass.call_constructor
File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception
jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException
Attributes loaded.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 68716.28it/s]
Process SpawnProcess-2:
Traceback (most recent call last):
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/workspace/src/tasks/webshop_docker/__init__.py", line 38, in predict
env = WebAgentTextEnv(observation_mode="text", human_goals=True)
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 61, in __init__
self.server = SimServer(
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 299, in __init__
self.search_engine = init_search_engine(num_products=num_products)
File "/root/workspace/src/tasks/webshop/web_agent_site/engine/engine.py", line 206, in init_search_engine
search_engine = LuceneSearcher(os.path.join(BASE_DIR, f'../search_engine/{indexes}'))
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pyserini/search/lucene/_searcher.py", line 51, in __init__
self.object = JLuceneSearcher(index_dir)
File "jnius/jnius_export_class.pxi", line 270, in jnius.JavaClass.__init__
File "jnius/jnius_export_class.pxi", line 384, in jnius.JavaClass.call_constructor
File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception
jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
Attributes loaded.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 72446.74it/s]
Process SpawnProcess-3:
Traceback (most recent call last):
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/workspace/src/tasks/webshop_docker/__init__.py", line 38, in predict
env = WebAgentTextEnv(observation_mode="text", human_goals=True)
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 61, in __init__
self.server = SimServer(
File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 299, in __init__
self.search_engine = init_search_engine(num_products=num_products)
File "/root/workspace/src/tasks/webshop/web_agent_site/engine/engine.py", line 206, in init_search_engine
search_engine = LuceneSearcher(os.path.join(BASE_DIR, f'../search_engine/{indexes}'))
File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pyserini/search/lucene/_searcher.py", line 51, in __init__
self.object = JLuceneSearcher(index_dir)
File "jnius/jnius_export_class.pxi", line 270, in jnius.JavaClass.__init__
File "jnius/jnius_export_class.pxi", line 384, in jnius.JavaClass.call_constructor
File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception
jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException
from agentbench.
zhc7
commented on July 19, 2024
Are you sure you've pulled the docker image? Based on the output, it seems that you are not using code in docker. The label of progress bar should be about 1 million instead of 1k.
from agentbench.
Z-ZHHH
commented on July 19, 2024
I have run
bash scripts/build_docker.sh
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e8fbc10eadab localhost/task:webshop "bash -c '\n …" 5 seconds ago Up 5 seconds interesting_dhawan
It seems that I've pulled the docker image. And there are files in the docker image:
LICENSE.md assets data run_envs run_web_agent_text_env.sh tests web_agent_site
README.md baseline_models requirements.txt run_prod.sh search_engine transfer
__pycache__ conftest.py run_dev.sh run_web_agent_site_env.sh setup.sh user_session_logs
from agentbench.
zhc7
commented on July 19, 2024
Based on what you provided in #42
Actually /root/webshop is soft linked to /root/workspace/src/tasks/webshop_docker to avoid overwriting files.
There are two strange things in the output:
- Label of the progress bar should be about a million not a thousand.
- Code executed should be /root/workspace/src/tasks/webshop_docker
The first problem is probably caused by the second. My guess is that you've missed some steps in tutorial.
from agentbench.
Z-ZHHH
commented on July 19, 2024
Thanks for your detailed reply!
I install the environment as follow:
pip install --upgrade pip
pip install -r requirements.txt
bash scripts/build_docker.sh
pip install -r src/tasks/os_interaction/requirements.txt
python src/tasks/os_interaction/images.py build -c configs/tasks/os_interaction/dev.yaml -r .
And I modify the llama2-chat agent file. The OS task could run successfully whereas the webshop meets the problem.
from agentbench.
zhc7
commented on July 19, 2024
The different between them is that OS task does not need to be executed in docker whereas webshop task locates in a docker image. I assume that you are trying to execute the webshop task outside of the docker. Please refer to https://github.com/THUDM/AgentBench/blob/main/docs/tutorial.md#how-to-run-all-tasks-in-agentbench. You may create an assignment with only webshop task.
from agentbench.
Related Issues (20)
- [Bug/Assistance] 测试os-std任务，提示Message: 0 samples remaining. HOT 6
- [Bug/Assistance] 为什么dbbench任务，在mysql数据库中指创建一个unkown数据库名，而且里面只有一张表名称也是unkown，是不是初始化有问题？ HOT 1
- [Bug/Assistance] 测试kg-std任务时，输出文件中全部状态都是task limit reached HOT 1
- [Bug/Assistance] kg-std任务运行的runs.jsonl文件中问题在数据集中找不到 HOT 4
- [Feature] Use for benchmarking agents like AutoGPT? HOT 1
- 我该怎么解决这个问题，跑mind2web，不太清楚该如何操作这个任务，能给出一些具体的指导吗，谢谢 HOT 17
- Card_Game这个任务跑不起来 HOT 4
- Benchmark for mistral models HOT 1
- Connection error HOT 3
- 增加对Cluade3的评测 HOT 2
- [Bug/Assistance] - Reproducing Results on Alfworld (HH) (vs. ReAct paper) HOT 4
- OS std 测试集结果 HOT 1
- [Bug/Assistance] mind2web的unknown是怎么回事？ HOT 1
- Excellent Job! Well, no offense, it seems LLM-Bench rather than AgentBench in essence. HOT 1
- 请问支持使用openai的tool_call接口进行测试吗？ HOT 1
- 请问如何使用本地的llama-2-hf模型进行测试呢，希望得到一些明确的指导！[Bug/Assistance] HOT 1
- [Feature] 请问每个任务的分是怎么计算的呢？比如OS任务中得到的只是一个准确率，但是在论文中Table3每个任务对应的都是分数，这中间的映射过程我在文中并没有找到，可以提示一下吗 HOT 1
- Would llama3 wizardlm2 and other latest models be tested and published in leaderboard? 请求添加llama3 wizardlm等24年4-5月大模型的测试结果 HOT 3
- [Feature] Add a LICENSE to the project HOT 2
- 请问trajectories有公开吗
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from agentbench.