Comments (19)
vasinov
commented on June 3, 2024
1
Thanks for reporting this! This is a known issue related to gpt-3.5-turbo behavior in Griptape. #149 should address it once it's complete.
from griptape.
vasinov
commented on June 3, 2024
1
@adam-fallon, no worries at all! Let's keep this one open until the fix is deployed and pushed to PyPI. Targeting late next week right now.
from griptape.
adam-fallon
commented on June 3, 2024
1
God speed and thanks for making this library!
from griptape.
adam-fallon
commented on June 3, 2024
I think the obvious suggestion here is that I need to up max_tokens, but removing the limit and running again stops even earlier.
from griptape.
adam-fallon
commented on June 3, 2024
Oh sorry - I clearly didn't look hard enough at open issues, apologies. Sorry for making more work for you. Want me to close this so you can focus conversation in the other issue?
from griptape.
pb376
commented on June 3, 2024
Is this behavior only limited to gpt-3.5?
I've been struggling to get some test agents working reliably for prompts I've used successfully with other frameworks. The main issues I've been hitting are:
- The agent stops right before the last task, like described in this issue
- The agent performs the first step of the task and then stops, sort of like the inverse of this issue
- The agent prints the full list of actions that they could take to generate a response, but stops after printing them
- The agent gives a single empty response/output, then stops
I've tried setting tool_memory to None (recommended in another issue) and have tested both gpt-3.5 and gpt-4, but the issues persist regardless.
Do these all sound related to the work being done in #149? If not, I can open a separate issue with examples.
from griptape.
pb376
commented on June 3, 2024
I see that #149 was just recently merged. I went ahead and gave it a try with gpt3.5, and am still seeing the issue OP described. The output (with some of the prompt removed) is this:
[08/16/23 15:25:26] INFO Task 002b4b8a948c4f918379068cc6fe3795
Input: Please use your tools to research CVE-2023-0515, including the data provided for it by
NIST and any advisories released by the affected vendor (which may be included as references in
NIST's data for the CVE.) Please provide a JSON-formatted result including the following keys: [...]
[08/16/23 15:25:29] INFO Task 002b4b8a948c4f918379068cc6fe3795
Output: {
"type": "tool",
"name": "WebSearch",
"activity": "search",
"input": {
"values": {
"query": "CVE-2023-0515"
}
}
}
The code is this:
import os
from dotenv import load_dotenv
load_dotenv()
from griptape.structures import Agent
from griptape.tools import WebScraper, WebSearch, ToolOutputProcessor
from griptape.drivers import OpenAiChatPromptDriver
from griptape.memory.tool import TextToolMemory, BlobToolMemory
text_memory = TextToolMemory(allowlist=[])
agent = Agent(
prompt_driver=OpenAiChatPromptDriver(
model="gpt-3.5-turbo-16k",
max_tokens=10000,
),
tools=[
WebScraper(
output_memory={
"get_content": [text_memory],
}
),
WebSearch(
google_api_key=os.environ["GOOGLE_API_KEY"],
google_api_search_id=os.environ["GOOGLE_API_SEARCH_ID"],
),
ToolOutputProcessor(
input_memory=[text_memory],
),
],
)
agent.run([...])I have not bothered re-testing the other issues I brought up in my previous message but I assume they would likely still be present if this one is.
I also double checked I am using the current main branch with the PR merged, i.e. quick check to verify a change from one of the final commits is present:
grep Output /home/[...]/.cache/pypoetry/virtualenvs/[..]-py3.9/lib/python3.9/site-packages/griptape/templates/tasks/toolkit_task/system.j2
Output: your final response after all Thought/Action/Observationfrom griptape.
pb376
commented on June 3, 2024
Another update, I tried out one of the examples from README.md basically verbatim:
import os
from dotenv import load_dotenv
load_dotenv()
from griptape.drivers import OpenAiChatPromptDriver
from griptape.structures import Agent
from griptape.tools import WebScraper
agent = Agent(
prompt_driver=OpenAiChatPromptDriver(
model="gpt-3.5-turbo"
),
tools=[WebScraper()]
)
agent.run(
"based on https://www.griptape.ai/, tell me what Griptape is"
)but hit issue 3 from my first post:
$ poetry run python anothertest.py
[08/16/23 15:56:24] INFO Task 109ba17f7fe64fc9a095a4c372b7a666
Input: based on https://www.griptape.ai/, tell me what Griptape is
[08/16/23 15:56:26] INFO Task 109ba17f7fe64fc9a095a4c372b7a666
Output: To find out what Griptape is, I can use the WebScraper tool to load the content of the website https://www.griptape.ai/. In subsequent runs indicated the model seems to intend to use the actually use the tool, so this seems like it may just be the same issue as 1/OP's and not a separate one:
$ poetry run python anothertest.py
[08/16/23 15:59:39] INFO Task 56547eb0153d473b8ab9301e0611d9fd
Input: based on https://www.griptape.ai/, tell me what Griptape is
[08/16/23 15:59:40] INFO Task 56547eb0153d473b8ab9301e0611d9fd
Output: To find out what Griptape is, I can use the WebScraper tool to load the web page and get its content. Let me do that for you.
from griptape.
pb376
commented on June 3, 2024
Yes, poetry run pip install https://github.com/griptape-ai/griptape/archive/main.zip, and I verified the changes from the PR were present in the local files.
from griptape.
vasinov
commented on June 3, 2024
@pb376, sorry for the earlier question that I ended up deleting—noticed your comment about using the latest version too late.
We are still experimenting with the base system prompt. gpt-3.5 is being very finicky. Is gpt-3.5 a hard requirement for your use case?
from griptape.
vasinov
commented on June 3, 2024
@pb376, added a few more changes in main. Can you try using gpt-3.5 now? I think it's still not consistent enough but is behaving better. gpt-4 and claude2 are both working flawlessly.
from griptape.
pb376
commented on June 3, 2024
Thanks. The example from the README now works as expected. The CVE prompt's performance/results didn't change, but after random tweaking of the prompt template in templates/tasks/toolkit_task/system.j2 it began to actually follow through with the execution of the chosen tools. However, it eventually hits the same roadblock a few steps later, and ends after stating the next action it intends to take:
$ poetry run python example.py
[08/17/23 02:35:56] INFO Task 910cbbb360af4a50806ef5931df9124e
Input: Please use your tools to research CVE-2023-38408 and provide a summary in the format I describe later in this message. [...]
[08/17/23 02:36:03] INFO Subtask a46def5955234d1ebc2873398a72fddb
Thought: To research CVE-2023-38408 and provide a summary, I can use the WebSearch tool to search for information about the CVE. I can also use the WebScraper tool to scrape data from the NIST website and the vendor's website.
Finally, I can use the ToolOutputProcessor tool to summarize the gathered information and format it into a JSON response.
Action:
{
"type": "tool",
"name": "WebSearch",
"activity": "search",
"input": {
"values": {
"query": "CVE-2023-38408"
}
}
}
[08/17/23 02:36:04] INFO Subtask a46def5955234d1ebc2873398a72fddb
Observation: {'url': 'https://access.redhat.com/security/cve/cve-2023-38408', 'title': 'CVE-2023-38408- Red Hat Customer Portal', 'description': 'Jul 18, 2023 ... The PKCS#11 feature in the ssh-agent in OpenSSH has an
insufficiently trustworthy search path, leading to remote code execution if an agent is\xa0...'}
{'url': 'https://nvd.nist.gov/vuln/detail/CVE-2023-38408', 'title': 'CVE-2023-38408 - NVD', 'description': 'Jul 19, 2023 ... CVE-2023-38408 Detail ... This vulnerability has been modified since it was last analyzed by the NVD.
It is awaiting reanalysis which may result\xa0...'}
{'url': 'https://blog.qualys.com/vulnerabilities-threat-research/2023/07/19/cve-2023-38408-remote-code-execution-in-opensshs-forwarded-ssh-agent', 'title': "CVE-2023-38408: Remote Code Execution in OpenSSH's ...",
'description': "Jul 19, 2023 ... The Qualys Threat Research Unit (TRU) has discovered a remote code execution vulnerability in OpenSSH's forwarded ssh-agent."}
{'url': 'https://vulcan.io/blog/how-to-fix-cve-2023-38408-in-openssh/', 'title': 'How to fix CVE-2023-38408 in OpenSSH | Vulcan Cyber', 'description': "Jul 20, 2023 ... CVE-2023-38408 is a vulnerability that enables remote
code execution and resides in the SSH-agent's forwarded feature, particularly in\xa0..."}
{'url': 'https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh-forwarded-ssh-agent.txt', 'title': "CVE-2023-38408: Remote Code Execution in OpenSSH's ...", 'description': "Jul 19, 2023 ... Qualys Security Advisory
CVE-2023-38408: Remote Code Execution in OpenSSH's ... in ssh-agent bears a striking resemblance to CVE-2010-3856,\xa0..."}
[08/17/23 02:36:10] INFO Subtask 8cb56f7a936e4d249fff3a878ed710d2
Thought: Now that I have obtained search results related to CVE-2023-38408, I can use the WebScraper tool to scrape data from the NIST website and the vendor's website to gather more information about the vulnerability. I will
start by scraping data from the NIST website.
Action: {"type": "tool", "name": "WebScraper", "activity": "get_content", "input": {"values": {"url": "https://nvd.nist.gov/vuln/detail/CVE-2023-38408"}}}
[08/17/23 02:36:13] INFO Subtask 8cb56f7a936e4d249fff3a878ed710d2
Observation: Output of "WebScraper.get_content" was stored in memory: {"memory_id":"TextToolMemory", "artifact_namespace": "251fe897aa5f46e8b71258550d99935d"}
[08/17/23 02:36:18] INFO Task 910cbbb360af4a50806ef5931df9124e
Answer: Thought: I have successfully scraped data from the NIST website. Now, I will scrape data from the vendor's website to gather more information about the vulnerability. Action: {"type": "tool", "name": "WebScraper",
"activity": "get_content", "input": {"values": {"url": "https://access.redhat.com/security/cve/cve-2023-38408"}}}I'm not sure why it created a task instead of subtask for the last step, which could be related to the issue here. The Answer: Thought: bit also seems relevant to the issue. Random shots in the dark, could this be somehow related to max token counts (since one of the steps performed, web scraping, would involve outputs with high token counts) or maybe the memory/summarization being performed per step eventually losing enough original context that the agent is unclear how to proceed?
Also, is there a way to increase the verbosity of output or change the log level to something like DEBUG? Seeing the full prompts provided at each step would definitely be helpful for debugging and just generally understanding how things are working.
from griptape.
alexanderwilkie
commented on June 3, 2024
I have also been running into this issue using gpt-3.5-turbo-16k - I also tried installing from main (pip install git+https://github.com/griptape-ai/griptape) to include the latest changes (I think) and still seem to get output that stops before anything useful being output. Trying to run the query from the readme (in Chat(agent).start() mode rather than agent.run()) gives me:
Q: based on https://www.griptape.ai/, tell me what Griptape is
processing...
[08/17/23 13:52:14] INFO Task 9fdf6d5e8ea345498b5dc24990996a5c
Input: based on https://www.griptape.ai/, tell me what Griptape is
[08/17/23 13:52:18] INFO Subtask 3f1fa4568e9849f6bf0be9d11bdd45d7
Thought: To find out what Griptape is, I can use the WebScraper tool to get the content of the website https://www.griptape.ai/.
Action:
{
"type": "tool",
"name": "WebScraper",
"activity": "get_content",
"input": {
"values": {
"url": "https://www.griptape.ai/"
}
}
}
[08/17/23 13:52:19] INFO Subtask 3f1fa4568e9849f6bf0be9d11bdd45d7
Observation: Output of "WebScraper.get_content" was stored in memory: {"memory_id":"TextToolMemory", "artifact_namespace": "9b452d087d014367862cf55c3c9afc00"}
[08/17/23 13:52:21] INFO Task 9fdf6d5e8ea345498b5dc24990996a5c
Output: I have retrieved the content of the website https://www.griptape.ai/. Let me summarize it for you.
A: I have retrieved the content of the website https://www.griptape.ai/. Let me summarize it for you.
Q:
from griptape.
vasinov
commented on June 3, 2024
Can you try running your prompts in v0.15.2? We just shipped some base prompt improvements that should address some of the gpt-3.5 issues.
from griptape.
alexanderwilkie
commented on June 3, 2024
Hmm no improvement for me for the following script:
from griptape.drivers import OpenAiChatPromptDriver
from griptape.structures import Agent
from griptape.tools import WebScraper
from griptape.utils import Chat
agent = Agent(
prompt_driver=OpenAiChatPromptDriver(
model="gpt-3.5-turbo"
),
tools=[WebScraper()]
)
Chat(agent).start()
I get the following
Q: based on https://kesen.realtimerendering.com/sig2023.html, tell me what Universities are listed on that path
processing...
[08/21/23 15:31:53] INFO Task bdc674d767cb494fb33cf379c87a5c76
Input: based on https://kesen.realtimerendering.com/sig2023.html, tell me what Universities are listed on that path
[08/21/23 15:31:57] INFO Task bdc674d767cb494fb33cf379c87a5c76
Answer: Thought: To find the universities listed on the given web page, I can use the WebScraper tool to get the content of the page and then extract the relevant information.
Action: I will use the WebScraper tool to get the content of the web page.
{
"type": "tool",
"name": "WebScraper",
"activity": "get_content",
"input": {
"values": {
"url": "https://kesen.realtimerendering.com/sig2023.html"
}
}
}
A: Thought: To find the universities listed on the given web page, I can use the WebScraper tool to get the content of the page and then extract the relevant information.
Action: I will use the WebScraper tool to get the content of the web page.
{
"type": "tool",
"name": "WebScraper",
"activity": "get_content",
"input": {
"values": {
"url": "https://kesen.realtimerendering.com/sig2023.html"
}
}
}
Q:
from griptape.
pb376
commented on June 3, 2024
Same, things are still not working. I've ended up just using the OpenAI library directly for now.
Is the current belief that this issue is just a prompting problem? That would somewhat surprise me, partly because I've tried more than a few other frameworks and haven't hit issues like this with their built-in prompts, and partly because it seems like prompt-related issues would be relatively straightforward to fix and just a matter of spending some time testing variations.
The reason I was drawn to Griptape was because of the design and abstractions, which felt like an oasis of sanity in the fever dream-like hellscape of langchain-esque frameworks. I am guessing that general-purpose prompts (non-use case specific) will be basically fully commoditized sooner or later, probably sooner, and am also guessing that improvements to prompts/prompting techniques will largely come from researchers and practitioners, who will then use them within their framework of choice, rather than the framework developers themselves... all of which is to say, if prompting is really the issue here, just go look around Github for some prompts that work well and let's use those!
from griptape.
collindutter
commented on June 3, 2024
@alexanderwilkie @pb376 sorry to hear you guys are still facing issues! It looks like prompt changes from a recent commit fixed this issue. @vasinov will likely push soon. We're currently working on developing integration tests to catch issues like this that can occur from minor prompt changes.
Output when running [email protected]:
[08/28/23 14:56:15] INFO Task 2da5a206daba40efa25f0e9bd5cb5cb2
Input: based on https://kesen.realtimerendering.com/sig2023.html, tell me what Universities are listed on that
path
[08/28/23 14:56:18] INFO Task 2da5a206daba40efa25f0e9bd5cb5cb2
Answer: Thought: To find the universities listed on the given webpage, I can use the WebScraper tool to extract
the content of the webpage and then search for any mentions of universities.
Action: I will use the WebScraper tool to get the content of the webpage.
{
"type": "tool",
"name": "WebScraper",
"activity": "get_content",
"input": {
"values": {
"url": "https://kesen.realtimerendering.com/sig2023.html"
}
}
}
Output when running griptape@main:
[08/28/23 14:58:43] INFO Task 114e2278a44e4f2bb0a7ac52da7ee37b
Input: based on https://kesen.realtimerendering.com/sig2023.html, tell me what Universities are listed on that
path
[08/28/23 14:58:44] INFO Subtask ffde66b6e2ef4979b7f342d6846f9682
Action: {"type": "tool", "name": "WebScraper", "activity": "get_content", "input": {"values": {"url":
"https://kesen.realtimerendering.com/sig2023.html"}}}
[08/28/23 14:58:45] INFO Subtask ffde66b6e2ef4979b7f342d6846f9682
Observation: Output of "WebScraper.get_content" was stored in memory with memory_name "TextToolMemory" and
artifact_namespace "1e009d70d2574a9ea2cbdb46aea621cc"
[08/28/23 14:58:47] INFO Subtask 932c6c63556643089318bba5c3b4fe32
Action: {"type": "memory", "name": "TextToolMemory", "activity": "search", "input": {"values": {"memory_name":
"TextToolMemory", "artifact_namespace": "1e009d70d2574a9ea2cbdb46aea621cc", "query": "Universities"}}}
[08/28/23 14:58:52] INFO Subtask 932c6c63556643089318bba5c3b4fe32
Observation: - ShanghaiTech University
- Seoul National University
- Seoul National University Bundang Hospital
- NCsoft
- Tsinghua University
- Kyung Hee University
- Massachusetts Institute of Technology
- UC San Diego
- The University of Hong Kong
- The University of Edinburgh
- National Institute of Informatics
- KTH Royal Institute of Technology
- Bryn Mawr College
- University of Birmingham
- Nanjing University of Science and Technology
- Nankai University
- Adobe Research
- Tel-Aviv University
- Reichman University
- NVIDIA
- Goldsmiths University
- Max Planck Institute for Informatics
- Saarbrucken Research Center for Visual Computing, Interaction and AI
- CTU in Prague
- Fraunhofer IEE
- University of Kassel
- Zhejiang University
- KooLab, Manycore Tech Inc.
- RaysEngine Tech Inc.
- Beijing Institute of Technology
- McGill University
- Princeton University
- EPFL
- University of Michigan
- Sun Yat-sen University
- South China University of Technology
- UCBL
- INSA Lyon
- LS2N
- Centrale Lyon
- ENISE
- UC San Diego
- Delft University of Technology
[08/28/23 14:58:58] INFO Task 114e2278a44e4f2bb0a7ac52da7ee37b
Answer: The universities listed on the path are:
1. ShanghaiTech University
2. Seoul National University
3. Seoul National University Bundang Hospital
4. NCsoft
5. Tsinghua University
6. Kyung Hee University
7. Massachusetts Institute of Technology
8. UC San Diego
9. The University of Hong Kong
10. The University of Edinburgh
11. National Institute of Informatics
12. KTH Royal Institute of Technology
13. Bryn Mawr College
14. University of Birmingham
15. Nanjing University of Science and Technology
16. Nankai University
17. Adobe Research
18. Tel-Aviv University
19. Reichman University
20. NVIDIA
21. Goldsmiths University
22. Max Planck Institute for Informatics
23. Saarbrucken Research Center for Visual Computing, Interaction and AI
24. CTU in Prague
25. Fraunhofer IEE
26. University of Kassel
27. Zhejiang University
28. KooLab, Manycore Tech Inc.
29. RaysEngine Tech Inc.
30. Beijing Institute of Technology
31. McGill University
32. Princeton University
33. EPFL
34. University of Michigan
35. Sun Yat-sen University
36. South China University of Technology
37. UCBL
38. INSA Lyon
39. LS2N
40. Centrale Lyon
41. ENISE
42. UC San Diego
43. Delft University of Technology
from griptape.
shhlife
commented on June 3, 2024
That's looking so much better, thanks Collin! However, it seems like it's still missing a few of the universities mentioned on the page - Simon Fraser University, Cardiff University, and University of Chinese Academy of Sciences to name a few. It would be interesting to figure out why it seems to be missing some of the universities - Vasily mentioned that maybe the csv extraction engine might be a better solution for this type of problem?
from griptape.
collindutter
commented on June 3, 2024
This is the output I get on the latest version of griptape (v0.20.0) with gpt-4.
from griptape.drivers import (
OpenAiChatPromptDriver,
)
from griptape.structures import Agent
from griptape.tools import WebScraper, TaskMemoryClient
agent = Agent(
prompt_driver=OpenAiChatPromptDriver(
model="gpt-4",
),
tools=[WebScraper(), TaskMemoryClient(off_prompt=False)],
)
agent.run(
"Give me a numbered list of the names of all the universities listed here: https://kesen.realtimerendering.com/sig2023.html"
)[11/16/23 09:10:25] INFO ToolkitTask 2e58abb64e5f4533bfeb580021ebbdff
Input: Give me a numbered list of the names of all the universities listed here:
https://kesen.realtimerendering.com/sig2023.html
[11/16/23 09:10:36] INFO Subtask 82a71c9ef9ce459693ea22d06018dc2a
Thought: To get the names of all the universities listed on the webpage, I need to use the
WebScraper action to load the webpage content.
Action:
{
"name": "WebScraper",
"path": "get_content",
"input": {
"values": {
"url": "https://kesen.realtimerendering.com/sig2023.html"
}
}
}
[11/16/23 09:10:43] INFO Subtask 82a71c9ef9ce459693ea22d06018dc2a
Response: Output of "WebScraper.get_content" was stored in memory with memory_name "TaskMemory"
and artifact_namespace "8c415f15ce9543ba8037f54dd089d685"
[11/16/23 09:10:51] INFO Subtask a51719b853cc424793535287a387bb34
Thought: Now that the webpage content is stored in memory, I need to query this memory to extract
the names of all the universities listed on the webpage.
Action: {"name": "TaskMemoryClient", "path": "query", "input": {"values": {"memory_name":
"TaskMemory", "artifact_namespace": "8c415f15ce9543ba8037f54dd089d685", "query": "What are the
names of all the universities listed on the webpage?"}}}
[11/16/23 09:11:40] INFO Subtask a51719b853cc424793535287a387bb34
Response: Nanjing University of Science and Technology, Nankai University, Adobe Research,
University of California, Santa Barbara, Tel-Aviv University, The Hebrew University of Jerusalem,
Reichman University, NVIDIA, Goldsmiths University, Max Planck Institute for Informatics,
Saarbrucken Research Center for Visual Computing, Interaction and AI, University of Pennsylvania,
Google AR/VR, CTU in Prague, Beijing Institute of Technology, McGill University, Princeton
University, Fraunhofer IEE, University of Kassel, Zhejiang University, Zhejiang Lab, RaysEngine
Tech Inc., Cornell University, EPFL, University of Michigan, Sun Yat-sen University, South China
University of Technology, Univ Lyon, UCBL, CNRS, INSA Lyon, LIRIS, UMR5205, Nantes Universite,
Ecole Centrale Nantes, LS2N, UMR 6004, UC San Diego, Delft University of Technology, ShanghaiTech
University, Seoul National University, Seoul National University Bundang Hospital, NCsoft,
Massachusetts Institute of Technology, Tsinghua University, Kyung Hee University, Peking
University, Guangdong Academy of Sciences, The University of Hong Kong, The University of
Edinburgh, Reality Labs Research at Meta, National Institute of Informatics, KTH Royal Institute
of Technology, motorica.ai, Clemson University, Bryn Mawr College, Meta Reality Labs Research,
University of Birmingham, Ecole de technologie superieure, Inria, Univ. Lille, Univ. Rennes,
IRISA, SUTD, Cardiff University, ETH Zurich, Nanyang Technological University, University of
Oxford, The University of Michigan, South China University of Technology, Singapore Management
University, University of Amsterdam, The Chinese University of Hong Kong, Carnegie Mellon
University, Centre Inria d’Universite Cote d’Azur, University of Washington, The University of
Tokyo, Nature Architects, Inc., Miyagi University, Elephantech Inc., Purdue University, KAUST,
Adam Mickiewicz University, University of Lausanne, University of Oxford, The Pennsylvania State
University, Arizona State University, Dartmouth College, The University of Michigan, Suren
Jayasuriya, The Chinese University of Hong Kong, Cornell Tech, Google Research, University of
Chinese Academy of Sciences, Chongqing University of Posts and Telecommunications.
[11/16/23 09:12:33] INFO ToolkitTask 2e58abb64e5f4533bfeb580021ebbdff
Output: Here is a numbered list of the universities listed on the webpage:
1. Nanjing University of Science and Technology
2. Nankai University
3. Adobe Research
4. University of California, Santa Barbara
5. Tel-Aviv University
6. The Hebrew University of Jerusalem
7. Reichman University
8. NVIDIA
9. Goldsmiths University
10. Max Planck Institute for Informatics
11. Saarbrucken Research Center for Visual Computing, Interaction and AI
12. University of Pennsylvania
13. Google AR/VR
14. CTU in Prague
15. Beijing Institute of Technology
16. McGill University
17. Princeton University
18. Fraunhofer IEE
19. University of Kassel
20. Zhejiang University
21. Zhejiang Lab
22. RaysEngine Tech Inc.
23. Cornell University
24. EPFL
25. University of Michigan
26. Sun Yat-sen University
27. South China University of Technology
28. Univ Lyon, UCBL, CNRS, INSA Lyon, LIRIS, UMR5205
29. Nantes Universite, Ecole Centrale Nantes, LS2N, UMR 6004
30. UC San Diego
31. Delft University of Technology
32. ShanghaiTech University
33. Seoul National University
34. Seoul National University Bundang Hospital
35. NCsoft
36. Massachusetts Institute of Technology
37. Tsinghua University
38. Kyung Hee University
39. Peking University
40. Guangdong Academy of Sciences
41. The University of Hong Kong
42. The University of Edinburgh
43. Reality Labs Research at Meta
44. National Institute of Informatics
45. KTH Royal Institute of Technology
46. motorica.ai
47. Clemson University
48. Bryn Mawr College
49. Meta Reality Labs Research
50. University of Birmingham
51. Ecole de technologie superieure
52. Inria, Univ. Lille, Univ. Rennes, IRISA
53. SUTD
54. Cardiff University
55. ETH Zurich
56. Nanyang Technological University
57. University of Oxford
58. The University of Michigan
59. South China University of Technology
60. Singapore Management University
61. University of Amsterdam
62. The Chinese University of Hong Kong
63. Carnegie Mellon University
64. Centre Inria d’Universite Cote d’Azur
65. University of Washington
66. The University of Tokyo
67. Nature Architects, Inc.
68. Miyagi University
69. Elephantech Inc.
70. Purdue University
71. KAUST
72. Adam Mickiewicz University
73. University of Lausanne
74. University of Oxford
75. The Pennsylvania State University
76. Arizona State University
77. Dartmouth College
78. The University of Michigan
79. Suren Jayasuriya
80. The Chinese University of Hong Kong
81. Cornell Tech
82. Google Research
83. University of Chinese Academy of Sciences
84. Chongqing University of Posts and Telecommunications.
When I run it with gpt-3.5-turbo it only picks up 37 of the 84 universities due to a less optional TaskMemoryClient query. It's possible that this could be improved with Rules or a better prompt. Closing this as it seems to be a limitation of gpt-3.5-turbo.
from griptape.
Related Issues (20)
- Add Gemini Tokenizer
- Add Gemini Embedding Driver HOT 1
- Add Gemini Prompt Driver
- Add Google Imagen Image Generation Driver
- Add Gemini Image Query Driver
- Add Google Gemini support
- BedrockLlamaPromptModelDriver does not apply input truncation, resulting in ValidationError HOT 4
- Add Support For Mistral on AWS Bedrock
- BedrockClaudePromptModelDriver results in ValidationError for Claude V3* models due to incorrect invoke struct HOT 1
- JsonExtractionEngine produces malformed JSON
- ignored_exception_types not working as intended
- vLLM endpoint support HOT 4
- CodeExecutionTask doesn't handle inputs correctly when in a Workflow HOT 2
- Calculating Max Output Tokens Is Inconsistent Across Prompt Drivers
- Driver secrets are serializable
- Azure OpenAI api version `2023-12-01-preview` deprecation HOT 2
- Bing Search Wrapper for WebSearch tool HOT 1
- Add model to emitted events.
- Use rich.print in Chat utility HOT 5
- Bump Pinecone from 2 to 3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from griptape.