Make sure if there are to many vectors process can be restarted

Include progress monitoring

Allows users to download and to use pre generated vector stores

I followed the instructions to train my own documents, but the generated index file was named "index.faiss" and "index.pkl" instead of "docs.index" and "faiss_store.pkl". After copying the generated files to the application directory and running app.py, I encountered the following error message when asking questions:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2548, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/Users/liam.mazy/Developer/DocsGPT/application/app.py", line 129, in api_answer
result = chain({"query": question})
File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 142, in call
raise e
File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 139, in call
outputs = self._call(inputs)
File "/usr/local/lib/python3.10/site-packages/langchain/chains/vector_db_qa/base.py", line 153, in _call
answer, _ = self.combine_documents_chain.combine_docs(docs, question=question)
File "/usr/local/lib/python3.10/site-packages/langchain/chains/combine_documents/map_reduce.py", line 139, in combine_docs
results = self.llm_chain.apply(
File "/usr/local/lib/python3.10/site-packages/langchain/chains/llm.py", line 91, in apply
response = self.generate(input_list)
File "/usr/local/lib/python3.10/site-packages/langchain/chains/llm.py", line 59, in generate
response = self.llm.generate(prompts, stop=stop)
File "/usr/local/lib/python3.10/site-packages/langchain/llms/base.py", line 128, in generate
raise e
File "/usr/local/lib/python3.10/site-packages/langchain/llms/base.py", line 125, in generate
output = self._generate(prompts, stop=stop)
File "/usr/local/lib/python3.10/site-packages/langchain/llms/openai.py", line 217, in _generate
response = self.completion_with_retry(prompt=_prompts, **params)
File "/usr/local/lib/python3.10/site-packages/langchain/llms/openai.py", line 178, in completion_with_retry
return _completion_with_retry(**kwargs)
File "/usr/local/lib/python3.10/site-packages/tenacity/init.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/usr/local/lib/python3.10/site-packages/tenacity/init.py", line 379, in call
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.10/site-packages/tenacity/init.py", line 314, in iter
return fut.result()
File "/usr/local/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/local/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/tenacity/init.py", line 382, in call
result = fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/langchain/llms/openai.py", line 176, in _completion_with_retry
return self.client.create(**kwargs)
File "/usr/local/lib/python3.10/site-packages/openai/api_resources/completion.py", line 25, in create
return super().create(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create
response, _, api_key = requestor.request(
File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 227, in request
resp, got_stream = self._interpret_response(result, stream)
File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 620, in _interpret_response
self._interpret_response_line(
File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 680, in _interpret_response_line
raise self.handle_error_response(
openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 8745 tokens (8489 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.

It seems that the application is not recognizing my custom index file. I'm not sure if I missed any steps or if there is a problem with the instructions.

Getting the following error when ingesting my documentation:

openai.error.RateLimitError: Rate limit reached for default-global-with-image-limits in organization org-Wge... on requests per min. Limit: 60.000000 / min. Current: 70.000000 / min. Contact [email protected] if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.

Additional context:

Number of Tokens = 538,902

I can't find any way to specify a backoff. I have a paid account, and apparently the rate limit is raised over time. However, I would like to be able to specify a rate limit.

After adding a file to /scripts/inputs I get the following error when running ingest_rst.py. Any ideas?

Traceback (most recent call last):
  File "/Users/ivan/Documents/dev/DocsGPT/scripts/ingest_rst.py", line 85, in <module>
    get_user_permission()
  File "/Users/ivan/Documents/dev/DocsGPT/scripts/ingest_rst.py", line 38, in get_user_permission
    call_openai_api()
  File "/Users/ivan/Documents/dev/DocsGPT/scripts/ingest_rst.py", line 21, in call_openai_api
    store = FAISS.from_texts(docs, OpenAIEmbeddings(), metadatas=metadatas)
  File "/Users/ivan/.local/lib/python3.8/site-packages/langchain/vectorstores/faiss.py", line 192, in from_texts
    index = faiss.IndexFlatL2(len(embeddings[0]))
IndexError: list index out of range

As a user of DocsGPT, I would like to have the ability to have multiple chats for different source documentation so that the app can remember what was discussed in the chat earlier and provide better subsequent answers. This would improve the user experience by giving the app more context about what the user is talking about, much like how different chats work in ChatGPT.

Proposed Solution
Implement a feature that allows the user to create multiple chats for different source documentation.
Store the history of each chat so that the app can provide better subsequent answers.
Update the UI/UX to make it easy for the user to switch between different chats and source documentation.

Or must all files exist locally ?

In real world of large enterprises, theres a confluence server, a jira server, sharepoint server that typically reside in a data center or as a SaaS cloud, and some on-prem custom html, excel files etc.

Would be great a ChatGPT type of ingest/compute searches across remote systems.. how feasible is this ?
Thanks

Both of the existing modal components have few things in common:
1]They both overlay over existing page. Both have similar CSS design(rounded corners, shadows).
2]Both have some similar functions as well, both will go away when "save" is clicked. As for now one of them have cancellation feature but as we add more modals, this too can be a common functionality.

These things should be moved to a generic component. This modal will accept :
1]save click handler as a prop
2]isCancellable boolean prop to decide whether to enable the cancellable feature or not.
3]The actually view to render as render prop

Make sure we can ingest code well

Javascript
Java
Python

Instead of keep images nested in folders we can keep the images in a parent level "assets" which will be referenced throughout the codebase. This will help us avoid duplication of image files that can increase the build size by a lot.

A .ENV file with your OpenAI key must also be in the scripts directory when you run ingest.py

Can be used with https://github.com/zealdocs/zeal
To feed it docs

Show how to switch in documentation

• Test on different embeddings and different llm (e.g Huggingface embeds and Cohere llm)
• Make sure different indexes can be used
• add all old vectorstores to new format in docshub
• Create documentation to show how to switch between LLM's
• Handle different embeddings in frontend
• Add embeddings path on DocsHUB (make it backwards compatable)

When visiting https://docsgpt.arc53.com/, it will prompt you to add an api key. If you skip that step and send a message, a server error is returned and no visual feedback.

Suggestion

• if api key is required, user shouldn't be able to send a message
• display an error message

Ingestion of a large text file emits a stream of warnings similar to the following:

Created a chunk of size 2799, which is longer than the specified 1500

I'm not sure what this means (what the implications are), and what I should do about this.

I read the comment in the source code that says:

# We do this due to the context limits of the LLMs.

Does this mean that uploading this data set will be a waste of money - that it won't work?
Is there something that I can do to mitigate this?

A more helpful message - maybe even with a link to more info - would be great.

Discussed in #5

Hello!

Would be great to have an option to run ingest script in dry run mode, so it will print how many tokens it will cost, before calling OpenAI API.

Also can you give any idea on how to estimate it based on docs and metadatas fields https://github.com/arc53/docsgpt/blob/main/scripts/ingest_rst.py#L36?

Thanks!

Add sources section to every reply by the AI in the UI
AI already generates sources. you need to only add it to the frontend. Required data is in the fetch request

For end users who are interested in the technology but are not developers and/or do not have experience with Github. The about link currently takes the user directly to the Github repo.

It isn't clear where to place the docs.index and faiss_store.pkl files to have the app recognize them as a new option in the drop-down.

Button to leave positive or negative ratings to answers.

As per the current code we are maintaining the view data as a state in parent component and passing it down to children using props which can get out of hand pretty quickly as soon as we have many component that have seperate views for mobile and desktop.

Instead we could let each component make their own independent decision whether or when to switch to mobile view or desktop view by encapsulating the decision within the component.

Since this is a common logic shared across multiple components, this can be moved into a custom hook which then can be used by multiple components.

Hi! I followed this guide to train the AI with my own documentation, which is a .md file.

I changed the .rst extensions I saw in ingest_rst.py and ingest_rst_sphinx.py to .md and that successfully generated the docs.index and faiss_store.pkl files wich I moved to the root of /application.

However, when using the Default option in the dropdown of the UI, I get a 500 response that says KeyError: 'source'.
Am I missing something?

When I installed the chrome extension, I got this error:

Manifest version 2 is deprecated, and support will be removed in 2023. See https://developer.chrome.com/blog/mv2-transition/ for more details.

I followed the instructions found here:
https://github.com/arc53/docsgpt/wiki#launch-chrome-extension

Getting this error out of the gate when trying to train on a single new .rst file. Any thoughts?

Traceback (most recent call last):
  File "/home/pshriwise/soft/doc-gpt/docsgpt/scripts/ingest_rst.py", line 36, in <module>
    store = FAISS.from_texts(docs, OpenAIEmbeddings(), metadatas=metadatas)
  File "/home/pshriwise/.pyenv/versions/3.9.1/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 192, in from_texts
    index = faiss.IndexFlatL2(len(embeddings[0]))
IndexError: list index out of range

Is there a way to update the index files (.pkl and .faiss) with new documents (they can be put in a separate folder) rather than starting the ingest.py over with all the files?

Hi everyone,

Just landed on this project, and i can already see the huge impact this project could have in software programming!

The idea would be to develop a community website where we could search for stores to download in order to import them directly in docsGPT. This would avoid people paying fees when a documentation for a given version has already been ingested by somebody else.

Another idea would be to have this community list directly integrated within the app. Or docsGPT could be hosted online directly.

Should be done in two parts

1. pass history with each api request
2. Add history section in the template and include it in the prompt template(limit it two two questions and answers)

The modal text should be updated to remove acronyms that non-developers might not understand.

• For users without pro accounts, requests to OpenAI API frequently time out due to calling rate limit.
• We could add a retry policy to increase stability.
  

  DocsGPT/scripts/parser/open_ai_func.py

  Lines 25 to 43 in 5883ce2

  	for i in tqdm(docs, desc="Embedding 🦖", unit="docs", total=len(docs), bar_format='{l_bar}{bar}| Time Left: {remaining}'):
  	try:
  	import time
  	store.add_texts([i.page_content], metadatas=[i.metadata])
  	except Exception as e:
  	print(e)
  	print("Error on ", i)
  	print("Saving progress")
  	print(f"stopped at {c1} out of {len(docs)}")
  	faiss.write_index(store.index, "docs.index")
  	store_index_bak = store.index
  	store.index = None
  	with open("faiss_store.pkl", "wb") as f:
  	pickle.dump(store, f)
  	print("Sleeping for 60 seconds and trying again")
  	time.sleep(60)
  	faiss.write_index(store_index_bak, "docs.index")
  	store.index = store_index_bak
  	store.add_texts([i.page_content], metadatas=[i.metadata])

The Guide for launching the chrome extension is blank.

https://github.com/arc53/docsgpt/wiki

OpenAPI3 (Swagger api) parsing of the json file would be extremely helpful as this is commonly published docs.

Use this to guide you
https://www.figma.com/file/OXLtrl1EAy885to6S69554/DocsGPT?node-id=0%3A1&t=hjWVuxRg9yi5YkJ9-1

maybe you're already going to do this but it'd be a great demo

In prompt https://github.com/arc53/docsgpt/blob/main/application/combine_prompt.txt we can see values like this: SOURCES: 28-pl 30-pl. What does it mean?

Does it related to https://github.com/arc53/docsgpt/blob/main/scripts/ingest_rst.py#L32?
And if so, how do I determine the right number if I want to generate similar prompt example file for my docs?

Thanks!

In a /scripts/parser add HTML parsing, you can use langchains module

getting a HTTP 500 when typing into the chat on https://docsgpt.arc53.com/