Collections that are missing embeddings can get stuck that way until an explicit re-index about gpt4all HOT 3 OPEN

I second this.

The program starts searching the selected collections...

tried this with 4 collections, to spot the fraction-of-a-second long text message "searching in localdocs:..."

...but immediately switches to the /default "generating response..." and "processing..." without parsing the collections which were however mentioned in the beginning but without them being really used (redundant here, but this is the idea :) )

from gpt4all.

I am able to reproduce this issue using a copy of some of 3Simplex's collections. It seems like the embeddings are missing for certain documents, due to the process getting interrupted somehow. These documents would have been re-indexed on every launch in previous versions of GPT4All because their modification timestamp did not match the database. Now they are only re-indexed the first time GPT4All v2.7.4 is started, and if that did not succeed then the collections will be broken until they are once again re-indexed (e.g. by changing the document snippet size) and it completes successfully.

We need to implement a way to know whether embeddings have been generated for a chunk so the program can continue where it left off.

from gpt4all.

I have also done as 3Simplex said, in the sense of changing a folder's contents as a collection, here's what I've done:

• deleted (Cut-and-Pasted it one level upwards) one file from a folder which was already known as a LocalDocs collection
• after deleting the file, the program did not reindex the collection
• after placing (Paste-d) the file back into its folder, the program started reindexing that collection

Done this with 3 distinct files in 3 distinct folders/categories.
The result was the same - those collections were reindexed.

However, the issue is still here, - of reindexing existing collections. I see several collections being indexed again, immediately after program start, which were created even before 2.7.3 (I can't remember, was it 2.6.1 or a 2.7.x) and stayed that way since then...

Edit :) - the explanation of cebtenzzre clarifies as to why this would happen. Indeed, a flag or something would be handy, like Windows which knows that it didn't shut down properly :)

from gpt4all.