[BUG] - Pinecone upsert creates multiple identical records about flowise HOT 6 CLOSED

You are right. We still don't yet implement a solution to just upsert the delta changes, everything will be re-upsert again. This is the area we need to improve. We can implement a logic to compare the new embeddings to the cached embeddings, and upsert the new ones. Since we already have embeddings caching:

from flowise.

@HenryHengZJ

Thank you very much for the honest response,

Do you have any estimation for the fix you were proposing?

By the way I read over pinecone's documentation they do support upsert, why not use their API I think they should handle the logic of upset once it exists and not flowise

https://docs.pinecone.io/reference/upsert

Thanks

from flowise.

we are using the upsert API under the hood via langchain abstraction. however the ids are different each time you upsert, that causes duplication.

dont have any estimation, but its one of the higher priority item we are taking a look at

from flowise.

Thanks for your honesty @HenryHengZJ

One last question please

Untill you'll fix this issue, if I'll reinsert the same vectors over and over again, is that something which may effect the performance or the accuracy of my bot?

Thanks

from flowise.

seems like it is a very hard task to achieve, I recommend not using upsert, but rather do delete all documents and re-insert these. @HenryHengZJ

from flowise.

soupsert cannot be used? im also noticing duplicates in my vector store. @HenryHengZJ your solution to use redis did not work

from flowise.