Specifying storage_location for local qdrant about langroid HOT 12 CLOSED

I think it's fine to do this in a colab notebook

from langroid.

How about instead -- when storage_path is specified anywhere in the code, we recreate the client (whether chroma or qdrant) ?

from langroid.

Actually this will lead to the "storage path already used" error I think

from langroid.

I think we should have a set_storage_path method on the VectorStore class, and it will do the right checks and prevent re-creation of client if it exists and the storage_path has not changed.

from langroid.

As for preventing setting member variable storage_class on a vecdb object, that is a separate problem. In theory one can try to set any member value of a class, and there are ways to prevent setting members other than those in an allowed list. I don't know if we should do that, as that is an issue for any class.

from langroid.

Yes, this approach set_storage_path will help to avoid the situation above.
But how this will fix the colab situation? this error RuntimeError: Storage folder .qdrant/data/ is already accessed by another instance of Qdrant client. If you require concurrent access, use Qdrant server instead.

from langroid.

In a notebook with multiple examples, each example will try to create a client anew. We should explicitly set a new storage path for each example, that should solve it?

from langroid.

yes, I was thinking about this, let me try it

from langroid.

In a notebook with multiple examples, each example will try to create a client anew. We should explicitly set a new storage path for each example, that should solve it?

Just tried this approach, works

from langroid.

Actually, doesn't solve the problem. If the user wants to run the same example again, the same error will show up, even if I every time create a new QdrantDBConfig (include changing storage_path), because the port is still allocated. We need to free the port every time we run the example.

from langroid.

That is expected behavior, since the notebook retains state across runs, unless we restart the kernel.

The only way to solve these "vecdb client creation" issues is to check if there is a client already, and I don't know if there is a way to do that .

from langroid.

I added this logic before creating any config to solve the problem

folder_path = '.data1/data1/'

if os.path.exists(folder_path) and os.path.isdir(folder_path):
    shutil.rmtree(folder_path)
    print(f"The folder '{folder_path}' and all its contents have been removed.")
else:
    print(f"The folder '{folder_path}' does not exist.")

from langroid.