Qdrant Scroll Method Timeout about qdrant-client HOT 8 CLOSED

@timvisee Please look at it and give me solution for it.

from qdrant-client.

@pseudo-usama scroll method is not what we call search, it is just iterating over the records in Qdrant with some optional restrictions like payload filters.
In order scroll to be fast, payload fields which are used in payload filters should have payload indices.
Limit does not depend on the size of your dataset, it's about the amount of results you want to have in return.
If you want to scroll over a huge amount of records, consider using pagination, which allows you to fetch the next records on demand instead of fetching all of them in one call.

from qdrant-client.

Hi @Faiee50 ,

scroll method returns a tuple of points and the next offset

you need to make calls to scroll with offset retrieved from the previous call (in the first call offset=None)

once returned offset value equals to None, it means that you've reached the end of the collection

from qdrant-client.

Hey @Faiee50, the solution for you would be to refer to the documentation https://qdrant.tech/documentation/concepts/points/#scroll-points

You are using very high value for the limit parameter, and it is absolutely no surprise that the endpoint gives you a timeout once you ask it to retrieve the whole dataset in one request.

You are welcome to use Pinecone if you find appropriate API there.

from qdrant-client.

I'm also facing the same issue. But I didn't find any solutions. Is there any easier solutions to search based on metadata over large dataset? In that case the limit should be very large. I also use offset but the process was very slow.

from qdrant-client.

Hi @joein ,
Thank you for the clarification regarding the scroll method in Qdrant. I understand that it is used for iterating over records with optional restrictions, such as payload filters, and that for efficient scrolling, the payload fields should have payload indices.

I am interested in learning how to implement pagination using the scroll method in Qdrant. Could you provide an example code snippet demonstrating this?

from qdrant-client.

@joein , thanks for your quick help.This is what i am searching for a long
:)

from qdrant-client.

you are welcome

p.s. it can be found here in the docs

from qdrant-client.