Comments (11)
@drittich see http://redisearch.io/Commands/#ftsearch
FT.SEARCH {index} {query} [NOCONTENT] [VERBATIM] [NOSTOPWORDS] [WITHSCORES] [WITHPAYLOADS]
[FILTER {numeric_field} {min} {max}] ...
[GEOFILTER {geo_field} {lon} {lat} {raius} m|km|mi|ft]
-----> [INKEYS {num} {key} ... ] <-------
[INFIELDS {num {field} ... ]
[SLOP {slop}] [INORDER]
[LANGUAGE {language}]
[EXPANDER {expander}]
[SCORER {scorer}]
[PAYLOAD {payload}]
[SORTBY {field} [ASC|DESC]]
[LIMIT offset num]
from redisearch.
Good idea, thanks!
If you're up for implementing it, I'll be happy to guide you. It really shouldn't be that hard.
from redisearch.
Another way to come at this would be the ability to access a Redis set from within a scoring function extension, so I could do intersection, etc., within RediSearch. It looks like the Redis context is not available in that function. Is that a design decision/requirement, or something that can be changed?
from redisearch.
The problem is that accessing redis data structures in a scoring function is extremely slow (given that it needs to happen for every document, a microsecond is a LOT of time), so I wanted to avoid this.
If this set is not big, you can send it as a query payload, for example.
Can you specify your use case in greater detail? Perhaps there's a more robust solution.
from redisearch.
Use case: 1,000,000 indexed documents, but user is only scoped to see 1,000 of them. (Scoping logic is complicated enough that it is not practical to stored the metadata needed for scoping in the FT index.) I have a Redis Set for each user with all of the document IDs that they have access to. So, the goal is to get the FT result set, intersect it with the Redis set of IDs scoped to that user, and then return a page of that data. We're currently trying a Lua solution for this. For some users though, they may be scoped to be able to see 900,000 of those 1,000,000 documents.
from redisearch.
If it's really in the order of 1000, you can use the INKEYS
option of the search and provide the limited ids in advance. (There is no limit for the number of keys in INKEYS but it's not practical to pass on a million keys, 1k or 10k are perfectly acceptable, and will even improve performance)
from redisearch.
@drittich did this solve your use case? in any way, your idea is nice, I'll keep this open
from redisearch.
I would try to implement this.
Whats the preferred way?
- FT.SEARCHSTORE {index} {set-id} {query}
or as additional Flag to FT.SEARCH: - FT.SEARCH {index} {query} STORE {set-id}
The last time I've written something in cpp is a long time ago, but I like the project and want to refresh my skills.
from redisearch.
Hey @salt4pommes. First of all I'd be glad if you tried, however, I tried thinking about it in greater detail and here are a few things to consider:
-
The structure of a search result doesn't map directly to any redis data structure. for later paging - ideally I would store it in a sorted set, and then we have more than cache - we have a prefetching cache.
-
So ideally I would create a new data type for this, which will allow paging and resorting, and be smart about expiration and not only be time based. This is not trivial even if you know the internals of the code. I'd be more than glad to guide you in doing that, but keep in mind it's going to take a bit of work to get right.
-
Two important things coming up are aggregation pipelines (already working in a different branch) and cursors. Cursors basically mean that we can read the results of a query in an iterative way, which is not that far from a cache. So I need to think if we're not duplicating things here.
-
While the open source version includes just a single shard mode of operation, we have a commercial version that scales to multiple servers, making caching a bit different perhaps.
So basically a bit of design is required here to get this right. The simplest idea would be to store just the top N ids from a search in a sorted set, and then we can read them and load the documents, and use it for paging as well as just plain caching.
I'd love to hear your thoughts on all these :)
from redisearch.
Hey Dvir!
I think the main idea behind this, is to store the results for later use or prevent duplicated search overhead.
Sure the fastest and simplest way is to store the results in a sorted set,
but I like the idea of the new data type and the possibilities it could offer.
As you said it is more a cache and a cache requires more maintenance.
That means, if we update or delete a document which is contained in the cache,
then we should also purge all caches which contains the document id.
So we need to maintain a ttl and a type of garbage collection when a document gets removed or updated.
I do not think that this feature will conflict with the cursors.
We could use the cached/stored document ids to fast fetch document data without a complex search and
we are also able to aggregate or query with new filters using the document ids from the cache.
Just another idea if we want a self maintaining cache:
When the cache gets deprecated by some changes in the index, we could save the query in the new data type to fetch the new results and update the cache automatically.
from redisearch.
Alright, let me think about it a bit and I'll draft a detailed design for the cache and invalidation process. Thanks!
from redisearch.
Related Issues (20)
- [BUG] Wildcard redisearch on TEXT field does not return result HOT 2
- [BUG] Unable to do full-text exact search with a colon in the text HOT 2
- [BUG] simple ft.create/ft.search with <100 bytes of data is leaking 1300 bytes of memory. HOT 3
- [BUG] FT.AGGREGATE performance problem HOT 6
- [BUG] Order of precedence not honored in APPLY functions with exponents HOT 1
- [Feature Request] Add FT.ALIASGET command
- L2 distance computation misunderstanding in documentation HOT 4
- [BUG] I can't run "make build" command successfully HOT 3
- Document Distributed Search (RSCoordinator) build/installation HOT 7
- Boost File Error when building 2.8.13 with Bullseye HOT 1
- [BUG] RediSearch HNSW indexing deadlock? HOT 2
- Facing build issue on PPC64LE architecture. HOT 3
- [BUG] APPLY substr function not using -1 count as documented - [MOD-6959]
- the results obtained after indexing are incomplete HOT 29
- Please Help Fix RSCoordinator So that Redis Search (RediSearch) Can Be Used Across Redis Cluster - module-oss.so initialization failed HOT 17
- Configuration of Custom Tokenizer HOT 1
- [BUG] Redis freezes and stops responding with 100% CPU Utilization while using redissearch with HNSW vector indexes HOT 2
- [Feature Request] int8 or float16 support for vector fields
- make cmd not working on linux system [RediSearch Installation] HOT 8
- [BUG] IDF change to 0 after writing the same text three times HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redisearch.