Comments (11)
Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).
Wow, I'm impressed by Your work. It worked out very good, first test that i run showed chunked out and highlighted content in fraction of seconds. Build index with those changes is only slightly larger.
Works perfect !
Keep on the good work, whoosh is for me the #1 library i discovered.
Regards
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
Well, I had to fix up the implementation of spans, but this should be a start...
This solution involves storing the characters at which each term occurs in each document as part of the index. Then when you go to search for a query, you have to use the low-level Matcher API to get the whoosh.spans.Span
objects representing matches in the document. The spans will give you the start and end character of each occurance in the text, so you can pull out a chunk around each occurance and highlight/display that.
#!python
# Instantiate a new field type that's like TEXT but uses the Characters
# format, which stores the start and end character of each term in the
# index
charfield = fields.FieldType(format=formats.Characters(analyzer),
scorable=True, stored=True)
# Use the custom field specification in the schema
schema = fields.Schema(title=TEXT(stored=True), content=charfield)
# Index the collection with this schema
# ...
# Now, to search for my_query.
# Use the lower-level Matcher API instead of a Results object
# to get the matching spans.
searcher = my_index.searcher()
matcher = my_query.matcher(searcher)
# Loop over the matching documents
while matcher.is_active():
# The document number of the current match
docnum = matcher.id()
# How to get the stored fields for the document number
d = searcher.stored_fields(docnum)
content = d["content"]
# Loop over all the places in the document that matched
# the query
for span in matcher.spans():
print " Match from %d to %d" % (span.startchar, span.endchar)
print " ", content[max(0, span.startchar-40) : span.endchar+40]
# Move to the next matching document
matcher.next()
searcher.close()
Of course this is not an ideal solution. It's low-level, and for a complex query (e.g. OR with lots of words) you'll get spans all over the place. The Spans class has a couple of simple methods to merge spans (Span.to()
and Span.merge()
), but it would be nice to integrate spans with highlighting so the highlighting system could find clusters of spans and use them to pick the best "chunks" to highlight. But hopefully this is enough to help you.
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
OK, I get it now. This is definitely possible, by storing character positions in the index, but it will increase the size of the index and requires using lower-level APIs. I'll work up an example and attach it.
from whoosh.
Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).
My use scenario is a full text search in source codes, and I have to use "heavy" tokenizers, since i would like to search on imports "from whoosh.example import something" which are divided by '.' in python.
Other programming languages are totally different so finding such light tokenizer would be difficult.
The problem in my case is that when searching for a specific pattern i accidentally hit a large file, the highlight freezes the application.
The perfect solution for me would be if whoosh could chunk the content for interesting pieces only, similar to SimpleFragmenter, and then highlighting would be super fast. I don't know if such solution is even possible. I image that many of codes would have log files people like to search in my app and those can be big.
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
Yes, tokenizing a 3.5 MB file like that, including all the codes, is just going to be slow in Python. Changing the regular expression to only parse the words at the start of each line at least reduces the time to 3s. What is it you want to do? Were you just using this file as a test?
from whoosh.
Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).
I don't know much about the internals, but I think it's just to much text to parse, maybe if there where info how to efficient chunk the content just for highlighting it would help allot.
from whoosh.
Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).
Try opening 'cmu_phonetic_dictionary'
i run Your code and got 20s on 3Ghz core2
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
On my first try, I can't reproduce your problem. This is what I tried:
#!python
from whoosh import analysis, highlight
analyzer = analysis.RegexTokenizer(expression=r"\w+") | analysis.LowercaseFilter()
formatter = highlight.HtmlFormatter('span', between='\n<span class="break">...</span>\n')
fragmenter = highlight.SimpleFragmenter(200)
#fragmenter = ContextFragmenter(search_items)
f = open("context.txt", "rb")
text = f.read().decode("utf8")
f.close()
from whoosh.util import now
from cgi import escape
t = now()
text = escape(text)
hl = highlight.highlight(text, ["abandon", "rocket"],
analyzer, fragmenter, formatter, top=5)
print repr(hl)
print now() - t
This prints:
u'Unfortunately, Klietz was eventually forced to <span class="match term0">abandon</span> his work. The\ncompany that originally owned the rights to Screenplay, Gambit, was\nsubsumed into a larger company, Interplay. Interplay later filed for'
0.0992741478549
This isn't especially fast, but it's definitely not a freeze. Is there something else I should try?
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
Thanks for your quick reply, I'll check this out.
from whoosh.
Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).
Here's the files that's causing this.
Try to search for example for "abandon rocket" it takes approx 20-30s to highlight those
from whoosh.
Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).
That's really strange. Is it possible to attach an example of the text you're highlighting (i.e. an example of res['content'])?
from whoosh.
Related Issues (20)
- Resolved: help changing operators HOT 7
- LockError on Windows HOT 1
- Please help Matt sync up the repos HOT 6
- Not possible to boost word adjacency in search results
- Current state of this project HOT 10
- Please do a release HOT 1
- Filtering Raising AttributeError only sometimes HOT 1
- can we have proximity within strict phrases?
- is literal HOT 1
- Invalid repo referenced in readme HOT 2
- TimeLimitCollector breaks filter of underlying FilterCollector
- 2.7.4: sphinx errors HOT 3
- 2.7.4: pytest error and warnings HOT 1
- 2.7.4: sphinx warnings `reference target not found`
- NestedParent query missing valid result
- NestedParent query is missing valid results HOT 1
- is there a way to add custom variations?
- <Schema: ['content']> is not a Schema
- Is AsyncWriter reliable to use?
- Sphinx: error: Invalid type
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from whoosh.