The subcommand should be hidden under something lik a dev feature. The subcommand should create a small index, entity index, autsuggest file etc. We need to host a truncated wikipedia dump and a single warc file on our servers that gets downloaded by the configure command.

The just local command can probably be removed after this has been implemented.

"Well, currently we don't. We are bootstraped and trying to keep costs low. In the future we will have, clearly labelled, contextual ads based on your current search query and a subscription option without ads. Just to re-iterate; we will only use your current search to match ads and will never track you across searches."
I hope you won't have to resort to the CPU cost of serving ads.

The user should be able to search for e.g. "test site:stackoverflow.com" and only websites containing "test" from stackoverflow should be returned.

To support this, we first need to re-write the query parser to Lalrpop as this will make it easier to parse the more advanced query syntax. It will also allow us to use the same parser library when we implement support for goggles.

Ideally we would support most of the operators mentioned here.

If one goes to the settings after performing a search, toggles safe-search, and then goes back to /search, then ?ss= is still set to the old value. However, any new search correctly uses the new value of safe-search.

This is in line what we expect when storing the query in sessions storage, but perhaps some parts of url.search should not be synced.

What are your thoughts on this?

We should calculate a centrality score for a fixed number of topics.
I think this can be done by considering a set of websites that has been classified as a specific topic, and only consider outbound links from those sites. Same principle as how personalized centrality will be implemented.

We can use the following datasets for site classification:
https://www.kaggle.com/datasets/shawon10/url-classification-dataset-dmoz
https://www.kaggle.com/datasets/hetulmehta/website-classification

I tried it, I think it's great and I like the unique results it's producing, but the chance I will continue to use it if it's not one of the search engines registered in my browser is exactly zero. I would love it if you made that possible or explained how we can do it, so that we can actually use this cool search engine instead of just seeing it once and forgetting it forever.

It would be nice to be able to restrict the search for sites that are similar to a given webpage(s). It can probably be implemented by doing a url search in the index, take the top N results and perform a MoreLikeThisQuery with Occur::Should between the N queries and an Occur::Must at the end.
This will require the parser to have acces to the index it builds the query for to perform the initial url search.

An empty search term crashes the page

Just like we support Han, Arabic etc. we should support more scripts. I think a full list can be found here.

there are some features that I miss in trystract.com

1. definitions(definitions for a query like 'define enthusiasm')
2. calculator widget
3. weather widget
4. instant answers

It would be nice to release stract as a docker image and to allow people to easily host stract with a docker compose

1. Create docker compose file
2. Host built images for dev and release channels

The answer cards should be present under most stackoverflow results

Could probably be implemented by performing a range query on the 'last updated' field

These are some places where byte offsets might be incorrectly used in place of char offsets or vice versa.

In this one the chars are counted, but char_indices produces byte offsets.

find returns a byte offset, but it is used as a char offset.

.len() returns a byte offset, but is used as a char offset.

.position() returns a char offset, but in .unwrap_or(query.len()) a byte offset is used. The resulting .take(idx) might use the byte offset given by query.len().

Searching for x-men does not currently return any sidebar as the query is tokenized as ['x-men'] whereas it should be ['x', 'men']. We should use the same tokenzier as the title-field

Noticed that the search preview of pages extract that are not Unicode (at least one that is in windows-1252) doesn’t display right:

There might be away to detect the page encoding and get the preview right?

Currently the mobile navigation drop down displays with a while background even in dark mode.

Some sites don't provide a description meta tag in the HTML. For these sites, we should take the description from DMOZ unless they have specified the NOODP tag.

1. Search for something (e.g. "Land Cruiser 80)
2. Optional: add optic
3. Click on first result
4. Return to search results
5. Search term (& optic) has (have) disappeared

edit: iOS 17 + Firefox

Searching for "linus torvalds" fails to highlight "torvalds" for some reason

Why, when I search for "Stract", none of the results include trystract, or even have stract in the title?

It would be fun to see if PGO can improve the search performance in the final application.

This leads to unambiguous grammar in the parser if we implement it naively.

We could probably just store a hashset of the image text + caption and check if they match during search. We will only need to deserialize the hashset for the results we show to the user, so it hopefully wouldn't be too expensive

Good starting points are this blog post from Peter Norvig and this blog post by Wolf Garbe. We can probably build the spell checker datastructure from all the terms in the index.

Tapping a search bar suggestion on iOS does nothing except hide the suggestions.

We already calculate the centrality for each specific webpage. This should be used during ranking and probably be prioritised higher than the host centrality.

Adding this should also give an extra boost to homepages since they tend to have a higher centrality score, and also mitigate the issue we have with multiple result from same site

• install rust
• install clang
• (optional) install cargo-watch
• (optional) install just
• (optional) install git-lfs and download data

The bang !wa 1 + 5 + 102 redirects to WolframAlpha correctly, but the expression does not seem to be encoded correctly, since the input query on WA is 1 5 102 missing the +'s.

The given URL query is ?i=1+++5+++102, while the expected is ?i=1+%2B+5+%2B+102.

I've compiled stract via cargo build --release. What do I do next?

How much disk space is required?

I can run the indexer / crawler / scraper via stract indexer, stract crawler and stract autosuggest-scrape.

• do you need to run the crawler first?

I can run the search servers via stract search-server and stract entity-search-server.

I can run the API server via stract api.

This issue tracks the problem identified in #97.

If a second search is performed without an optic, the optic selector gets empty whereas it should have "No Optic" still selected.

~19% of time during indexing seems to be used to detect the language at various stages. Most of these can probably be omitted.

Hello, this project looks interesting. As a fellow open source developer I wanted to ask for permission to add your main instance (stract.com) to Hearchco meta-search engine. If it is fine by you for us to scrape it, if not I won't be adding it to respect your contributions to the open source community.

We can get the data from Who Tracks Me and store it as a fastfield in the inverted index

• The input should have type=url or similar for correct keyboard on mobile
• A loading indicator should be shown on changes

Also have a merge policy that merges M segments into N of approximately equal size

Currently "example.com" get's tokenized to ["example", "com"]. This should be tokenized to ["example", ".", "com"] instead as this will give far better search results for navigational searches and searches like "c++". The tokenizer needs to be changed both in the indexing and search stage.

Text containing whitespaces such as "example query" should still be tokenized as ["example", "query"]

I’m guessing you use "…" to enable the search for fixed terms, which is great.

There’s a funny side effect when you use non-English style quotation marks, where they are recognised as part of the search term(s). (E.g. „…“, «…», …)

Still not sure whether a bug or a feature. :-)

We should strive to have a diverse set of results in the search from multiple sites. The ranking for subsequent results from the same site should therefore be penalised.

I think this can be done with a custom collector implementation and pre-hash the site name for every website and store it in a fast field

Should prevent "the the the..." DoS attack.

Retry all types of image downloads (sidebar, primary image, favicon etc.). Use exponential backoff with a limit.

Show entities from wikipedia in the sidebar during search.

"By default, we do store some usage statistics in order to improve the search results. Specifically the following information is stored for each search:"
I wonder how do you manage to store the data (using spinning rust for example)