I cannot install tinysearch 0.8.1 via cargo

You would assume this works, but it does not:

cargo install tinysearch

warning: none of the package's binaries are available for install using the selected features
  bin "tinysearch" requires the features: `bin`
Consider enabling some of the needed features by passing, e.g., `--features="bin"`

Instead you need to do this, but it does not work either:

cargo install tinysearch --features=bin

error: couldn't read /home/jieiku/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tinysearch-0.8.1/assets/crate/Cargo.toml: No such file or directory (os error 2)
 --> /home/jieiku/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tinysearch-0.8.1/src/bin/utils/assets.rs:1:37
  |
1 |   pub static CRATE_CARGO_TOML: &str = include_str!(concat!(
  |  _____________________________________^
2 | |     env!("CARGO_MANIFEST_DIR"),
3 | |     "/assets/crate/Cargo.toml"
4 | | ));
  | |__^
  |
  = note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)

error: couldn't read /home/jieiku/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tinysearch-0.8.1/assets/crate/src/lib.rs: No such file or directory (os error 2)
 --> /home/jieiku/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tinysearch-0.8.1/src/bin/utils/assets.rs:5:33
  |
5 |   pub static CRATE_LIB_RS: &str = include_str!(concat!(
  |  _________________________________^
6 | |     env!("CARGO_MANIFEST_DIR"),
7 | |     "/assets/crate/src/lib.rs"
8 | | ));
  | |__^
  |
  = note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `tinysearch` (bin "tinysearch") due to 2 previous errors
error: failed to compile `tinysearch v0.8.1`, intermediate artifacts can be found at `/tmp/cargo-installZTcieL`

Would be great if we could make installation work without cargo.
Distributions, roughly ordered by complexity:

• binary releases on Github
• homebrew formula
• npm package

Feel free to pick any task from the list.

Any plans on adding new features?

E.g. adding new fields (like product image),
make body field optional,
adding search filters (e.g. size between/less/bigger than),
booleans (has feature X: yes/no)
or this point in the readme "Only searches for entire words. There are no search suggestions (yet)."

I have no experience building search engines so not really aware what is easy or good idea to build or not.

At the moment, we download the engine from crates.io whenever we build a wasm module.

This worked as an initial proof-of-concept, but it is not ideal for a few reasons:

• Network access is required. This might cause slow and flaky builds.
• The download from crates.io is quite fragile and relies on an internal API. The downloader code is quite hacky and and contains a few unwrap()s.
• Whenever we build a new version of the engine, we have to publish it to crates.io. This is an additional step and the engine on its own is quite useless, which defeats the point of publishing the crate to crates.io in the first place.

After thinking about it for a bit, I think the way forward is to bundle the engine code as a static binary blob inside of the tinysearch CLI tool.
This way we can avoid the above mentioned problems and also get one step closer toward publishing the tool as a node module in the future.

Looking around for crates that allow for including a whole directory, I found the following options, which need to be evaluated:

• https://github.com/tilpner/includedir (relatively mature, supports compression, no updates since one year)
• https://github.com/Michael-F-Bryan/include_dir (relatively mature, doesn't support compression, recently updated, issue for Dir::extract: Michael-F-Bryan/include_dir#41

We might have to bundle up one of the above crates, to support unrolling all the included files into a directory. Was thinking of dump_crate or include_crate for a name.

tinysearch is already quite fast when indexing the posts of my homepage. Doesn't mean we can't make it even faster! That would help with indexing bigger corpa.

The idea is to use rayon for that, which provides out-of-the-box parallelization by using par_iter() instead of iter() on the input files.

Was thinking a lot about the project name lately.
I picked it, because it felt like an obvious name for a... tiny search.

Now I have my doubts:

• Not the the most creative of names.
• Is it tinysearch or tiny-search or tiny_search on crates.io?
• It's kinda tedious to type out. Shorter is better.

Was thinking of something short and snappy.
The two candidates I have in mind right now are ty (first and last letter of the current project name; a symbol for the project's tiny footprint) and ivy (can you tell I like the letter y?).
There is already a popular ivy project on Github here, but it's in a super unrelated field, so I think it's fair game.
Open for other suggestions, too!

Would like to get an opinion from the other two core developers.
@jorgelbg, @fluential what's your take on this? Do you share my desire for a rename? 😆
Everyone else is invited to discuss, too of course.

Hey @mre !

An interesting project! I'm working on a related effort in a CNCF special interest group and would love to exchange notes. The Layer5 open source community has a growing focus on WASM / Rust. I welcome you to join - http://slack.layer5.io. You might find use in some of the ongoing initiatives in the community. Ping me when you do - @kanishkarj !

Now, each time you invoke tinysearch it create many files :

• package.json
• tinysearch_engine_bg.d.ts
• tinysearch_engine_bg.wasm
• tinysearch_engine.d.ts
• tinysearch_engine.js

Generating (or cleaning every files except the wasm) only the wasm could be nice to integrate this process in an automated publish workflow.

Maybe another way to see it would be to only create the wasm by default, and an option to also generate the js, demo.html…

wasm-pack is a way to compile Rust to WebAssembly.
Would be nice to use it for this project, because we could automate the (currently manual) build process.

Here are some instructions on how to get started.

I've mostly built tinysearch for my own private website and I'm not planning to actively maintain it in the future. If someone would like to take over from here, please leave a comment.

Maintenance entails:

• Responding to new issues.
• Keeping the dependencies up-to-date.
• Optionally adding new features in the future.

If you're unsure, feel free to ask here. 😊

People are interested in a size-comparison between

• tinysearch
• lunr.js
• elasticlunr
• flexsearch
• fuse.js

If anyone wants to do a comparison, post a comment here.
Some ideas:

• Use an open dataset, e.g. Shakespeare texts https://shakespeare.folger.edu/download/
• Create a size histogram of the results that we can include into the README.
• Add the code to the repository under a bench folder and create a pull request. This will make the benchmark reproducible in the future.

Feel free to ask questions here before starting.

I am wrapping up a refactor on Abridge.

I needed to test that tinysearch still works, and I ran into an error I have not seen before:

I then see its also an issue on endler.dev just click into the search box with the developer console open and you will see it:

So I then tried the endler.dev demo file and it is working properly, so it seems the actual wasm is OK? but the way that we load it in JS is the problem?

I am not yet sure what to do to fix it. It seems its related to loading the wasm on demand, when a user clicks into the search box, ideally we want to keep the ability to do that.

How difficult would it be to also return the page description in the results?

I have been implementing tinysearch into the abridge zola theme.

here is a demo using tinysearch: https://jieiku.github.io/abridge-tinysearch/

here is the normal demo which is using elasticlunr (zola's default): https://abridge.netlify.app/

abridge tinysearch branch: https://github.com/Jieiku/abridge/tree/tinysearch

Here is how elasticlunr looks:

I have modified the tinysearch js example to more tightly integrate with abridge, so it looks VERY similar, but I don't have a description to go along with the title:

The author of failure himself says so here https://internals.rust-lang.org/t/failure-crate-maintenance/12087/3.

Hi! Can you please add the tag v0.7.0 to point to 5d98c35 ?

This would enable bumping the Homebrew formula to the latest version! Thank you!

This change will reduce the number of dependencies.

After struggling mightily with tinysearch installed via Homebrew and direct downloads, this one was installed via cargo install tinysearch, cargo install wasm-pack

Help please, I am still stuck :(

Why is tinysearch's invocation of wasm-pack looking for a Cargo.toml file in the temp directory it created?

tinysearch --optimize --path static public/data_tinysearch/index.html
Unpacking tinysearch WASM engine into temporary directory "/var/folders/nx/7ys_xr014yg41_mw1zn_wvb00000gn/T/.tmpMxhsOO"
Starting unpack
Copying index into crate
Compiling WASM module using wasm-pack
Error: crate directory is missing a `Cargo.toml` file; is `/var/folders/nx/7ys_xr014yg41_mw1zn_wvb00000gn/T/.tmpMxhsOO/engine` the wrong directory?
Error: failed to execute "wasm-pack" "build" "/var/folders/nx/7ys_xr014yg41_mw1zn_wvb00000gn/T/.tmpMxhsOO/engine" "--target" "web" "--release" "--out-dir" "/Users/name/blog/static"
status: exit status: 1

Would be great to build the project on every new commit / pull request.
For that, we could add a Travis configuration.

Trying to run tinysearch on Windows 11 (with a Hugo blog):

> tinysearch.exe .\public\index.json
Unpacking tinysearch WASM engine into temporary directory "C:\\Users\\stop_\\AppData\\Local\\Temp\\.tmpZx9gHh"
Starting unpack
Name: \\?\C:\Users\stop_\(redacted)\blog\.git
Name: \\?\C:\Users\stop_\(redacted)\blog\.gitmodules
Name: \\?\C:\Users\stop_\(redacted)\blog\.hugo_build.lock
Name: \\?\C:\Users\stop_\(redacted)\blog\archetypes
Name: \\?\C:\Users\stop_\(redacted)\blog\config.toml
Name: \\?\C:\Users\stop_\(redacted)\blog\content
Name: \\?\C:\Users\stop_\(redacted)\blog\data
Name: \\?\C:\Users\stop_\(redacted)\blog\go.mod
Name: \\?\C:\Users\stop_\(redacted)\blog\go.sum
Name: \\?\C:\Users\stop_\(redacted)\blog\layouts
Name: \\?\C:\Users\stop_\(redacted)\blog\public
Name: \\?\C:\Users\stop_\(redacted)\blog\resources
Name: \\?\C:\Users\stop_\(redacted)\blog\static
Name: \\?\C:\Users\stop_\(redacted)\blog\storage
Name: \\?\C:\Users\stop_\(redacted)\blog\themes
Error: Engine directory could not be created at C:\Users\stop_\AppData\Local\Temp\.tmpZx9gHh\engine

This is a quite incomplete error message in my opinion...

File handling and index generation are cobbled together at the moment in the generate() method:
https://github.com/mre/tinysearch/blob/master/src/main.rs#L58-L77

Would be better if we iterated over all files outside of generate(), probably here:
https://github.com/mre/tinysearch/blob/master/src/main.rs#L34.

This way, we could later parallelize the corpus generation using rayon.

Currently when I build for a Zola site I do this:

tinysearch --optimize --path static public/data_tinysearch/index.html

The result is that 7 files are generated:

I would like a way for all those files to go to a temp/build directory, and only output the tinysearch_engine_bg.wasm, as far as I can tell that is the only file I require for the search to work. (that and my tinysearch.js file)

I debated even opening this report because I can probably just specify /tmp as the directory and then copy the wasm file from there... but I am curious what others think.

Edit: Here is what I am now doing instead:

cd ~/.dev/abridge
zola build
mkdir -p tmp && tinysearch --optimize --path tmp public/data_tinysearch/index.html && rsync -avz tmp/*.wasm static/ && rm -rf tmp

https://github.com/BurntSushi/fst
https://blog.burntsushi.net/transducers/#common-crawl

Expect that this blows up the index size significantly, but it might be an alternative backend that could be enabled with a feature flag, e.g. tinysearch --engine=fst or so.
@BurntSushi, do you think that a corpus like that would compress nicely with fst? https://github.com/tinysearch/tinysearch/blob/master/fixtures/index.json
Think I'll play around with your sample code on the main repository when I find the time.

I was confused for a minute when I noticed https://www.npmjs.com/package/tinysearch was 3 years old but then I checked it 0.8.2 was released last week.

Would it be possible to update npm with a new version only to point out that it's not updated on npm anymore to avoid confusion and point back to cargo/rust or github directly. Thanks!

I guess I have pushed capabilities of tinysearch to the limits. The site I'm indexing is by no means tiny. It's over 1700 pages albeit with very little text. The compiled wasm binary is 343k . Amazingly it still works very fast, though in case of some queries I'm getting a lot of false positives, like over 50%. I was wondering if it might be possible to make it more precise at the cost of a larger binary. I could live up with a ca 1MB binary as this site definitely does not have to load under 3s.
I was using the docker image to compile the binary.

docker run -v $PWD:/tmp tinysearch/cli index.json

Could tiny search run in IOS and Android WebView browsers?

I'm evaluating Tinysearch for my hugo blog.
I built the json by tweaking a little bit the example given and compiled tinysearch from git.

I installed it here for now so you can try it.
The index is here if you want to find something in here :-)

• It finds results from words which doesn't exist at all ( try "xugn.ui").
• It finds pages without the term (try "morse").
• it can't find pages containing the term (try "midsommar")

Is the index not well built or is a bug or else ?

Does tinysearch support CJK languages? (Chinese, Japanese, and Korean)

Also what about stemmers and stopwords?

I am interested in using tinysearch for Zola. I proposed it here: getzola/zola#1849

It was mentioned that there may not be specific stemmers/stopword lists for languages other than English?

I was reading over this https://endler.dev/2019/tinysearch/ and I seen removing stopwords mentioned, but it was not clear if this was something you manually did or something that is supported by tinysearch when it builds the index.

Provide a basic Github Action CI pipeline that replaces Travis.

let engine_dir = temp_dir.path().join("engine");
if !engine_dir.exists() {
  fs::create_dir_all(&engine_dir)?;
}
if !engine_dir.exists() {
    for path in fs::read_dir(out_path)? {
        println!("Name: {}", path.unwrap().path().display())
    }
    bail!(
        "Engine directory could not be created at {}",
        engine_dir.display()
    );
}

println!("Copying index into crate");
// fs::rename("storage", engine_dir.join("storage"))?;
fs::copy("storage", engine_dir.join("storage"))?;

两个物理硬盘，不能使用 fs::rename

OS: ArchLinux
Tinysearch version: Tried 0.6.2 and 0.6.0

What I did

• Installed wasm-pack, terser, binaryen, and finally tinysearch
• copied the given example file to index.json
• ran the following variations of commands

tinysearch index.json
tinyseach "index.json"
tinesearch /absolute/path/to/index.json

All of them gave the following output

Unpacking tinysearch WASM engine into temporary directory "/tmp/.tmpk894cL"
Start unpack
Copying index into crate
Error: No such file or directory (os error 2)

Kindly advice as to what other info would be useful

We need to add some usage examples for

• Jekyll
• Hugo
• Cobalt

For the most part, this means showing how to create an index.json as shown in /fixtures/index.json from this repo.

For example, I'm using the following template file for zola:

{% set section = get_section(path="_index.md") %}

[
{%- for post in section.pages -%}
{
"title": {{ post.title | safe | striptags | json_encode }},
"url": {{ post.permalink | safe | json_encode }},
"body": {{ post.content | safe | striptags | json_encode }}
}{% if not loop.last %},{% endif %}
{%- endfor -%}
]

If someone wants to tackle this, feel free to add a comment here so that others know you're working on this. You can also create a pull request straight away.

Great looking project! Exciting.

I tested it on the demo site and it's failing with errors in the console.

Hi! I'm using Github actions to build and deploy a website, and I thought it'd be cool to integrate tinysearch. I didn't find any github action to do that so I made one: https://github.com/marketplace/actions/tinysearch-action It's a simple wrapper of the docker image.

I use it in https://github.com/leonhfr/blog to build the wasm, though the search is still pending theme integration.

Just thought I'd put it here in case it's useful to someone. Thanks for the work!

The tinysearch fixtures/index.json command in the "Usage" section of the readme isn't working. I think this is because the Cargo build for tinysearch hasn't been changed since the fix in this PR: #154.

When I ran the local project using cargo run fixtures/index.json, the generation worked as expected.

At the moment, it takes ages to generate the Wasm binary.
That's because we download an entire crate and compile it whenever we run tinysearch corpus.json. Maybe anyone has an idea on how to improve that?

tempdir crate has been deprecated; use tempfile instead

The tempdir crate has been deprecated
and the functionality is merged into tempfile.

See advisory page for additional details.

Error

./tinysearch public/index.json
Error: NotEnoughSpace

Index file

 ls -lha public/index.json
-rw-r--r-- 1 q q 6,7M Jun 11 22:46 public/index.json

strace results

strace ./tinysearch public/index.json -p /tmp/

...

getrandom("", 0, GRND_NONBLOCK)         = 0
getrandom("\x1d\xee\x9e\xa5\x0c\x2d\xf4\x99\x9c\x9a\xe5\xb8\xc4\x52\x05\xb7\x4c\x3c\x2a\x79\xe4\xee\x5f\x24\xaf\x1e\xcf\x62\xb8\x49\x35\x97", 32, GRND_NONBLOCK) = 32
write(2, "Error: ", 7Error: )                  = 7
write(2, "NotEnoughSpace", 14NotEnoughSpace)          = 14
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7f7fffdd5000, 12288)           = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Some bigger websites may have high number of pages to index.

Making the body field optional would make the index more manageable and performant at a bigger scale.

For some use-cases, Xor filters are supposed to be faster and smaller than Bloom and Cuckoo filters.
Recently, support for strings was added to the xorfilter Rust crate:
https://github.com/bnclabs/xorfilter/blob/master/tests/xorfilter.rs#L56
We should test that to see if they make our wasm binary smaller.

If someone wants to give this a shot, please write a short comment here.
I'd be happy to provide mentoring if needed.

Hello,

I am trying to test the docker but when running docker build --build-arg RUST_IMAGE=rustlang/rust:nightly-alpine -t tinysearch/cli:nightly .

at step 19 from 28, the build stops with the error =>

Step 19/28 : RUN wasm-pack --version
 ---> Running in d0787575a530
The command '/bin/sh -c wasm-pack --version' returned a non-zero code: 139

I tried with others docker images like docker build --build-arg WASM_BRANCH=master --build-arg TINY_MAGIC=64 -t tinysearch/cli . or other solutions from this thread => https://github.com/tinysearch/tinysearch/issues/111

But it always lead to this error.

Does anyone know why this happened?.

Thanks for the help.

We could need some automated check for the size of the wasm file for every change.
It would report the size of the wasm file that gets generated when we call target/release/tinysearch --optimize fixtures/index.json for every pull request.

We could write a Github action for that, similar to ember-asset-size-action, which measures the size of CSS and JS fixtures like that.
See #18 for the original discussion.

For Zola sites, the tinysearch json index gets included in the sitemap.xml file.

https://endler.dev/sitemap.xml

I found a report that suggests not rendering the section will resolve the issue, but thats not the case for me: getzola/zola#604

There is currently a pull request to allow more formats, maybe then we could output/use json instead of html? getzola/zola#1998

Guys, can anyone help me with thisProblem?
I can't seem to install pop normally.. :(
I use 32GB thumb drive to install this.
:(

Hello, under what license is this project released under ? Thank-you.

When trying to build nightly with docker build --build-arg RUST_IMAGE=rustlang/rust:nightly-alpine -t tinysearch/cli:nightly . I'm getting the following error:

thread 'main' panicked at 'crate directory should exist', src/readme.rs:11:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Well, this is embarrassing.

wasm-pack had a problem and crashed. To help us diagnose the problem you can send us a crash report.


We have generated a report file at "/tmp/report-2ec64b30-c702-4581-a790-5d71fb2f66fe.toml". Submit an issue or email with the subject of "wasm-pack Crash Report" and include the report as an attachment.

- Authors: Ashley Williams <[email protected]>, Jesper Håkansson <[email protected]>

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!
Error: failed to execute "wasm-pack" "build" "/tmp/.tmpzklwBA/engine" "--target" "web" "--release" "--out-dir" "/tmp"
status: exit status: 101

In spite of the message no report file was generated at tmp/report-2ec64b30-c702-4581-a790-5d71fb2f66fe.toml