To make the app more efficient, add an option to only OCR the current window.

Currently the search UI does not show matched text / highlight matched words, as rem does.

Would improve the search experience to show why a particular frame is matching your query.

Screenshot from rem

Overlay transparent text on image so as to be selectable

(potentially works with less effort on Mac due to webview, but won't work elsewhere currently)

When scrolling through the timeline, it's very expensive to keep opening videos and pulling out frames.

We need to pull all frames for a video chunk into memory and make them available via a cache.

It would be better to load the "next chunk" for the timeline as well to keep in front of the user so there's no waiting time.

Ideally we'd keep around a bunch of frames in the cache so the user could scroll back and forth and it is instant / snappy.

Leverage something like sqlite-vss to add natural language search (in addition to) FTS.

The embedding function via candle already exists in the code.

Hi. I came across your project on HN a while ago. It's nice that you created rem without xcode+swift because I do not want to go through apple signing process. I really appreciate this repo.

While trying to run xrem, I met this error message. I am a noob to rust and svelte. I suspect that it failed to get a max frame from sqlite database.

% npm run tauri dev -- --release
(...)
warning: `xrem` (bin "xrem") generated 22 warnings (run `cargo fix --bin "xrem"` to apply 3 suggestions)
    Finished release [optimized] target(s) in 6.07s
starting app...
starting server...
started server...
9:36:06 AM [vite-plugin-svelte] /Users/noris/workspace/xrem/src/routes/search/ThumbnailCard.svelte:24:0 A11y: visible, non-interactive elements with an on:click event must be accompanied by a keyboard event handler. Consider whether an interactive element such as <button type="button"> or <a> might be more appropriate. See https://svelte.dev/docs/accessibility-warnings#a11y-click-events-have-key-events for more details.
9:36:06 AM [vite-plugin-svelte] /Users/noris/workspace/xrem/src/routes/search/ThumbnailCard.svelte:24:0 A11y: <div> with click handler must have an ARIA role
9:36:06 AM [vite-plugin-svelte] /Users/noris/workspace/xrem/src/routes/search/ThumbnailCard.svelte:70:4 Unused CSS selector ".thumbnail-card .app-name"
9:36:06 AM [vite-plugin-svelte] /Users/noris/workspace/xrem/src/routes/search/ThumbnailCard.svelte:82:4 Unused CSS selector ".thumbnail-card .app-name"
thread 'tokio-runtime-worker' panicked at src/server.rs:93:14:
Failed to get max frame: InvalidColumnType(0, "MAX(id)", Null)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at src/server.rs:88:48:
called `Result::unwrap()` on an `Err` value: PoisonError { .. }
thread 'tokio-runtime-worker' panicked at src/server.rs:88:48:
called `Result::unwrap()` on an `Err` value: PoisonError { .. }

I believe that it refers to async fn get_max_frame_handler() in src-tauri/src/server.rs.

(Don't know if it helps) I am running

• Vertura on M2 mac
• rustc 1.76.0

I ran npm i and npm run tauri dev -- --release

also had to upgrade ffmpeg rust deps to latest version because it couldn't compile

and my frames look like this:

note this log:

[swscaler @ 0x300218000] No accelerated colorspace conversion found from yuv420p to rgb24.

maybe something related to my local env?

mac book pro m3

node -v
v20.9.0

rustup -V
rustup 1.27.1 (54dd3d00f 2024-04-24)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.79.0 (129f3b996 2024-06-10)`

rem has settings. Would be great to introduce them to xrem

Is there a plan to incorporate image embeddings along with OCR and metadata-based retrieval? Utilizing the CLIP model from Candle to generate image embeddings could provide clearer context and improve the accuracy of xrem’s results. If performance is a concern, downscaling images before embedding could be a viable solution.

if you're looking for help, you might want to define some Need-help issues. I think many people like to contribute.