How to integrate DuckDB for text-based search & filtering about deepscatter HOT 3 OPEN

There's also some movement here I should mention.

Transformations on a working branch are allowed to return promises. Since duckdb-wasm queries are async, that means you could package an Arrow record batch on a tile, send it to the duckDB with arbitrary SQL, and then get a record batch back.

from deepscatter.

So this is a little more straightforward now, although the API is still in flux. Here's what the steps would be on this new branch.

1. Define a transformation function that works on Tile objects with a signature (tile : Tile) => Promise<Float32Array>, where the array is the same length as the tile. For example, you could do:

[...]
scatterplot._root.transformations['has dog in it'] = async function(tile) {
  const output = new Float32Array(tile.record_batch.numRows)
  // the column being searched.
  const all_rows = tile.record_batch.getChild("full_text")
  const all_strings = all_rows.toArray(); // The reason to use duckdb is that this function, deserializing a lot of text from arrow UTF-8 to javascript UTF-16 strings, is *extremely* slow.
  let i = 0;'
  for (let string of all_strings) {
     if (string.match(/dog/) {
        output[i] = 1;// Store a match as a float.
     }
  }
  return output // This array will be attached to the record batch at render-time, lazily.
}

(It's allowed for the promise not to be async, which is what you'd want in this simple JS regex case. But with duckdb you would ship the array of full_text to the db, and then get back the results of a db query as arrow.)

And then actually plot it, which causes the transformation to be run on tiles as they're needed.

scatterplot.plotAPI({encoding: {
foreground: {
field: 'has dog in it',
op: 'eq',
a: 1
}}})

from deepscatter.

If anyone's following along at home, David and I got a prototype of this running over the weekend. https://observablehq.com/d/cae8e4a3a8b7d4db The hardest part turned out to be misalignment of Arrow versions among arrow-js, duckdb, and deepscatter.

from deepscatter.