FastVS (Fast Vector Search) is a Python library designed for exact vector search in tables. It provides functionality to work with both PyArrow Tables and Pandas DataFrames, allowing users to perform nearest neighbor searches using various distance metrics. It is most optimized for PyArrow Tables, as it uses the Rust Arrow library under the hood for zero-copy and vectorized computation.

Note: FastVS only supports f64 (64-bit floating point) vectors as of now.

Here is a rough comparison of the performance of FastVS vs. numpy and scipy.spatial.distance.cdist for cosine similarity (over 5 trials, on my Mac M1):

You should probably not use FastVS if you are working with large datasets and want approximate nearest neighbor search.

FastVS can be installed using pip:

pip install fastvs

FastVS offers the following main functions:

Searches a PyArrow Table for the k nearest neighbors of a query point.

• table (pyarrow.Table): The table to search.
• column_name (str): This column should be a list or np array type column, where each element is a vector of floats.
• query_point (list or np array): The query point.
• k (int): The number of nearest neighbors to return.
• metric (str): The metric to use for the search (e.g., "euclidean", "manhattan", "cosine_similarity", "inner_product").

• Tuple[List[int], List[float]]: The indices and distances of the k nearest neighbors.

import pyarrow as pa
from fastvs import search_arrow

indices, distances = search_arrow(your_pyarrow_table, "your_column", [1.0, 2.0], 5, "cosine_similarity")

Searches a Pandas DataFrame for the k nearest neighbors of a query point. This function uses search_table under the hood. Note that this function is slower than search_arrow due to the copying of data from the DataFrame to the Arrow table format.

• df (pandas.DataFrame): The DataFrame to search.
• column_name (str): The column name to search. This column should be a list or np array type column, where each element is a vector of floats.
• query_point (list or np array): The query point.
• k (int): The number of nearest neighbors to return.
• metric (str): The metric to use for the search.

• Tuple[List[int], List[float]]: The indices and distances of the k nearest neighbors.

import pandas as pd
from fastvs import search_pandas

df = pd.read_csv("your_dataset.csv")
indices, distances = search_pandas(df, "your_column", [1.0, 2.0], 5, "cosine_similarity")

Applies a distance function to a PyArrow table and returns an array of distances.

• table (pyarrow.Table): The table to search.
• column_name (str): The column name to search. This column should be a list or np array type column, where each element is a vector of floats.
• query_point (list or np array): The query point.
• metric (str): The metric to use for the search.

• pyarrow.Array: The distances in the order of the table.

import pyarrow as pa
from fastvs import apply_distance_arrow

table = pa.Table.from_pandas(your_dataframe)
distances = apply_distance_arrow(table, "your_column", [1.0, 2.0], "euclidean")

Applies a distance function to a Pandas DataFrame and returns a Series of distances. Uses apply_distance_arrow under the hood.

• df (pandas.DataFrame): The DataFrame to search.
• column_name (str): The column name to search. This column should be a list or np array type column, where each element is a vector of floats.
• query_point (list or np array): The query point.
• metric (str): The metric to use for the search.

• pandas.Series: The distances as a pandas Series.

import pandas as pd
from fastvs import apply_distance_pandas

df = pd.read_csv("your_dataset.csv")
distances = apply_distance_pandas(df, "your_column", [1.0, 2.0], "euclidean")

FastVS supports various distance metrics, including:

• Euclidean ("euclidean")
• Manhattan ("manhattan")
• Inner Product ("inner_product")
• Cosine Similarity ("cosine_similarity")

The Euclidean distance between two points $P$ and $Q$ in $N$-dimensional space, with $P = (p_1, p_2, ..., p_N)$ and $Q = (q_1, q_2, ..., q_N)$, is defined as:

The Manhattan distance (also known as L1 norm) between two points $P$ and $Q$ in $N$-dimensional space is the sum of the absolute differences of their Cartesian coordinates:

Cosine similarity measures the cosine of the angle between two vectors $P$ and $Q$ in an $N$-dimensional space. It is defined as:

where$P \cdot Q$is the dot product of vectors $P$ and $Q$, and $|P|$ and $|Q|$ are the magnitudes (Euclidean norms) of vectors $P$ and $Q$, respectively.

The inner product (or dot product) between two vectors $P$ and $Q$ in an $N$-dimensional space is defined as the sum of the products of their corresponding components:

Contributions to FastVS are welcome! Please submit your pull requests to the repository or open an issue for any bugs or feature requests.

To Dos:

• Clean up rust code
• Support f32

FastVS is released under the MIT License. See the LICENSE file in the repository for more details.