epsilla-cloud / vectordb Goto Github PK

Describe the feature

• to be filled
  This new feature for epsilla cloud would help users to create and search within image video content databases.
  Motivation and use case
• to be filled
  So first you should create a new smart Search Database then select image data content search add on then import your
  Data content from your local Files drive and then name your image data content search database using smart search
  For yourself or your team and then run your reverse image or input keyword into the search bar and run a few image searches
  Either way by keyword or by uploaded image or video.

Additional context

• to be filled
  Any further than less suggestions please let me know if this helps with the future of vector databases for personal use or public use.

Describe the feature

• to be filled

Motivation and use case

• to be filled

Additional context

• to be filled

Describe the feature

• Support Jina embedding: https://jina.ai/embeddings/

Motivation and use case

• to be filled

Additional context

• to be filled

Describe the feature

• Set auto incremental ID assignment, so that users don't need to manually manage the ID in application layer

Motivation and use case

• From Discord customer feedback: I also find that supporting auto incremental ID might be helpful (at least for me). Since I spent more of my time focusing on managing the metadatas dan embedding.

Additional context

Describe the feature

• Clear data support in table level and in DB level

Motivation and use case

• Keep the schema, but clear data

Additional context

• to be filled

When using Epsilla database with the docker image, we are required to use load_db method to load the database. The returned status code is HTTP 500 when the db is already loaded by the server. This could happen during normal operations, when multiple clients connect to the same db server.

It causes the clients to handle it with string matches on the error message.

if status_code != HTTPStatus.OK:
                if status_code == HTTPStatus.INTERNAL_SERVER_ERROR and (
                    "Database catalog file is already loaded" in response["message"]
                    or "DB already exists" in response["message"]
                ):
                    self._logger.info(f'{self._db_config.db_name} already loaded.')
                else:
                    raise IOError(
                        f"Failed to load database {self._db_config.db_name}. "
                        f"Error code: {status_code}. Error message: {response}."
                    )

It is better if the server / client library returns HTTP 200. Alternatively, the right HTTP status code (HTTP 409) should be used to avoid string matches on the error message.

It should require a simple change to this section of the code:

Describe the bug

I'm referring to the documentation at the following URL: https://epsilla-inc.gitbook.io/epsilladb/vector-database

1. Missing documentation on querying existing databases
2. Missing documentation on querying existing tables
3. Missing documentation on querying fields of existing tables
4. Missing documentation on filtering syntax
5. Missing documentation on indexing (see additional context).

Additional context

There is some rudimentary documentation on indexing, but some key points are missing and some are unclear:
a) No information on how to create the table with an index on the embedding field with VECTOR_FLOAT dataType, which is not created by "model", but provided as part of the data during insert.

My use case: inserting billions of language sentences (STRINGs), with their embeddings, and query them with embedding vector later on to retrieve a sentence.

b) It is not clear if the "Embedding" name of the field in the table is a keyword or if the embedding vector column can have an arbitrary name.
c) It is also unclear if externally built embedding is inserted along with the data into the table, will it be indexed automatically (by its name "Embedding" or by its type "VECTOR_FLOAT").
d) Is it possible to index any other dataType then STRING? From the documentation:
When creating tables, you can define indices to let Epsilla automatically create embeddings for the STRING fields
And then later on:
Then you can insert records in their raw format and let Epsilla handle the embedding followed by an example with insert of the text data and their embeddings, though the "Embedding" column is not defined in the table (in the previous code snippet) and despite the fact that Epsilla is promised to create the embeddings automatically.

The table used is the format with Python

client.create_table(
    table_name='MyTable',
    table_fields=[
        {'name': 'MessageID', 'dataType': 'BIGINT', 'primaryKey': True},
        {'name': 'PixelVec' , 'dataType': 'VECTOR_FLOAT', 'dimensions': 100*100*3}
    ]
)

After calling

status, response = client.query(
    table_name='MyTable',
    query_field='PixelVec',
    query_vector=myVec,
)

the response[0]['MessageID'] is some float or double
It is 1.157567399403463e+18 when it should be 1157567399403462676.
They are very close, just not quite there. After calling int(1.157567399403463e+18) I get 1157567399403462912. Only the last 3 digits are off, but it makes that makes it unusable.

I wanted to use the MessageID to retrieve the actual message, but this just isn't possible with such a response..

If you need/want any more information, please let me know.

Describe the feature

• Close the database from the Python client.

Motivation and use case

• I have a huge file with sentences (CCMatrix.en), and I want to put each sentence with its vector in a table. However, I do not have enough RAM to put 1.3B sentences into a single database. The OOM-Killer will wreak havoc on the Docker container and some other stuff. I can have several database instances where I can put these sentences and work with one database at a time. However, I can't be sure that the next database I open will have enough RAM if I didn't close the previous one.

Additional context

• Sure, I can start/stop the containers for each database and manage these starts/stops from the Python code, but this does not feel right.

Hi there,
I installed oat++ from git so I set the cmakelists like this
IMPORTED_LOCATION "/usr/local/lib/oatpp-1.3.0/liboatpp.a" INTERFACE_INCLUDE_DIRECTORIES "/usr/local/include/oatpp-1.3.0/oatpp/"

48%] Linking CXX shared library libvectordb_dylib.so /usr/bin/ld: /usr/local/lib/oatpp-1.3.0/liboatpp.a(Environment.cpp.o): relocation R_X86_64_TPOFF32\ against symbol `_ZN5oatpp4base11Environment25m_threadLocalObjectsCountE' can not be used when mak\ing a shared object; recompile with -fPIC /usr/bin/ld: failed to set dynamic section sizes: bad value collect2: error: ld returned 1 exit status make[2]: *** [CMakeFiles/vectordb_dylib.dir/build.make:533: libvectordb_dylib.so] Error 1 make[1]: *** [CMakeFiles/Makefile2:115: CMakeFiles/vectordb_dylib.dir/all] Error 2 make: *** [Makefile:91: all] Error 2

Describe the feature

• insert API: report how many records being inserted, how many skipped

{
    "statusCode": 200,
    "message": "Insert data to MyTable1291 successfully. successfully inserted 3 records. 2 records skipped ...",
    "result": {
        "inserted": 3,
        "skipped": 2
    }
}

• delete API: how many records being deleted

{
    "statusCode": 200,
    "message": "successfully deleted 2 records.",
    "result": {
        "deleted": 2
    }
}

Motivation and use case

• Better message for distributed dispatch aggregation, and telemetry

Additional context

• to be filled

Describe the feature

• Add a parameter in insert API to support upsert behavior
  What is upsert? For now, when inserting records, the records whose primary key already exist will be skipped.
  We want to support another behavior to update the existing records instead of skipping them

Motivation and use case

• Easier document update

Additional context

• to be filled

The server says its is listening on localhost but it is listining on 0.0.0.0 not 127.0.0.1 that is confusing.
std::cout << "Server running on http://localhost:" << port << std::endl;

Hi, are there plans for the API to support similarity search for multiple vectors in a single request? Afaict this is not currently possible, and it's not in the roadmap.

I'm currently benchmarking Epsilla and I would imagine batch queries would improve performance.

Describe the bug
Delete by filter is not always working as expected
Root cause need deep dive, reproduce step not standardized yet

Screenshots

• If applicable, add screenshots to help explain your problem.

Additional context

• Add any other context about the problem here.

Describe the feature
Step 1: docker support

• Add a new API to export db as a tarball and download to local:

GET /api/<DBName>/export

• Add a new API to import db from a tarball, post as payload:

POST \
     -H "Content-Type: multipart/form-data" \
     -F "file=@/path/to/your/tarball" \
     -F "json={\"name\": \"the DB name\", \"path\": \"path/on/docker/disk\"};type=application/json" \
     /api/import

Step 2. cloud support

• TBD

Motivation and use case

• To support easy db migration between docker images

Additional context

• to be filled

Describe the feature

• to be filled
  Similarity search is a another tool for searching for similar results from within your vector database.
  Motivation and use case
• to be filled
  How does this feature works so all you have to do is click on the bottom left corner of the image or video that has a icon of a magnifying 🔍 glass and should show tremendous results from within the vector database.
  Additional context
• to be filled
  Any questions about this matter please let me know if you have any interest.

Describe the feature

• Support variable dimension embedding model. If the embedding model vendor supports variable dimensions, we can let use specify a smaller or equal dimension of the model during index creation

Motivation and use case

• OpenAI text-embedding-3 supports this new feature

Additional context

• to be filled

Describe the bug
When 2 threads do search at the same time, one thread search with limit = 5, 2nd thread search with limit = 10. The @distance will return 0 for 6th - 10th element in 2nd search thread

Screenshots

• If applicable, add screenshots to help explain your problem.

Additional context

• Add any other context about the problem here.

Describe the feature

• In search filter condition, add support for @distance

{
    "table": "MyTable",
    "query": "This is a document",
    "filter": "@distance < 0.35",
    "limit": 3
}

Motivation and use case

• To filter out unpromising results from the retrieval

Additional context

• to be filled

Describe the feature

• Expose the 2 parameters for supporting high filtering case

Motivation and use case

• VectorBench benchmark use case

Additional context

• to be filled

Describe the feature

• Support OpenClip embedding to unify text and image vector embedding space, and support cross modality search

Motivation and use case

• Multi-modal search

Additional context

• to be filled

Describe the feature

• Create a new API to report statistics, start with total number of records

GET /api/<DBName>/statistics
{
  statusCode: 200,
  result: [
    {
      tableName: <tableName>
      totalRecords: 234
    },
    {
      tableName: <tableName>
      totalRecords: 23456
    },
    ...
  ]
}

Motivation and use case

• For telemetry

Additional context

• to be filled