jaeyson / ex_typesense Goto Github PK

From @javiertoledo #32

Hi @jaeyson,

I added this implementation to enable the multi_search endpoint for vector searches. When using vector embeddings, the request size tends to be much larger, and the Typesense documentation recommends using the multi-search endpoint to avoid potential limits using GET requests.

I've implemented a small test for the vector embedding scenario and drafted the @doc comments, trying to mimic your code style. However, I haven't implemented an integration with Ecto because I didn't need it. Please let me know your thoughts, and feel free to close the PR and rework it in your way.

Thanks again for this nice library!

Hi there, first of all, thanks for this library, it's being quite useful!

I'm working on a small internal tool that basically keeps track of our different database instances and performs health checks across our company, and I wonder if there is a way to configure the connection parameters dynamically from a connections database in which I store the URL and encrypted API keys for each connection. I saw in the code that you're loading the connection parameters from the Application environment in the HttpClient module, so I'm guessing that the answer is no, but I'm not an expert in Elixir, and I wanted to ask anyway, just in case there's some elegant way to override the default behavior of the HttpClient that I don't know.

If there's no other way, I guess I could always fork this repository and add the feature myself. Would you be interested in incorporating this feature into this codebase?

Thanks!

Would it possible to replace the existing API related code in https://github.com/jaeyson/ex_typesense/tree/main/lib/ex_typesense with generated code using the OpenAPI spec, https://raw.githubusercontent.com/typesense/typesense-api-spec/master/openapi.yml, using from https://github.com/aj-foster/open-api-generator?

Not sure if we want to put the generated code within the same repository or into two separate repositories or packages.

WDYT?

I would like to know if this is a work in progress? Or am I not finding the right code.

Description:

Hi @jaeyson,

I would like to request an enhancement to the ExTypesense.HttpClient module to allow developers to configure the options for Req.Request through the application configuration. This would provide greater flexibility and allow developers to tailor the HTTP client to their specific needs.

For example, it would be beneficial to configure the options like so:

config :ex_typesense,
  api_key: "XXXXXX",
  ...
  options: [finch: MyApp.Finch]

This change would enable developers to leverage the various options available in Req.Request, such as:

• :finch - the name of the Finch pool.
• :connect_options - dynamically starts (or re-uses already started) Finch pool with the given connection options.
• :inet6 - if set to true, uses IPv6.
• :pool_timeout - pool checkout timeout in milliseconds.
• :receive_timeout - socket receive timeout in milliseconds.
• :unix_socket - if set, connect through the given UNIX domain socket.
• :finch_private - a map or keyword list of private metadata to add to the Finch request.
• :finch_request - a function that executes the Finch request.

For more details on the available options, please refer to the Req documentation.

Reason for Request:

The primary reason for this request is to provide developers with the ability to configure the HttpClient to meet their specific requirements. By exposing the options for Req.Request as a configurable value, developers can easily adjust the behavior of the HTTP client without modifying the library code.

Use Case:

One specific use case for this enhancement is the need to use a different Finch adapter other than the default one started by Req. For instance, in a scenario where an application has multiple Finch pools configured for different services, a developer might want to specify a particular Finch pool for the HttpClient to use. This can be achieved by configuring the options as follows:

config :ex_typesense,
  api_key: "XXXXXX",
  ...
  options: [finch: MyApp.CustomFinch]

In this example, MyApp.CustomFinch is a custom Finch pool that the developer has configured with specific connection options or other settings that differ from the default Finch pool. This flexibility allows the developer to optimize the HTTP client's performance and behavior according to the application's unique requirements.

Thank you for considering this enhancement. I believe it will greatly improve the flexibility and usability of the ExTypesense library.

elixir version: 1.15.4
ex_typesense version: 0.4

I configure ex typesense into runtime.exs. But when I run test all of data from ExTypesense.Connection.new is nil
I put the configuration into test.exs but no effect

#33 (review)

Using the id from document is much better than using foreign key for clearing up all documents in a collection.

Update:

To delete all documents in a collection, you can use a filter that matches all documents in your collection. For eg, if you have an int32 field called popularity in your documents, you can use filter_by=popularity:>0 to delete all documents.

Another discussion: https://threads.typesense.org/t/deleting-typesense-documents-with-specific-field-using-filter-by-/2J3912e

I'll open this issue if Typesense added a wildcard for filter_by in id

Create unit tests for ci

Adding mix task like mix phx.new as a convenience command when using shell, particularly when one is testing in localhost. Scope for the mix task will be the following:

• creating collection, where it takes the name of module
• alias pointing to the collection name
• importing all records to the collection

E.g. we have a module called Person (persons table) where the collection name will be persons, then documents will be all records in the persons table. Ideally the mix task will look like:

mix typesense create --schema <MODULE_NAME>
mix typesense create --schema <MODULE_NAME> <CUSTOM_COLLECTION_NAME>
mix typesense create --alias <MODULE_NAME> <ALIAS>
mix typesense create --alias <COLLECTION_NAME> <ALIAS>

• mix typesense create --schema App.Accounts.Person <-- where the schema name will be persons.
• mix typesense create --schema App.Accounts.Person persons_dec_2023 <-- where persons_dec_2023 is the custom name linked to that module.
• mix typesense create --alias persons_dec_2023 persons <-- where persons is the collection alias
• mix typesense create --alias App.Accounts.Person persons_alias

Hey @jaeyson Would be possible to release the latest changes from the repository.
There have been a few updates merged recently that would be beneficial to have available in the latest version. Thanks

Hi,
I would like to request a feature to clear all documents from a collection without dropping the collection and its schema. This can be useful in scenarios where we want to reset the data in a collection but retain the schema for future use.

Based on docs, there are few points for making backups/restores (aka snapshots):

Backup steps

It is unsafe to directly archive/backup Typesense's data directory, since Typesense might have open files that it's writing to, as > the backup is being taken. Instead, you want to do the following:

• Call the Snapshot API endpoint, specifying the directory to which to write the snapshot to on the server.
• Backup this directory to remote storage, using say tar -czvf backup.tar.gz -C /tmp/typesense-data-snapshot .

Restore steps

• Stop any running Typesense processes.
• Delete the contents of the existing data dir. Eg: rm -rf /var/lib/typesense/*
• Extract the tar gzip backup file to the data directory. Eg: cd /var/lib/typesense ; tar -xf backup.tar.gz .
• Start the Typesense process again.

Typesense will then read the snapshot from the data dir and build the in-memory index.

Manage and load balance typesense nodes for high availability.

Scenario: distribute load on instances when there's no external load balance request.

Accompanying docs:

• High availability
• Running in production
• Syncing data