Git Product home page Git Product logo

meilisearch-go's Introduction

Meilisearch-Go

Meilisearch Go

GitHub Workflow Status Test License Bors enabled

⚡ The Meilisearch API client written for Golang

Meilisearch Go is the Meilisearch API client for Go developers.

Meilisearch is an open-source search engine. Learn more about Meilisearch.

Table of Contents

📖 Documentation

This readme contains all the documentation you need to start using this Meilisearch SDK.

For general information on how to use Meilisearch—such as our API reference, tutorials, guides, and in-depth articles—refer to our main documentation website.

⚡ Supercharge your Meilisearch experience

Say goodbye to server deployment and manual updates with Meilisearch Cloud. Get started with a 14-day free trial! No credit card required.

🔧 Installation

With go get in command line:

go get github.com/meilisearch/meilisearch-go

Run Meilisearch

There are many easy ways to download and run a Meilisearch instance.

For example, using the curl command in your Terminal:

# Install Meilisearch
curl -L https://install.meilisearch.com | sh

# Launch Meilisearch
./meilisearch --master-key=masterKey

NB: you can also download Meilisearch from Homebrew or APT or even run it using Docker.

🚀 Getting started

Add documents

package main

import (
	"fmt"
	"os"

	"github.com/meilisearch/meilisearch-go"
)

func main() {
	client := meilisearch.NewClient(meilisearch.ClientConfig{
                Host: "http://127.0.0.1:7700",
                APIKey: "masterKey",
        })
	// An index is where the documents are stored.
	index := client.Index("movies")

	// If the index 'movies' does not exist, Meilisearch creates it when you first add the documents.
	documents := []map[string]interface{}{
        { "id": 1, "title": "Carol", "genres": []string{"Romance", "Drama"} },
        { "id": 2, "title": "Wonder Woman", "genres": []string{"Action", "Adventure"} },
        { "id": 3, "title": "Life of Pi", "genres": []string{"Adventure", "Drama"} },
        { "id": 4, "title": "Mad Max: Fury Road", "genres": []string{"Adventure", "Science Fiction"} },
        { "id": 5, "title": "Moana", "genres": []string{"Fantasy", "Action"} },
        { "id": 6, "title": "Philadelphia", "genres": []string{"Drama"} },
	}
	task, err := index.AddDocuments(documents)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	fmt.Println(task.TaskUID)
}

With the taskUID, you can check the status (enqueued, canceled, processing, succeeded or failed) of your documents addition using the task endpoint.

Basic Search

package main

import (
    "fmt"
    "os"

    "github.com/meilisearch/meilisearch-go"
)

func main() {
    // Meilisearch is typo-tolerant:
    searchRes, err := client.Index("movies").Search("philoudelphia",
        &meilisearch.SearchRequest{
            Limit: 10,
        })
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    fmt.Println(searchRes.Hits)
}

JSON output:

{
  "hits": [{
    "id": 6,
    "title": "Philadelphia",
    "genres": ["Drama"]
  }],
  "offset": 0,
  "limit": 10,
  "processingTimeMs": 1,
  "query": "philoudelphia"
}

Custom Search

All the supported options are described in the search parameters section of the documentation.

func main() {
    searchRes, err := client.Index("movies").Search("wonder",
        &meilisearch.SearchRequest{
            AttributesToHighlight: []string{"*"},
        })
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    fmt.Println(searchRes.Hits)
}

JSON output:

{
    "hits": [
        {
            "id": 2,
            "title": "Wonder Woman",
            "genres": ["Action", "Adventure"],
            "_formatted": {
                "id": 2,
                "title": "<em>Wonder</em> Woman"
            }
        }
    ],
    "offset": 0,
    "limit": 20,
    "processingTimeMs": 0,
    "query": "wonder"
}

Custom Search With Filters

If you want to enable filtering, you must add your attributes to the filterableAttributes index setting.

task, err := index.UpdateFilterableAttributes(&[]string{"id", "genres"})

You only need to perform this operation once.

Note that Meilisearch will rebuild your index whenever you update filterableAttributes. Depending on the size of your dataset, this might take time. You can track the process using the task status.

Then, you can perform the search:

searchRes, err := index.Search("wonder",
    &meilisearch.SearchRequest{
        Filter: "id > 1 AND genres = Action",
    })
{
  "hits": [
    {
      "id": 2,
      "title": "Wonder Woman",
      "genres": ["Action","Adventure"]
    }
  ],
  "offset": 0,
  "limit": 20,
  "estimatedTotalHits": 1,
  "processingTimeMs": 0,
  "query": "wonder"
}

🤖 Compatibility with Meilisearch

This package guarantees compatibility with version v1.x of Meilisearch, but some features may not be present. Please check the issues for more info.

💡 Learn more

The following sections in our main documentation website may interest you:

⚙️ Contributing

Any new contribution is more than welcome in this project!

If you want to know more about the development workflow or want to contribute, please visit our contributing guidelines for detailed instructions!


Meilisearch provides and maintains many SDKs and Integration tools like this one. We want to provide everyone with an amazing search experience for any kind of project. If you want to contribute, make suggestions, or just know what's going on right now, visit us in the integration-guides repository.

meilisearch-go's People

Contributors

42atomys avatar alallema avatar alexisvisco avatar aooohan avatar azanul avatar b4sen avatar bidoubiwa avatar bors[bot] avatar brunoocasali avatar curquiza avatar dependabot-preview[bot] avatar dependabot[bot] avatar eskombro avatar gillesfabio avatar ginglis13 avatar gorymoon avatar jackielii avatar joel-nickson avatar luigibarbato avatar meili-bors[bot] avatar meili-bot avatar mmachatschek avatar paulkovalov avatar penthaapatel avatar qdequele avatar roelofjan-elsinga avatar theag3nt avatar thearas avatar trim21 avatar zinovik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

meilisearch-go's Issues

Missing code samples.

An update has been made on the code-samples list of samples.

The following samples are missing from this repository .code-samples.meilisearch.yaml

 settings_guide_synonyms_1
 add_movies_json_1

To know what we expect from these specific samples, please visit the cURL sample file.

Exemple for settings_guide_synonyms_1:

  $ curl \
    -X POST 'http://localhost:7700/indexes/tops/settings' \
    --data '{
        "synonyms": {
            "sweater": ["jumper"],
            "jumper": ["sweater"]
        }
    }'

Infinite loop in AwaitAsyncUpdateID function

If I have well understood, we are stuck in an infinite loop, if for any reason, the status of the update is stuck to enqueued.

meilisearch-go/client.go

Lines 241 to 256 in f9427a2

// AwaitAsyncUpdateID check each 16ms the status of a AsyncUpdateID.
// This method should be avoided.
// TODO: improve this method by returning a channel
func (c Client) AwaitAsyncUpdateID(indexID string, updateID *AsyncUpdateID) UpdateStatus {
apiUpdates := c.Updates(indexID)
for {
update, err := apiUpdates.Get(updateID.UpdateID)
if err != nil {
return UpdateStatusUnknown
}
if update.Status != UpdateStatusEnqueued {
return update.Status
}
time.Sleep(time.Millisecond * 16)
}
}

We should define a default value of timeout.
This value could be change when passing an optional timeOut parameter in this function.

Plus, the name of the updateID parameter is not really accurate: the function expects an update "object" and not a updateID.

Moving tests to testify and testcontainers

Translating tests to testify and testcontainers.

Testify will allow you to write unit tests and is also very convenient if you need to set up an environment. This is a very good JUnit counterpart for Golang. Testcontainers will allow us to raise the container with Meilisearch in runtime and test environment, which is good news in my opinion. Together with this task, it will probably be possible to close issue #16 .

Delete indexes after tests

Currently, if I have already a MeiliSearch isntance running locally, and if I run the tests ./run_tests.sh, there are several indexes still running in my MeiliSearch instance.

We should provide tests that clean all the indexes created for tests purpose.

Per index stats

It seems like the API for per-index stats information is currently not available with this client library, is that right? client.Stats().Get() always returns an empty stats data.

Using SearchRequest.FacetsDistribution creates invalid request

Hello,

I'm running into an issue with adding FacetsDistribution to my SearchRequest object. It appears a change made recently is generating an invalid value.

I'm running go 1.13.6 on linux, pulling from master. Though it seems the following change has been implemented for all of 0.11.x.

values.Add("facetsDistribution", fmt.Sprintf("[%q]", strings.Join(request.FacetsDistribution, "\",\"")))

The problem seems to be %q is escaping the prequoted list, resulting in an empty FacetsDistribution in the response.

Example usage:

facets := []string{"city", "state"}
results, _ :=client.Search(idx).Search(meilisearch.SearchRequest{
	Query: "bob",
	FacetsDistribution: facets,
})
// results.FacetsDistribution is empty

The problem can be replicated with

facets := []string{"city", "state"}
fmt.Printf("[%q]", strings.Join(facets, "\",\""))
// Result: ["city\",\"state"]

https://play.golang.org/p/Gh2USjlJvsG

Thanks!

Migrating 40M records from Postgres to Meilisearch takes too long

I am trying to migrate 40M Postgres records (user comments) to Melisearch. Each time, I am querying 10K records from Postgres and writing to Meilisearch. The migration script has been completed. But it has been more than 3 days, only 4.2 Million records only migrated. The count is increasing very slowly.

I also see the following error messages in the error log:

[2021-02-18T12:48:44Z INFO  ureq::unit] sending request POST https://api.amplitude.com/httpapi
[2021-02-18T13:14:58Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T13:43:50Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T13:48:45Z INFO  ureq::unit] sending request POST https://api.amplitude.com/httpapi
[2021-02-18T14:13:03Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T14:42:05Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T14:48:45Z INFO  ureq::unit] sending request POST https://api.amplitude.com/httpapi
[2021-02-18T15:12:06Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T15:41:40Z ERROR meilisearch_core::database] commit nested transaction failed: Input/output error (os error 5)
[2021-02-18T15:48:46Z INFO  ureq::unit] sending request POST https://api.amplitude.com/httpapi

Also, I tried to get updates using the following curl

curl 'http://localhost:7700/indexes/comments/updates' | jq

It is running forever, not getting any result from the server.

  1. What is commit nested transaction failed and How to fix it?
  2. Is there any best practice to write large amounts data to meilisearch?
  3. Is there any limitation in the amount of data to be handled by meilisearch?
  4. Is there an option to disable amplitude data?

Search parameters should be optional

The search returns no result if I don't pass the Limit parameter. It does not fail. It just returns empty Hits.

_, err := client.Search("indexUID").Search(meilisearch.SearchRequest{
	Query: "harry pottre",
})

=> no result

_, err := client.Search("indexUID").Search(meilisearch.SearchRequest{
	Query: "harry pottre",
        Limit: 10,
})

=> result

We should be able to pass only the Query parameter

Edit

I understand that 2 parameters at least are needed because

searchRes, err := client.Search("indexUID").Search(meilisearch.SearchRequest{
	Query: "harry pottre",
	AttributesToHighlight: "*",
})

returns results as well.

My bad, it does not even run at all, I open an issue for that: #7

Version

v0.9.0 of MeiliSearch.

Change master branch to main

Let's be allies and make this change that means a lot.

Here is a blog post that explain a little more why it's important, and how to easily do it. It will be a bit more complicated with automation, but we still should do it!

Make sdk faster

As I think http/client is a rather slow library. I would like to use something faster as the httpclient, for example the fasthttpclient. It also probably makes sense to add methods for sending requests with a raw body ([] byte). I can take it on myself

A basic error handler

Create a basic error handler, with these 3 custom errors:

  • MeiliSearchApiError
  • MeiliSearchCommunicationError
  • MeiliSearchTimeoutError

MeiliSearch API has an error handler, and returns this kind of JSON when an error is found:

{
  "message": "Index movies not found",
  "errorCode": "index_not_found",
  "errorType": "invalid_request_error",
  "errorLink": "https://docs.meilisearch.com/errors#index_not_found"
}

The complete list of errorCode is available in the documentation

It could be great to let the user access message, errorCode, errorType and errorLink to a better understanding an error.

Access _formatted in a search response

Currently, we do not have a way to access the _formatted key in a search response if we ask for an AttributesToHighlight.

meilisearch-go/types.go

Lines 157 to 163 in f9427a2

type SearchResponse struct {
Hits []interface{} `json:"hits"`
Offset int64 `json:"offset"`
Limit int64 `json:"limit"`
ProcessingTimeMs int64 `json:"processingTimeMs"`
Query string `json:"query"`
}

client.List() parameters need to be optional

Since the MeiliSearch HTTP server changed from tide to atix they way it handles parameters that are empty has changed. As Go initializes every variable to a Default Zero value, when a parameter is not set to a value by the user, the Go SDK will send the defult zero value to the server. This leads to problems like #37

For the client List method, the 3 parameters (limit, offset and attributesToRetrieve) are then initialized to their default zero values. If the user does not specify, for example some attributesToRetrieve, Go SDK is making a request with an empty list of attibutesToRetrieve. This parameters need to be added to the request only if the user specifies a value for them.

Test on master are failing because of this, as we can see for example here

GitHub actions arre nonetheless accepting those tests and validatign the check, which is a different issue and should be handled in #51

AddOrUpdate API error

When i update products with the following code;

_, err := conn.Client.Documents(Index+sort).AddOrUpdate(jsonParsed.Data()) // => { "updateId": 0 }
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

same times I get the following error? could it be load?

unable to execute request (path "PUT /indexes/productPT/documents" with method "Documents.AddOrUpdate"): Put "https://meili.nobrinde.online/indexes/productPT/documents": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

golint

In the development workflow part there is two points about golint:

# Install golint
$ go get -u golang.org/x/lint/golint
# Use golint
$ golint

2 issues:

  • The first command modifies the go.sum and the go.mod files. Is it expected?
  • Despite I ran the first command, the golint tool is still unavailable for me.

When I run go get -u golang.org/x/lint/golint I get:

go: found golang.org/x/lint/golint in golang.org/x/lint v0.0.0-20200302205851-738671d3881b
go: golang.org/x/tools upgrade => v0.0.0-20200325010219-a49f79bcc224

And the command returns 0 (success).

AttributesToHighlight example does not work

When I try the example in the readme with the AttributesToHighlight parameter:

resp, err := client.Search(indexUID).Search(meilisearch.SearchRequest{
    Query: "harry pottre",
    AttributesToHighlight: "*",
})

it does not run, with this error:

# command-line-arguments
./test.go:47:3: cannot use "*" (type string) as type []string in field value

Tests with a master key

Currently, the tests run with a MeiliSearch instance that does not provide any master key.

Can we launch a MeiliSearch instance with the option --master-key=masterKey to test this package? And for course, adapt the tests.

Getting weird JSON error while creating a document

I see a weird JSON error when adding a document to Meilisearch server. This may not be a bug, and is probably caused by wrong usage, but I couldn't find any useful troubleshooting documents for this.

2020/09/13 05:58:39 unaccepted status code found: 400 expected: [202], message from api: 'Invalid JSON: invalid type: map, expected a sequence at line 1 column 0', request: {"content":"testing 1234","createdAt":"2020-09-13T05:58:39.362533+09:00","id":"btejcvrpc98mn373um10","image":"https://google.com","link":"","tags":["beans","bananas","tag1"],"title":"OOOH lala","type":"POST","updatedAt":"2020-09-13T05:58:39.362888+09:00","urlSlug":"test-slug"} (path "PUT /indexes/articles/documents" with method "Documents.AddOrUpdate")

# maybe the problem has been caused by wrong conversion method, tried using third-party conversion module and it shows the similar sort of error
# github.com/mitchellh/mapstructure

2020/09/13 06:28:32 unaccepted status code found: 400 expected: [202], message from api: 'Invalid JSON: invalid type: map, expected a sequence at line 1 column 0', request: {"content":"testing 1234","createdAt":{},"deletedAt":{"Time":{},"Valid":false},"id":"btejr03pc98g0r8l6l80","image":"https://google.com","tags":["beans","bananas","tag1"],"title":"OOOH lala","type":"POST","updatedAt":{},"urlSlug":"test-slug"} (path "PUT /indexes/articles/documents" with method "Documents.AddOrUpdate")

no matter how I construct map[string]interface{} I keep seeing Invalid JSON error message.

import (
  "gorm.io/gorm"
  "log"
  "github.com/meilisearch/meilisearch-go"
  "github.com/novalagung/gubrak/v2"
)

type Article struct {
	ID        string `gorm:"primaryKey"`
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt gorm.DeletedAt `gorm:"index"`
	Type      string
	Title     string
	Content   string
	Image     string
	Link      string
	Tags      []Tag `gorm:"many2many:article_tags;"`
	URLSlug   string
}

type Tag struct {
  Value string;
  ID       string;
}

func TagsToStringArray(tags []Tag) []string {
	return gubrak.From(tags).Map(func(t Tag) string {
		return t.Value
	}).Result().([]string)
}

func ArticleToSearch(article *Article) map[string]interface{} {
	tagValues := TagsToStringArray(article.Tags)
	return map[string]interface{}{
		"id":        article.ID,
		"content":   article.Content,
		"createdAt": article.CreatedAt,
		"updatedAt": article.UpdatedAt,
		"title":     article.Title,
		"type":      article.Type,
		"image":     article.Image,
		"link":      article.Link,
		"urlSlug":   article.URLSlug,
		"tags":      tagValues, // at first I assumed this might be the problem, but it persists even without this item
	}
}

func AddToMeili() {
        article := Article{}
        getArticleFromDB(&article)
	meiliDoc := scheme.ArticleToSearch(article)
	log.Printf("meili doc %v", meiliDoc)
	_, err := meiliClient.Documents("articles").AddOrUpdate(meiliDoc)
        log.Printf("%v", err) // <-- the error is detected from here
}

Searchable Attributes not persisting

Hi, I'm having an issue with UpdateSearchableAttributes not saving after a document insert. To reproduce:

  1. Create new directory, then run go mod init and go get github.com/meilisearch/meilisearch-go
  2. Paste code below
  3. Start Meilisearch
  4. go run main.go, you should see [Name] printed. That is the searchable attribute.
  5. Comment out line 42 (createIndex() in main) and run again
  6. You should see [Name Year] printed, meaning the settings created in createIndex were not persisted.
Code
package main

import (
	"fmt"
	"time"

	"github.com/meilisearch/meilisearch-go"
)

type Movie struct {
	Name string
	Year int
	ID   int
}

var uid = "movies4"

var client = meilisearch.NewClient(meilisearch.Config{
	Host: "http://127.0.0.1:7700",
})

func createIndex() {
	// Create index
	client.Indexes().Create(meilisearch.CreateIndexRequest{
		UID:        uid,
		PrimaryKey: "ID",
	})

	// Update searchable attirbutes
	_, err := client.Settings(uid).UpdateSearchableAttributes([]string{"Name"})
	if err != nil {
		fmt.Println(err)
	}

	time.Sleep(time.Second)
}

func insert(id int) {
	// Insert document
	movies := []Movie{Movie{Name: "The Shining", Year: 1980, ID: id}}
	_, err := client.Documents(uid).AddOrUpdate(&movies)
	if err != nil {
		fmt.Println(err)
	}

	// Get searchable attributes
	a, err := client.Settings(uid).GetSearchableAttributes()
	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(*a)
}

func main() {
	createIndex()
	insert(1)
}


Get Total Number of hits for SearchResponse

We don't get the Total Size of the Search Response, but we do get it as nbHits in the web-interface.
To fix that, could we add :
TotalHits int64 json:"nbHits" to the struct SearchResponse in types.go?

Typo in documentation

on the page of Quick start guide, in the create your index section, under the Go tab

Quick Start Guide | Create your index

in the third square box, the example given contains

var client = NewClient(meilisearch/Config{
    Host: "http://127.0.0.1:7700",
})

it should be a . and not /

var client = NewClient(meilisearch.Config{
    Host: "http://127.0.0.1:7700",
})

Use URL query parameter builder

This code is unsafe for the developer who is using the library.

Query parameters should be built with the URL type. (https://stackoverflow.com/a/30657518/5894301)

Why is it unsafe ?

Imagine the developer who is using the APIDocuments.List set AttributesToRetrieve to an input from an user of his website. A malicious user can give an input like that: ?limit=300000.

What could be cool ?

With the current internalRequest type you can't set query parameters. I think we could add a QueryParams field, build the URL and add thoses query parameters in the Client.sendRequest method. That will avoid to have a special case in this method too (which is not pretty nice).

Change the usage

Description

In the continuity of this discussion. The Meilisearch Teams has decided to try to bring documentation, logic, and structure of this SDK closer to the others to be more developer-friendly and to have a usage that seems to us more pleasant and more accessible.

Steps

Changing the usage

package main

import (
	"fmt"

	"github.com/meilisearch/meilisearch-go"
	"github.com/meilisearch/meilisearch-go/client"
)
func main() {
	client = meilisearch.NewClient(client.ClientConfig{
                Host: "http://127.0.0.1:7700",
                MasterKey: "masterKey",
        })
	// An index is where the documents are stored.
	index := client.Index("indexUID")

	// If the index 'books' does not exist, MeiliSearch creates it when you first add the documents.
	documents := []map[string]interface{}{
		{"book_id": 123,  "title": "Pride and Prejudice"},
		{"book_id": 456,  "title": "Le Petit Prince"},
		{"book_id": 1,    "title": "Alice In Wonderland"},
		{"book_id": 1344, "title": "The Hobbit"},
		{"book_id": 4,    "title": "Harry Potter and the Half-Blood Prince"},
		{"book_id": 42,   "title": "The Hitchhiker's Guide to the Galaxy"},
	}
	_, err := index.addDocuments(documents)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	index.Delete()
}

Restructure the file hierarchies

New package structure inspired by what stripe does and following @eskombro guidance:

  • A Client package, which is responsible for:

    • Index creation, update, and retrieval
    • Global MeiliSearch keys handling
    • Server stats, health, sys-info, and version
  • An Index package, responsible for:

    • Documents
    • Updates
    • Search (internal to each index)
    • Index settings
    • Index stats
  • A Custom Error package widely used in the project following previous error handling discussions

    • which will be made in a next issue

Tests

All methods must be well tested a lot

To do

  • Client package
  • Index package
  • Error package
  • Tests

Test helPR

Create issue, link a PR and automate labeling

A Go SDK Refactor?

Every language has it's own specifics, its own rules and good practices. Go is clearly not an exception. But Maintaining several SDKs requires a lot of conciliation between those specific characteristics from one side, and the bigger project (MeiliSearch itself) integrations flow, logic and structure.

This is why I consider it really important to try to find as much coherence as possible between the different SDKs ans integrations. As for now, I feel that the Go SDK is very often considered as the exception, a 'separate case', in every discussion. But this makes maintainability not really cost efficient. Also, as it evolves, it will just keep growing further and further in the future from it's SDK brothers and sisters, and I don't want it to end up alone :)

I spent some time looking to how the other SDKs are built, and why is it that makes it so specific, because I would like to try to make some effort to bring this SDK closer to the other SDKs documentation, logic and structure, to make it easier to evolve, to be used and documented. The work that has been done on it until now is very good (special thanks to @alexisvisco) and I think we can totally keep building over it, and this is why I would like to propose a little refactor for the whole project, keeping as much of this progress as we can, but without fear of exploring new paths.

In my opinion, there is one big thing that can totally give it a new workflow and look, and is not tied to any specificity of Go, but would make it more coherent with the rest of the MeiliSearch project and easier to maintain and live with the other SDKs: the project structure.

  • Most MeiliSearch SDKs are using an POO paradigm until now. Go doesn't really exactly implement POO principles, but structures, interfaces and methods tied to them offer us more or less the same functionality, inheritance aside. This is something that is pretty much exploited everywhere in this SDK already. Following that tought, I would like to suggest, first of all, a new Package structure:

    • A Client package, which is responsible for:

      • Index creation, update and retrieval
      • Global MeiliSearch keys handling
      • Server stats, health, sys-info and version
    • An Index package, responsible for:

      • Documents
      • Updates
      • Search (internal to each index)
      • Index settings
      • Index stats
    • A Custom Error package widely used in the project following previous error handling discussions

    • Tests, tests everywhere!

I am picturing a repository having one folder at it's root (and all the config files that you normally find in a go repo) called meilisearch containing the whole sources of the meilisearch package. Inside this folder, something that looks like this (more or less)

Screenshot 2020-06-16 at 03 07 57

This would be a very fast scketch of how a Possible Getting started would look like

package main

import (
	"fmt"

	"github.com/eskombro/meilisearch-go/meilisearch/client"
	"github.com/eskombro/meilisearch-go/meilisearch/index"
)

func main() {

	meiliHost := "http://localhost:7700"
	masterKey := "masterKey"
	indexUid := "indexUID"

	// Create an index WITH Master Key
	msClient = client.Create(client.Client{Host: meiliHost, MasterKey: masterKey})
	msIndex = index.Create(msClient, index.IndexConfig{Uid: indexUid})

	documents := []map[string]interface{}{
		{"book_id": 123,  "title": "Pride and Prejudice"},
		{"book_id": 456,  "title": "Le Petit Prince"},
		{"book_id": 1,    "title": "Alice In Wonderland"},
		{"book_id": 1344, "title": "The Hobbit"},
		{"book_id": 4,    "title": "Harry Potter and the Half-Blood Prince"},
		{"book_id": 42,   "title": "The Hitchhiker's Guide to the Galaxy"},
	}
	update := msIndex.addDocuments(documents)
	msIndex.Delete()
}

Methods can then be clearly separated, and use the same logic we use in other SDKs. This structure and example of a getting started is for a general idea. It is not only keeping the same logic and structure than the other SDKs, but also closer to what they do at Algolia (which I bring into the discussion because it seems to offer a very good experience for the developer):

Screenshot 2020-06-16 at 03 20 24

I would really like to have your opinion in this subject, my only goal with this is to start a general discussion of where the SDK should be heading to, about what it's goal should be, and start having a roadmap for it, that will then transform into issues, and then peoples work and contributions for a great MeiliSearch Go Experience! 🎉

I would really like to have your opinion on this, @curquiza, @tpayet , @bidoubiwa, @qdequele, @alexisvisco, @derekahn => If you guys have some time to read and give some insigth, I would be very grateful!

Typo In 'Add documents' Code Example In README.md File

Hello,

The 'Add documents' code example in the README.md file contains a typo, which results in a compile error. The code:

var client = meilisearch.NewClient(meilisearch.Config{
        Host: "http://127.0.0.1:7700",
        APIKey: "masterKey"
    })

Is missing a comma at the end of "masterKey", example:

var client = meilisearch.NewClient(meilisearch.Config{
        Host: "http://127.0.0.1:7700",
        APIKey: "masterKey", // <- comma needed here :)
    })

Without the comma you get a compiler error:

syntax error: unexpected newline, expecting comma or }

Great project!

Thanks.

Create and fill sample file to display GO examples in MeiliSearch Documentation

Code Samples for MeiliSearch documentation

Introduction

As MeiliSearch grows so does its SDK's. More and more SDK's rises from the ground and they all deserve to be as well documented as the core engine itself.

The most common way for a user to understand MeiliSearch is to go to its official documentation. As of yesterday all examples in the documentation were made with cURL. Unfortunately, most of our users do not communicate with MeiliSearch directly with cURL. Which forces them to search for the specific references somewhere else (in the readme's, in the sdk's code itself,..). This makes for unnecessary friction.

Goal

We want our documentation to include all SDK's

As a first step we want all examples to be made using the most possible SDK's. As did Stripe and Algolia.

sdk_sample

examples with curl, javascript and soon enough this SDK too!

To achieve this it is expected from this SDK to create a sample file containing all the code samples needed by the documentation.

These are the steps to follow:

  • Create your sample file
  • Fill your sample file
  • Add your samples to the documentation

Create your sample file

The sample file is a yaml file added at the root of each MeiliSearch SDK.
Sample files are created based on the sample-template.yaml file.

sample-template file:

get_one_index_1: |-
list_all_indexes_1: |-
create_an_index_1: |-
...

This template is accessible publicly here or in the public directory : .vuepress/public/sample-template.yaml of the documentation.

The name of the file should be .code-samples.meilisearch.yaml

Fill your sample file

After creating the sample file with the content of the sample template you should have a file containing all the sampleId's.

get_one_index_1: |-
list_all_indexes_1: |-
create_an_index_1: |-
...

By the name of the different sampleId you should know where that code sample will be added to the documentation.

For example, create_an_index_1 is the first example in the API References of the index creation.

Using the already existing cURL example in the documentation you should see what is expected from that sample. It is very important that you look at the already existing samples in the documentation as it gives you the parameters to use in your sample to match with the rest of the documentation.

Good sample based on documentation:

create_an_index_1: |-
  client.createIndex({ uid: 'movies' })

Bad sample that does not match with the response.

create_an_index_1: |-
  client.createIndex({ uid: 'otherName' })

Each sample is expected to be written in the respective SDK language.

Javascript example:

get_one_index_1: |-
  client.getIndex('movies').show()
list_all_indexes_1: |-
  client.listIndexes()
create_an_index_1: |-
  client.createIndex({ uid: 'movies' })
  ...

The complete cURL sample file is available at the root of the documentation repository.
Every other SDK sample file should be available at the root of their respective repository.

Formatting Exception

There is one exception to the formatting rule.
When the sampleId finishes with _md it means it is expected to be written in markdown format.

JavaScript sample id with _md extension:
yaml-js-example

Add your samples to the documentation

Once your sample file is filled with the code samples, you will need to create a pull request on the documentation repository.

Open the following file in your IDE:
.vuepress/code-samples/sdks.json

And add your sample file to the list:

[
  ...
  {
    "language": "sdk-language",
    "label": "sdk-label",
    "url": "url to yaml file"
  }
]

The language key expect a supported language for the code highlight.

The label key is the name of the tab. While label and language could have been the same, it created some conflict (i.e: bash and cURL).

The url is the raw link to access your sample file. It should look like this:
https://raw.githubusercontent.com/[PROJECT]/[REPO]/.code-samples.meilisearch.yaml

lint review

Golint is a linter for Go source code - https://goreportcard.com/report/github.com/meilisearch/meilisearch-go#golint

        meilisearch-go/apis.go
        Line 10: warning: comment on exported type ApiIndexes should be of the form "ApiIndexes ..." (with optional leading article) (golint)
        Line 78: warning: comment on exported type ApiSearch should be of the form "ApiSearch ..." (with optional leading article) (golint)
        Line 89: warning: exported type ApiSynonyms should have comment or be unexported (golint)
        Line 101: warning: comment on exported type ApiStopWords should be of the form "ApiStopWords ..." (with optional leading article) (golint)
        Line 124: warning: comment on exported type ApiUpdates should be of the form "ApiUpdates ..." (with optional leading article) (golint)
        Line 142: warning: comment on exported type ApiKeys should be of the form "ApiKeys ..." (with optional leading article) (golint)
        Line 163: warning: exported type ApiSettings should have comment or be unexported (golint)
        Line 201: warning: exported type ApiSystemInformation should have comment or be unexported (golint)

        meilisearch-go/error.go
        Line 11: warning: exported type ErrCode should have comment or be unexported (golint)
        Line 14: warning: exported const ErrCodeUnknown should have comment (or a comment on this block) or be unexported (golint)
        Line 118: warning: exported method Error.WithMessage should have comment or be unexported (golint)
        Line 128: warning: exported method Error.WithErrCode should have comment or be unexported (golint)
        Line 138: warning: exported method Error.ErrorBody should have comment or be unexported (golint)

        meilisearch-go/types.go
        Line 5: warning: exported type Unknown should have comment or be unexported (golint)
        Line 11: warning: exported type SchemaAttributes should have comment or be unexported (golint)
        Line 14: warning: exported const SchemaAttributesDisplayed should have comment (or a comment on this block) or be unexported (golint)
        Line 20: warning: exported type Attributes should have comment or be unexported (golint)
        Line 22: warning: exported type RawAttribute should have comment or be unexported (golint)
        Line 27: warning: exported type RawSchema should have comment or be unexported (golint)
        Line 32: warning: exported type Schema should have comment or be unexported (golint)
        Line 34: warning: exported type Index should have comment or be unexported (golint)
        Line 41: warning: exported type Settings should have comment or be unexported (golint)
        Line 47: warning: exported type Synonym should have comment or be unexported (golint)
        Line 52: warning: exported type ACL should have comment or be unexported (golint)
        Line 55: warning: exported const AclIndexesRead should have comment (or a comment on this block) or be unexported (golint)
        Line 65: warning: exported type APIKey should have comment or be unexported (golint)
        Line 76: warning: exported type Version should have comment or be unexported (golint)
        Line 82: warning: exported type Stats should have comment or be unexported (golint)
        Line 88: warning: exported type SystemInformation should have comment or be unexported (golint)
        Line 104: warning: exported type SystemInformationPretty should have comment or be unexported (golint)
        Line 120: warning: exported type UpdateStatus should have comment or be unexported (golint)
        Line 123: warning: exported const UpdateStatusUnknown should have comment (or a comment on this block) or be unexported (golint)
        Line 129: warning: exported type Update should have comment or be unexported (golint)
        Line 138: warning: exported type AsyncUpdateId should have comment or be unexported (golint)
        Line 146: warning: exported type CreateIndexRequest should have comment or be unexported (golint)
        Line 152: warning: exported type CreateIndexResponse should have comment or be unexported (golint)
        Line 161: warning: exported type SearchRequest should have comment or be unexported (golint)
        Line 175: warning: exported type SearchResponse should have comment or be unexported (golint)
        Line 183: warning: exported type ListDocumentsRequest should have comment or be unexported (golint)
        Line 189: warning: exported type ListSynonymsResponse should have comment or be unexported (golint)
        Line 191: warning: exported type BatchCreateSynonymsRequest should have comment or be unexported (golint)
        Line 193: warning: exported type CreateApiKeyRequest should have comment or be unexported (golint)
        Line 200: warning: exported type UpdateApiKeyRequest should have comment or be unexported (golint)

Client.Timeout exceeded error when adding a few thousand documents to the index

I am trying to add around 25K documents to the index and getting the following error when doing so.

unable to execute request (path "PUT /indexes/players/documents" with method "Documents.AddOrUpdate"): Put "http://localhost:7700/indexes/players/documents": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When I use the functions, DefaultWaitForPendingUpdate or , I get the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12e23d4]

goroutine 1 [running]:
github.com/meilisearch/meilisearch-go.Client.WaitForPendingUpdate(0x15f2cc8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3b9aca00, ...)
	/Users/deshetti/go/pkg/mod/github.com/meilisearch/[email protected]/client.go:257 +0x104
github.com/meilisearch/meilisearch-go.Client.DefaultWaitForPendingUpdate(0x15f2cc8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3b9aca00, ...)
	/Users/deshetti/go/pkg/mod/github.com/meilisearch/[email protected]/client.go:239 +0x115
main.main()
	/Users/deshetti/Projects/playground/player-database/index-players/main.go:149 +0x406
exit status 2

Here is how I am using the above in my code. I am new to both Golang and Meilisearch, excuse me if I am missing something simple.

	// Creates an index in Meilisearch with the name players
	createIndex("players")

	// Index the players
	var client = meilisearch.NewClient(meilisearch.Config{
		Host: meiliURI,
	})

	updateIDRes, err := client.
		Documents("players").
		AddOrUpdate(articles)

	client.WaitForPendingUpdate(context.TODO(), 60*time.Second, "players", updateIDRes)
	// client.DefaultWaitForPendingUpdate("players", updateIDRes)

	if err != nil {
		log.Fatal(err)
	}

Any help is greatly appreciated. Thanks.

Tests failed on master

$ sh run_tests.sh
testmeili
758e874a9ab7fe96ae6c034c7ef6e2032693c4c048a7b6756aa886e84a07271e
=== RUN   TestClientDocuments_Get
--- PASS: TestClientDocuments_Get (0.08s)
=== RUN   TestClientDocuments_Delete
--- PASS: TestClientDocuments_Delete (0.15s)
=== RUN   TestClientDocuments_Deletes
--- PASS: TestClientDocuments_Deletes (0.15s)
=== RUN   TestClientDocuments_List
    TestClientDocuments_List: client_documents_test.go:171: expected to return the document[1]
--- FAIL: TestClientDocuments_List (0.08s)
=== RUN   TestClientDocuments_AddOrReplace
    TestClientDocuments_AddOrReplace: client_documents_test.go:215: number of doc should be at least 1
--- FAIL: TestClientDocuments_AddOrReplace (0.08s)
=== RUN   TestClientDocuments_AddOrUpdate
    TestClientDocuments_AddOrUpdate: client_documents_test.go:259: number of doc should be at least 1
--- FAIL: TestClientDocuments_AddOrUpdate (0.09s)
=== RUN   TestClientDocuments_DeleteAllDocuments
--- PASS: TestClientDocuments_DeleteAllDocuments (0.09s)
=== RUN   TestClientHealth_Get
--- PASS: TestClientHealth_Get (0.00s)
=== RUN   TestClientHealth_Set
--- PASS: TestClientHealth_Set (0.01s)
=== RUN   TestClientIndexes_Create
--- PASS: TestClientIndexes_Create (0.02s)
=== RUN   TestClientIndexes_Get
--- PASS: TestClientIndexes_Get (0.02s)
=== RUN   TestClientIndexes_Delete
--- PASS: TestClientIndexes_Delete (0.03s)
=== RUN   TestClientIndexes_List
--- PASS: TestClientIndexes_List (0.07s)
=== RUN   TestClientIndexes_UpdateName
--- PASS: TestClientIndexes_UpdateName (0.02s)
=== RUN   TestClientIndexes_UpdatePrimaryKey
--- PASS: TestClientIndexes_UpdatePrimaryKey (0.02s)
=== RUN   TestClientKeys_Get
--- PASS: TestClientKeys_Get (0.00s)
=== RUN   TestClientSearch_Search
--- PASS: TestClientSearch_Search (0.33s)
=== RUN   TestClientSettings_GetAll
--- PASS: TestClientSettings_GetAll (0.02s)
=== RUN   TestClientSettings_UpdateAll
--- PASS: TestClientSettings_UpdateAll (0.08s)
=== RUN   TestClientSettings_ResetAll
--- PASS: TestClientSettings_ResetAll (0.08s)
=== RUN   TestClientSettings_GetRankingRules
--- PASS: TestClientSettings_GetRankingRules (0.02s)
=== RUN   TestClientSettings_UpdateRankingRules
--- PASS: TestClientSettings_UpdateRankingRules (0.09s)
=== RUN   TestClientSettings_ResetRankingRules
--- PASS: TestClientSettings_ResetRankingRules (0.08s)
=== RUN   TestClientSettings_GetDistinctAttribute
--- PASS: TestClientSettings_GetDistinctAttribute (0.02s)
=== RUN   TestClientSettings_UpdateDistinctAttribute
--- PASS: TestClientSettings_UpdateDistinctAttribute (0.08s)
=== RUN   TestClientSettings_ResetDistinctAttribute
--- PASS: TestClientSettings_ResetDistinctAttribute (0.08s)
=== RUN   TestClientSettings_GetSearchableAttributes
--- PASS: TestClientSettings_GetSearchableAttributes (0.02s)
=== RUN   TestClientSettings_UpdateSearchableAttributes
--- PASS: TestClientSettings_UpdateSearchableAttributes (0.08s)
=== RUN   TestClientSettings_ResetSearchableAttributes
--- PASS: TestClientSettings_ResetSearchableAttributes (0.08s)
=== RUN   TestClientSettings_GetDisplayedAttributes
--- PASS: TestClientSettings_GetDisplayedAttributes (0.02s)
=== RUN   TestClientSettings_UpdateDisplayedAttributes
--- PASS: TestClientSettings_UpdateDisplayedAttributes (0.09s)
=== RUN   TestClientSettings_ResetDisplayedAttributes
--- PASS: TestClientSettings_ResetDisplayedAttributes (0.09s)
=== RUN   TestClientSettings_GetStopWords
--- PASS: TestClientSettings_GetStopWords (0.02s)
=== RUN   TestClientSettings_UpdateStopWords
--- PASS: TestClientSettings_UpdateStopWords (0.08s)
=== RUN   TestClientSettings_ResetStopWords
--- PASS: TestClientSettings_ResetStopWords (0.08s)
=== RUN   TestClientSettings_GetSynonyms
--- PASS: TestClientSettings_GetSynonyms (0.02s)
=== RUN   TestClientSettings_UpdateSynonyms
--- PASS: TestClientSettings_UpdateSynonyms (0.08s)
=== RUN   TestClientSettings_ResetSynonyms
--- PASS: TestClientSettings_ResetSynonyms (0.08s)
=== RUN   TestClientSettings_GetAcceptNewFields
--- PASS: TestClientSettings_GetAcceptNewFields (0.02s)
=== RUN   TestClientSettings_UpdateAcceptNewFields
--- PASS: TestClientSettings_UpdateAcceptNewFields (0.08s)
=== RUN   TestClientSettings_GetAttributesForFaceting
--- PASS: TestClientSettings_GetAttributesForFaceting (0.02s)
=== RUN   TestClientSettings_UpdateAttributesForFaceting
--- PASS: TestClientSettings_UpdateAttributesForFaceting (0.08s)
=== RUN   TestClientSettings_ResetAttributesForFaceting
--- PASS: TestClientSettings_ResetAttributesForFaceting (0.14s)
=== RUN   TestClientStats_Get
--- PASS: TestClientStats_Get (0.07s)
=== RUN   TestClientUpdates_List
--- PASS: TestClientUpdates_List (0.02s)
=== RUN   TestClientVersion_Get
--- PASS: TestClientVersion_Get (0.00s)
=== RUN   Test_deleteAllIndexes
--- PASS: Test_deleteAllIndexes (0.24s)
FAIL
ok  	github.com/meilisearch/meilisearch-go	(cached)

Tasks

  • Fix the code or the tests
  • Why do the tests script returns 0 (= SUCCEED) when the tests fail? This should be fixed as well because it cannot currently be used like that in a CI

Need synchronous API or at least give what's the sent request on /indexes/[table]/updates

Not printing the the requested json makes it harder to debug (I'm using go client btw)

already sent this:

"[{\"id\":20,\"author\":\"Prof. Laury Herzog\",\"title\":\"Aut perferendis accusantium sit voluptatem consequatur.\",\"body\":\"Consequatur accusantium voluptatem sit perferendis aut. Sit perferendis accusantium consequatur aut voluptatem. Perferendis accusantium aut sit voluptatem consequatur. Accusantium sit consequatur aut perferendis voluptatem. Sit accusantium aut perferendis consequatur voluptatem. Perferendis consequatur accusantium sit aut voluptatem. Perferendis sit voluptatem consequatur accusantium aut. Perferendis voluptatem sit aut consequatur accusantium. Voluptatem perferendis sit aut accusantium consequatur.\",\"created\":1606484683456729702}]"

but still got

{"status":"failed","updateId":19,"type":{"name":"DocumentsAddition","number":1},"error":"document id is missing","errorType":"invalid_request_error","errorCode":"missing_document_id","errorLink":"https://docs.meilisearch.com/errors#missing_document_id","duration":0.000806693,"enqueuedAt":"2020-11-27T13:44:43.459032791Z","processedAt":"2020-11-27T13:44:43.462177939Z"}

apparently I was wrongly set the primary key in the past (because I was calling .Documents(space).AddOrReplaceWithPrimaryKey(row, row.Id)) -- because I thought the 3rd parameter is ID of that specific document instead of the primary key column name.

curl -X GET 'http://localhost:7700/indexes/articles'        
{"name":"articles","uid":"articles","createdAt":"2020-11-27T11:47:01.834279018Z","updatedAt":"2020-11-27T13:44:38.401930845Z","primaryKey":"5"}

so I have to drop it first..

Add CI tests on different go versions

As for now, go.mod defines go version to 1.12. But the CI can test if this builds and runs on different versions in a very simple way, by using an Strategy Matrix in GitHub Actions:

jobs:
  build:
    runs-on: ubuntu-16.04
    strategy:
      matrix:
        go: [ '1.15', '1.12' ]
    name: Go ${{ matrix.go }} tests
    steps:
      - uses: actions/checkout@v2
      - name: Setup go
        uses: actions/setup-go@v1
        with:
          go-version: ${{ matrix.go }}
      - run: go test -v ./...

This should be tested in https://github.com/meilisearch/meilisearch-go/blob/master/.github/workflows/go.yml

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.