philippus / elastic4s Goto Github PK

http://www.elasticsearch.org/blog/1-0-0-beta2-released/#aggs

With version 1 due out soon, I want to lock down the DSL for our own 1.0 release.
Does anyone have any suggestions for improvements, things they'd like to change, breaking backwards compatibility.

I can make the change in build.sbt but do I need to update the pom.xml as well?

As described here (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-dynamic-mapping.html) there is an ability to disable dynamic creation of mappings for unmapped types, i.e;

curl -XPUT 'http://localhost:9200/twitter/tweet/_mapping' -d '
{
    "tweet" : {
        "dynamic": "strict",
        "properties" : {
            "message" : {"type" : "string", "store" : "yes"}
        }
    }
}
'

Proposed syntax:

create index "twitter" mappings(
  "tweet" as (
      //...
   ) dynamicDisabled true
)

Was at the London Elasticsearch meetup last night and Clinton said 0.90.4 is 2 weeks away. This issue is to prep a release for that version, with the new suggestors and function scorer (need to confirm these are 0.90.4 features and not 1.0.0).

I am trying to index a document which includes a completion type field named "ac". I am yet to index it in a convenient way using the index DSL. Any suggestions?

{
  "name": "Fyodor Dostoevsky",
  "ac": {
    "input": [
      "fyodor",
      "dostoevsky"
    ],
    "output": "Fyodor Dostoevsky",
    "payload": {
      "books": [
        "123456",
        "123457"
      ]
    }
  }
}

I want to you use (fantastic) client in a play app and for simplicity want to run an embedded (local) ES node. However, upon reloading (and soft restarting) my play app it throws me an exception saying it could not get a lock on the lucene files.

I think this is because the local node created is never closed. Only restarting my JVM (the sbt instance) fixes it.

Would it make sense to return a tuple containing the node and the client?

The documentation still refers to client.bulk whereas it seems that anything in the same client.execute block will be executed in bulk.

It does not seem possible to map nested objects[1] using the index creation DSL. If that's out-of-scope for the DSL, allowing the mapping to be specified in JSON would be handy.

[1] http://www.elasticsearch.org/guide/reference/mapping/object-type/

Apparently, it is possible to write this:

  search in "foo" query {
      matchall
  } facets {
      facet terms "facet1" field "field1"
      facet terms "facet2" field "field2"
  }

(notice the lack of comma between the two FacetDefinitions)

This will result in a query that only contains the last facet. Trying to write it with , won't work (haven't quite figured out why, given there's an appropriate method for this notation in the code).

So this is a little confusing. Writing the above by wrapping the definitions in a Seq works.

It looks like elastic4s doesn't support docasupsert yet.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html

or at least the DSL doesn't mention it and there are no tests for it.

Looks like a nice DSL otherwise. Hopefully there'll be some other places where I can use it.

Hi, I would like to see a BoostFactorScoreDefinition in addition to RandomScore, ScriptScore etc.

Here it is in the ES docs:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_boost_factor

Cheers,
-Fayvor

Deprecate numeric_range

i think it lacks import to
import com.sksamuel.elastic4s.ElasticClient

and it should be
val client = ElasticClient.local

In some cases we'd like to pass through "raw" queries (for example, queries stored externally in their JSON form). I can do this via the java API, but haven't found a way to do it in the elastic4s DSL.

e.g. I'd like to do this but stay in the elastic4s DSL:

val q = """{ "match": { "drummer" : "will champion" } }"""
val r = client.java.prepareSearch("music").setQuery(q).execute().actionGet()
println(r)

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/mapping-core-types.html#_doc_values_format

Add a way to set a type mapping without creating an index. http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping/

Hi,

Thanks for this fantastic work :)

I would like to ask for a new feature. In elastic search, as part of search query, user can specify the fields to be returned, it appears this feature is not yet implemented and when a search performed using elastic4s, all the fields of matching documents are returned.

It appears there is no way to specify a scroll duration when executing ElasticClient.searchScroll(scrollId : String). This makes elasticsearch skip the scrolll id in the result which is needed to fetch the next batch. I guess the most elegant solution would be extending the dsl to create a SearchScrollRequestBuilder, but a simple additonal Scroll parameter would also suffice.

I tried to create the nested object in mappings like the following

create index "myindex" mappings (
  "mytype" as (
      field("problem") nested (
          field("attr1") typed StringType,
          field("attr2") typed StringType
      )
  )
)

I got error
org.elasticsearch.index.mapper.MapperParsingException: No type specified for property [problem]

So, I tried to put type StringType on field("problem") and it worked but the result
was not correct. Actually, it was correct because the StringType overrides the nested type. My problem is that nested object doesn't work.

Did I miss something?
Thank you.

http://www.elasticsearch.org/blog/elasticsearch-0-90-12-1-0-1-released/

Hi!
Thanks for the amazing work you're doing!
Could it be possible to specify the "completion" type useful for the completion suggester (http://www.elasticsearch.org/blog/you-complete-me/). Or is there another way to do it? I need to write a mapping that looks like (see the field "name_autocomplete"):

{
    "products": {
        "properties": {
            "id": {
                "type": "string"
            },
            ....,
            "name_autocomplete": {
                "type": "completion",
                "payloads": true
            }
        }
    }
}

Thanks!

val client = ElasticClient.remote("localhost", 9300)
  val responseF = client execute { index into "bands/singers" fields "name"->"chris martin" }
  val response = Await.result(responseF, 3 seconds)
  println(response.getId())
  val singersF = client execute { search in "bands/singers" query "Chris" }
  val singers = Await.result(singersF, 3 seconds)
  println(singers);

It seems like it should work but unfortunately I got IndexMissingException.
I used exactly same syntax as it appears in README.md. However, if I omit
the type name "singers" from the "search in" like the following

 val singersF = client execute { search in "bands" query "Chris" }

The query will work just fine.

My environment is
OS = OSX 10.8.5
scala = 2.10.2
elastic4s = 0.90.5.1
elasticserver = 0.90.5

Current implementation have nested wrapper classes in public traits, which prevents proper equality test (even though it's seldom used) and implicit conversions in companion object.

Before covering all ES requests, I'd like to suggest:

• change object ElasticDsl to package dsl and bring inner classes to top-level
• give an ActionRequest instance to the default constructors of RequestDefinitionLike subclasses, and move auxiliary constructors into companion objects
• move implicit conversions to companion objects
• provide implicit conversions wrapping raw ES request for more generic execute

Benefits:

• provide consistent rule to deal with every request
• some wrappers can be value classes which decreases runtime overhead
• implicit conversion without import tax
• a few more...

Hi, can it be posible to have the "text" query? I have this issue, where I need to filter bi certain phrases, the search should be what ever field that with one of those phrases in certain field. I think I can only solve it with "text"

thanks

I'd like to add support to filter executed queries in percolate API as described here - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html#_filtering_executed_queries

Any suggestions / considerations before I code it?

There is no need to have a hard dependency on Jackson. Also alternatively you can completely remove the Jackson dependency and use the Jackson that's embedded in ElasticSearch.

Hello, I only have found the script option for score queries, but it would be nice to be able to do this:

"sort" : {
    "_script" : { 
        "script" : "Math.random()",
        "type" : "number",
        "params" : {},
        "order" : "asc"
    }
  }

Any idea of a workaround?

Hi, I'm looking for an ES client and your DSL-based approach looks very promising. Thanks in advance :)

One thing makes me worry is about the current execute implementation.

https://github.com/sksamuel/elastic4s/blob/master/src/main/scala/com/sksamuel/elastic4s/Client.scala#L39-L41

It takes per-client ExecutionContext and returns a Future by wrapping blocking actionGet() with a future {} block. It doesn't block the current thread but actually blocks the dedicated one, so it might reduce overall throughput and be a bottleneck.

Suggestion is attaching an ActionListener[T] using this and returning a Future from our own Promise, for example (not tested, just a concept):

def execute[A <: ActionRequest, B <: ActionResponse, C <: ActionRequestBuilder[A, B, C]](action: Action[A, B, C], req: A): Future[B] = {
  val p = Promise[B]()
  client.execute(action, req, new ActionListener[B] {
    def onResponse(response: B) = p.success(response)
    def onFailure(e: Throwable) = p.failure(e)
  })
  p.future
}

It'll make the API require an ExecutionContext for every request. If you accept this change I'll open a PR when time allows.

I can't connect to a local node using val client = ElasticClient.remote("localhost", 9300). When I try to I get a org.elasticsearch.client.transport.NoNodeAvailableException: No node available error. I am not sure what is causing this but the node does not seem to even see the connection attempt.

http://www.elasticsearch.org/blog/percolator-redesign-blog-post/

I'm trying to follow through the examples in the README and I am running into trouble creating an index with mappings. The below code throws compile errors with Scala 2.10.2 and using version 0.90.2.8 of elastic4s.

import com.sksamuel.elastic4s.ElasticClient
import com.sksamuel.elastic4s.ElasticDsl._
object Hello extends App {
    val client = ElasticClient.local
    client.execute {
        create index "places" mappings (
            "cities" as (
                "id" typed IntegerType,
                "name" boost 4,
                "content" analyzer StopAnalyzer
            )
         )
    }
}

The errors I'm getting are:

[error] .../src/main/scala/example/Hello.scala:14: type mismatch;
[error]  found   : String("name")
[error]  required: ?{def boost(x$1: ? >: Int(4)): ?}
[error] Note that implicit conversions are not applicable because they are ambiguous:
[error]  both method string2query in trait QueryDsl of type (string: String)com.sksamuel.elastic4s.StringQueryDefinition
[error]  and method field in trait CreateIndexDsl of type (name: String)com.sksamuel.elastic4s.ElasticDsl.FieldDefinition
[error]  are possible conversion functions from String("name") to ?{def boost(x$1: ? >: Int(4)): ?}
[error]                 "name" boost 4,
[error]                 ^
[error] .../src/main/scala/example/Hello.scala:13: not found: value IntegerType
[error]                 "id" typed IntegerType,
[error]                            ^
[error] two errors found
[error] (compile:compile) Compilation failed

Works:

update id 5 in "scifi/startrek"

Doesn't work:

update id 5 in "scifi" -> "startrek"

According to Fuzzy Query The default of prefix_length should be set to something other than 0 to allow for better performance. Although FuzzyDefinition exposes max_expansions, prefix_length is not exposed. A workaround is to use the elasitcsearch builder directly, but exposing this through the query dsl would be preferable.

Hi, I haven't been able to find how to do this:

{
    "bool" : {
        "must" : {
            "term" : { "user" : "kimchy" }
        },
        "must_not" : {
            "range" : {
                "age" : { "from" : 10, "to" : 20 }
            }
        },
        "should" : [
            {
                "term" : { "tag" : "wow" }
            },
            {
                "term" : { "tag" : "elasticsearch" }
            }
        ],
        "minimum_should_match" : 1,
        "boost" : 1.0
    }
}

Any suggestion of how to achieve that request?

Hi,
Is there a way to specify the routing for a mapping during index creation? Looked through the source code and didn't see anything, but could be missing. I suppose that would go in the mapping definition if I wanted to add it?

Thanks!
-PW

Is it possible to add the functionality to do a phrase_match query as outline at http://www.elasticsearch.org/guide/reference/query-dsl/match-query/?

I love the DSL of this client but I haven't been able to connect to a remote cluster over HTTP.

As far as I can determine this client does NOT work over HTTP but joins the cluster over TCP. This requires an extra listening port to be opened, which is not allowed on Heroku.

Is this assumption correct? If so, it would be nice if that was a bit more explicitly documented...

If this assumption is not correct, I'd value an example of how you can connect to a remote cluster that you have only a HTTP connection string for.

When creating a mapping with nested fields the following error is generated, i.e:

curl -XPUT 'http://localhost:9200/twitter/' -d '
{
    "settings": {
        "index": {
            "number_of_shards": 2,
            "number_of_replicas": 1
        }
    },
    "mappings": {
        "tweets": {
            "_source": {
                "enabled": true
            },
            "properties": {
                "_id": {
                    "type": "string",
                },
                "user": {
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}
'

Fails with an error:

org.elasticsearch.index.mapper.MapperParsingException: mapping [trucks]
    at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:311)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:298)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:680)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: No type specified for property [menu]
    at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:255)
    at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:219)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:177)
    at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:387)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:195)
    at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:308)
    ... 5 more

Fix: adding "type": "nested" to users the request succeeds, i.e:

curl -XPUT 'http://localhost:9200/twitter/' -d '
{
    "settings": {
        "index": {
            "number_of_shards": 2,
            "number_of_replicas": 1
        }
    },
    "mappings": {
        "tweets": {
            "_source": {
                "enabled": true
            },          
            "properties": {
                "_id": {
                    "type": "string",
                },
                "user": {
                    "type": "nested",
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}
'

In docs:

// single node
val client = ElastiClient.remote("host1", 9300)

But in fact this doesn't work because #remote expects paramters of another type. Tried to pass a tuple as follows ElastiClient.remote(("host1", 9300)) it compiles but throws an error

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:221)
at com.sksamuel.elastic4s.ElasticClient$.remote(Client.scala:228)
at com.sksamuel.elastic4s.ElasticClient$.remote(Client.scala:226)

Same error if I'm trying to create a client like this: ElastiClient.remote("host1" -> 9300).

Hello, I haven't been able to do a query like this:

"query_string" : {
        "fields" : ["content", "name^5"],
        "query" : "this AND that OR thus",
        "use_dis_max" : true
    }

the closest thing I've found is:

          "multi_match": {
            "query": "pants blue",
            "fields": [
              "specs.color^12",
              "title^8",
              "description^7",
              "keywords^6",
              "brand_name^3",
              "_all^1"
            ],
            "analyzer": "whitespace"
          }

Is there a way to make a query_string with multiple fields?

thanks

Is there any way of creating an index that uses an edgeNGram tokenfilter? I want to be able to search for users, which have username and name, both by full words and prefixes. So far I've managed only to create the index:

client.execute {
    create index "things" mappings (
      "users" as (
        field("username") typed StringType,
        field("name") typed StringType
      )
    )
}

I believe what I need to use is the Edge N-Gram token filter but I can't figure out how to inject that with the elastic4s DSL.

Hi, I can't see support for the enabled field in the mappings definition. Have I missed it or is it NYI?

Quick question: you use req._builder in SearchDslTest to get the json string from the query. I'd like to do this in my tests. Is there another way to get the json that doesn't use a private method?

https://github.com/sksamuel/elastic4s/blob/6d4b9e906880dc7b2c479d2a2e64ab61a133af7d/src/test/scala/com/sksamuel/elastic4s/SearchDslTest.scala

Thanks-
-f

It's not possible to define a boolean type in mapping.

Thanks for this library!

Please how do I delete an index programmatically? Can't seem find a the api for this.

It'd be great to have an OSGi friendly distribution of elastic4s to allow deployment in an OSGi container.

Just figured I'd drop this here since it'll probably affect you:

elastic/elasticsearch#4634

Things are fixed/changed very frequently and we need to use/test those fixes in our projects. What do you think about doing a SNAPSHOT release and updating it frequently?