elastic / elasticsearch-java Goto Github PK
View Code? Open in Web Editor NEWOfficial Elasticsearch Java Client
License: Apache License 2.0
Official Elasticsearch Java Client
License: Apache License 2.0
DO we have documentation for this?
Hello,
by documentation of SignificantTermsAggregation should support regex aggregation
Documentation
The object new TermsAggregation.Builder() is missing the include method that supports it
it only has
public final Builder include(String value) when for exclude the currect implemtation exists (TermsExclude exclude)
it's expected that the object will have
private final TermsInclude include;
as in TermsAggregation.Builder
not having it causes the wrong behavior and elastic to not use regex for filtering
Hi gyus!
I know that project is still in progress, but got some problems in simple usages.
Gets NPE in ElasticsearchError.class constructor when requesting nonexistent doc through get. Fails on Objects.requireNonNull(builder.status, "status")
validation
client.get(s -> s.index(index).id("asd"), Document.class)
Elasticsearch version (7.16.2):
Plugins installed: []
JVM version (16.0.2):
OS version (MacOS Montery 12.1):
Description of the problem including expected versus actual behavior:
The elastic java client CombinedFieldsQuery has field mimimumShouldMatch instead of minimumShouldMatch
Steps to reproduce:
Use the elastic java client co.elastic.clients.elasticsearch._types.query_dsl.CombinedFieldsQuery to build a query with a combined fields query that has any minimumShouldMatch
Confirm that CombinedFieldsQuery has mimimumShouldMatch and an error is returned when searching any index
Provide logs (if relevant):
{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:1366] [combined_fields] unknown field [mimimum_should_match] did you mean [minimum_should_match]?"}],"type":"x_content_parse_exception","reason":"[1:1389] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1389] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1366] [combined_fields] unknown field [mimimum_should_match] did you mean [minimum_should_match]?"}}},"status":400}
Thanks for providing this, I just wanted to do a first test. Maybe I am missing something, but I don't get the dependency, I am getting an error about not being authorized:
$ ./gradlew classes
> Task :compileJava FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
> Could not resolve co.elastic.clients:elasticsearch-java:8.0.0-SNAPSHOT.
Required by:
project :
> Could not resolve co.elastic.clients:elasticsearch-java:8.0.0-SNAPSHOT.
> Unable to load Maven meta-data from https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml.
> Could not get resource 'https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml'.
> Could not GET 'https://maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml'. Received status code 401 from server: Unauthorized
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 2s
1 actionable task: 1 executed
$ https maven.pkg.github.com/elastic/elasticsearch-java/co/elastic/clients/elasticsearch-java/8.0.0-SNAPSHOT/maven-metadata.xml 1 ↵
HTTP/1.1 401 Unauthorized
Access-Control-Allow-Origin: *
Content-Length: 0
Content-Security-Policy: default-src 'none';
Date: Thu, 03 Jun 2021 15:52:04 GMT
Server: GitHub Registry
Strict-Transport-Security: max-age=31536000;
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-GitHub-Request-Id: FC2A:7EF3:13680D:168AD5:60B8FAA4
X-XSS-Protection: 1; mode=block
access-control-allow-methods: GET, HEAD, OPTIONS
www-authenticate: Basic realm="GitHub Package Registry"
My setup for gradle is as in the documentation, when I try to open the link to the metada file, my browser wants me to login as well.
Trying a SearchMvtRequest
gives an error as we are trying to serialise the response to Json:
RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200)).build());
Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
ElasticsearchClient client = new ElasticsearchClient(transport);
SearchMvtResponse response = client.searchMvt(
(builder) -> builder.zoom(0).x(0).y(0).addIndex("points").field("location")
);
Gives the following error:
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 26)): only regular white space (\r, \n, \t) is allowed between tokens
When calling the method ElasticsearchIndicesClient.getSettings(...)
method, it is possible to add the includeDefaults
parameter and set that to true:
GetSettingsResponse getSettingsResponse = client.indices()
.getSettings(srb -> srb.index(index).includeDefaults(true));
But the GetSettingsResponse
that is returned does not contain this information, although it is returned from the server (the following json was extracted from an intercepting proxy):
{
"testindex" : {
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "testindex",
"creation_date" : "1636661459560",
"number_of_replicas" : "1",
"uuid" : "EYazsV63SXmlmFcNhRFvRw",
"version" : {
"created" : "7150299"
}
}
},
"defaults" : {
"index" : {
"flush_after_merge" : "512mb",
"final_pipeline" : "_none",
"max_inner_result_window" : "100",
"unassigned" : {
"node_left" : {
"delayed_timeout" : "1m"
}
},
"max_terms_count" : "65536",
"rollup" : {
"source" : {
"name" : "",
"uuid" : ""
}
},
"lifecycle" : {
"name" : "",
"parse_origination_date" : "false",
"step" : {
"wait_time_threshold" : "12h"
},
"indexing_complete" : "false",
"rollover_alias" : "",
"origination_date" : "-1"
},
"routing_partition_size" : "1",
"force_memory_term_dictionary" : "false",
"max_docvalue_fields_search" : "100",
"merge" : {
"scheduler" : {
"max_thread_count" : "1",
"auto_throttle" : "true",
"max_merge_count" : "6"
},
"policy" : {
"floor_segment" : "2mb",
"max_merge_at_once_explicit" : "30",
"max_merge_at_once" : "10",
"max_merged_segment" : "5gb",
"expunge_deletes_allowed" : "10.0",
"segments_per_tier" : "10.0",
"deletes_pct_allowed" : "33.0"
}
},
"max_refresh_listeners" : "1000",
"max_regex_length" : "1000",
"load_fixed_bitset_filters_eagerly" : "true",
"number_of_routing_shards" : "1",
"write" : {
"wait_for_active_shards" : "1"
},
"verified_before_close" : "false",
"mapping" : {
"coerce" : "false",
"nested_fields" : {
"limit" : "50"
},
"depth" : {
"limit" : "20"
},
"field_name_length" : {
"limit" : "9223372036854775807"
},
"total_fields" : {
"limit" : "1000"
},
"nested_objects" : {
"limit" : "10000"
},
"ignore_malformed" : "false"
},
"source_only" : "false",
"soft_deletes" : {
"enabled" : "true",
"retention" : {
"operations" : "0"
},
"retention_lease" : {
"period" : "12h"
}
},
"max_script_fields" : "32",
"query" : {
"default_field" : [
"*"
],
"parse" : {
"allow_unmapped_fields" : "true"
}
},
"format" : "0",
"frozen" : "false",
"sort" : {
"missing" : [ ],
"mode" : [ ],
"field" : [ ],
"order" : [ ]
},
"priority" : "1",
"codec" : "default",
"max_rescore_window" : "10000",
"max_adjacency_matrix_filters" : "100",
"analyze" : {
"max_token_count" : "10000"
},
"gc_deletes" : "60s",
"top_metrics_max_size" : "10",
"optimize_auto_generated_id" : "true",
"max_ngram_diff" : "1",
"hidden" : "false",
"translog" : {
"generation_threshold_size" : "64mb",
"flush_threshold_size" : "512mb",
"sync_interval" : "5s",
"retention" : {
"size" : "-1",
"age" : "-1"
},
"durability" : "REQUEST"
},
"auto_expand_replicas" : "false",
"mapper" : {
"dynamic" : "true"
},
"recovery" : {
"type" : ""
},
"requests" : {
"cache" : {
"enable" : "true"
}
},
"data_path" : "",
"highlight" : {
"max_analyzed_offset" : "1000000"
},
"routing" : {
"rebalance" : {
"enable" : "all"
},
"allocation" : {
"include" : {
"_tier" : ""
},
"disk" : {
"watermark" : {
"ignore" : "false"
}
},
"exclude" : {
"_tier" : ""
},
"require" : {
"_tier" : ""
},
"enable" : "all",
"total_shards_per_node" : "-1"
}
},
"search" : {
"slowlog" : {
"level" : "TRACE",
"threshold" : {
"fetch" : {
"warn" : "-1",
"trace" : "-1",
"debug" : "-1",
"info" : "-1"
},
"query" : {
"warn" : "-1",
"trace" : "-1",
"debug" : "-1",
"info" : "-1"
}
}
},
"idle" : {
"after" : "30s"
},
"throttled" : "false"
},
"fielddata" : {
"cache" : "node"
},
"default_pipeline" : "_none",
"max_slices_per_scroll" : "1024",
"shard" : {
"check_on_startup" : "false"
},
"xpack" : {
"watcher" : {
"template" : {
"version" : ""
}
},
"version" : "",
"ccr" : {
"following_index" : "false"
}
},
"percolator" : {
"map_unmapped_fields_as_text" : "false"
},
"allocation" : {
"max_retries" : "5",
"existing_shards_allocator" : "gateway_allocator"
},
"refresh_interval" : "1s",
"indexing" : {
"slowlog" : {
"reformat" : "true",
"threshold" : {
"index" : {
"warn" : "-1",
"trace" : "-1",
"debug" : "-1",
"info" : "-1"
}
},
"source" : "1000",
"level" : "TRACE"
}
},
"compound_format" : "0.1",
"blocks" : {
"metadata" : "false",
"read" : "false",
"read_only_allow_delete" : "false",
"read_only" : "false",
"write" : "false"
},
"max_result_window" : "10000",
"store" : {
"stats_refresh_interval" : "10s",
"type" : "",
"fs" : {
"fs_lock" : "native"
},
"preload" : [ ],
"snapshot" : {
"snapshot_name" : "",
"index_uuid" : "",
"cache" : {
"prewarm" : {
"enabled" : "true"
},
"enabled" : "true",
"excluded_file_types" : [ ]
},
"repository_uuid" : "",
"uncached_chunk_size" : "-1b",
"index_name" : "",
"partial" : "false",
"blob_cache" : {
"metadata_files" : {
"max_length" : "64kb"
}
},
"repository_name" : "",
"snapshot_uuid" : ""
}
},
"queries" : {
"cache" : {
"enabled" : "true"
}
},
"shard_limit" : {
"group" : "normal"
},
"warmer" : {
"enabled" : "true"
},
"max_shingle_diff" : "3",
"query_string" : {
"lenient" : "false"
}
}
}
}
}
With the old RestHighLevelClient
the org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse
contained two entries: indexToSettings
for the custom settings made by the user and indexToDefaultSettings
for the default settings.
I have set up a minimal Java application to test the client (https://github.com/sothawo/elastic-java-test) It's basically the code from the readme showing how to use the Elastisearch Client.
I have Elasticsearch 7.12.1 running in docker on port 9002 and am using an intercepting proxy on port 8080 to check what's going over the wire.
So this simple code
public class App {
public static void main(String[] args) throws IOException {
// Create the low-level client
RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200))
.setHttpClientConfigCallback(builder -> {
builder.setProxy(new HttpHost("localhost", 8080));
return builder;
})
.build();
// Create the transport that provides JSON and http services to API clients
Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
// And create our API client
ElasticsearchClient client = new ElasticsearchClient(transport);
// Search all items in an index that contains documents of type AppData
SearchResponse<AppData> search = client.search(s -> s
.index("appdata-index"),
AppData.class
);
if (search.hits().hits().isEmpty()) {
System.out.println("No match");
} else {
for (Hit<AppData> hit : search.hits().hits()) {
processAppData(hit._source());
}
}
}
private static void processAppData(AppData appData) {
if (appData != null) {
System.out.println(appData.toString());
}
}
}
leads to this error:
20:06:06: Executing task 'App.main()'...
> Task :compileJava UP-TO-DATE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :App.main()
Exception in thread "main" java.lang.NullPointerException: took
at java.base/java.util.Objects.requireNonNull(Objects.java:233)
at co.elastic.clients.elasticsearch._global.SearchResponse.<init>(SearchResponse.java:105)
at co.elastic.clients.elasticsearch._global.SearchResponse$Builder.build(SearchResponse.java:381)
at co.elastic.clients.elasticsearch._global.SearchResponse$Builder.build(SearchResponse.java:365)
at co.elastic.clients.json.JsonpObjectBuilderParser.parse(JsonpObjectBuilderParser.java:43)
at co.elastic.clients.json.JsonpValueParser.parse(JsonpValueParser.java:91)
at co.elastic.clients.base.RestClientTransport.getHighLevelResponse(RestClientTransport.java:175)
at co.elastic.clients.base.RestClientTransport.performRequest(RestClientTransport.java:65)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1227)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1246)
at com.sothawo.elasticjavatest.App.main(App.java:39)
That's when parsing the response, not even on parsing the returned data.
In the proxy I see the following request
POST http://localhost:9200/appdata-index/_search HTTP/1.1
Content-Length: 2
Content-Type: application/json; charset=UTF-8
Proxy-Connection: Keep-Alive
User-Agent: elasticsearch-java/7.12.1-SNAPSHOT (Java/16.0.1)
X-Elastic-Client-Meta: es=7.12.1-SNAPSHOT,jv=16,t=7.12.1-SNAPSHOT,hc=4.1.4
Host: localhost:9200
{}
and this response:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 408
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "appdata-index",
"_type": "_doc",
"_id": "5sko2HkB1yiMr4eS5Dmo",
"_score": 1.0,
"_source": {
"foo": "foo1",
"bar": "bar1"
}
},
{
"_index": "appdata-index",
"_type": "_doc",
"_id": "58ko2HkB1yiMr4eS-jlu",
"_score": 1.0,
"_source": {
"foo": "foo2",
"bar": "bar2"
}
}
]
}
}
So the request and response over the wire are ok. Taking the proxy out yields the same result.
What could be the reason for this error?
Not sure if this is intentional, but the PR to add links to the Javadoc and Github repo added a link to the javadoc that doesn't work. Not sure how link templating works, but it isn't properly formatting the links. Happy to open a PR if it makes sense that this should be fixed
Hi,
The msearch request must be terminated by a newline [\n]
As I understand it, the problem is in the serialization of the request - instead of json separated new line, the client sends an array of objects
Steps to reproduce:
IndexResponse ir = doIndex(new IndexRequest(new IndexRequest.Builder()
.index(INDEX)
.type(DOC_TYPE)
.id(DOC_ID)
.refresh(JsonValue.TRUE)
.document(Map.of(FOO, BAR))));
PutScriptRequest putScriptRequest = new PutScriptRequest(builder -> builder
.id("elastic-search-template")
.script(new StoredScript(scriptBuilder ->
scriptBuilder.lang(ScriptLanguage.Mustache)
.source("{" +
" \"query\": { \"term\" : { \"{{field}}\" : \"{{value}}\" } }," +
" \"size\" : \"{{size}}\"" +
"}")
)));
client.putScript(putScriptRequest);
MsearchTemplateRequest multiRequest = new MsearchTemplateRequest(builder -> builder.searchTemplates(
new TemplateItem(itemBuilder -> itemBuilder.index(INDEX)
.id("elastic-search-template")
.params(Map.of("field", JsonData.of(FOO), "value", JsonData.of(BAR), "size", JsonData.of(5))))
));
try {
MsearchTemplateResponse<Map> response = doMultiSearchTemplate(multiRequest, Map.class);
} catch (ApiException e) {
// reason -> {JsonStringImpl@7318} ""The msearch request must be terminated by a newline [\n]""
LOGGER.error("Error", e);
}
Addition logs:
2183 [main] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:r]: Event set [w]
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Request ready
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Attempt 1 to execute request
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Target auth state: UNCHALLENGED
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Proxy auth state: UNCHALLENGED
2183 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: Set timeout 30000
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> POST /_msearch/template HTTP/1.1
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 103
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: application/json; charset=UTF-8
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: localhost:49297
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4
2183 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: Event set [w]
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Output ready
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] produce content
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Request completed
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] [content length: 103; pos: 103; completed: true]
2184 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][rw:w]: 452 bytes written
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /_msearch/template HTTP/1.1[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 103[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localhost:49297[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
2184 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[{"id":"elastic-search-template","index":["my-index"],"params":{"size":5,"field":"foo","value":"bar"}}]"
2185 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Request ready
2185 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:w]: Event cleared [w]
2188 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:r]: 696 bytes read
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "X-elastic-product: Elasticsearch[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "content-type: application/json; charset=UTF-8[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "content-length: 249[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
2188 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"},"status":400}"
2188 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 400 Bad Request
2188 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << X-elastic-product: Elasticsearch
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << content-type: application/json; charset=UTF-8
2189 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << content-length: 249
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE(249)] Response received
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Response received HTTP/1.1 400 Bad Request
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE(249)] Input ready
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Consume content
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient - [exchange: 4] Connection can be kept alive indefinitely
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 4] Response processed
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient - [exchange: 4] releasing connection
2189 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:59098<->127.0.0.1:49297[ACTIVE][r:r]: Remove attribute http.nio.exchange-handler
2190 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-0][route: {}->http://localhost:49297][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2190 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection [id: http-outgoing-0][route: {}->http://localhost:49297] can be kept alive indefinitely
What am I doing incorrectly?
MsearchRequest multiSearchRequest = new MsearchRequest(builder -> builder
.index(INDEX)
.searches(
Json.createObjectBuilder()
.add("index", Json.createArrayBuilder().add(INDEX))
.add("types", Json.createArrayBuilder())
.add("search_type", "query_then_fetch")
.add("ccs_minimize_roundtrips", true)
.build(),
Json.createObjectBuilder()
.add("query", Json.createObjectBuilder()
.add("match", Json.createObjectBuilder()
.add(FOO, Json.createObjectBuilder()
.add("query", BAR)
.add("operator", "OR")))
.build()).build())
.typedKeys(true));
ApiException apiException = null;
try {
MsearchResponse response = doMultiSearch(multiSearchRequest, Map.class);
} catch (ApiException e) {
apiException = e;
}
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /my-index/_msearch?typed_keys=true HTTP/1.1[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 159[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localhost:49173[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: elasticsearch-java/7.15.0-SNAPSHOT (Java/11.0.11)[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-Elastic-Client-Meta: es=7.15.0-SNAPSHOT,jv=11,t=7.15.0-SNAPSHOT,hc=4.1.4[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Authorization: Basic ZWxhc3RpYy11c2VyOmVsYXN0aWMtcGFzcw==[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[{"index":["my-index"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true},{"query":{"match":{"foo":{"query":"bar","operator":"OR"}}}}]"
1836 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Request ready
1836 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:60408<->127.0.0.1:49173[ACTIVE][r:w]: Event cleared [w]
1840 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:60408<->127.0.0.1:49173[ACTIVE][r:r]: 696 bytes read
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "X-elastic-product: Elasticsearch[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "content-type: application/json; charset=UTF-8[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "content-length: 249[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
1840 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 << "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The msearch request must be terminated by a newline [\n]"},"status":400}"
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 400 Bad Request
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << X-elastic-product: Elasticsearch
1840 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 << Warning: 299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."
TermQuery
or MatchQuery
to JsonValue
?.searches(Json.createArrayBuilder().add(Json.createObjectBuilder()
.add("query", Json.createObjectBuilder().add("match", Json.createObjectBuilder().add(FOO, BAR)))
.build()).build()))
Object Builders for maps use an InternalMap
which is a sub-class of a hash map:
Hash maps are unordered, the order is defined by the internal hashing function and can be different for different runtime environments. Although the JSON standard does not define dictionaries to be ordered, it would be good to switch to a linked hash map with insertion order:
Because this does not seems performance-critical, I suggest to switch to a linked hash map with insertion order
If I'm understanding the syntax correctly, this should enable the write block on an index -
client.indices().addBlock(b -> b.index(sourceIndexName).block(IndicesBlockOptions.Write));
It results in a PUT
operation with the enum name rather than the value.
From Apache http wire tracing -
http-outgoing-7 >> "PUT /tmp_public_id_and_name/_block/Write HTTP/1.1[\r][\n]"
This causes an exception -
"type":"illegal_argument_exception", "reason":"No block found with name Write"
It works like this, so maybe this is the correct way?
client.indices().putSettings(b -> b.index(sourceIndexName).settings(s -> s.blocks(w -> w.write(true))));
How do you implement the equivalent of
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}
with this library?
Then the following code:
client.clearScroll(new ClearScrollRequest.Builder().scrollId(scrollId).build());
where scrollId is not null, raises an ApiException with root cause error:
Validation Failed: 1: no scroll ids specified;"
Why?
It seems the client is not parsing/returning sub aggregations that appear under filter aggregation. Here is a sample query and response.
Request:
{ "_source": { "excludes": [ "attachmentContents.*" ], "includes": [ "*" ] }, "aggregations": { "count": { "value_count": { "field": "messageStorageId.messageId" } }, "login_filter": { "aggregations": { "date": { "terms": { "field": "year", "order": [ { "_key": "desc" } ] } }, "to_domain": { "terms": { "exclude": [ "unknown" ], "field": "to.domain" } }, "file_ext": { "terms": { "exclude": [ "" ], "field": "attachments.fileExt" } }, "has_att": { "terms": { "field": "hasAttachments" } }, "from": { "terms": { "exclude": [ "unknown" ], "field": "sender.name" } }, "from_domain": { "terms": { "exclude": [ "unknown" ], "field": "sender.domain" } }, "to": { "terms": { "exclude": [ "unknown" ], "field": "to.name" } } }, "filter": { "bool": { "must": [ { "bool": { "minimum_should_match": "1" } } ] } } } }, "from": 0, "highlight": { "fields": { "body": { "fragment_size": 0, "number_of_fragments": 0 }, "attachments.fileName": { "fragment_size": 0, "number_of_fragments": 0 }, "attachmentContents.*": { "fragment_size": 200, "number_of_fragments": 2 }, "subject": { "fragment_size": 0, "number_of_fragments": 0 } }, "post_tags": [ "</span>" ], "pre_tags": [ "<span class=\"blue-hilite\">" ] }, "query": { "bool": { "must": [ { "bool": { "must": [ { "query_string": { "default_operator": "and", "query": "Tsang" } } ] } }, { "bool": { "minimum_should_match": "1", "should": [ { "match_all": {} } ] } } ] } }, "size": 10, "sort": [ { "messageDate": { "order": "asc" } } ], "stored_fields": [ "_source" ] }
Response:
{ "took": 190, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": null, "hits": [] }, "aggregations": { "count": { "value": 1 }, "login_filter": { "meta": { }, "doc_count": 1, "to_domain": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "cnn.com", "doc_count": 1 } ] }, "date": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "2008", "doc_count": 1 } ] }, "has_att": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": 1, "key_as_string": "true", "doc_count": 1 } ] }, "file_ext": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "doc", "doc_count": 1 } ] }, "from_domain": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "gmail.com", "doc_count": 1 } ] }, "from": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "John Tsang", "doc_count": 1 } ] }, "to": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "John Smith", "doc_count": 1 } ] } } } }
Specifically, sub aggregations above under "login_filter" section are not visible in the java client:
For my example, let's take a look at ElasticsearchAsyncClient#index
:
public <TDocument> CompletableFuture<IndexResponse> index(IndexRequest<TDocument> request)
throws IOException, ElasticsearchException {
@SuppressWarnings("unchecked")
JsonEndpoint<IndexRequest<?>, IndexResponse, ErrorResponse> endpoint = (JsonEndpoint<IndexRequest<?>, IndexResponse, ErrorResponse>) IndexRequest._ENDPOINT;
return this.transport.performRequestAsync(request, endpoint, this.transportOptions);
}
Is it true that this method can actually throw an IOException? I'd expect that the CompletableFuture
might return a failure (essentially wrapping the exception), but it's not actually throwing said exception. Am I missing some other checked IOException
further down in the call chain? In the above example, RestClientTransport#performRequestAsync
doesn't throw an IOException
.
Elasticsearch version (bin/elasticsearch --version
): 7.16.2
Plugins installed: []
JVM version (java -version
):
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.21.10)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.21.10, mixed mode, sharing)
OS version (uname -a
if on a Unix-like system):
Linux pop-os 5.15.8-76051508-generic #202112141040
163950527821.10~0ede46a SMP Tue Dec 14 22:38:29 U x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
When performing a snapshot using the new Java client, it fails due to missing X-Elastic-Product
header sent by Elasticsearch.
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
final CreateSnapshotRequest request = new CreateSnapshotRequest
.Builder()
.repository(repository)
.snapshot(name)
.waitForCompletion(waitForCompletion)
.build();
client.create(request);
Client being used (maven dep):
<elasticsearch.version>7.16.2</elasticsearch.version>
...
<dependency>
<groupId>co.elastic.clients</groupId>
<artifactId>elasticsearch-java</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
Provide logs (if relevant):
co.elastic.clients.transport.TransportException: [es/snapshot.create] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.
at co.elastic.clients.transport.rest_client.RestClientTransport.checkProductHeader(RestClientTransport.java:340)
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:250)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
at co.elastic.clients.elasticsearch.snapshot.ElasticsearchSnapshotClient.create(ElasticsearchSnapshotClient.java:142)
I am trying to implement a generic deep pagination function that takes a SearchRequest, creates a point-in-time and then paginates through all the values using search_after.
This requires modifying the search request I am given to include the PIT and set the search_after field correctly. However, using the new client, I am unable to modify an existing request due to all data being immutable and all mutability being contained within the corresponding builder classes.
A solution to this problem could be adding a generated method that creates a builder from the values contained within the object. Alternatively, a new constructor could be added to the builder classes that take a constructed object and initialize their values from the object.
Since some 7.x service do not support these headers, how can I disable this in Java client?
When running some tests using this client we run into "jar hell". That is, when bringing in the elasticsearch-java dependency it's runtime classpath includes duplicate classes from the jakarta.json
package. This seems to be due to us pulling in both jakarta.json-api
and the glassfish imlementation jakarta.json
dependencies.
There's a few things going on here:
implementation
it's only added to the runtime classpath for consumers. Since the client exposes types from json-api through it's own API (for example) this means users will run into compile errors if trying to use those types. They can "fix" this by manually adding json-api as a compile dependency.Two options to fix here:
json-api
dependency and make the glassfish impl api
instead of implementation
. This means we also leak all the org.glassfish.json
implementation classes onto the compile classpath of any consumers.json-api
a compileOnly
dependency and the glassfish impl runtimeOnly
. This should result in only the API classes being on the compile classpath, and both API and implementation classes available at runtime.I only found a method for adding single doc: co.elastic.clients.elasticsearch.core.BulkRequest.Builder::addDocument
with elasticsearch-java:7.16.0,
query:
var suggest = Suggester.of(builder -> {
builder.text(request.prefix);
for (String field : request.fields) {
builder.suggesters(field, s -> s.completion(c -> c.field(field).skipDuplicates(true).size(request.limit)));
}
return builder;
});
var response = client.search(builder -> builder.index(index).suggest(suggest).source(s -> s.fetch(false)), documentClass);
response from es (successful and expected response):
{
"took": 67,
"timed_out": false,
"_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
"hits": {"total": {"value": 0, "relation": "eq"}, "max_score": null, "hits": []},
"suggest": {
"completion#completion:completion1": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete1", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete1", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}],
"completion#completion:completion2": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete2", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete2", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}]
}
}
but the client failed to parse with this exception, it looks like it tried to parse it as TermSuggestOption, but it's actually CompletionSuggestOption
co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'TermSuggestOption.score'
at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:66)
at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:52)
at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:167)
at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:126)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:86)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:75)
at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:292)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:341)
at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:327)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1504)
I'm trying to build a Bulk request with some upsert operations, but it seems not supported in this version's Java Client yet.
I've seen the docAsUpsert and Script in the UpdateRequest. Will these options be supported at UpdateOperation in the next version?
final IndexResponse ir = client.index(builder -> builder.index("test").id("1") .value(new JsonMap("name", "alex1")) .refresh(Json.createValue("wait_for")));
fails with
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Unknown value for refresh: [\"wait_for\"]."}],"type":"illegal_argument_exception","reason":"Unknown value for refresh: [\"wait_for\"]."},"status":400}
I am not sure I am passing refresh parameter correctly but I could not find any examples
on another note. is this client ready for production use or it is in experimental state?
I really would like to get off low level REST client but not sure what this client's status is
would appreciate any info regarding this project status
tested with 7.16.0-SNAPSHOT (20.11.2021, 05:44)
In the TermVectorOption enum
https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_types/mapping/TermVectorOption.java the value with_positions_payloads is missing (https://www.elastic.co/guide/en/elasticsearch/reference/current/term-vector.html)
7.16,Elasticsearch Java API Client
SearchTemplateResponse<Object> response = ElasticSearchClient.searchTemplate(request, Object.class);
The response
just have the hits
property to get the search result, but how to get aggregation result ?
I am trying to create a point-in-time (code is in Kotlin)
client.openPointInTime { builder ->
builder.index(index)
.keepAlive { timeBuilder -> timeBuilder.time("5m") }
}
As can be seen, I have to specify the time and the unit as part of a string. This is neither very elegant nor type safe. Is this currently the only way to set these time values or am I missing something?
The old high-level client accepted a TimeValue, which was both a long and an associated TimeUnit. However, while the new client seems to also have TimeUnit class, it seems to be entirely disconnected from the Time class.
The only usage of TimeUnit I can find is in the DiskUsageRequest, but it seems to be used incorrectly there (TimeUnit does not contain a duration, but it tries to parse the timeout duration as a TimeUnit).
I also found a TimeBuilders class in the same package, but it is empty.
Hello, I'm trying to use the elasticsearch-java client in a new kotlin spring boot 2.7.0-SNAPSHOT project. As soon as I add the import for JacksonJsonpMapper() I get the following error:
at com.codinghumans.framework.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.kt:21) ~[main/:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[na:na]
at kotlin.reflect.jvm.internal.calls.CallerImpl$Constructor.call(CallerImpl.kt:41) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:108) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:159) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112) ~[kotlin-reflect-1.6.0.jar:1.6.0-release-798(1.6.0)]
at org.springframework.beans.BeanUtils$KotlinDelegate.instantiateClass(BeanUtils.java:867) ~[spring-beans-5.3.14.jar:5.3.14]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:196) ~[spring-beans-5.3.14.jar:5.3.14]
... 54 common frames omitted
Caused by: java.lang.ClassNotFoundException: jakarta.json.spi.JsonProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[na:na]
... 66 common frames omitted```
This is my build.gradle.kts
```import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.7.0-SNAPSHOT"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
id("org.flywaydb.flyway") version "8.2.0"
kotlin("jvm") version "1.6.0"
kotlin("plugin.spring") version "1.6.0"
}
group = "com.codinghumans"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
maven { url = uri("https://repo.spring.io/milestone") }
maven { url = uri("https://repo.spring.io/snapshot") }
}
dependencies {
implementation("co.elastic.clients:elasticsearch-java:7.16.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.13.1")
implementation("com.ibm.icu:icu4j:70.1")
implementation("mysql:mysql-connector-java:8.0.27")
implementation("org.flywaydb:flyway-mysql:8.3.0")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-validation")
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
testImplementation("org.springframework.boot:spring-boot-starter-test")
developmentOnly("org.springframework.boot:spring-boot-devtools")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "17"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
I'm quite new to java, so I'm a bit lost...Any idea what I might be doing wrong?
Elasticsearch is a big user of randomized testing. The library however features a lot more, like thread leak detection in test cases. So a test suite fails, if a thread is kept running - for more info check the annotation party in ESTestCase.
If you stick with junit4, this might make sense, if you upgrade to junit jupiter, I don't think there is a port just yet due to changes in the test infrastructure.
When the track_total_hits
property of a query is set to false
like this:
client.search(srb -> srb
.index(index)
.trackTotalHits(thb -> thb.enabled(false)), Entity.class);
The client fails on parsing the response because it expects the response to contain the total
element which is not returned in this case:
co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'HitsMetadata.total'
at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
at co.elastic.clients.elasticsearch.core.search.HitsMetadata.<init>(HitsMetadata.java:68)
at co.elastic.clients.elasticsearch.core.search.HitsMetadata.<init>(HitsMetadata.java:53)
at co.elastic.clients.elasticsearch.core.search.HitsMetadata$Builder.build(HitsMetadata.java:220)
at co.elastic.clients.elasticsearch.core.search.HitsMetadata$Builder.build(HitsMetadata.java:139)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1504)
at org.springframework.data.elasticsearch.backend.elasticsearch.DevTests.someTest(DevTests.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Not sure if this is an issue with the generated code, my IntelliJ setup or perhaps a bug in IntelliJ I am using the current version 2021.1.1).
I just want to write the code to create a SearchRequest.Builder
and add some values.
When I have entered
SearchRequest.Builder builder = new SearchRequest.Builder();
builder.i
just after entering the i
after the period, when IntelliJ IDEA is starting to autocomplete, the UI freezes and in the IntelliJ logs I can see, that the JVM is running out of heap space. Happens on two machines, the first has IDEA configured with 8GB heap, the second with 16GB.
Is this working for other people using IntelliJ IDEA?
Going on with my tests I wanted to create a simple match
query for a call like this:
GET /test-index/_search
{
"query": {
"match": {
"message": {
"query": "this is a test"
}
}
}
}
I try to use this code:
SearchRequest searchRequest = new SearchRequest.Builder()
.index("test-index")
.query(b -> b
.match(
// what to use here????
))
.build();
The QueryContainer.Builder
has these 2 match
methods:
public ObjectBuilder<QueryContainer> match(NamedQuery<JsonValue> v) {
this.$variant = v;
this.$tag = QueryContainer.Tag.match;
return new Constant(this.build());
}
public ObjectBuilder<QueryContainer> match(Function<co.elastic.clients.elasticsearch._types.query_dsl.NamedQuery.Builder<JsonValue>, ObjectBuilder<NamedQuery<JsonValue>>> f) {
return this.match((NamedQuery)((ObjectBuilder)f.apply(new co.elastic.clients.elasticsearch._types.query_dsl.NamedQuery.Builder())).build());
}
what would I need to put there? I am missing a MatchQuery
class here. Or something like
.match(m -> m.field("message").query("this is a test").build()
The same problem comes up when trying to build a simple terms query.
I have created a Search Request with an Aggregation on a nested object. ES server returns the correct response (as seen from the logs / trace). However, the client does not parse the response completely, and the sub aggregations are missing
http://localhost:9200/my_providers/_search?typed_keys=true' -d '{"aggregations":{"agg_services":{"aggregations":{"grp_by_service_slug":{"aggregations":{"nested_agg_attributes":{"aggregations":{"grp_by_service_attr_name":{"terms":{"field":"providerServiceDTOList.providerServiceAttrValDTOSet.attrName"}},"grp_by_service_attr_value":{"terms":{"field":"providerServiceDTOList.providerServiceAttrValDTOSet.attrValValue"}}},"nested":{"path":"providerServiceDTOList.providerServiceAttrValDTOSet"}}},"terms":{"field":"providerServiceDTOList.serviceSlug"}}},"nested":{"path":"providerServiceDTOList"}}},"query":{"term":{"marketplaceId":{"value":1}}}}
I have removed the data in the hits[] for brevity
{"took":3,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[]},"aggregations":{"nested#agg_services":{"doc_count":2,"sterms#grp_by_service_slug":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"appliances-repair","doc_count":1,"nested#nested_agg_attributes":{"doc_count":2,"sterms#grp_by_service_attr_name":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Product Type","doc_count":2}]},"sterms#grp_by_service_attr_value":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Refrigerator","doc_count":1},{"key":"Washing Machine","doc_count":1}]}}},{"key":"carpenter","doc_count":1,"nested#nested_agg_attributes":{"doc_count":2,"sterms#grp_by_service_attr_name":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Service Type","doc_count":2}]},"sterms#grp_by_service_attr_value":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Doors and Windows","doc_count":1},{"key":"Modular Kitchen","doc_count":1}]}}}]}}}}
When deserialize the json to SearchRequest
{
"size": 9999,
"query": {
"match_all": {}
},
"sort": [
{
"modify_time": {
"order": "desc"
}
}
],
"track_total_hits": 2147483647
}
Array deserializer use function deserialize(parser, mapper, event) to deserialize each item in sort block, it will be failed because SortOptions not implement this function
In version 7.16.2, unlike most of builders, the fluent API does not work for SourceConfig.Builder.
This code does not compile:
SourceConfig.Builder builder = new SourceConfig.Builder().filter(b -> b);
Error is "Type mismatch: cannot convert from ObjectBuilder to SourceConfig.Builder"
This code compiles:
SourceConfig.Builder builder = new SourceConfig.Builder();
builder.filter(b -> b);
I think the api document of java high level client is easy to understand. But the java api client is simple and missing a lot of detail about how to use it to communicate with the elastic. Thanks.
in the old RestHighLevelClient
, I can execute a bulk request with an update-using-script operation. It doesn't seem to work
in this new client.
hi,
before used "elasticsearch-rest-high-level-client.jar", in SearchSourceBuilder.java can print the DSL query
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(boolQuery);
...
logger.info("=====DSL====="+searchSourceBuilder.toString());
I‘m trying "elasticsearch-java.jar" , please tell me how to print the DSL SearchRequest query?
SearchRequest searchRequest = SearchRequest.of(s -> s
.index(indexName)
.query(q -> q
.term(t -> t
.field("goodsId")
.value(v -> v.stringValue("1"))
)));
i',m debugging. no found the record。 please help!
jakarta.json.stream.JsonParsingException: Jackson exception: Numeric value (-9223372036854775808) out of range of int (-2147483648 - 2147483647)
ClusterStatsResponse stats = cluster.getClient().cluster().stats();
Request:
curl http://localhost:60512/_cluster/stats\?pretty
Response:
{ "_nodes" : { "total" : 2, "successful" : 2, "failed" : 0 }, "cluster_name" : "testCluster", "cluster_uuid" : "mFsnsmy6R4mIEyAcxOdRGg", "timestamp" : 1640886234666, "status" : "green", "indices" : { "count" : 0, "shards" : { }, "docs" : { "count" : 0, "deleted" : 0 }, "store" : { "size_in_bytes" : 0, "total_data_set_size_in_bytes" : 0, "reserved_in_bytes" : 0 }, "fielddata" : { "memory_size_in_bytes" : 0, "evictions" : 0 }, "query_cache" : { "memory_size_in_bytes" : 0, "total_count" : 0, "hit_count" : 0, "miss_count" : 0, "cache_size" : 0, "cache_count" : 0, "evictions" : 0 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 0, "memory_in_bytes" : 0, "terms_memory_in_bytes" : 0, "stored_fields_memory_in_bytes" : 0, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 0, "points_memory_in_bytes" : 0, "doc_values_memory_in_bytes" : 0, "index_writer_memory_in_bytes" : 0, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 0, "max_unsafe_auto_id_timestamp" : -9223372036854775808, "file_sizes" : { } }, "mappings" : { "field_types" : [ ], "runtime_field_types" : [ ] }, "analysis" : { "char_filter_types" : [ ], "tokenizer_types" : [ ], "filter_types" : [ ], "analyzer_types" : [ ], "built_in_char_filters" : [ ], "built_in_tokenizers" : [ ], "built_in_filters" : [ ], "built_in_analyzers" : [ ] }, "versions" : [ ] }, "nodes" : { "count" : { "total" : 2, "coordinating_only" : 0, "data" : 2, "data_cold" : 2, "data_content" : 2, "data_frozen" : 2, "data_hot" : 2, "data_warm" : 2, "ingest" : 2, "master" : 2, "ml" : 0, "remote_cluster_client" : 2, "transform" : 2, "voting_only" : 0 }, "versions" : [ "7.16.1" ], "os" : { "available_processors" : 16, "allocated_processors" : 16, "names" : [ { "name" : "Mac OS X", "count" : 2 } ], "pretty_names" : [ { "pretty_name" : "Mac OS X", "count" : 2 } ], "architectures" : [ { "arch" : "aarch64", "count" : 2 } ], "mem" : { "total_in_bytes" : 34359738368, "free_in_bytes" : 118784000, "used_in_bytes" : 34240954368, "free_percent" : 0, "used_percent" : 100 } }, "process" : { "cpu" : { "percent" : 0 }, "open_file_descriptors" : { "min" : 337, "max" : 338, "avg" : 337 } }, "jvm" : { "max_uptime_in_millis" : 51439, "versions" : [ { "version" : "17.0.1", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "17.0.1+12", "vm_vendor" : "Eclipse Adoptium", "bundled_jdk" : true, "using_bundled_jdk" : true, "count" : 2 } ], "mem" : { "heap_used_in_bytes" : 824238880, "heap_max_in_bytes" : 17179869184 }, "threads" : 72 }, "fs" : { "total_in_bytes" : 994662584320, "free_in_bytes" : 722047512576, "available_in_bytes" : 722047512576 }, "plugins" : [ ], "network_types" : { "transport_types" : { "netty4" : 2 }, "http_types" : { "netty4" : 2 } }, "discovery_types" : { "zen" : 2 }, "packaging_types" : [ { "flavor" : "default", "type" : "tar", "count" : 2 } ], "ingest" : { "number_of_pipelines" : 2, "processor_stats" : { "gsub" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 }, "script" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 } } } } }
I create a client to perform a search and most of the builder types build a type of query except for range:
client.search({ s ->
s.index("myindex")
.query { q -> q.range(someJsonValue) }
Most of the functions on the "query builder" allow something to be built, where range
wants a JsonValue
, how do we build that JsonValue?
When a request is sorted on fields that may have null
values, the responses's hits.hits.sort
array will contain null
values that cause a parsing failure as arrays generally don't accept null
values in the API spec.
This is related to elastic/elasticsearch-specification#1109 and will be solved once sort
is considered as an opaque JSON blob.
/cc @ilgrosso
Trying to use client.search().documents()
, but I got an empty list:
List<Major> majorList = this.elasticsearchClient.search(s ->
s.size(RECOMMEND_SIZE)
.index("major")
.query(q ->
q.bool(bl ->
bl.must(mst ->
mst.match(match ->
match.field("name")
.query(qr -> qr.stringValue(name))
.fuzziness("AUTO"))))),
Major.class).documents();
The majorList
is empty. But when I use the client.search().hits().hits()
the result contains what I need.
Am I using it the wrong way?
The unwrap()
method may return a null deserialiser if two threads access the method simultaneously.
Stacktrace:
Caused by: java.lang.NullPointerException: null
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144) ~[elasticsearch-java-7.16.0.jar:na]
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487) ~[elasticsearch-java-7.16.0.jar:na]
Debugging screenshot where you can see deserializer
is set, but d
is null
:
Looking at the article linked to in the source code, it looks like the "Compliant Solution (Immutable)" is desired. In this case, there is one assignment (d = deserializer
) missing just before the null check inside the synchonized
block.
So the code below should work better, making sure d
is getting assigned with deserializer
, for the cases where it has been assigned by another thread after the current thread has entered the synchronized block:
protected JsonpDeserializer<T> unwrap() {
// See SEI CERT LCK10-J https://wiki.sei.cmu.edu/confluence/x/6zdGBQ
JsonpDeserializer<T> d = deserializer;
if (d == null) {
synchronized (this) {
d = deserializer;
if (d == null) {
d = ctor.get();
deserializer = d;
}
}
}
return d;
}
Version: 7.16.0
execute simple test case as follow:
import org.junit.jupiter.api.Test;
public class TestEsAliasCommands extends AbstractEsCallerTest {
@Test
void testGetNotExistingAlias() {
try {
getEsClient().indices().getAlias(g -> g.name("not-existing-alias")).toCompletableFuture().get();
} catch (Throwable e) {
e.printStackTrace();
}
}
}
where getEsClient()
return an ElasticsearchAsyncClient
instance. error as follow:
co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'VALUE_STRING' instead of '[START_OBJECT, KEY_NAME]'
at co.elastic.clients.json.JsonpUtils.ensureAccepts(JsonpUtils.java:68)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:74)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:276)
at co.elastic.clients.transport.rest_client.RestClientTransport.access$200(RestClientTransport.java:60)
at co.elastic.clients.transport.rest_client.RestClientTransport$1.onSuccess(RestClientTransport.java:165)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:649)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:383)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:833)
environment:
docker.elastic.co/elasticsearch/elasticsearch:7.16.2
Hi,
I've tried to replace deprecated high level elasticsearch client with this client and have couple questions.
Using version 7.15.2.
When an index mapping contains a property that is an object,
co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient#getMapping(co.elastic.clients.elasticsearch.indices.GetMappingRequest)
crashes on parsing the result.
This can be tested with the following code:
class Name {
String first;
String last;
public Name(String first, String last) {
this.first = first;
this.last = last;
}
// getter+setter
}
class Person {
String id;
Name name;
public Person(String id, Name name) {
this.id = id;
this.name = name;
}
// getter+setter
}
ElasticsearchClient client = ... // setup the ElasticsearchClient
String index = "testindex";
Person person = new Person("42", new Name("Ford", "Prefect"));
client.index(b -> b.index(index).id(person.id).document(person));
GetMappingResponse getMappingResponse = client.indices().getMapping(mrb -> mrb.index(index));
This will produce the following error:
jakarta.json.stream.JsonParsingException: Property 'type' not found
at co.elastic.clients.json.JsonpUtils.lookAheadFieldValue(JsonpUtils.java:139)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:128)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
at co.elastic.clients.json.JsonpDeserializer$StringMapDeserializer.deserialize(JsonpDeserializer.java:461)
at co.elastic.clients.json.JsonpDeserializer$StringMapDeserializer.deserialize(JsonpDeserializer.java:447)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
at co.elastic.clients.base.DictionaryResponse.lambda$setupDictionaryResponseDeserializer$0(DictionaryResponse.java:148)
at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:150)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:120)
at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95)
at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42)
at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205)
at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102)
at co.elastic.clients.base.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:242)
at co.elastic.clients.base.rest_client.RestClientTransport.performRequest(RestClientTransport.java:104)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getMapping(ElasticsearchIndicesClient.java:1044)
at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getMapping(ElasticsearchIndicesClient.java:1061)
The mapping returned from the server:
{
"testindex" : {
"mappings" : {
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"properties" : {
"first" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"last" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
I debugged through the code; when reading the properties, the parser comes to the name
entry and then on finding a START_OBJECT tries to get the type
property of the object, basically expecting something like
{
"type": "object", <=== Elasticsearch does not send a type entry here
"properties": {
"first": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"last": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
Even when the mapping is stored explicitly in Elasticsearch with the type object, it is not returned on getting the mapping.
The co.elastic.clients.json.JsonpMapper class already uses the "new" jakarta.json.*
classes (which are implemented in org.glassfish:jakarta.json). This causes an issue when trying to use the elasticsearch client in an environment which still has the "old" implementation org.glassfish:javax.json on the classpath (which is the case for most if not any Java EE 8 application server).
This is because there is a class "org.glassfish.json.JsonProviderImpl" in both org.glassfish:javax.json and org.glassfish:jakarta.json. One uses javax.json
classes and the other one uses jakarta.json
classes.
The elastic client now (in class JsonValueParser
) simply calls jakarta.json.spi.JsonProvider.provider()
which basically calls:
Class.forName("org.glassfish.json.JsonProviderImpl");
However, in an Java EE 8 environment, this will return the JsonProviderImpl from org.glassfish:javax.json because the classes unfortunately share the same name and package.
During runtime, this will lead to an exception:
jakarta.json.JsonException: Provider org.glassfish.json.JsonProviderImpl could not be instantiated: java.lang.ClassCastException: org.glassfish.json.JsonProviderImpl cannot be cast to jakarta.json.spi.JsonProvider
I have created a post on StackOverflow to ask if there is any possibility to use both in the same application. However, I do not expect there to be a solution unless the implementation of the "new" "org.glassfish.json.JsonProviderImpl" (in org.glassfish:jakarta.json) is moved to a different package or renamed. (I am also not sure why the authors of org.glassfish:jakarta.json chose to reuse the same package and class name for the new version).
However, in order to use the Elastic Client in Java EE 8 environments, there should be some sort of workaround for the time being: Maybe there could be two builds - one using the jakarta.json.* classes and one using the javax.json.* classes (or potentially using the maven shade plugin).
Any ideas?
Client version 7.16.1:
It seems that the co.elastic.clients.elasticsearch._types.query_dsl.Query
is missing the wrapper query type (https://www.elastic.co/guide/en/elasticsearch/reference/7.16/query-dsl-wrapper-query.html).
The enum in the DataStreamHealthStatus contains the wrong statuses, elastic return the status of the data stream in capitals. Currently the parsing of the object return a exception.
package co.elastic.clients.elasticsearch.indices.DataStreamHealthStatus
Green("green"),
Yellow("yellow"),
Red("red");
jakarta.json.stream.JsonParsingException: Invalid enum [GREEN] at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:75) at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:61) at co.elastic.clients.util.StringEnum$Deserializer.deserialize(StringEnum.java:45) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95) at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42) at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205) at co.elastic.clients.json.JsonpDeserializer$ArrayDeserializer.deserialize(JsonpDeserializer.java:430) at co.elastic.clients.json.JsonpDeserializer$ArrayDeserializer.deserialize(JsonpDeserializer.java:409) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:68) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:122) at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:95) at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:42) at co.elastic.clients.json.JsonpDeserializer$LazyDeserializer.deserialize(JsonpDeserializer.java:205) at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:102) at co.elastic.clients.base.RestClientTransport.getHighLevelResponse(RestClientTransport.java:231) at co.elastic.clients.base.RestClientTransport.performRequest(RestClientTransport.java:93) at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.getDataStream(ElasticsearchIndicesClient.java:838)
by the documentation a function score can contain only weight and no other inner functions
in the java api in order to build the FunctionScore we must "hold" a "ContainerBuilder" object, that is only provided as
a response to exp, gauss etc, the weight method (FunctionScore.java:306) returns builder instead of returning ContainerBuilder making this use case impossible to invoke within the java Api
Since upgrading from the 7.15.x client to the 7.16.0 client I am getting a TransportException
:
co.elastic.clients.transport.TransportException: [es/search] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.