[es/snapshot.create] Missing [X-Elastic-Product] header about elasticsearch-java HOT 21 CLOSED

I ran into the same exact problem with just a search call against ElasticSearch 6.8. Could you perhaps make this check configurable in the client instead of hard-coding for the snapshot case like you did in the workaround?

Otherwise, is there an older version of the client that does not perform this check that I could use?

from elasticsearch-java.

Hi,

Elasticsearch is running on my local machine (docker container). The response doesn't contain the X-Elastic-Product header when debugging the java sdk. There is no proxy / no intermediate. Everything runs on my local computer.

Same issue when disabling "gzip compression" on client side:

Debugging RestClientTransport:

I'm not sure to understand why this header has to be sent by Elasticsearch.

from elasticsearch-java.

Thanks for the additional details. I was able to reproduce it, and this is an issue in Elasticsearch for this endpoint that may also affect some others. We'll fix Elasticsearch and if necessary add some workarounds in the client library.

from elasticsearch-java.

@swallez Yes, sorry, I realized after I asked that the client version has to match the server version. Thanks for responding!

from elasticsearch-java.

@mertkinag please read the compatibiity policy: https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/introduction.html

The Elasticsearch Java client is forward compatible; meaning that the client supports communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

So you cannot use the Java API client with Elasticsearch 6.x. The High Level Rest Client (version 6) works fine though.

from elasticsearch-java.

Pinging @elastic/es-data-management (Team:Data Management)

from elasticsearch-java.

I've transferred this to the elasticsearch-java repo, since this mentions using the new Java client rather than the old HLRC.

from elasticsearch-java.

@jloisel can you check that the X-Elastic-Product HTTP header is effectively received by the machine running your application? Is there an intermediate proxy between your application and the Elasticsearch server that would filter out this header?

from elasticsearch-java.

@jloisel can you provide us the command line used to start the Docker container, so that we can try to reproduce the issue?

from elasticsearch-java.

Here is the docker-compose.yml I use:

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    stdin_open: true
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    tty: true
    environment:      
      - path.repo=/tmp
      - transport.host=0.0.0.0
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - 9200:9200/tcp
      - 9300:9300/tcp
    restart: always
  kibana:
    image: docker.elastic.co/kibana/kibana:7.16.2
    stdin_open: true
    tty: true
    ports:
      - 5601:5601/tcp
    restart: always

volumes:
  elasticsearch-data:
    driver: local

Client config using Spring Boot:

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch.cluster.HealthResponse;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientOptions;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.base.MoreObjects;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpHost;
import org.elasticsearch.client.HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.io.IOException;
import java.net.InetAddress;
import java.util.concurrent.atomic.AtomicInteger;

import static java.time.Duration.ofSeconds;
import static org.elasticsearch.client.RequestOptions.DEFAULT;
import static org.elasticsearch.client.RestClient.builder;

@Slf4j
@Configuration
class ElasticSearchConfig {
  static final AtomicInteger PORT = new AtomicInteger(9200);

  @Bean(destroyMethod = "close")
  RestClient restClient(
    @Value("${elasticsearch.hostname:localhost}") final String hostname,
    @Value("${elasticsearch.compression:true}") final boolean isCompression) throws IOException {
    final InetAddress[] addresses = MoreObjects.firstNonNull(
      InetAddress.getAllByName(hostname),
      new InetAddress[0]);

    final HttpHost[] hosts = new HttpHost[addresses.length];
    for(int i = 0; i< addresses.length; i++) {
      final InetAddress addr = addresses[i];
      hosts[i] = new HttpHost(addr.getHostAddress(), PORT.get());
    }

    return builder(hosts)
      .setCompressionEnabled(isCompression)
      .build();
  }

  @Bean
  ElasticsearchClient elasticsearchClient(
    final ObjectMapper mapper,
    final RestClient restClient,
    @Value("${elasticsearch.client.buffer-limit:104857600}") final int bufferLimit) {

    final RequestOptions.Builder options = DEFAULT.toBuilder();
    options.setHttpAsyncResponseConsumerFactory(new HeapBufferedResponseConsumerFactory(bufferLimit));

    final ElasticsearchTransport transport = new RestClientTransport(
      restClient,
      new JacksonJsonpMapper(mapper)
    ).withRequestOptions(new RestClientOptions(options.build()));

    return new ElasticsearchClient(transport);
  }
}

from elasticsearch-java.

Thanks @jloisel. I used your docker compose file and initialization code, and everything works fine on my side.

First of all, curl show that the X-Elastic-Product header is correctly returned:

% curl -v http://localhost:9200
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
> GET / HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 546
<
{
  "name" : "7c1b69ce0ceb",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "OjvSBJCyTCK7x9jpM8lKtw",
  "version" : {
    "number" : "7.16.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
    "build_date" : "2021-12-18T19:42:46.604893745Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host localhost left intact
* Closing connection 0

I then used your initialization code with this simple main method to fetch Elasticsearch node information:

    public static void main(String[] args) throws Exception {
        RestClient http = restClient("localhost", true);
        ElasticsearchClient client = elasticsearchClient(new ObjectMapper(), http, 104857600);

        System.out.println(client.info().name());
        System.out.println(client.info().clusterUuid());
        System.out.println(client.info().version().number());
        System.out.println(client.info().version().buildDate());

        client._transport().close();
    }

The request succeeded and printed the same information as returned by curl.

Can you check that the header is returned by running curl -v http://localhost:9200?

from elasticsearch-java.

Try to create a snapshot. That's the failing part, other requests are working fine. See logs when trying to create a snapshot using the Java SDK:

co.elastic.clients.transport.TransportException: [es/snapshot.create] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

	at co.elastic.clients.transport.rest_client.RestClientTransport.checkProductHeader(RestClientTransport.java:340)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:250)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.snapshot.ElasticsearchSnapshotClient.create(ElasticsearchSnapshotClient.java:142)
	at io.vavr.control.Try.mapTry(Try.java:634)
	at com.octoperf.database.client.elasticsearch.ElasticsearchSnapshots.create(ElasticsearchSnapshots.java:55)
	at com.octoperf.database.client.elasticsearch.ElasticsearchSnapshotSpringTest.shouldTakeSnapshot(ElasticsearchSnapshotSpringTest.java:34)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	....
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://0:0:0:0:0:0:0:1:9200], URI [/_snapshot/test/1641413019710?wait_for_completion=true], status line [HTTP/1.1 200 OK]

Creating a snapshot with curl:

curl -v -XPUT "http://localhost:9200/_snapshot/test/1641413019710?wait_for_completion=true"

And the command result:

*   Trying ::1:9200...
* Connected to localhost (::1) port 9200 (#0)
> PUT /_snapshot/test/1641413019710?wait_for_completion=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 2011
< 
{"snapshot":{"snapshot":"1641413019710","uuid":"L80vX4ulS2StSR0wvwukQw","repository":"test","version_id":7160299,"version":"7.16.2","indices"...
.... ]}}

No X-elastic-product header seen in response.

from elasticsearch-java.

@dakrone FYI, this is a bug in the create snapshot API. I opened elastic/elasticsearch#82358 for this.

from elasticsearch-java.

Thanks for diagnosing this @swallez, sorry to transfer it here only to transfer it back

from elasticsearch-java.

@luispollo the Java API client (i.e. this new client) is compatible with Elasticsearch 7.14 and beyond. For older releases, and in particular 6.8 which is a different major version, please use the High Level Rest Client.

from elasticsearch-java.

@luispollo Is there a way or workaround to make Java API client (new client) compatible with old elastic search servers .? I am using Elasticsearch 7.6

from elasticsearch-java.

@tejadandu Not that I could find as the versions are generally incompatible. I had to downgrade to a matching version of the Java API client.

from elasticsearch-java.

@tejadandu the Java API client will work with 7.14 and up, but not with 7.6

from elasticsearch-java.

Upgraded to 7.17.6 client requests still get X-Elastic-Product errors

from elasticsearch-java.

@qurenneng what version of Elasticsearch server are you using? And is there a proxy between your client application and the Elasticsearch server that could remove this header?

from elasticsearch-java.

@swallez hey, i have also same issue when my Es client trying to send request to Es server but my new clients version is 8.4.2 and es server version is 6.8.4. Could this be the reason ?

My client dependency :

co.elastic.clients elasticsearch-java 8.4.2

from elasticsearch-java.