Change of behaviour for the `"query_string": "*"` filter after upgrading to 8.14 about elasticsearch HOT 15 CLOSED

@dej611 for the index settings, is there a default_field supplied?

from elasticsearch.

Pinging @elastic/es-analytical-engine (Team:Analytics)

from elasticsearch.

I've been trying for a bit, and I can't get this to reproduce with a more minimal example. Here's the script I just used:

PUT http://localhost:9200/test

{
  "mappings": {
    "properties": {
      "number": {
        "type": "long"
      },
      "text": {
        "type": "keyword"
      }
    }
  }
}

PUT http://localhost:9200/test/_doc/1

{ "number": 100, "text": "foo" }

PUT http://localhost:9200/test/_doc/2

{ "number": 101, "text": "foo" }

PUT http://localhost:9200/test/_doc/3

{ "number": 11, "text": "bar" }

POST http://localhost:9200/test/_search

{
    "size": 0,
        "query": {
            "match_all": {}
        },
        "aggregations": {
            "match_star": {
                "filters": {
                    "filters": {
                        "Status": {
                            "bool": {
                                "must": [],
                                "filter": [
                                {
                                    "query_string": {
                                        "query": "*"
                                    }
                                }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        }
                    }
                }
            }
        }
}

Which returns exactly what I would expect:

  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "match_star": {
      "buckets": {
        "Status": {
          "doc_count": 3
        }
      }
    }
  }

i.e. one bucket with three matches. So I don't think query_string: query: * is the problem.

@dej611 can you trim this query down to the minimum needed to reproduce the issue, please? If you need help doing that, please reach out to me next week and I can find some time to pair with you on it.

Alternatively, that query_string: query: * clause should be functionally a no-op; you could just omit the whole thing. It shouldn't make a difference, and my testing indicates that it won't, but if that fixes your chart I don't see that it would hurt anything.

from elasticsearch.

I still can't reproduce this locally, but @dej611 was able to get me access to the cluster where this is occurring. Based on analysis on that cluster, I now believe this is an issue with the query_string search, that has nothing to do with the aggregation. I ran two queries:

POST /[REDACTED]/_search
{
  "profile": true, 
  "query": {
    "query_string": {
      "query": "*"
    }
  }
}

POST /[REDACTED]/_search
{
  "profile": true, 
  "query": {
   "bool": {
      "must": []
    }
  }
}

I would expect that these would return the same results, however the query_string version returns no hits, while the empty boolean query returns ~1100 results. Profiling the query_string version, it rewrites to a MatchNoDocs query:

           "query": [
              {
                "type": "MatchNoDocsQuery",
                "description": """MatchNoDocsQuery("unmapped fields [*]")""",

I'm going to pull the search team in on this to get some domain expertise about what might be happening here.

from elasticsearch.

Pinging @elastic/es-search (Team:Search)

from elasticsearch.

@dej611 another setting that could change this behavior is index.query_string.lenient. Did default_field or index.query.default_field or any index.query_string setting change in kibana between the two versions?

from elasticsearch.

I can see a default_field configured in settings:

GET /[REDACTED]/_settings
...
"query": {
  "default_field": [
    "message"
  ]
}
...

We've checked also any change in the index template used for the integration, and the final index template is derived from this https://github.com/elastic/elasticsearch/blame/main/x-pack/plugin/core/template-resources/src/main/resources/metrics%40settings.json (I see the default_field setting here has been changed last time 4 years ago).

from elasticsearch.

The profile output says it rather clearly: the field target by the query string is not mapped. Can you check the mapping for the message field on the index being queried?

Just in case, what happens if you specify message:* in the query string query?

from elasticsearch.

I managed to have a 8.13.1 instance with the same integration without the issue.

Checking it I see that some indexes on the 8.13.1 have a different default_field configuration than its 8.14 counter-part...

The profile output says it rather clearly: the field target by the query string is not mapped.

Tried to run the profile on both 8.13.1 and 8.14 and I see both MatchNoDocsQuery on different indexes.

Can you check the mapping for the message field on the index being queried?

In both cases there's:

"message": {
  "type": "match_only_text"
},

Just in case, what happens if you specify message:* in the query string query?

0 results in both instances.

from elasticsearch.

@gizas found this discussion here about removing the default_field for logs-*: #99872 (PR #102456 )
That discussion led to the removal of the default_field in the index template in Kibana for Fleet, fallback to the default [*] which affects the index used in this issue: elastic/kibana#177605 ( PR elastic/kibana#178020 )

from elasticsearch.

So, elastic/kibana#178020 is most likely the cause here? Do those vector changes explain the discrepancy?

@dej611 what happens if you flip back in 8.14 the default field calculation (just do it manually and update the index settings directly), do you get hits again like you expect?

from elasticsearch.

I've manually changed the default_field : ["*"] to indexes in the 8.14.0 and results started to flow again.
If I understood it correctly the changes in elastic/kibana#178020 were expecting to fallback default_field to ["*"] ES value, but because of the default index template for metrics in https://github.com/elastic/elasticsearch/blame/main/x-pack/plugin/core/template-resources/src/main/resources/metrics%40settings.json that limited the actual search space when using the wildcard.
Is that correct?

from elasticsearch.

If multiple templates are being utilized and the default_field value was previously overwritten so that this metrics template was ignored, that is likely the cause of this failure.

from elasticsearch.

@dej611 this seems like an unintended behavior change on the Kibana side. No behavior changed directly in the Elasticsearch server core.

Do you think we can close this?

from elasticsearch.

Yes, I think so. Thanks for your help 👍

from elasticsearch.