`date_range` with `gte` lower than `lte` (<24h) but errors with min value greater than max value about elasticsearch HOT 9 CLOSED

@boesing you are using YYYY instead of yyyy for your date parsing. These are two different things.

YYYY represents "week of year", which is a tricky definition in the ISO year-week calendar. You almost never want this.

yyyy represents the calendar year. What folks normally consider year.

FYI, I did your test with yyyy and things were indexed just fine.

these dates are not integer, though elastic parses these dates to integer and I guess there is a bug.

We parse all values and index them into long values representing the milliseconds since the epoch. So, even though you are not supplying a number, all dates are technically numbers and we store them as such.

from elasticsearch.

Pinging @elastic/es-search (Team:Search)

from elasticsearch.

Did you check the local time of the ES nodes ? My guess is , some of them having different timezones contributing to this issue of < 24 hr durations.

from elasticsearch.

I did not, as I can reproduce it in a single node already. even tho, the local timezone should be ignored since the timezone is required to be submitted in the documents value. thats why I explicitly disabled date_detection in the mapping and defined the format of the date_range to contain a timezone.
Therefore, I would not expect to have any sideeffects on es server timezone at all. But maybe my assumption is incorrect 🤷🏼‍♂️

from elasticsearch.

@boesing The timestamps in the exception are as expected. The min value is indeed greater than the max value, and this is not allowed when indexing in a date range field

min: 1704128073000
max: 1704124499000

Note the seventh digit in min is an 8, but its a 4 in max

Maybe I don't understand the issue?

from elasticsearch.

@benwtrent Could you check the data I am sending to elastic? these dates are not integer, though elastic parses these dates to integer and I guess there is a bug.

There is a difference in those dates round about 23 hours and a couple of minutes. Something within elasticsearch does not convert those strings into the correct integers and thus - ofc - the values in the error are reflecting an issue but that is not caused on client side.

if that would be a client issue, I do not understand why the same request works within elastic cloud which is on 8.11.0 with lucene 9.7.0.

from elasticsearch.

Ah, wasn't aware of that. So its the format which introduces the problem.
I will try that, thanks for clarification! That might fix our problem.

from elasticsearch.

@benwtrent What I still do not understand is, that if elasticsearch uses SimpleDate as described here and here, how does that lead to issues for dates in April?

I've adapted our mapping but I still don't get the underlying issue with my concrete example.

• Why is it valid in elastic.cloud but not when running docker?
• How is the year of the week of week 18 not equal (both dates are in the same calendar week, but even if they weren't in the same week, the earlier date would be in a lower week)?

So I would expect issues end of december where the week of the year could become 2025 on december 30th, 2024 but not end of april 🤔

from elasticsearch.

Why is it valid in elastic.cloud but not when running docker?

I don't think it is? I got the exact same error as you when testing in cloud, however its obvious that the resulting min and max are incorrect for a range mapping.

How is the year of the week of week 18 not equal (both dates are in the same calendar week, but even if they weren't in the same week, the earlier date would be in a lower week)?

I honestly don't know.

But we are not using SimpleDate. We do utilize the underlying Java date parsing, but it is much more complicated.

Here is the parsing once we have the temporal accessor:

Here is the code once we have parsed out the TemporalAccessor for WeekOfYear:

    private static LocalDate localDateFromWeekBasedDate(TemporalAccessor accessor, Locale locale) {
        WeekFields weekFields = WeekFields.of(locale);
        if (accessor.isSupported(weekFields.weekOfWeekBasedYear())) {
            return LocalDate.ofEpochDay(0)
                .with(weekFields.weekBasedYear(), accessor.get(weekFields.weekBasedYear()))
                .with(weekFields.weekOfWeekBasedYear(), accessor.get(weekFields.weekOfWeekBasedYear()))
                .with(TemporalAdjusters.previousOrSame(weekFields.getFirstDayOfWeek()));
        } else {
            return LocalDate.ofEpochDay(0)
                .with(weekFields.weekBasedYear(), accessor.get(weekFields.weekBasedYear()))
                .with(TemporalAdjusters.previousOrSame(weekFields.getFirstDayOfWeek()));

        }
    }

from elasticsearch.